Release UPX 3.91 for OpenPandora


FBnil

They'll own everything and be miserable.
Joined
Dec 14, 2012
Messages
5,118
Location
Yurp
TL;DR: Packer for executables, executables are smaller while still executable. unpacker is small and adds almost no startup time. RAM usage is a little bit bigger, but not much.

I was playing around, trying to get a Webbrowser icon.png to the Pandora, with the advent of the Pyra, we need it to keep it alive. Now, I needed some setuid things, which I forgot how to set, and I stumbled upon an old UPX project I had... and here is the binary compiled for the Open Pandora: upx-armel-bin.tar.gz

edit: requires a newer libstdc++.so.6 file (we have v10, it needs v20) that is automatically available if you open your codeblocks CLI. (no need to be inside that CLI environment). Alternatively you can  put libstdc++.so.6 somewhere in your lib path. or relink the symlink.

Compilation Procedure for in the future (pretty straight forward):

1. Download the LZMA SDK (not sure if it was needed): http://sourceforge.net/projects/sevenzip/files/LZMA%20SDK/

note: it unpacks NOT to a subdirectory, make sure you are in an empty dir first.

2. Download UCL form http://www.oberhumer.com/opensource/ucl/

3. Download UPX sources from http://upx.sourceforge.net/

4. Open up CodeBlocks CLI

5 Compile UCL:

  •  ./configure
  • make
  • (make install I think is not needed)
6. LZMA is source only, no need to compile, but UPX needs its location, define these variables:

  • UPX_LZMADIR=/media/<sdcard>/CODE/lzma  (that "empty directory before unpacking it")
  • UPX_LZMA_VERSION=0x918
  • UPX_UCLDIR=/media/<sdcard>/CODE/ucl-1.03  (current version)
7. Now compile UPX

  • make
  • make all (not sure if this is needed, it took a while to figure out that I needed to look for the *.out file)
8. In the src there is an upx.out ready for use. (I renamed it to /usr/bin/upx to have it ready for use)

The first thing I did was compress the upx binary itself, it shrunk to 22.7% of its original size. The next, Firefox (why is there a firefox AND a firefox.bin? 2 identical files in the pnd!)reduced to 52%

And yes... still runs... :p next to its other instance... with _wb_'s Sysinfo, they are close, but because one has a cache, the original uses more ram (the other creates an empty profile).

edit: HASE: hase 50.74% , libsparrow3d.so 36.37%... that is more than a meg less space it uses....
 

Attachments

  • upx-armel-bin.tar.gz
    357.6 KB · Views: 193
Last edited by a moderator:
In this day of age of SD cards hitting hundreds of gigabytes and most people having at least a megabit of download speed, I strongly recommend against executable compression (or even stripping them) unless you really need it (like participating in 4K/64K demo competition or something).

From OS maintainer's point of view it will only make it harder to maintain compatibility with such PND or add some compatibility layer to Pyra (for example) because most tools won't show useful info on compressed binaries.
 
Wouldn't any gains be offset by less compression in the squashfs image anyway?  I wouldn't expect this to have much impact on PND size if the PND is already compressed.

- Neelix
 
As a dialup user, i like it.
 
Last edited by a moderator:
Wouldn't any gains be offset by less compression in the squashfs image anyway
Squashfs has compression settings for fast decompression, which compresses a little less well than upx (but they use both LZMA, so I assume their compression ratios are very similar). So, yes, your reasoning is valid, but there is still some space to gain, not much though.

Unfortunately, the down side is higher CPU usage during decompression (that of Squashfs AND of UPX), thus more draining of the battery. nothing is "free" my friend.

because most tools won't show useful info on compressed binaries.
'comon Notaz. "file" still gives info. And if you mean linking and such, that should still work.

Let me reverse your logic: I think removing debugging symbols in .so makes them faster, but not debuggable...so you either get a faster game... or a game that is easier to debug.

So stripped .so's wont show useful info and make it harder for a maintainer.... Same argument...  besides, UPX is reversible... its just a packer/depacker... it is like flac... but it is also like mp3 at 128bits... you can go overboard and kill the binary...

As a dialup user, i like it.
Thanks for the thumbs up. Unfortunately, most big games are made up of highres texture images and hifi sound... the binaries are often not that big. But for libreoffice, and firefox, those are good candidates.
 
Last edited by a moderator:
because most tools won't show useful info on compressed binaries.
'comon Notaz. "file" still gives info. And if you mean linking and such, that should still work.
Work yes, have easily readable useful info (like full list of function dependencies) - no.
Let me reverse your logic: I think removing debugging symbols in .so makes them faster
Wrong. Debug information is not even loaded to memory in normal use, only debugger reads it when it needs it. Stripping does not make anything faster.
but not debuggable...so you either get a faster game... or a game that is easier to debug.
Nope, you only lose ability to debug.
besides, UPX is reversible... its just a packer/depacker... it is like flac... but it is also like mp3 at 128bits... you can go overboard and kill the binary...
Still a complete waste of time - my time if I have to debug your PND, startup time and memory (small but still a waste). Except for a few people like Elw3, who are more or less without Internet anyway, it is a lose-lose situation.
 
Last edited by a moderator:
@Notaz:


We are talking about two different things:
Debug symbols and compiler optimization.
If you use the strongest optimization settings (-O3), you will of lose symbols that are useful in debugging.
If you need debugging symbols, you gain them at the expense of losing compiler optimizations, hence speed.
Can't have both, afaik.

You are stating that you are afraid that others will start using this and you will waste time debugging their PND's. I am glad you are passionate about things. Just don't make ik personal.
If you have some spare time, read this:
http://blog.bassemdy.com/2015/04/21/problem-solving-in-the-corporate-world/
And understand that you are blocking. Which is not a good approach at the current stated problem.

Anyway, once the Nand starts failing bad, and I want the distro to use less space, Ill be using UPX. Others may want to use symbolic links to the SD card.

And as stated. At the moment I do not know if it is better to upx AND squashfs. Having the same algorithm, a recompression might do no good. Perhaps using a bigger lookup table for squashfs is better... who knows. Need to test things out first.
 

Oh, and Notaz: I still love you man!  #nohomo.
 
If you use the strongest optimization settings (-O3), you will of lose symbols that are useful in debugging.


If you need debugging symbols, you gain them at the expense of losing compiler optimizations, hence speed.


Can't have both, afaik.
Wrong again. With -O3 you still get all the symbols, but you lose 1:1 mapping to them because the optimizer is throwing some of your code out or morphing it to something else. When you run strip or gcc -s, then the symbols are gone for good. What I was saying is that people should avoid running strip on their binaries, using -O3 is fine and still leaves heaps of useful debug info.


There are of course exceptions for huge programs like firefox where debug symbols may take hundreds of megabytes, then it's reasonable to strip them.

I don't think this applies to our niche community, we are not trying to make money here or take over the world.

And understand that you are blocking. Which is not a good approach at the current stated problem.
Which is? If we were targeting GP32 which has 32MB (or similar, I forget now) limit on it's cards, then using upx would be really great. As it is now it's just a waste of time.

Anyway, once the Nand starts failing bad, and I want the distro to use less space, Ill be using UPX.
Sure fine, but I'm quite sure there are lower hanging fruits there.
 
Last edited by a moderator:
In my experience, a lot of compressed exectuables pop up in virus detection as some "generic dropper", which is never helpful
 
edit: HASE: hase 50.74% , libsparrow3d.so 36.37%... that is more than a meg less space it uses....
Easy explained: Sparrow3d has lot of very similar code, which differs only in details: alpha test on or off, z test an or off and so on. It is faster to have a big library for different usecases than making the library small e.g. for cache hit optimization.
 
Back
Top