Beta (De)Compressors

[joseluisjazz]

I compiled unrar (nonfree) and p7zip for the Pandora and I thought it would be a good idea to create a thread to test them and the rest of the (de)compressors as they keep rolling out. So here they are.

UPDATED [post='902417']2010-08-08[/post]: Added UPX & unace.


upx (The Ultimate Packer for eXecutables)

Note 1:
UPX is an advanced executable file compressor. UPX will typically reduce the file size of programs and DLLs by around 50%-70%, thus reducing disk space, network load times, download times and other distribution and storage costs.

Programs and libraries compressed by UPX are completely self-contained and run exactly as before, with no runtime or memory penalty for most of the supported formats.

Note 2:
The precompiled UPX versions are linked against the NRV compression library instead of the UCL library. Using the same compression algorithms, NRV achieves a better compression ratio. NRV is not publicly available, though, and probably never will be.

Note 3:
This is the version for using from the Pandora. You could download binaries for other platforms that use the NRV library and compress the packages there. This would be the best option if Openpandora decides to compress the firmware ipks and repackage them again, in order to save space in the NAND.

• Version: 3.05. Source code downloaded from upx.sourceforge.net
• Can pack: Linux Armel Executables.
• Adjustements to the source tree: None.
• Binaries included: upx.out
• Compilation options:-Os -mcpu=cortex-a8 -mfpu=neon -ftree-vectorize -mfloat-abi=softfp -ffast-math -fsingle-precision-constant
• Binary download: upx-3.05-Os-upx-pandora.tar.gz (82 KiB)
• Installation to root:

  $ sudo tar xzvf upx-3.05-Os-upx-pandora.tar.gz -C /

unrar

• Version: 3.9.10. Source code downloaded from www.rarlab.com
• Can unpack: RAR. (more? check)
• Adjustements to the source tree: None.
• Binaries included: unrar
• Compilation options:-O3 -mcpu=cortex-a8 -mfpu=neon -ftree-vectorize -mfloat-abi=softfp -ffast-math -fsingle-precision-constant
• Binary download: unrar-3.9.10-Os-upx-pandora.tar.gz (82 KiB)
• Installation to root:

  $ sudo tar xzvf unrar-3.9.10-Os-upx-pandora.tar.gz -C /

b7zip

• Version: 9.13. Source code downloaded from p7zip.sourceforge.net
• Can pack: 7z, ZIP, GZIP, BZIP2 and TAR.
• Can unpack: 7z, ZIP, GZIP, BZIP2, TAR, ARJ, CAB, CHM, CPIO, DEB, DMG, HFS, ISO, LZH, LZMA, MSI, NSIS, RAR, RPM, UDF, WIM, XAR, Z.
• Adjustements to the source tree: Personalized makefile for pandora system.
• Binaries included: 7za, 7z (and its plugins), 7zr, 7zCon.sfx
• Compilation options: Made 5 optimization variants by only changing the "OPTFLAGS" line in the makefile from -00 -O1 -O2 -O3 to -Os. The results vary in speed & binary size. I found this interesting in order to have a model for comparison, and a better perspective on the impact of optimizations for the Pandora.
• I've run integrity tests: Yes.
• Benchmarks: p7zip has a benchmark command-line parameter, so I made some benchmarks tests at 500MHz and at 800MHz for each version. Also here is the filesize list of the binaries.
• Benchmark summary (total avg. MIPS rating at 500MHz): -O0:124 -O1:259 -O2:288 -O3:291 -Os:299
• Benchmark summary (total avg. MIPS rating at 800MHz): -O0:194 -O1:398 -O2:439 -O3:442 -Os:456
• File size summary (in bytes): -O0:5932186 -O1:4309758 -O2:4290074 -O3:5172678 -Os:3196830.
• Binary download (only best variant): p7zip_9.13-Os-upx-pandora.tar.gz (1.24 MiB)
• Installation to root:

  $ sudo tar xvf p7zip_9.13-Os-upx-pandora.tar.gz -C /

• Choice: In this case is pretty clear. -Os and -O3 versions are equally fast for packing but -Os version is noticeably faster for unpacking, better average and it's the smallest one in size by far so that's the best choice.

Ancient or arcane pieces of software (the museum part) ---------------------------------------

unace

• Version: 1.2b. Source code downloaded from debian.org
• Can unpack: ACE. (version 1 only)
• Adjustements to the source tree: None.
• Binaries included: unace
• Compilation options:-Os -mcpu=cortex-a8 -mfpu=neon -ftree-vectorize -mfloat-abi=softfp -ffast-math -fsingle-precision-constant
• Binary download: unace-1.2b-Os-upx-pandora.tar.gz (X KiB)
• Installation to root:

  $ sudo tar xzvf unace-1.2b-Os-upx-pandora.tar.gz -C /

Miscellaneus:

Small reading about Options That Control Optimization in gcc.

EDIT 20101027: fixed links

 

[Pleng]

Might be worth alerting ED to this so they can be included in HF4?

 

[joseluisjazz]

I think you're right Pleng, it would be useful to do that.

On a side note, I didn't compile the p7zip GUI since it uses WxWidgets, and that library is not (yet) available in the pandora. But that's another software I wanna try to port.

 

[fusion_power]

Might be worth alerting ED to this so they can be included in HF4?

+1
I would like to have File archivers onto the Pandora but I'm not that Linux-terminal-guy so a PND or better directly included into the Firmware would be very userfriendly. Do this programs come with a normal GUI or is packing/unpacking still a Terminal-operation?

 

[sebt3]

Do this programs come with a normal GUI or is packing/unpacking still a Terminal-operation?

Command-line driven.

 

[joseluisjazz]

Following unix tools philosophy, they are used from terminal, but they can be called too from other programs, like GUIs.

But later today I maybe bring a surprise in the form of a pretty GUI. Stay on tune.

Ninja'd!

 

[Hitnrun]

I think you're right Pleng, it would be useful to do that.

On a side note, I didn't compile the p7zip GUI since it uses WxWidgets, and that library is not (yet) available in the pandora. But that's another software I wanna try to port.

I successfully compiled wxWidgets and my wxWidgets application for the Pandora, if you feel that this GUI is interesting enough to make a PND I can give you the directions on how to do it.

 

[Pleng]

I would like to have File archivers onto the Pandora but I'm not that Linux-terminal-guy so a PND or better directly included into the Firmware would be very userfriendly. Do this programs come with a normal GUI or is packing/unpacking still a Terminal-operation?

I believe once unrar is installed, Pandora's built in archiving gui (squeeze), is able to decompress RAR files.

 

[Prometheus]

I'm pretty sure that Pleng is correct. To anecdotally back that up, this is what it also does on my main computer, which is running Xubuntu 10.04 - I had to add support for RAR and 7z to it recently, and once I did, Squeeze was immediately able to work with those formats.

 

[milkshake]

I would like to have File archivers onto the Pandora but I'm not that Linux-terminal-guy so a PND or better directly included into the Firmware would be very userfriendly. Do this programs come with a normal GUI or is packing/unpacking still a Terminal-operation?

I believe once unrar is installed, Pandora's built in archiving gui (squeeze), is able to decompress RAR files.

this is correct see here...
http://www.gp32x.de/board/index.php?/topic/55730-unrar-not-available/

 

[joseluisjazz]

good news everyone!

I have a grave announcement. Anyone with a weak heart had better leave now. Goodbye.

(...)

Uhm... where was I? Oh yes. I've ported the very ancient and very pointles unace decompressor version 1.2b!! No, wait a moment.. Those were the crappy news!!

The good news are that I've also ported the fabulous, the fantastic nand-precious-space-saver UPX, hurrai!!!

What? You don't know what that is? Well, it is a compressor, but not any compressor. It compresses ELF ARMEL executables and libraries, and it does it so well that I'm completely blown. The best part is that you can use those executables compressed by UPX normally without noticing any performance loss! And the other best part is that the size is reduced about a 50% and sometimes much more !


So to I made some tests and all the binaries worked perfectly in all cases. I give you two examples. First the mplayer binary (8.5MB size)

With the fastest compression option (1):

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   8874752 ->   4493556   50.63%   linux/armel   mplayer

With the best compression option (9):

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   8874752 ->   3946540   44.47%   linux/armel   mplayer

After those promising results I needed to defy the universe by compressing the very upx binary with itself! And I did it comparing the ratio on the normal version and the stripped version. Here are the results:

$ cp /usr/bin/upx.out upx.out-original
$ cp /usr/bin/upx.out upx.out-stripped
$ strip upx.out-stripped
$ upx.out -9 upx.out-*
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1604395 ->    375108   23.38%   linux/armel   upx.out-original              
   1420944 ->    323412   22.76%   linux/armel   upx.out-stripped              
   --------------------   ------   -----------   -----------

Fantastic!

After these test and more, I know that from now on I'll compile everything for the Pandora with -Os, then strip it and finally compress it with UPX.

Here you can download it: upx-3.05-Os-upx-pandora.tar.gz (bin & docs). Test it and tell me your opinion.

Now I'll go make some benchmarks and then update the first post

 

[Pleng]

After some extensive testing it might be a good idea to include compressed apps in the firmware image. Could give us, what, another 100Mb or so to play with?

 

[trevormacro]

Hello,
I have pandora.
I install upx ( upx-3.05-Os-upx-pandora.tar.gz) and p7zip ( p7zip_9.13-Os-upx-pandora.tar.bz2 ) from links.
I want to extract my files in sd directory " 7z files".
What is the command line to extract my files please ?
thanks

 

[fettouhi]

p7zip link is dead.

 

[mcobit]

Use xarchiver!

 

[joseluisjazz]

fixed the p7zip link

 

[mcobit]

Hi, I am working on file-roller as a new gui for packaged files. XArchiver is relatively outdated and can't even use p7zip.
Can I incorporate your binaries into the pnd so the program can use them, when starting?
And one more question: Do they have to be installed to /usr/bin to start or can they be used from everywhere?

Thanks!

 

[joseluisjazz]

yes of course, you can use those binaries or compile them yourself as I did. It's easy.
And I believe they can be used from everywhere, if they are on path, but you'll just have to try it.

regards

 

[mcobit]

Thanks, everyone works, only 7z misses its libraries in /usr/bin so i will compile this myself.

Well, I ran into other issues with gnomeui. For now we have to live with xarchiver