PND criticism

[craigix]

An IPA file is effectively a zip file with a very specific file structure. Data and binaries must go in preset directory names. It also contains an info.plist file which is basically an XML file containing all the information for the application.


That IPA file is then mounted as a filesystem. This can also be done in Linux (as well as Windows and on Mac) but it is not as flawless as Craig says. It looks flawless to the end user, but it is actually quite a bit slower than it needs to be: ZIP files were not designed with random access of files in mind, and file lookups need to be performed sequentially through the archive. They chose ZIP because it was easier on the developer, not because it was better for the end user.


A better solution would be to use squashFS, or similar compressed filesystem designed for random access. To the end user, it would have functioned basically the same, with the added bonus of file reads being slightly faster. The developers would have required a program other than some zip program, adding a small amount of additional complexity, but that's easy enough to come by, provided you have no problems with open source.


So, to summarize, Apple is using a compressed archive format (ZIP) with a very specific directory structure, containing a single XML file with basic instructions on how that application is run.


PND uses a superior compressed archive format (squashFS) with no limitations on directory structure, except that it must contain a single XML file with basic instructions on how that application is run. As a bonus, that XML is also appended to the end of the archive for even faster access: no need to scan the archive and decompress it.


Apple's IPA format is loaded as a filesystem and executes binaries within it, but to the end user looks like a single file.


Pandora's PND format is loaded as a filesystem and executes binaries within it, but to the end user looks like a single file.


tl;dr: PND is IPA with a few extra features.

Thanks for clearing that up, I personally wish the Pandora had used Zip as everyone can make/open/edit zip files and it's an established format. (keep it simple and all that).


I'm also sad that people instantly assumed I must be wrong about the iPhone using zip files, even though they know I develop for it, and the iphone Zip format is now probably the most popular way of handling this in the world.


Does the PND format compress as well as Zip files?

 

[ivanovic]

Thanks for clearing that up, I personally wish the Pandora had used Zip as everyone can make/open/edit zip files and it's an established format. (keep it simple and all that).


I'm also sad that people instantly assumed I must be wrong about the iPhone using zip files, even though they know I develop for it, and the iphone Zip format is now probably the most popular way of handling this in the world.


Does the PND format compress as well as Zip files?

Since plain zip compression is rather bad it does compress even better than plain zip.


Just as a sample:


I created the wesnoth pnd using plain pnd_make.sh with the compress param, so that a compressed squashfs is created. This is 310MB large for the 1.9.6 release. Packaging the same data (minus the appended PXML and icon) as a zip using the command zip -9 -r wesnoth-1.9.6.zip wesnoth-1.9.6 I get a zipfile which is 319MB large. Since Wesnoth consists of mixed content, meaning images, as well as music files (ogg-vorbis) and text files it probably shows some average compression over generic pandora applications. So a compressed squashfs saves about 3% space compared to a zip with highest compression rate.


The default compression used in mksquashfs is "gzip" btw, so it is rather close to "normal" zip. In theory xz compression would be possible, too, but I don't know if the pandora ships new enough tools to really accept this. Using xz it should be possible to gain some more space, though it would cost CPU time.


And yeah, this would still be better than what you ship for the iphone if plain zip is used there. You basically get faster access to the files compared to zip and have at least the same compression. The only "downside" is that the devs have to install the squashfs tools to package stuff this way. Those users that want to extract the content of a pnd can also do so using the squashfs tools by calling "unsquashfs". Honestly, this is not harder than using a zip program, so no real difference in convenience, right?

 

[craigix]

Thanks for clearing that up, I personally wish the Pandora had used Zip as everyone can make/open/edit zip files and it's an established format. (keep it simple and all that).


I'm also sad that people instantly assumed I must be wrong about the iPhone using zip files, even though they know I develop for it, and the iphone Zip format is now probably the most popular way of handling this in the world.


Does the PND format compress as well as Zip files?

Since plain zip compression is rather bad it does compress even better than plain zip.


Just as a sample:


I created the wesnoth pnd using plain pnd_make.sh with the compress param, so that a compressed squashfs is created. This is 310MB large for the 1.9.6 release. Packaging the same data (minus the appended PXML and icon) as a zip using the command zip -9 -r wesnoth-1.9.6.zip wesnoth-1.9.6 I get a zipfile which is 319MB large. Since Wesnoth consists of mixed content, meaning images, as well as music files (ogg-vorbis) and text files it probably shows some average compression over generic pandora applications. So a compressed squashfs saves about 3% space compared to a zip with highest compression rate.


The default compression used in mksquashfs is "gzip" btw, so it is rather close to "normal" zip. In theory xz compression would be possible, too, but I don't know if the pandora ships new enough tools to really accept this. Using xz it should be possible to gain some more space, though it would cost CPU time.


And yeah, this would still be better than what you ship for the iphone if plain zip is used there. You basically get faster access to the files compared to zip and have at least the same compression. The only "downside" is that the devs have to install the squashfs tools to package stuff this way. Those users that want to extract the content of a pnd can also do so using the squashfs tools by calling "unsquashfs". Honestly, this is not harder than using a zip program, so no real difference in convenience, right?

Take the dev hat off. This does not make it easy for users.


Zip would have worked, not required any extra work, and everyone would have been able to open and edit them. That's why it works so well on iPhone.


I now see I'm never going to get my ideas for ease of use being the most important thing on the Pandora across. There is too much resistance to it, I predict this won't go anywhere good.

 

[Bosbeetle]

What does opening up a PND have to do with ease of use, i have no clue how to do it but i don't need to open it up :blink: can't think of any reason i would want to look 'inside' a pnd.


And Ive never had a devhat

 

[notaz]

Take the dev hat off. This does not make it easy for users.

How come? Please explain. You want to imply iPhone users often modify their app packages?

 

[WizardStan]

Take the dev hat off. This does not make it easy for users.

Let's find out.


-------------------


*dev hat off*


I'm a user. I don't care what format it is in so long as it just works. PND just works. ZIP would have also just worked. Doesn't matter to me which is used.


*dev hat on*


I'm a dev. It matters to me what format is used. I want the most suitable compression format available. SquashFS and ZIP both compress files about the same, but SquashFS is specifically designed for random file access so will actually perform better than regular zip. It's a little harder on us devs, but we've got to do what's right for the end user.


*dev hat off*


Wait, so this SquashFS is better than ZIP?


*dev hat on*


Yup. It's faster for sure, and anecdotally makes smaller files.


*dev hat off*


But to me, Johnny Apple the end user, it doesn't matter what you choose because they'll both function the exact same way?


*dev hat on*


That's correct. No matter what format we choose, to you it will still be one file that simply works. The only difference between the file formats is which one is fastest. That's why we spent so much time debating what formats to use, so you the end user could get the best experience we could manage.


*dev hat off*


Wow! Thanks for thinking about these underground things that I would never have been aware of.


-------------------


Looks like it doesn't not make things easy for the user. In fact, the user appreciates the devs going to the hard work of handling all the stuff they normally wouldn't see so that the end experience is even better.

I now see I'm never going to get my ideas for ease of use being the most important thing on the Pandora across.

Is it possible that you can't get your idea across because your idea is actually wrong? The end user doesn't care what format the file is in, so long as it works. If Apple had chosen some custom compression format instead of ZIP, it would have made absolutely zero difference to the end user. I honestly don't know why they picked ZIP since it is not the best tool for the job, but it must have had some merit they decided was worth it. Maybe the ability for anyone to just be able to create or unpack on a regular computer was part of their design goals, but that doesn't help the end user at all. How many end users want to open the file and see the inner workings? Not many, I would suspect. For the Pandora, anyone who is interested in the inner workings should not have any problems coming to the forum, asking, and being directed to the free SquashFS tools that exist for unpacking the file.


If you really, really want to use zip, despite the fact that it isn't the best choice, there is absolutely nothing stopping you. I can build the Fuse-ZIP filesystem, talk to ED about having it included in hotfix 6, then ZIP files will be mountable as well. That's the best part of PND: it doesn't restrict the developer. It lets the developer choose what he or she thinks is best (within reason, new file formats need to be added to the system first) but it all looks and behaves the same to the end user: one file that just runs, easy peasy.

 

[EvilDragon]

Well, seriously - how often has the standard iPhone user opened the zip-like archive they use?


For what reason? I cannot think of a single one.

 

[Asiyura]

I'm not a dev, well, I could if I had my Pandora

But I'm no tech guru (I mainly learnt to make java softwares).


And to be honest, I don't care which compression system is used in PND, as long as it's fast.

 

[ZXDunny]

If you really, really want to use zip, despite the fact that it isn't the best choice, there is absolutely nothing stopping you. I can build the Fuse-ZIP filesystem, talk to ED about having it included in hotfix 6, then ZIP files will be mountable as well. That's the best part of PND: it doesn't restrict the developer. It lets the developer choose what he or she thinks is best (within reason, new file formats need to be added to the system first) but it all looks and behaves the same to the end user: one file that just runs, easy peasy.

You know what really, really needs to be done? The same thing that .zip files have on larger PCs. This, I'm convinced, is what Craig wants. He wants to be able to click on a .zip file and have an app (like WinZIP or gui-Arc for example) list the files, so he can drag and drop stuff in and out. I use it all the time in windows (less so in linux, because linux is a pain) and it would be great to have some sort of window/linux/pandora client app that mimics winzip/winrar but for PND files.


PND files are hard to handle if you're not a top-flight linux developer. I can't work with them at all in windows, and if I can I suspect it's only with some really obscure command-line utils. There seems to be no "double click" facility to open them, inspect them, replace files within them... that sort of thing. PND is terribly confusing to me. I figured out how to mount them on the pandora and look at their contents, but I'm buggered if I can get them to re-pack with my own shit in there.


They're easy to download and run, that's simple. But work with them? Nah, I don't code in C, sorry.


D.

 

[Bosbeetle]

If you really, really want to use zip, despite the fact that it isn't the best choice, there is absolutely nothing stopping you. I can build the Fuse-ZIP filesystem, talk to ED about having it included in hotfix 6, then ZIP files will be mountable as well. That's the best part of PND: it doesn't restrict the developer. It lets the developer choose what he or she thinks is best (within reason, new file formats need to be added to the system first) but it all looks and behaves the same to the end user: one file that just runs, easy peasy.

You know what really, really needs to be done? The same thing that .zip files have on larger PCs. This, I'm convinced, is what Craig wants. He wants to be able to click on a .zip file and have an app (like WinZIP or gui-Arc for example) list the files, so he can drag and drop stuff in and out. I use it all the time in windows (less so in linux, because linux is a pain) and it would be great to have some sort of window/linux/pandora client app that mimics winzip/winrar but for PND files.


PND files are hard to handle if you're not a top-flight linux developer. I can't work with them at all in windows, and if I can I suspect it's only with some really obscure command-line utils. There seems to be no "double click" facility to open them, inspect them, replace files within them... that sort of thing. PND is terribly confusing to me. I figured out how to mount them on the pandora and look at their contents, but I'm buggered if I can get them to re-pack with my own shit in there.


They're easy to download and run, that's simple. But work with them? Nah, I don't code in C, sorry.


D.

Didnt somebody make that program? If not they should

 

[ZXDunny]

If you really, really want to use zip, despite the fact that it isn't the best choice, there is absolutely nothing stopping you. I can build the Fuse-ZIP filesystem, talk to ED about having it included in hotfix 6, then ZIP files will be mountable as well. That's the best part of PND: it doesn't restrict the developer. It lets the developer choose what he or she thinks is best (within reason, new file formats need to be added to the system first) but it all looks and behaves the same to the end user: one file that just runs, easy peasy.

You know what really, really needs to be done? The same thing that .zip files have on larger PCs. This, I'm convinced, is what Craig wants. He wants to be able to click on a .zip file and have an app (like WinZIP or gui-Arc for example) list the files, so he can drag and drop stuff in and out. I use it all the time in windows (less so in linux, because linux is a pain) and it would be great to have some sort of window/linux/pandora client app that mimics winzip/winrar but for PND files.


PND files are hard to handle if you're not a top-flight linux developer. I can't work with them at all in windows, and if I can I suspect it's only with some really obscure command-line utils. There seems to be no "double click" facility to open them, inspect them, replace files within them... that sort of thing. PND is terribly confusing to me. I figured out how to mount them on the pandora and look at their contents, but I'm buggered if I can get them to re-pack with my own shit in there.


They're easy to download and run, that's simple. But work with them? Nah, I don't code in C, sorry.


D.

Didnt somebody make that program?

I don't know - I tried double clicking on a PND just now on my windows box and it didn't do anything, just asked me what I wanted to open it with... Any suggestions as to what might work? It's a new install of Windows7.


D.

 

[Bosbeetle]

Didnt somebody make that program?

I don't know - I tried double clicking on a PND just now on my windows box and it didn't do anything, just asked me what I wanted to open it with... Any suggestions as to what might work? It's a new install of Windows7.


D.

Well i think I am mistaken but Im sure somebody could make it. I still don't know why you would want to.

 

[skeezix]

As noted -


Zip drivers suck righ now . They keep open files in RAM! So no useful for us .. Unless someone is writing drivers


Thanks for correction; I though ipa was uncompressed for performance; earlier prototypes actuallu unpack everything. Does it purely mount and run now? (I'd just assumed all that 'installing..' bit in appstore was unpacking.) they've rejectex 3 Pps of mine due to similar ones existing so I've given up on Apple appstore


Jeffphone

 

[Caine]

I don't know - I tried double clicking on a PND just now on my windows box and it didn't do anything, just asked me what I wanted to open it with... Any suggestions as to what might work? It's a new install of Windows7.


D.

I think most PNDs are using squashfs. A quick google of windows squashfs yields this site. I don't have a windows box to test it on though.

 

[WizardStan]

I don't know - I tried double clicking on a PND just now on my windows box and it didn't do anything, just asked me what I wanted to open it with... Any suggestions as to what might work? It's a new install of Windows7.

I've found these


Command line tools, unfortunately, but they let you pack and unpack a squashFS PND file.


You can also make PND files with ISO, which can be done with winrar, I think. If not, free CD burning software can make ISOs. Not quite as easy in Windows as just using ZIP, but squashFS is the superior format for a lot of reasons. I'll keep an eye out and see if I can't find a GUI that handles it. If nothing else, maybe I can come up with a crappy interface that just wraps these command line tools. And believe me, it would be crappy, I suck at GUI development.


edit: actually, I don't think those will work. They're two years old, and I'm pretty sure we're using a more recent version. Use these ones instead.

 

[Caine]

According to the website, 7-zip has support for both ISO and squashfs. It was my preferred compression tool when I still used windows and I highly recommend it.

 

[ZXDunny]

I don't know - I tried double clicking on a PND just now on my windows box and it didn't do anything, just asked me what I wanted to open it with... Any suggestions as to what might work? It's a new install of Windows7.

I've found these


Command line tools, unfortunately, but they let you pack and unpack a squashFS PND file.


You can also make PND files with ISO, which can be done with winrar, I think. If not, free CD burning software can make ISOs. Not quite as easy in Windows as just using ZIP, but squashFS is the superior format for a lot of reasons. I'll keep an eye out and see if I can't find a GUI that handles it. If nothing else, maybe I can come up with a crappy interface that just wraps these command line tools. And believe me, it would be crappy, I suck at GUI development.


edit: actually, I don't think those will work. They're two years old, and I'm pretty sure we're using a more recent version. Use these ones instead.

My point was to highlight what Craig is on about - PND files are not as accessible as .ZIP files, which is what Craig wants. Yes, for the newbie there's no need to access the inside of a PND file (actually not true, I know I wanted to replace some crappy graphics files on a PND but that's almost impossible), but for the developer it would be amazingly handy not to have to use the command line - I don't use the command line to develop for the Pandora, I use a point-and-click IDE. In fact, I'd like to avoid the command line where possible as it's just a horrid interface.


Until everyone can access the internals of a PND file using drag-and-drop for both read and write operations, then the PND format will be forever too technical for most people to bother to try to understand.


D.

 

[ivanovic]

Yes, for the newbie there's no need to access the inside of a PND file (actually not true, I know I wanted to replace some crappy graphics files on a PND but that's almost impossible)

There is no need to enter the file itself if it is for local purposes. You can just "replace" files by "overloading" them from appdata. If you have a file that is eg called "run.sh" (which many pnds have!) you can use your own version of this by just placing the replacement file at the same position in appdata as it is inside the pnd itself. No need to alter the pnd itself to get changes applied for some quick&dirty testing!

 

[ZXDunny]

Yes, for the newbie there's no need to access the inside of a PND file (actually not true, I know I wanted to replace some crappy graphics files on a PND but that's almost impossible)

There is no need to enter the file itself if it is for local purposes. You can just "replace" files by "overloading" them from appdata. If you have a file that is eg called "run.sh" (which many pnds have!) you can use your own version of this by just placing the replacement file at the same position in appdata as it is inside the pnd itself. No need to alter the pnd itself to get changes applied for some quick&dirty testing!

But how do I know where to put them? How do I know which directory my improved graphics need to go in without opening up the PND file? it's all very well just popping them in appdata/PNDname, but which subdirectory should they go in?


This was one if the first things I wanted to do when I first got my Pandora. This community and OPT had already told me that I could do whatever I wanted with it, but when it came to doing what I wanted there was very little info on how to get it done, and no tools available on OPTs website to allow me to do what I wanted.


D.

 

[WizardStan]

But how do I know where to put them? How do I know which directory my improved graphics need to go in without opening up the PND file? it's all very well just popping them in appdata/PNDname, but which subdirectory should they go in?

7zip can read squashFS files no problem. You can see the structure, extract the originals you want, and edit them in your graphics tool of choice. Can't replace them, 7zip can't write squashFS, but as Ivanovic says, you can dump replacement files into appdata.


Does that help?