The New Pxml Format

[skeezix]

Redid the sample apps in apps/ app2/ and pndsample/ so they conform to the new PXML, so discotest and pndnotifyd actually do useful things with em

Now I just need to add pnd_mount and pnd_unmount for cpas, so he can pull preview pics from pnd files, and we're gold. (Lots of little things to add, but this is all pretty usable imho.)

Need to re-test the hell out of things now that we've got such a 'big' change t the pxml in there (and yes, we have a messy API now, but we can trim it down.)

Thanks mate,

jeff

 

[parag0n]

Well, yeah, I know (from experience) that premature optimization is the root of all evil and all that.
I was more referring to the actual problems with the current algorithm.

For example, if your PXML has the languages "en_GB" and "de_DE" in it, and you ask libpnd to give the "en_US" description, it will return the "de_DE" description instead of the "en_GB" one.
I could figure out how to do it but it would be a suboptimal solution (but I will if the situation requires it and I don't have so many other things to do).

split at the _, then do the distance thing on the first part, then any matching second parts?

 

[dflemstr]

Hey, good to hear!
Sorry if I didn't do alot of testing; there was no obvious way to unit-test things (I made my own raw unit test as you can see) and as I run the daemon, everyhting seemed to work, at least with gdb.

And I haven't done professional C/C++ in ages, so Java has swamped my brain a bit, but yeah. Hope it isn't too horrible.

BTW: Hey, you do really like merging with patches, dont you (or did you merge with git and then rebase)?
Oh well, it's not like I will contribute with something else in the near future, so there probably won't be any problems.

@parag0n yes, but as it stands now, the language can be specified like "en" or "en_US" or anything (both in the file, and the app can ask for the lang in both formats). How do you match things, then? And if you want a "en"-language, and there is none available, which of the others do you pick?
The algorithm I built in is far from ideal and buggy like hell, but it's something. I didn't think it was worth committing my "improvements", since they had other problems.

 

[skeezix]

dflem -- you might contribute more in the future no? As Sean Connery said so long ago "never say never"

Theres probably a bunch of segfaults lurking in your code and in the whole base, but as long as we can test it out we'll find 'em.

I merged by hand, as I wanted to see bit by bit whats changed*, so I could move it to fit or alter it to fit style, etc.. in the end I didn't have to massage it much as you see, just copied it verbatim in, but moved a few files around and so on. Auto-merges can be risky, and I'm keeping a tight eye on this stuff for now, so did it by hand. (Now, if it was a modified sert of 50 files, automerge .. 3 files, I can deal with

I'll add a few more pnd-file manipulators, and then cpas will be re-pulling to sync up with the latets libpnd for his launcher, and then we'll see it all in action.

jeff

I'm not sure what the language picking issue could be, but maybe we should just hard-define the language list to root languages. "en" is better than "en_GB" in practice Iimagine.. few woudl differentiate between en_CAN en_US en_GB and en_AFRICAN (however their iso encoding is.) Just 'en' maybe, by standard. *shrug*

* I don't mean to act as a gatekeeper, but I guess I am right now, to try and keep it all together and timed to fit with what else is going on and so on; once the device ships (or before that?..) I'm going to just step back and not worry about it. The more hands the merrier, ajnd it is open source and community stuff.. let everyone else fix it/improve it.. there is a lot to do

 

[dflemstr]

Nah, I don't have any problem with manual merges. You have every right to act as a gatekeeper; to go wild on a beta piece of software is just doomed to fail. I'ts just that now my clone of the repo is incompatible (I can't get your changes that you did while merging) and I have to make a new clone of the official one (which might be a good thing, after all; but I'd lose my branches and revisions, metadata etc), or alternatively merge your tree with mine

And, yea, I might do more stuff; just not in the near future probably. Because if I were to change things right now, Git would get really confused when merging since our repos are out-of-sync. This problem solves itself most probably, if i remove my clone and clone your tree again (which is what I will do, most probably...) but yeah... I don't know what needs doing at the moment, really (except making a new ISO language algorithm and making some functions of pnd_pxml_* return other things than strings, such as ints etc, to gain better compatibility)

EDIT:
CODE

mv pandora-libraries pandora-libraries.old
git clone git://git.openpandora.org/pandora-libraries.git pandora-libraries
cd pandora-libraries
git remote add github git@github.com:dflemstr/pandora-libraries.git
git push github master

 

[Cpasjuste]

Thanks you for your help dflemstr, it's very appreciated
I must also thanks, of course, skeezix and vimacs who past a lot of time on libpnd (I should also thanks notaz and djwillis .. but it's another story).

All seems to be going like we wanted, so we should soon be able to show some cool demo of the pandora launching ".pnd" packaged applications, and i must say it's really cool. Just put the single file on your SD, insert it and voila. It will also be very easy to write a "pnd package manager" with all that work done.

Of course we still have a lot of work to do for the pandora but it's really going in the right way, i'm very excited

Thanks again guys.

 

[dflemstr]

Hm, OK, since I am the topic starter, am I allowed to hijack this thread? (if not, and you, the reader, are a mod, feel free to spilt this topic)

Was just generally wondering how the package management is going to work.
PND-files are easy to understand how they work, and it is as you say easy to make a package manager for downloading them.

(Warning: these are alot of questions. Feel free to treat some of the more redundant ones as rhetorical)

But what about core updates? Will they happen automatically through angstrom packages, and have a nifty update-system so that you don't have to care about them? Will they preserve any settings you modified on the NAND in conf files?
Will it be possible to (easily) overlay the NAND with another partition on, say, a SD-card, and have the core-updates write themselves to it (so as to save on NAND writes)?

Will it be possible to install PND-files to the NAND? (I know that they can be extracted so this question is kinda dumb) Will it be possible to treat PND-files exactly like core updates and have them install like normal packages?

What about other distros, is there a repository for Ubuntu, etc? A repo for PND-files?(except the GP32x Archive, which is kind of bad as a repository; maybe the Archive could serve as a mirror?)

etc etc...
Package management is important to me

 

[Cpasjuste]

I will try to remember what was said so i may say some mistake.


I think we dont want anything on the NAND except our core distro. All applications/configurations files should leave on the SD card with the application, which is handled by libpnd/unionfs.

There will be sometime, a package which can be installed from SD card which will update the NAND core dirsto, either with some small update if critical, or which will replace all the system including user configurations. This should not happen too much.

Overlay the NAND for installaing core update wont do the trick, since the system could be broke if you remove/replace your SD, and it's the same problem for a "pnd package manager". By package manager i should have said a package downloader. But of course you will be able to do what you want finally, if you want some partition to be on the SD you will be able to do it by yourself, same for the whole system (pandora one or any distro) which can be booted from nand. It's linux after all.

Other distro are not planned officially, but a lot of people will work on that when the pandora gets released.

Finally, gp32x archive will be a pnd repo, but of course we could (and we should) work on a system to have some up-to-date mirror, updating themselve from a master server, serving file via http and for exemple, with a mysql database accessible by any application that could be used to download pnd games/app from the application.

I repeat, i may be wrong on some point since we didnt talk about all that stuff since a while, we are currently working hard on the kernel/distro/games support, the rest will be easier.

 

[Vorporeal]

There are ways to have something written to the SD as an update to the data on the NAND - there are things like UnionFS which will take two partitions and make them appear like one (you can probably have it take all the conflicts and have it use the versions on the SD instead of the NAND). As Cpasjuste said, this could cause problems if you boot without the SD (you might revert back to the non-unified, totally base, original distro or you might not be able to turn on the Pandora if it complains enough). So yeah, there are ways of doing that, but you probably don't want to.

 

[dflemstr]

Ok, thank you very much for the information
Yes, I know of UnionFS, was just wondering if there would be some built-in interface for doing all this or if I would have to edit fstab myself (which should be easy peasy, when thinking about it)

Not many core updates, you say? So we won't keep up with the kernel updates? Pity

And what about core libs? We don't want to compile everything statically. What if the community wants to include a library in the core so that everyone has it (and can dynlink it), would that be possible through an update? (I guess that's what the <osversion>-tag is for in the PXML files...)

 

[Cpasjuste]

Ok, thank you very much for the information
Yes, I know of UnionFS, was just wondering if there would be some built-in interface for doing all this or if I would have to edit fstab myself (which should be easy peasy, when thinking about it)

Not many core updates, you say? So we won't keep up with the kernel updates? Pity

And what about core libs? We don't want to compile everything statically. What if the community wants to include a library in the core so that everyone has it (and can dynlink it), would that be possible through an update? (I guess that's what the <osversion>-tag is for in the PXML files...)

I didn't said there wont be much update, i wanted to said that i think not a lot of full NAND erase/update should happen.

For the libraries, a lot will be included by default. I think that each time we exec a game/app, the pandora librarie will add the game/app path to the ld preload path, so you should just need to put your special libraries with your application. There is a second advantage with this, even if the dev don't update his application, the user will still be able to update the game/app libraries.

We will also maybe add a default libraries path on the SD, maybe it's already planned, skeezix and vimacs are working on this sort of stuff.

 

[dflemstr]

Yeah, ok, it all sounds very well planned.
I'm just worried about that developers will have to include a full python interpreter with every python app they write, just because PND-files can't handle dependencies (and the core doesn't have an interpreter inclued(ok, python is a bad example, but let's instead say Java, for example)).

A feature of the PXML format would be that the "unique ID" could be used to import stuff from other PNDs, like, say, if I needed libpulseaudio and the java core for my app, I would write the following in my ID:
"dflemstr.myapp.2:someone.libpulseaudio.3:someuser.java.core.1"

...it would be possible, and then everything wouldn't have to be so big.

 

[EvilDragon]

Well, all important libraries and runtimes should be included in the core system on the NAND, so you won't need a python interpreter each time as it is already there within the system

 

[dflemstr]

Well, all important libraries and runtimes should be included in the core system on the NAND, so you won't need a python interpreter each time as it is already there within the system

Yes, well, ok, but which are these libraries?
Say that I wanted to have "libxml.so.1" because my application wants to use XML. How do I know if it will be there? Am I guaranteed that the library will be there in all future versions of the NAND-image? What if a new version of libxml comes out, "libxml.so.2"? Will both versions be included? etc... etc...

Maybe I just worry too much

 

[DasFool]

Well, all important libraries and runtimes should be included in the core system on the NAND, so you won't need a python interpreter each time as it is already there within the system

Awesome, so Python is already included? Do you know if anyone is working on Pygame yet?

 

[dflemstr]

Pygame is just SDL + a python compatibility library + a layer of awesomeness, so it should be ready to run already.
Question is if it will be included or not (aka see my previous post).

 

[Cpasjuste]

Yes, all the libraries you should need are already there, including libxml

If you really need to use a specific version of a library, you'll have to include it with your app (which should not happen a lot, i'v ported a lot of thing without the need of other libs). For now i see if a library is missing when i run my program

 

[Butterman]

Bah, I should really read.

 

[dflemstr]

A list somewhere with all the libraries that are included (including Python modules/eggs and Java JARs) would be very VERY helpful

And if you REALLY want to, is it possible to make a PND with, for example, lwjgl in it (which I guess won't be included in the core), and access that PND from another PND? (so that not every PND is huge because they have to include lwjgl)

 

[Username]

All this makes me think, if the programs that come with it (Firefox, Gimp) are stored on the NAND, will all of their settings be stored on the NAND? And will you be able to save any new settings?