The New Pxml Format
- Thread starter dflemstr
- Start date
dflemstr
It's a ball.
'skeezix' said:Unique-ID format has not been hard defined yet, fwiw; maybe "project-dev-optional-123", ie:
MAME-Franxis-123
SquidgeSNES-Squidge-
(no versioning on that last one)
Anyway, we can worry about that later
Last edited by a moderator:
Squidge
Certified Guru
'Kramy' said:
Last edited by a moderator:
lulzfish
Pandora Defense Squad
Wait, it's used to access local data or something?
I'm confused now.
I'm confused now.
Last edited by a moderator:
urjaman
"I Know. We're going for a ride."
Yeah, i think (somebody correct me if wrong) that this ID is what enables the system to union the writable overlay for the application (if the app modifies itself or its configuration files the changes go there) with the filesystem in the .pnd (the original "virgin" application & data) ...'lulzfish' said:
Last edited by a moderator:
lulzfish
Pandora Defense Squad
Oh, so you can keep savegames and the like on the Pandora rather than the SD card? I suppose that makes sense.
urjaman
"I Know. We're going for a ride."
Found where i read that; see this: (long post, explained in the end) www.gp32x.de/board/index.php?s=&showtopic=46797&view=findpost&p=702956'lulzfish' said:
But yeah, I get it so that the point is that the .pnd is never modified so it's always original.
By default the writes will be to the same SD (on which the pnd is): www.gp32x.de/board/index.php?s=&showtopic=46797&view=findpost&p=703134
But as always with open source software - I think it wont be hard to change that to be to the NAND if you want it so.
Last edited by a moderator:
dflemstr
It's a ball.
How would that work? To create the hash, you would have to have a finished .pnd file, and to finish the .pnd file, you have to have a PXML in it with the hash'sindbad' said:
Anyways, it's too early imho to decide such things. The PXML seems to me to still be a WIP (there's a lot to do to add multi-app support and fix some internal bugs etc). This means that the PXML will likely break backwards-compatibility with my proposed standard (and also with the old v1.5 standard, if mine gets rejected), just because it's not a finished format yet.
If I were to decide, I wouldn't use an UID at all. I would just take the "author.name", "title" and "version" tags out of the PXML file and smush them together in a function in libpnd. Then the end user wouldn't have to care about generating a UID, but instead to think about chosing a good app name etc.
What do you think?
Ah, and the real reason of this post's existance:
I've put together a human-readable spec for the new format.
Get it while i'ts hot:
github.com/dflemstr/pandora-libraries/raw/75cc4064762e0ffee90ea6f85a37c240fea372d3/meta/pxml-info/PXML_spec.html
(You must download this. Open the link, then use your browser to save the page. Otherwise, your browser will probably show it as a text document, and not as a HTML file.)
Last edited by a moderator:
skeezix
Internal Development
We've discussed it endlessly internally, but I've always been pushign for 'reasonably unique'; you can do it the 'hard unique' way like Palm, where you hit a CGI and obtain a unique-id fromthe official DB. It can work, but it doesn't really help if someone copies the PXML.xml and forgets to fetch a new ID, etc.
The other school is.. consider java; no central authority, but you put your organizations domain name up front; poof, 'reasonably unique'.
Thats the angle I've been pushing; put a dev name or a app name, and some identifiers in there, and you're almost certainly guaranteed to be unique, unless someone wants to make a mess of things, or copy/pastes. None of which a formal unique-id solves anyway.
Using a hash or the like is just 'overkill', and no one wants to see that nasty name on their SD card.
(ie: if we redirect pnd or pxml.xml appdir writes back to (say) SD's /pandora/appdata/UNIQUE_ID directory, then no one wants that to be 300 chars long
jeff
Scanned a bit of your document, it is not quite right in one part. Very well done though, and thanks
The pnd_make.sh is in the libpnd git for reference
An app is a dir containing a PXML.xml file (And whatever else it wants), or its one of those bundled up into a .pnd file; a .pnd file is essentially a .iso file with the PXML.xml appended to the end (and optionally an icon.)
That is fail-safe -- if they forget to append the PXML.xml, the fact an iso is 'plaintext' (not a .cso), the search for the PXML will find it (albeit slowly if its a multimegabyte iso.) IDeally the PXML is appended (pnd_make.sh will do that of course), and the PXML is findable easily that way (and the icon following it.)
An application need not be a .pnd -- it can be unpacked as a dir, no problem. (This already works.. libpnd returns a discovered app and does not distinguish between app-dir or app.pnd, and supports execution and mounting/unmounting of both kinds of apps with the same API.)
(Naturally we'll be pushing for .pnd distribution to keep things in a single file and easy on the user, but I like to be clear it is not the only way.)
jeff
(and before you ask, we did consider a lot of variations; appending indexes, using zip files with their built in TOC even if uncompressed, cramfs and so on. Given that we want PXML appended, we can always consult the PXML to identify the package type (if not .iso) up front, and so on. Thats all future.)
The other school is.. consider java; no central authority, but you put your organizations domain name up front; poof, 'reasonably unique'.
Thats the angle I've been pushing; put a dev name or a app name, and some identifiers in there, and you're almost certainly guaranteed to be unique, unless someone wants to make a mess of things, or copy/pastes. None of which a formal unique-id solves anyway.
Using a hash or the like is just 'overkill', and no one wants to see that nasty name on their SD card.
(ie: if we redirect pnd or pxml.xml appdir writes back to (say) SD's /pandora/appdata/UNIQUE_ID directory, then no one wants that to be 300 chars long
jeff
Scanned a bit of your document, it is not quite right in one part. Very well done though, and thanks
The pnd_make.sh is in the libpnd git for reference
An app is a dir containing a PXML.xml file (And whatever else it wants), or its one of those bundled up into a .pnd file; a .pnd file is essentially a .iso file with the PXML.xml appended to the end (and optionally an icon.)
That is fail-safe -- if they forget to append the PXML.xml, the fact an iso is 'plaintext' (not a .cso), the search for the PXML will find it (albeit slowly if its a multimegabyte iso.) IDeally the PXML is appended (pnd_make.sh will do that of course), and the PXML is findable easily that way (and the icon following it.)
An application need not be a .pnd -- it can be unpacked as a dir, no problem. (This already works.. libpnd returns a discovered app and does not distinguish between app-dir or app.pnd, and supports execution and mounting/unmounting of both kinds of apps with the same API.)
(Naturally we'll be pushing for .pnd distribution to keep things in a single file and easy on the user, but I like to be clear it is not the only way.)
jeff
(and before you ask, we did consider a lot of variations; appending indexes, using zip files with their built in TOC even if uncompressed, cramfs and so on. Given that we want PXML appended, we can always consult the PXML to identify the package type (if not .iso) up front, and so on. Thats all future.)
Last edited by a moderator:
Squidge
Certified Guru
'dflemstr' said:
Last edited by a moderator:
dflemstr
It's a ball.
@Squidge
OK, I didn't think that the UID would be used for storage separation, but that the app would create it's own overlay directory in "/etc" etc.
Thought the UID would only be used internally, for the hashing of opened overlays or similar.
I now realize that it makes sense to add an id-value tot he PXML.
@skeezix
The PXML spec as I wrote it is included in my pandora-libraries repo and will be included in your merge. Feel free to change it after that.
Ah, and btw: I might change the spec or code with bugfixes (until you, skeezix, either merge my changes or decide not to include them) so I won't update the URL up there with the spec etc. If you are interested in the up-to-date contents, follow the URL in the first post.
OK, I didn't think that the UID would be used for storage separation, but that the app would create it's own overlay directory in "/etc" etc.
Thought the UID would only be used internally, for the hashing of opened overlays or similar.
I now realize that it makes sense to add an id-value tot he PXML.
@skeezix
The PXML spec as I wrote it is included in my pandora-libraries repo and will be included in your merge. Feel free to change it after that.
Ah, and btw: I might change the spec or code with bugfixes (until you, skeezix, either merge my changes or decide not to include them) so I won't update the URL up there with the spec etc. If you are interested in the up-to-date contents, follow the URL in the first post.
skeezix
Internal Development
I'll probabyl merge it in Wednesday, and then you can suck down a new official clone and go from there. Theres been a pile of changes the lst week, but none in the pnd_pxml/tinyxml stuff, so should still be mergable easily.
A few more things I want to add ASAP, and then it can rest a little while, while we fiddle with your stuff etc. Lots of little things to do (Add more device specific stuff like flashing the LEDs etc, tuning the pndnotifyd daemon behaviour, and so on.. but nothing urgent per se, I think.)
jeff
A few more things I want to add ASAP, and then it can rest a little while, while we fiddle with your stuff etc. Lots of little things to do (Add more device specific stuff like flashing the LEDs etc, tuning the pndnotifyd daemon behaviour, and so on.. but nothing urgent per se, I think.)
jeff
Last edited by a moderator:
dflemstr
It's a ball.
I actually merge your repository every time you update it (I track your HEAD) so there will simply not be any merging conflictsskeezix said:
(My "gitk" graph looks quite interesting right now)
And also; just wanted to know about a new deadline for the merging (if one exists), because I might start on more extensive documentation work and optimization(and a writer, maybe) which might take a while and be non-interruptable, if there's time enough for it
I don't want to stress you or something, I am just asking whether I should expect it soon or not
EDIT: I might aswell add some info about my repository for merging or branching or forking etc (for everyone), since I've gotten PMs from GIT newbies:
1. You are interested in the "new-pxml-format"-branch. Nothing else, because nothing else is up-to-date.
2. You can get the repository by cloning it. Clone the git url linked to in the first post. "git clone <git url>"
3. If you have the official repository or an old version, do a "git pull <git url> new-pxml-format" to get my branch merged in. Finish with a commit.
4. Attention when merging: I have removed everything that becomes obsolete with the new format. This includes some example PXML files that existed before and that were written in the old format. Be prepared
.If you for some reason need these files, that's what revision control is for. "git checkout <old version>"
5. I have introduced a .gitignore file. The role this file plays is quite important and I have adjusted some other parts of the repository to use this file better.
6. Check with "gitk" that everything was done correctly, always. Very handy GIT gui.
Last edited by a moderator:
skeezix
Internal Development
Soon, promise
I got hit with a need to move a couple of my domains ASAP; woosh
jeff
I got hit with a need to move a couple of my domains ASAP; woosh
jeff
Last edited by a moderator:
dflemstr
It's a ball.
Well, that's fine with me.
Because I use the extra time as much as I can
I for example now finished "char *pnd_pxml_get_app_name ( pnd_pxml_handle h, char *iso_lang )" that I've been working on for a while. (And no, I haven't broken the existing API)
But, it currently has a very rudimentary algorithm for matching the 'best match' language.
I would very much appreciate any ideas on how to improve it (I have some myself but they all aren't efficient enough) from all of you forum members:
CODE
char *pnd_pxml_get_app_name ( pnd_pxml_handle h, char *iso_lang ) {
/*
* NOTE that this will return the closest language available.
* So, if "en_US" isn't available, it will return "en_GB" instead,
* and if that doesn't exist, it will return "de" or "it" or whatever.
* This might or might not be desirable.
* Note also that this algorithm isn't perfect yet; it needs improvement.
*/
pnd_pxml_t *p = (pnd_pxml_t *) h;
int similarity_weight = 0xff; /*Set to something Really Bad in the beginning*/
char *best_match = NULL;
int i;
for (i = 0; i < p->titles_c; i++)
{
int new_weight = abs(strcmp(p->titles.language, iso_lang));
if (new_weight < similarity_weight)
{
similarity_weight = new_weight;
best_match = p->titles.string;
}
}
return best_match;
}
titles[] is an array of structs with "string" and "language" fields. "language" is the ISO-language of the string in "en_US" or "en"-format.
Any ideas appreciated as stated previously.
Because I use the extra time as much as I can
I for example now finished "char *pnd_pxml_get_app_name ( pnd_pxml_handle h, char *iso_lang )" that I've been working on for a while. (And no, I haven't broken the existing API)
But, it currently has a very rudimentary algorithm for matching the 'best match' language.
I would very much appreciate any ideas on how to improve it (I have some myself but they all aren't efficient enough) from all of you forum members:
CODE
char *pnd_pxml_get_app_name ( pnd_pxml_handle h, char *iso_lang ) {
/*
* NOTE that this will return the closest language available.
* So, if "en_US" isn't available, it will return "en_GB" instead,
* and if that doesn't exist, it will return "de" or "it" or whatever.
* This might or might not be desirable.
* Note also that this algorithm isn't perfect yet; it needs improvement.
*/
pnd_pxml_t *p = (pnd_pxml_t *) h;
int similarity_weight = 0xff; /*Set to something Really Bad in the beginning*/
char *best_match = NULL;
int i;
for (i = 0; i < p->titles_c; i++)
{
int new_weight = abs(strcmp(p->titles.language, iso_lang));
if (new_weight < similarity_weight)
{
similarity_weight = new_weight;
best_match = p->titles.string;
}
}
return best_match;
}
titles[] is an array of structs with "string" and "language" fields. "language" is the ISO-language of the string in "en_US" or "en"-format.
Any ideas appreciated as stated previously.
Last edited by a moderator:
skeezix
Internal Development
Remember, optimization leads to complexity and as a rule you optimize things that need optimization. The core of an emu, graphics blitting, audio rendering, the good stuff. Now, while PXML might be raided a lot (a thousand apps on an SD card, say), it probably doesn't need to be super efficient.
So I say reference implementations first, improve as one can. (Hence why a lot of libpnd is quick stab, but designed to be replaced. Admittedly some of the API, notabnly PXML stuff is cruddy, but we were in a hurry
I'm with you there.
jeff
So I say reference implementations first, improve as one can. (Hence why a lot of libpnd is quick stab, but designed to be replaced. Admittedly some of the API, notabnly PXML stuff is cruddy, but we were in a hurry
I'm with you there.
jeff
Last edited by a moderator:
dflemstr
It's a ball.
Well, yeah, I know (from experience) that premature optimization is the root of all evil and all that.
I was more referring to the actual problems with the current algorithm.
For example, if your PXML has the languages "en_GB" and "de_DE" in it, and you ask libpnd to give the "en_US" description, it will return the "de_DE" description instead of the "en_GB" one.
I could figure out how to do it but it would be a suboptimal solution (but I will if the situation requires it and I don't have so many other things to do).
I was more referring to the actual problems with the current algorithm.
For example, if your PXML has the languages "en_GB" and "de_DE" in it, and you ask libpnd to give the "en_US" description, it will return the "de_DE" description instead of the "en_GB" one.
I could figure out how to do it but it would be a suboptimal solution (but I will if the situation requires it and I don't have so many other things to do).
skeezix
Internal Development
hey,
I'm integrated things now.
You did make it pretty painless
I have moved the pnd_pxml_names.h to the include/ dir, since it might be of value as an interface (ie: for others who make their own parser, say) and not purely in the realm of private to the API.
There are a bunch of segfaults that can occur in the face of bad data (the previous code probably had the same problem), so we'll have to attack them sometime, but I'll fix up the shitty PXML 'sample apps' included and see how it goes.
I've also ditched your 'meta' dir and pushed things into the docs directory.
Do a fresh pull (in a bit) and ditch your existing branch, to keep it clean I guess..
jeff
I'm integrated things now.
You did make it pretty painless
I have moved the pnd_pxml_names.h to the include/ dir, since it might be of value as an interface (ie: for others who make their own parser, say) and not purely in the realm of private to the API.
There are a bunch of segfaults that can occur in the face of bad data (the previous code probably had the same problem), so we'll have to attack them sometime, but I'll fix up the shitty PXML 'sample apps' included and see how it goes.
I've also ditched your 'meta' dir and pushed things into the docs directory.
Do a fresh pull (in a bit) and ditch your existing branch, to keep it clean I guess..
jeff
Last edited by a moderator:
skeezix
Internal Development
Man, you didn't even try running the other apps as a regression test,m ya bastard
(discotest and pndnotifyd are the main apps run after every change
discotest failed in 3 places, do a diff..
Added two "if pelem -> gettext fails" checks, and added a handler for a null ptr you didn't catch.
Seems to work pretty well.
I'm glad you put in the iterator for 'best match', though we may have to tweak that more, but I've not had to deal with that much before so we need to fiddle. But I'm glad you put that in there day zero
jeff
(discotest and pndnotifyd are the main apps run after every change
discotest failed in 3 places, do a diff..
Added two "if pelem -> gettext fails" checks, and added a handler for a null ptr you didn't catch.
Seems to work pretty well.
I'm glad you put in the iterator for 'best match', though we may have to tweak that more, but I've not had to deal with that much before so we need to fiddle. But I'm glad you put that in there day zero
jeff
Last edited by a moderator:
