Further Information About The Pxml Format Needed


dflemstr

It's a ball.
Joined
Jul 31, 2008
Messages
2,514
Location
Stockholm, Sweden
Website
Visit site
(tl;dr part below)
So, I was having thoughts about package management, PNDs and the like, thinking about how packages are to be managed on the Pandora.

As you might or might not know, the PXML format contains <version> information. Now, the sole purpose of versioning information in a package would be to make it upgradable, right? Why would you otherwise store information about versions in the package...
So, auto-upgrades for PND-packages would be pretty nifty.
And don't say that an auto-upgrade system for the Pandora would be impossible; I've tested package upgrades based on XML metadata already (see this) so it is possible.

Sounds perfect, right? You can download a PND, put it in some folder, and a daemon can scan your PND folders for you, and notify you of new upgrades for your packages. Awesome.

There's one slight problem preventing this, though (well, if you read the topic title you know there had to be a problem): even though versioning information is available, it is currently impossible to actually see if one package is the successor of another.
Why? Because there is no information available in the PXML format that actually tells us what "series" of packages the package belongs to.

In big package management systems such as RPM, etc, every package has a package identifier; "firefox" would for example be the identifier for Firefox, and "kdebase-runtime" would contain the KDE runtime libraries. It is then of course possible to upgrade these packages, since you just look for a package with the same name, and a higher version number, and replace the old package. This identifier, however, is not available int he current PXML format.

→→tl;dr part starts here (but it still is a lot of reading ;))
Now, we have discussed this before (I wrote the whole introduction above just to inform uninformed people); we have discussed various definitions for the "id" attribute in the PXML tag that would enable us to avoid storage conflicts and introduce some kind of identification system, thus enabling us to do auto-updating etc. Such a definition was never created, though, and I think it's time to look into this now (because soon it will be too late).

The definition has to contain package identification data for an update system while still being changeable between versions for the persistent storage system.

My proposed standard for this id attribute would therefore be as follows:
  • The overlay/persistent storage system uses the whole id as its "hash value"; it doesn't have to care about the standard at all and just uses the whole id for overlay identification (→ we thus remain backwards-compatible as this is how the system looks like now! No code changes needed!)
  • The id attribute is split up into three parts: "name:last-modifier:overlay-number":
    1. The name is the package identification name. This is the unique package name, like "firefox" or "kdebase-runtime" as explained above, thus enabling package upgrades.
    2. The last-modifier is the user name of the person that last changed the package so that it required a new overlay (explained below)
    3. The overlay-number is a number increased every time the application needs a new overlay. This doesn't have to be a number, but can be anything the author wants it to be. Explained below.
Now, it should be pretty clear how the name part works; I still need to clarify how the other fields work though.
Let's take an example: Someone wants to package Firefox (let's say, I do). I therefore create a PND with a PXML file containing the following ID:
CODE
firefox:dflemstr:1

...because the package is a "firefox" package, and it needs a new overlay (since it's the first release of the package) so I write my user name and the number 1 to identify this overlay.
Now, let's say that some releases later (note: the overlay won't change every version most probably!), I decide that Firefox needs a new overlay because of new configuration file formats etc. I then increase the overlay number like so:
CODE
firefox:dflemstr:2

And let's say that I am releasing a development version of Firefox every now and then (IRL I would name the package something else like "firefox-svn", so as to not trigger the auto-upgrade of everyone's stable firefox versions but this is hypothetical), and need a separate overlay for that. In that case I can simply add one by specifying some string that makes sense to me (it can be anything; it just has to be unique amongst my own overlay identifiers):
CODE
firefox:dflemstr:dev-1

...and when I release a new development version of Firefox that also needs a new overlay I can specify something else:
CODE
firefox:dflemstr:dev-2-experimental

etc.

Now, let's say that someone thinks I'm a lousy packager and wants to release his/her own version of Firefox, with a new overlay. Then this person creates a PND with the following ID:
CODE
firefox:somedude:whatever-53

...and thus no overlay collisions can occur!
If that user still remains compatible with the old overlay, he/she can of course use my old overlay by specifying my old ID attribute; you only have to change the attribute when you need a new overlay.

Opinions? Any flaws that I have missed?
 
Last edited by a moderator:
I thought we had already decided on this, and that the uid was already used to store pxml information on the nand, and refer to that data when a newer version of an application was installed, to ensure the same configuration data can be used?
 
I thought we had already decided on this, and that the uid was already used to store pxml information on the nand, and refer to that data when a newer version of an application was installed, to ensure the same configuration data can be used?
Yes that part was decided on, but what if one needs to auto-upgrade a package but not to use the same configuration data, there's now a format for the id that allows to define this - with a package name - and also i think it's a much better thing for a package management thing to come at you and say "There's a new version of firefox" than "There's a new version of firefox<something gibberish looking>"

@dflemstr: i see no problems in your suggestion, good to define a format for the id before too late.
 
Last edited by a moderator:
We have always defined the unique-id as a 'relatively unique string' to be used in various nefarious ways. I don't think we should stick version and meta-package information into it and thus require discombobulation ..

ie: An easy way to construct the uique-id is to bundle app, dev and version numbre together, but it does not have to be so; could just be face-mash-gobldeyguk, as long as it honours the filesystem rules and such so that pnd-run can mount it, and SDs can have it, etc.

I thought PXML already had a 'package name' and a 'package version' separate values, so that if we wanted, we could write an app to say 'yes, newer version exists', and do whatever.

Myself, having investigated the inner workings of RPM and .deb know they are almost total garbage; in general, we all know that auto-updating packages is a nightmare, and I doubt we want to attewmpt it for our little platform.. I'd rather just say 'hey, new versin available, go get it; or I can get it for you'.

(Remember, for us, an update is jyust a new .pnd file, no installing or uninstallation though, so it could be a lot easier for us than for .deb and .rpm)

(The best I had was with an rpm once, where I said I wanted to ditch firefox and it knew X11 was required, so it ditched X11 on me. FAIL!)

Anyway, I thought we had..

- unique-id (which might look like package name and version, but does not have to be)
- package-name
- package-version

jeff
 
Maybe my post was a bit unclear. The unique ID tag will not contain any information at all about upgrades, we all agree on that, don't we. It's primary purpose will still be to identify overlays.
However, the problem I've been trying to highlight is that there isn't any way for us to tell if one package is the successor of another (@skeezix, in other words, there is no "package-name" information), and that you could use the unique ID for this (sorry for repeating myself, I thought this was clear).

Maybe you get a better picture of the situation if I put it like this:
Scenario: Let's say we use the current system and release Firefox with the following (partial) PXML:
CODE
<version major="3" minor="1" release="1" build="213"/>
<title lang="en_US">firefox, the internet browser</title>

...and let's say we release an update with the following info:
CODE
<version major="3" minor="5" release="1" build="226"/>
<title lang="en_US">Firefox, the ultimate internet browser</title>


Now tell me: how are we supposed to know if the one package is the successor of the other? They have no common identifier that tells us that they are of the same package series. We could rely on the various title translations to identify the packages, but as you can see in my example, there is a difference (in the first package, the title is something completely different than in package 2) and there is no guarantee that the two packages will share any title translation at all (titles change, and in one package, "en_US" might not be available, or a letter might have been capitalized etc: unreliable).

There thus has to be a way to solve this problem.

Another solution would of course be to introduce a "<package-name>" or "<name>" tag. It would just be there to identify the package, basically.

EDIT: just to make this clear again: There's no guarantee that versions of the same package will have the same overlay, so we cannot rely on this, as explained above. The only reason why it exists is to identify overlays, and if a new version makes the old overlay obsolete because of new config formats, it should be possible to change it (and this is why it was introduced).
 
If that is the packager's intent for it to be a successor then wouldn't they be a total moron for changing the title to something different.
Let me put it like this:
Let's say you are a bank manager, and want a way to manage all the bank accounts in your bank, then what of the following would be the most sensible solution for you:
  1. When a customer creates an account, let him/her write a sentence about why they like your bank so much. Use this sentence in your internal databases, as a global account identifier, for transactions, etc. When the customer needs to make transactions, he/she needs to write the whole sentence on the sheet of paper provided for the transaction. If the customer makes one mistake by putting one too many spaces, changing a capital letter into a small one, or misspelling something, then the transaction can't be made because the new sentence refers to another bank account. And when the bank employees handle lots and lots of accounts, they will have to handle lots and lots of sentences written by different customers. Also, the opinion of the customer might change (relate to: "Azureus" → "Vuze", "KDE" → "K Desktop Environment"; changes that have happened IRL), and when the customer next writes his/her opinion about the bank on his/her transaction sheet, it suddenly won't work because the bank doesn't know which account this opinion refers to.
  2. When a customer creates an account, the account gets an identifier, in shape of a number. This number will never change, is short, concise, and descriptive, and can easily be jotted down when making transactions. Because the number has to be a certain length etc, and follow certain rules, it is more difficult to miswrite it and it becomes much easier to handle.
Personally, the 2nd solution sounds best to me. Now, for packages you wouldn't use numbers but instead concise strings (that contain no spaces, letters a-z, dashes and numbers only) like "firefox" or "konqueror" or "kdebase-runtime" but it's the same principle.

EDIT: you would of course still DISPLAY the contents of <title>; the name tag is just there to be able to upgrade packages easily, and maybe also to allow packages to be referred to in a more concise way. For example, what of the following would you prefer to have as an URL for a PND:

  • http://mysite.com/pnds/Firefox,%20the%20ultimate%20internet%20browser%20version%203.3.9
  • http://mysite.com/pnds/9589836287129
  • http://mysite.com/pnds/firefox/3.3.9
 
Last edited by a moderator:
If that is the packager's intent for it to be a successor then wouldn't they be a total moron for changing the title to something different.
Let me put it like this:
Let's say you are a bank manager, and want a way to manage all the bank accounts in your bank, then what of the following would be the most sensible solution for you:
  1. When a customer creates an account, let him/her write a sentence about why they like your bank so much. Use this sentence in your internal databases, as a global account identifier, for transactions, etc. When the customer needs to make transactions, he/she needs to write the whole sentence on the sheet of paper provided for the transaction. If the customer makes one mistake by putting one too many spaces, changing a capital letter into a small one, or misspelling something, then the transaction can't be made because the new sentence refers to another bank account. And when the bank employees handle lots and lots of accounts, they will have to handle lots and lots of sentences written by different customers. Also, the opinion of the customer might change (relate to: "Azureus" → "Vuze", "KDE" → "K Desktop Environment"; changes that have happened IRL), and when the customer next writes his/her opinion about the bank on his/her transaction sheet, it suddenly won't work because the bank doesn't know which account this opinion refers to.
  2. When a customer creates an account, the account gets an identifier, in shape of a number. This number will never change, is short, concise, and descriptive, and can easily be jotted down when making transactions. Because the number has to be a certain length etc, and follow certain rules, it is more difficult to miswrite it and it becomes much easier to handle.
Personally, the 2nd solution sounds best to me. Now, for packages you wouldn't use numbers but instead concise strings (that contain no spaces, letters a-z, dashes and numbers only"] like "firefox" or "konqueror" or "kdebase-runtime" but it's the same principle.

EDIT: you would of course still DISPLAY the contents of <title>; the name tag is just there to be able to upgrade packages easily, and maybe also to allow packages to be referred to in a more concise way. For example, what of the following would you prefer to have as an URL for a PND:

  • http://mysite.com/pnds/Firefox,%20the%20ultimate%20internet%20browser%20version%203.3.9
  • http://mysite.com/pnds/9589836287129
  • http://mysite.com/pnds/firefox/3.3.9
[/quote]Maybe I just don't get it, thinking that this string is not manipulated or typed in by any human once it's created by the package author so your analogy would not apply. It's a unique string of crap, whether the string be made of digits, 7 or even 8 bit characters should not matter in the slightest. All a package needs is a unique id of some sort, thought title was it. If that unique id changes, it's not the same application.
 
Last edited by a moderator:
I had thought we did have a package-name field in there; I've been killed the last 2 months so I've not looked at things at all, but I had thought we did for this purpose.

(Really, PXML spec needs a refresh, as we did a lot of changes there, back with your updates before and various discussions we've had.)

I would not be surprised if things have fallen through the cracks .. people are busy :)

(For my part.. amid layoffs, and yet twice the workloads.. yay! :/)

Anyway, I'll be pounding through a bunch of libpnd changes soon, so this is a good time to bring it up.

I would rather, as mentioned above, have separate fields; i) package name, ii) package versioning, and iii) unique-id -- they're all separate; it might well be the unique-id also includes that info, but theres no guarantee it has to (its free form, lets keep it like that.) Version could be multiple splits, but I'm not a fan of hard-defining such things since everyone likes to do it differently. (Likewise with 'release date' might be a good one, since this way we coudl tell a verison is newer and just sort by latest release-date, but it could be weird for project forks and such.)

So..

- all other fields
- package-name (not the package _title_, but the internal name; 'firefox' as your example)
- package-version
- package-release
- unique-id

jeff
 
PXML:
CODE
<version major="3" minor="1" release="1" build="213"/>
<title lang="en_US">firefox, the internet browser</title>

...and let's say we release an update with the following info:
CODE
<version major="3" minor="5" release="1" build="226"/>
<title lang="en_US">Firefox, the ultimate internet browser</title>
Now tell me: how are we supposed to know if the one package is the successor of the other? They have no common identifier thatWell, since the uid field is used by the file system to store configuration information, the uid field of the newer version will be the same so that the newer version of the browser uses the same configuration data. Therefore, it will also know that since they both use the same configuration data and the second has a higher version number than the first that the second is the successor of the first.
 
Last edited by a moderator:
Well, since the uid field is used by the file system to store configuration information, the uid field of the newer version will be the same so that the newer version of the browser uses the same configuration data. Therefore, it will also know that since they both use the same configuration data and the second has a higher version number than the first that the second is the successor of the first.
Is this really the case? The whole purpose of the UID is to make it possible for one chain of packages to have multiple overlays, or this is at least what skeezix has told me. So, if I make an app, and some guy decides to fork my app and requires a new overlay, or I change things about my app so that it needs a new overlay, then there should be a possibility for us to do this, and the upgrade path still has to work in such a case, doesn't it?

Is it really as you say, that one package chain == one overlay? So we could theoretically just put "firefox" as the overlay ID and all problems/this discussion would be gone, and we would all get much less headaches because we won't need to add another field to the PXML? Everything that was said about the UID in the original PXML remake thread a while ago points in the other direction, so are you sure that what you say is correct?

The thing is also that, as mentioned in my last post, we don't just need a package series identifier for upgrades, but also to be able to keep up some kind of package handling. It's impractical to make a database of packages by using "The ultimega über Firefox browser" as a key, or even an overlay ID that, with the current standard, can be anything from "firefox" to "Hello, World!" to "þ®←ðªþœ@”ß". So if the above is true, that you can identify a series by overlay, can't we at least say that it has to consist of "[-_a-zA-Z0-9]+" or something so as to make it manageable?

PS. I'm sorry for taking up so much of your time; I had kind of expected that you would see my point instantly(when having read my above post again, I see this as an impossibility now;)), and then either would've told me that it's redundant/wrong/crazy (which, as I see it, is not yet the case since no reason has been given) or would've said "OK" and let me get on with my package management app :p
 
Last edited by a moderator:
The unique-id is used for seevral things, one of which is for uniqueness of writebacks to the SD, but also for naming icons on NAND caches and other stuff. (mountpoint names, whaytever.)

The unique-id is a unique-id, and devs/users should feel free to alter it to change that association. ie: You want a new set of data for your app, change it. You want 5 versions of the app to use the same data dir (as I expect people to do), don't change it.

I don't expect people to alter unique id for every version of an app, but possibly for major changes that don't work with older versions of the data that they cannot handle gracefully in an app.

ie: For OutcaST, I'd never have changed the unique-id.. I handle savestate changes with my own version number embedded on the 'disk'. I'd rather people just upgrade freely and it uses whatever data it already sees, so I'd never alter it. But other devs may want to update it as they see fit.

Application version, package name, etc, that should be something else. ie: Don't "overload" things. (We can get tricky with pointer arithmetic, but when it comes to XML, lets keep it simple and obvious and not require string manipulation just to get a useful value out of a field.)

jeff
 
OK, this at least creates the need for SOME kind of standard, since you cannot create files with UTF-8 names on FAT32 file systems and MOUNT doesn't accept certain characters for mount points. And I would really like to have the possibility of putting the uid as part of an URL. With "€ðđ©“”@øðĸ", this is not possible.
EDIT: agreed that we shouldn't put the package series id inside the uid, and not versioning info, either (something that I never proposed in the first place ;)), but there:
  1. Should be a standard for the uid, at least to limit the allowed characters. UTF-16 control characters are nasty, and can be used for UI and memory exploits.
  2. MUST be some way to identify a package series, be it with the uid or with yet another field introduced (such as <name>)
 
Last edited by a moderator:
, but PS could partially read some SS configuration data and vice versa.
[*]if your update your app and change it so as old configuration is invalid, the application should gracefully accept this and either use as much as possible from the old config, or delete it and start again, after prompting the user that there settings will be wiped out. I wouldn't expect the app to take the easy way out and change it's uid to get a fresh dir to play in. After all, it's unlikely that the user will be wanting to play with two different versions of your app on a regular basis. This is the way things work on a desktop pc - newer software can read and update old configs, but old software just ignores newer configs.
[/LIST]
I do agree that the characters should be limited to [-_a-zA-Z0-9].
Another example using your firefox example: One author distributes version 3.1 of firefox. Another author releases his own version with some of his own 'edits', and calls this 3.2, and changes the overlay so not to conflict with 3.1. The original author updates firefox to a proper version of 3.1.5 with the original overlay. We don't want the update system to give us 3.2, but 3.1.5, as that is the true successor. The second author could change his series name to ensure this doesn't help, but since he's changed the overlay anyway, why have to change two things? People will forget, change the series (or the overlay), forget about the other and end up with corrupt configuration directories and incorrect versions updated. For this reason, I think only one thing should change for forks.
 
Last edited by a moderator:
dflem -- the specification is of course 'loose' since we're all hacking in spare time :) The definition has always been "relatively unique" (so we don't have to have a CGI up on the pandora site to generate them, knowing people would copy/paste them _anyway_ and thus break the CGI's purpose), and should be resitrcted to alphanum-ASCII really, but is more permissive than that. Ultimately it has to be usable on SD and in other points, as you say.

Its come up a few timesto 'hard define', but as I say ... someone needs to rewrite and harden down the PXML spec. Do you want to take on that role? I'm working 60+ hours too much for it :)

But basicly agreed,its just not well published info.

jeff

Note that it should really be considered case insensitive and should really be all lowercase; we're probably loking at FAT filesystems after all.
 
OK, thank you all for your time, now at least I know what we're working with here.
Maybe we should clarify in the pxml documentation, then, that the id attribute also will be used for package series identification (and it already has an attribute name fitting for the purpose, how convenient!) and thus should be something sensible like "firefox" or "my-splendid-app" or something (not forced, of course; it can still be anything, but I don't think that two people both will call their application e.g. "firefox" completely independent of each other, and my PND repository app will check for name collisions anyways, so esoteric random appended numbers should not be necessary(or at least I won't suffer from the problems arising ;) )), and matching the "[a-z0-9-_]+" regex so as to make it saveable on FAT32. I'll make the patch for you, skeezix, if you want to and don't have time yourself.
 
Are you making a repository app (ie: dpkg, etc) of some sort?

jeff

Which PXML spec are you referring to; one of EDs, or the one you made for your pxml patch a couple months back?

As to patches.. you mean, to add a package-name and package-version and package-release date set of fields?

Which fields are you thinking of adding, specifically?

jeff
 
Which PXML spec are you referring to; one of EDs, or the one you made for your pxml patch a couple months back?
As to patches.. you mean, to add a package-name and package-version and package-release date set of fields?

Which fields are you thinking of adding, specifically?

jeff

1. Both.
2. I thought about updating the specfile I wrote a couple of moths back, because it's the only documentation out there about the PXML format afaik
3. I don't want to change any code, just tell people that the overlay-id should match "[a-z0-9-_]+" (because, as you both say, it has to be FAT32-compatible and because I would want to include the ID in an URL), and that the id should make some kind of sense, because it will be exposed as an URL by me as I said (so people can choose to have "dflemstr-jlhdjh2983" as their ID if they want to bot it's not recommended since it might be used for something beyond a hashing key)

of some sort?
jeff

Yes. And no. Currently, I'm just making a server-side repository manager for PNDs (because, no offense, I think that the File Archive sucks for package management and I need something more future-proof for my personal package management at least; the situation is similar to the one when I wrote the PXML patch because the "old" PXML format was broken). Because, seriously, the File Archive is OLD, and you can't create a frontend (like yum, dpkg, etc) easily for it; and since we have CRUD, AJAX, Comet, jQuery and so on available today, why not use these technologies?

So, I made a webapp that dynamically generates an XML of its inventory, including metadata, to allow me to write frontends for it (frontends download the metadata and let the users search packages by description etc, and directly download, install and upgrade packages).
At the moment, I will be the only one using it, probably by mirroring the File Archive with a spider (yes, that's how important I think PND auto-upgrades are ;) ), but more people can join if they want to, of course, if I get access to a reasonably fast server (I have an account system set up with OpenID so really, ANYONE can join).

BTW, my app has a killer feature: It reads uploaded PNDs, parses the PXML automatically, and makes PNDs searchable by description etc by using the acquired information. This is not even remotely possible with the File Archive.

EDIT: It's not finished yet, so don't ask for the code. An old version is available on GitHub but don't download it, it's not worth it.
 
Last edited by a moderator:
Ask ED for webspace :p

BTW, I think unique-id and package-id should be [a-z0-9-_]+ .. FAT doens't really handle case sensitivity well, so lets aim for lowercase.

Don't call it 'overlay', thats a complex name.

jeff
 
Back
Top