Further Information About The Pxml Format Needed

[dflemstr]

Ask ED for webspace

BTW, I think unique-id and package-id should be [a-z0-9-_]+ .. FAT doens't really handle case sensitivity well, so lets aim for lowercase.

Don't call it 'overlay', thats a complex name.

jeff

Yeah, Squidge already told me that [a-z0-9-_]+ would be better and I totally agree (that's what I wrote in my last post )

So it's decided then, that the "id" attribute represents the package id and must match that regex? Cool, I'll continue working on my app

 

[skeezix]

I need to check the various specs (but I won't , but I'm assuming there is a unique-id (already there) and also a package-id ("firefox"); I do not believe they dshoudl be one and the same, as I've repeated above.

Not entirely sure what squidge things, or what you think above, but I don't like to overload function into one field

jeff

 

[dflemstr]

; I do not believe they dshoudl be one and the same, as I've repeated above.

Not entirely sure what squidge things, or what you think above, but I don't like to overload function into one field

jeff

Oh, how complicated things can be...
Squidge means that if you fork an app, thus needing a new overlay, you should change the app name anyways (if you fork "firefox", then you shouldn't call your fork "firefox" too but "firefox-dflemstrmod" or something), and if your application needs a new overlay because of an upgrade, the application should take care of upgrading its config files; thus an UID is unnecessary and you'll only ever need a package ID, so we can forget the UID and just use the "id" field in my spec or (if you decide to use the, to me, deprecated v1.5 "official" format) the "unique_id" tag for package identification.
Thus, we don't overload anything, but instead give a PND an identifier that is shared between versions and that also is used in any way the system wants to use it, for example for overlay identification.
I have to say that I agree with Squidge; all the other packaging systems have had packages handle the upgrading of their config files themselves, so why shouldn't we?

 

[skeezix]

The rational I've always had for 'unique-id' is that it is easy to make something up reliably, with a fairly high degree of not running into someone else. If you make a package 'firefox', it is pretty likely someone else will, and there won't be an easy way (barring your tool, see what I did there .. besides, there is also the case where people might like to share data for different packages -- possibly two emus with the same data, etc. My thought being -- if we join the two, then we necessarily link and overload the two. If we keep them separate, they can be the same or not, and we gain flexibility for nearly no cost.

*shrug*

jeff

 

[dflemstr]

.. besides, there is also the case where people might like to share data for different packages -- possibly two emus with the same data, etc. My thought being -- if we join the two, then we necessarily link and overload the two. If we keep them separate, they can be the same or not, and we gain flexibility for nearly no cost.

*shrug*

jeff

Well, I don't really care; as long as there is some value that is common amongst all packages of the same type (eg. "firefox"), and that value is semi-URL-encodable (aka doesn't contain "đªµ"), I'm content.
However, there is no package-name field yet so I am thus not yet content We only have the unique-id in the format ,which was the reason why I brought all of this up in the first place.

Oh well, I'll just use the "id"-attribute as identification for now, and if you guys decide to change something, I just have to change one line of code so...

 

[Squidge]

Jeff, I would assume your API already takes care of the fact if two applications want to share configuration data? I was under the impression that the directory created for configuration data would be unique to each application. I don't want two application happily sharing information, and then another version of one of the apps comes out, and the other app can't read its configuration files properly anymore.

Maybe one of the directories should be called "SharedDataStorage" or somesuch if we want to allow such shared data, but I still think apps should also have there own private directory.

We know why the uid is in the pxml, but the name of the subdir in the SharedDataStorage directory could be hardcoded into the app and not be a part of the pxml, as it wouldn't be required by anything other than the application itself?

I don't suppose there's any real reason not to add a timestamp to the uid to make it more unique, so "firefox" becomes "firefox-a10fd042" and is retained for all proper versions, and so forks would just change the timestamp. It would be better than adding the authors name, as it could then be carried on by someone else without leaving the previous authors name in there (for example, if the previous author decides to pass the development to someone else)

 

[skeezix]

unique-id is 'relatively unique', where it shoudl be as mentioned before sufficientlky obfuscated within reason to be unique; but I like to reserve the option of the _DEVELOPER_ to make it as less-unique as he wants.

ie: If theres a Franxis MAME with package id "mame4all" and unique-id "mame4all-base-12344543212"; and then someone forks it, to have a binary that is just compiled with a different compiler say, but no other changes.. then I would think it find for that developer to keep the same unique-id so it mounts in the same place for data/roms/etc .. why not?

Likewise the user could use a PXML-override tool to say suggest something else. (One problem here is that I rely on th eunique-id for PXML-overrides, which might be a problem in this case, bu tI doubt there is a better solution. I don't anticipate many packages to do this sort of colliding-unique-id trickery.)

I don't think any "Shared Data" is needed; most apps should not collide the unique-id, and those that do shoud be orchesytrated and probably are just one developer anyway (ie: version upgrades, same app.)

So in my view, package-id (thought we had it in there is the 'shortname' of an app ("firefox", "mame4all", "outcast", etc) and used for package update type operations. (One trick is from deb's, where they make 5 different werbservers all use the same package name, so that other related packages can say "depend on webserver" without carying _which_ one, but I've not gone down that road of thought here.)

Likewise "unique-id" is just a "developer made up relatively unique id", couild be "dev name . package name" say .. "franxis.mame4all" if he wanted it to be specific to his builds, and not bloody likely to collide with anyone (aside from copy/paste lazy people, but that is unavoidable in all cases anyway.) He could also elect to just use 'mame4all' or 'mame4all-328403214324532akjfhksahfkewr432' if he wanted. I say leave it up to developer to be _as specific or not_ as he wants, but keep package id separate..

.. most flexible, no real work.

*shrug*

jeff

 

[dflemstr]

So your solution would be to add a package-name field? Sure, why not, as long as it works I'm content. I can write the code (it's, like what, 8 lines?) if no one else is up for it; I just need to have *something* to use for my package-name field in my webapp so if you approve I would like to make such a field into reality asap.

 

[skeezix]

I thought it was there; I can add it perhaps, I wil lbe doign some changes soon.

I forget if we have a PXML-validator test tool in libpnd; We've had a few hacks, but maybe I should make sure thetres a basic 'official' one in there, even if it only checks 1 field.. we can grow it later.

ie: Sure, XML can be self-validating, but like most people are going to do that; having a PXML validator tool may be helpful, and certainly it means in pndnotifyd/etc we can just emit error for bad PXML. (there is a funciton to check pxml, but it just returns 'good!' nowadays.. it used to chekc, but we dropped it out during the changes you added I think.)

Field name recommendation?

- package-id <- something short, up to 20 chars or something ideally, the lowercase alphanum -_ type regex
- package-release-date

I forget if we have version in there now; I dislike including 'complex 4-5 part version numbers' in there, since that rather binds people to one versioning method (which can be handy .. I prefer just 'version' and let the dev do whatever they prefer, or no version at all.

package-release-date is handy, since regardless of version, it is easy to say 'this one is newer', without understanding the many version patterns people use.

jeff

 

[dflemstr]

Field name recommendation?

- package-id <- something short, up to 20 chars or something ideally, the lowercase alphanum -_ type regex
- package-release-date

I forget if we have version in there now; I dislike including 'complex 4-5 part version numbers' in there, since that rather binds people to one versioning method (which can be handy .. I prefer just 'version' and let the dev do whatever they prefer, or no version at all.

package-release-date is handy, since regardless of version, it is easy to say 'this one is newer', without understanding the many version patterns people use.

jeff

Dude, please don't change the version field into something arbitrary too; upgrading will become even more impossible It's good now as it is (and it's possible to store it in a 32-bit integer (provided that no app ever comes above version 255) or a 64-bit int (max version = 65535 ) which I like a lot)

I'd prefer <package-name>, <package-series> or <package> as the field name, because <package-id> can easily be confused with <PXML id="..."> which is the uid.

My webapp automatically tracks package upload dates so this isn't a problem for me (at least with my webapp ) but it might also be handy. Of course, if you keep the current version format, we can just use that for upgrades.

And the XML spec/validator that I uploaded seems to have been borked up when merging or something (I shouldn't have used so many Git branches ) because it doesn't validate the example PXMLs any longer → we're better off making a validator application anyways because XML specs are a pain to write.

BTW, I looked at the libpnd code now and saw some very serious bugs with the PXML-in-PND finder; it would for example find a PXML inside the ISO and use that (since ISO is uncompressed... brr), or it would fail to parse a PXML file containing "<PXML" somewhere inside of it besides at the beginning. And if a PXML has trailing newlines or something at the end, or the word "PNG", we get problems with the screenshot. There's a lot to be done it seems

EDIT: interesting fact: if I make an ISO containing only an empty text file, 99% of the ISO will be filled with 0x00; you waste about 1 mb of space with the ISO format in some cases. Who was responsible for choosing that format instead of GZip or LZMA?

 

[skeezix]

Actually..

- PXML embedded within an iso beingfound is by-design; if someone forgets to append it then a slow-back-seek location of it within the iso works as well
- locating something else with <PXML literal string (like in a descriptiuon .txt file) is an accepted risk (currently); we can add more protection later; hey, I hacked this stuff together quickly.. it shoudl be relatively reliable stuff, but it is certainly no where near bullet proof Most apps will include 'PXML' but not 'PXML' hopefully :/
- <PXML should not occur within a PXML file; that'd make it invalid anyway
- With regard to png/etc, theres a lot of compromises made to handle that we might allow .iso or many other formats, and jjust cat things together; for awhile we had an index up front or at the end to point to different parts explicitly, so we could have more arbitreary parts than just pxml and screenshot (such as man screenshots, or audio clips, etc) and locagte them fast (skip right to chunk, not seeking.) In the end we deemed it better to keep a simple format (that even works when format details are left off), and then rely on 'mount' operations -- if someone wants to get audio clips or sample screenshots (other than the icon), then they can pnd_mount the pnd file, get their goodies (some as define din the PXML.xml) and then pnd_unmount it. (pnd_mount going in soon, but essentially is the same as pnd_run but without the run-part.)

So yeah, lots of potential issues and such, but what can you do in a few hours here or there. We've endlessly debated things, but we're on purpose trying to keep simple; overdoing it is usually bad for adoption

"in some cases"; a minimal empty .iso should not be 1MB (wtf did you use tro make that iso?!); further, .iso is not the only format that might be used down the road; iso was chosen because it mounts and unmounts very fast and is easy to support, and easy for everyone to use; you have to consider uses under load and what peopel will be doing with it, and what drivers/etc people will be using, and speed on various media, etc.

We might well have blown everything, but we can at least say we came up with some ideas that are pretty damned awesome (run from a SD card with no installation!), and as you can see.. pretty simple stuff, and hey .. the code is all up and people can go to town with it.

I mean, I certainly don't intend on standing in the way of patches

jeff

 

[skeezix]

I wonder if we should have somethign like..

<package>
<shortname>firefox</shortname>
<release-date>whatever</release-date?
</package>

Or more tighter..

<package shortname="firefox" released="somedate"/>

Or just <package released="somedate">firefox</package>

Pick one and I'll go with it.

jeff

 

[dflemstr]

I wonder if we should have somethign like..

<package>
<shortname>firefox</shortname>
<release-date>whatever</release-date?
</package>

Or more tighter..

<package shortname="firefox" released="somedate"/>

Or just <package released="somedate">firefox</package>

Pick one and I'll go with it.

jeff

<package name="firefox" released="somedate"/>
Why short?
And I don't care about the released field; I won't be using it because it just isn't worth it (for me) so do as you wish with that.

EDIT: oh and my current PXML extraction algorithm currently works about the same way as the libpnd one, except that I limit the seek range to 32kb (because I just don't want to have the server hung for 1 minute crunching through a 100mb PND file) but we really need to find something sustainable that cannot be exploited...

The code for those who are interested; sorry for the lack of comments and the chaos:
CODE
def getPXMLFromPND(pnd: Array[Byte]): NodeSeq = try {
val reversed = pnd.projection.reverse //don't actually create new arrays here, just use a projection
val reversedStartPattern = "<PXML".getBytes.reverse
val endString = "</PXML>"

def cutEnd(pxml: String): String = {
val parts = pxml.split(endString)

if(parts.length >= 2 || (pxml endsWith "</PXML>"))
parts(0) + endString
else error("Could not find end of PXML file")
}

//yeah, yeah, this is more CPU intensive than it has to be... I don't care, it's concise
val candidates = for {
i <- 0 until Math.min(reversed.length, 32 * 1024).toInt //scan max 32 kb backwards
candidate = reversed take i
if(candidate endsWith reversedStartPattern)
} yield candidate

val uncutPXML = new String(candidates.last.reverse)
XML.loadString(cutEnd(uncutPXML))
} catch { case _ => NodeSeq.Empty }

 

[skeezix]

"shortname" is an old traditional field name I've been using for a looong time, but I use it as a trigger in my brain to mean 'one word'; I worry if we put name it'll just get "franxis mame4all ported by dcemu guys!" or something

Just hacked a basic validator that does nothing in, will check it in soon; hopefully we can add some actual validatoin or regexp's or something to it soon

jeff

Added <package name="foo" released="bar" />, will check in soon

 

[dflemstr]

Oh yeah in case I forget: make the releasedate in the international format (YYYY-MM-DD hh:mm:ss:µµµµ TIME_ZONE) otherwise you'll get problems with parsers.
Or you could just use an unix time number, which also is an international standard and readable everywhere (however you'll have to convert it to UTC every time)).

 

[skeezix]

yep, agreed. But again, I'm Mr. Copout; I'll help with the specs and check in generic code.. we'll flehs in details later .(And by 'we' I mean 'everyone else'). Damn, I wish I had free time like until a couple years ago

jeff

 

[skeezix]

A few checkins lately; I think I'll put in a 'field list' container (Rather than all those silly hardcoded structs for pxml and disco-t contents), and get in a category-mapper for .desktop files emitted, and other stuff. Gotta get ship-shape soon!

jeff

 

[dflemstr]

, and get in a category-mapper for .desktop files emitted, and other stuff. Gotta get ship-shape soon!

jeff

Keep us updated
Especially if you do some fancy stuff with that PXML-in-PND locator; I'm currently using a completely different algorithm for my needs so it would be good if we could agree on one algorithm for finding PXMLs, so that we don't find different PXMLs depending on application.

 

[dflemstr]

Hey, quick question again (slightly off-topic but meh):
What would be the preferred MIME-type for PND-files?
(And this question only regards web transfers BTW; I don't care about how the file gets recognized on your specific platform (Windows lacks libmagic and thus doesn't care about MIME-types anyways), but I do care about how it's seen before being downloaded (Save As.../Open With... dialogs should behave correctly), so when I say "the file can be opened with X" below, I mean "the browser has the possibility of associating the file with X")

• "application/octetstream" → Generic binary file
  Advantage: The file will be treated as a binary (aka a file without structure) and won't be opened by applications that could be harmful to the file (except maybe hex editors )
• Disadvantage: You can't open the file in an ISO editor (without setting up special file associations)

[*]"application/x-cd-image" → ISO file

• Advantage: Can easily be opened by ISO editors on Linux, and on non-IE-using Windows PCs
• Disadvantage: Some ISO editors might truncate the file and remove the PXML information at the end if it thinks that the file is an ordinary ISO file, which could be a problem when giving people the possibility of opening the file with an ISO editor.

[*]"application/x-pandora-pnd" → PND File (custom MIME)

• Advantage: You can make custom apps, for example a PND installer app (that basically just moves the PND to some folder but could have the feature that it sorted the file in categories for you) because the browser has a chance to identify the file as a PND file and nothing else
• Disadvantage: The file cannot be opened by default on any platform.

 

[skeezix]

I'd either go with octetstream (is what I use now), or x-pandora-pnd .. I've mulled it over a tough, and that seems the ideal approach but I forget the behaviour for stock-browsers -- will they download it, or dump it to the display as binary garbage (which isn't really ideal.) If most browsers will just download the unknown type, it might be good.

Still, octetstreeam is a known good quantity so pretty tempting.

We definately don't want .iso I think .. ie: A lot of peeps have .iso opening software like isobuster, and it even ships with a lot of boxen. Making it iso is good so peopel who have half a clue can fiddle, but I prefer to make it not totally obvious to screw it up by the noobs Also, .pnd implies that it mnight be a .iso or something else (we could use .zip too, it works. Appending pxml to a zip is just like a comment to a zip, no problem.)

So x-pandora-pnd if we can get away with it, but if its trouble a pure binatry stream is the way to go imho.

jeff