The New Pxml Format

[dflemstr]

OK, you are all probably wondering what this thread is about.

If you are, read this thread:
www.gp32x.de/board/index.php?showtopic=46903

This post is mainly targeted at pandora devs, but might be interesting for the rest of the devs around here that will be using PXML.

Basically, I have made a XML compliant version of the PXML format, and fixed some of the "format bugs" that the old format had (for example, it only supported 2 category trees etc.)

This is a WIP (I started with it on Sunday XD), but if you're interested in getting the code, it can be found here (a cloned git tree):
github.com/dflemstr/pandora-libraries/tree/new-pxml-format
The branch that you are interested in is the "new-pxml-format" branch.

Clone that repo, or pull from it if you already have an old version of pandora-libraries laying around.

Current implementation (in code):
- The format can be read by libpnd, with the existing api. No existing code that uses libpnd has to be changed for the new format to work.
- The format can be generated by any editor that supports XSD-generators

What has to be done (in code):
- A writer in libpnd has to be written
- The libpnd api should be extended/changed to support some of the new features that this format provides.

-------------------------------------------------------------

Some features that the format currently HAS:
- It is fully XML compliant and has a XSD spec. This means that any library you have that supports XML can make a PXML file for you, or read one (with some effort on your part, of course). This also means that PXML files can be validated (aka "syntax checked") for you.
- It is alot shorter and imho more intuitive than the old version was. You don't need tags for everything, and things were put in attributes.
- The format can and probably will be extended (it's eXtensible Markup Language, after all). By, for example, putting the application ID in an attribute, the format could potentially support multiple apps in one PXML (this would require breaking backwards compatibility, however)

Some new features that this format has/CAN have, but that aren't supported by the backend (yet):
- Description and title could be specified any language known to man. This includes languages that differ a only a little from each other, like UK english and US english ("en_GB" and "en_US"). The old format currently only supports "en"-style country codes, aka not variations.
- There can be an unlimited number of categories that the app can be sorted into. Currently, the backend supports 2.
- The filetypes could be specified as MIME types instead of by "*.suffix"
- The file association launch lines could support not having to have the filename at the end (like it has to have now)
- The format COULD support basic HTML in description and title. Might be difficult for the title, though.
- The format COULD be extended to support preview picture slideshows (by introducing timing). This will probably not happen, though.
- There's a possibility for the app author to add an email address. Yes, this isn't supported currently by the old format, even if the spec says so.
- As said before, the format could be able to store multiple application data in one file, by extending the current spec.

-------------------------------------------------------------

If you want an example file to see how the format is like, either clone the GIT repository up there, or tell me about it so that I can upload one and link here.
Also, a "spec" for the format (like the one ED did) is being worked on, to make the format understandable by developers

EDIT: Fight vs forum bugs

 

[skeezix]

Are you 'done'?

ie: "has to be done" items you don't plan on doing, or may do, but aren't related to this work per se

Specifically, as time is very tight for me at least, I'd like ot only merge it once (and after that it shoudl be easier if anyone finds bugs in it since it'll be the main tree.)

Should I work on a merge soon, or wait a day or two?

I've not looked at it at all yet, sorry.

jeff

(btw, woot

 

[Vimacs]

QUOTE
- The format can and probably will be extended (it's eXtensible Markup Language, after all). By, for example, putting the application ID in an attribute, the format could potentially support multiple apps in one PXML (this would require breaking backwards compatibility, however)

Multiple apps per pxml is kinda required to be in the format before release, it is a pretty important feture.
While it doesnt have to be actually implemented by the time the final image gets baked it as to be in the specification in a way that it wont brake backwards compatibility later on.

Also uniqueid is missing, we need that.

 

[dflemstr]

Well, the problems that I have listed remain. If you want to call this "done" is up to you to decide.

If yes:
The problems need a solution, and if you want to call my part done, you'd have to fix them (Or leave the problems as they are: not advised)

If no:
If you use git, you can just make a pull+merge once (which, by the way, shouldn't produce alot of conflicts at all, since you haven't changed anything on pxml yourself (hopefully) and therefore there's nothing that conflicts),
and you could then pull from the repo at github and get my changes when I add them (which also won't create any conflicts), and I could fix the problems listed.

I always stay up-to-date to your repository at git.openpandora.org, so I would take care of the merging of anything you change.

Git isn't SVN, I have made 5 branches and 45 commits during the development of the new PXML format (you won't see them because I rebased), and I never had any merge conflicts at all

If you don't use git, then
1. Why do you have a git repo?
2. You should consider using git, since this would allow you pulling from me, thereby giving you less to do, thereby lightening your burden.
3. If you don't want to use git, it might be a PITA to get my changes, so you should consider "calling this done"

 

[skeezix]

Put in unique-id; it is vital and absolutely required before a PXML.xml can be useful. (you can put that in the xsd, fail if no unique-id. unique-id is a string, not a number.)

My definion unique-id, btw is 'mostly unique', same sort of process as with naming java classes. (Rather than run a CGI that spits out unique-ids, and peopel could sitll break everythign by copy/pasting them, I'd rather people just guess at a reasonably unique value. Palm had peopel use aCGI and it didn't help much.) So an ID as far as I'm concernd is 'dev name + project name + whatever they want"). PErhaps we need to spec that out more.. ie: "project name COLON dev name COLON optional version if they wish each update to orphan previous update"

I'm not (at this point anyway) going to set up a automerge to anyone -- it creates a nice backdoor into the device which would be uncool; for now I'm going to vet any submissions so I'll be careful about it. (Not sure if that is the right or best approac, I know, but with time being tight its the best I can manage. Opening fully up with 50 peopel submitting changes will just mean we can't keep it together

Once things are 'final enough', I can step back and let it go nuts, just right now I don't think I can 'manage' with a lot of activity.

As such, can you make a 'done-ish' one, and I'll merge it, and we can go with it, and you can let me know every once in awhile when you've got some good stuff I need to repull.

Fair?

(I don't meen to sound like I'm overlording... hmm I hope thats nt how this sounds.

jeff

 

[dflemstr]

Multiple apps per pxml is kinda required to be in the format before release, it is a pretty important feture.
While it doesnt have to be actually implemented by the time the final image gets baked it as to be in the specification in a way that it wont brake backwards compatibility later on.

I could start working on it (aka changing the specification and adjusting the format so that multiple apps are possible), BUT that would require changing of the pxml API (it is currently not capable of handlig multiple apps without severe hacks), thus I do not know what I'm allowed to change.

However, if you could change pnd_pxml.h so that the API is to your liking, and has the features that you want, including those addressed above (only the accessor functions, not the internal struct as I would take care of that), THEN I could change the spec, parser, the implementation of the functions in your pnd_pxml.h etc. so that it just works.

OR you could of course do all that by yourself. I don't claim copyright to the spec or anything (let's say its GPL, shall we?), if you think it needs changing, go for it!

Also uniqueid is missing, we need that.

It's not. But I notice now that I didnt provide one in my example file; I wrote it by hand since I've been stuck on a machine for the last day with barely more than "vim" and "gcc" on it (aka no XSD checker) and to err is human .

However, the XSD and the "PND" test data pxml file has the id, and the parser of course:
CODE
<PXML xmlns="http://openpandora.org/namespaces/PXML" id="123456x86ls">
.............

I have committed a correction to the public repo.

If you think the ID location should be changed, give me an updated "example.PXML.xml" with your wanted structure and I will update the XSD, the parser and all other PXMLs.

 

[skeezix]

Its been a nutty couple of days ofr me so I'v enot had a lot of time, but will soon. Let me go with things as is for now, so we don't break "too much" right now. In a bit (week or so) I'll put in the API change for multiple applications and see about a quick mod to make it work, and we can go from there. (ie: you can do xsd changes, and/or you can do pnd_pxml.[ch] or whatever.)

Lets play it by ear.

Right now I'd like to keep code working day to day as we're working on window manager setup and such right now so pndnotifyd is handy to have working

jeff

Thanks mate

As to PXML PDF, ED owns that, once we merge everything up he can work on reworking that.

Unique-ID format has not been hard defined yet, fwiw; maybe "project-dev-optional-123", ie:

MAME-Franxis-123
SquidgeSNES-Squidge-
(no versioning on that last one)

Anyway, we can worry about that later

jeff

 

[dflemstr]

@skeezix EDIT: yea, do as you say, basically. Remember that its the new-pxml-format branch that is important.
I didn't mean an automerge(wow, haven't used that in ages), I ment that you could just pull from my repo whenever you need any new version of the PXML related things.
But of course, I agree, it could be unsafe if 1. someone was to steal my ssh key or 2. I went nuts, committed some dangerous stuff and you were to pull it without looking.

So, the 'semi-finished' version you want is already there, go for it, make the merge now.

It has the problems mentioned in post #1 but if you want to take care of it yourself I won't hold it against you (I have less to do then so hey, won't complain)
It is very important that these things are fixed (they are mostly not because of the new format btw, they all existed before, too); you WILL need to make it possible for apps to be in some other language than "en", "de, "fr", "it". The parser is written, as you can all see, with some small hacks to still allow it to populate the old internal structure; these hacks will have to be changed for your new API.

So yeah, pull the repo mentioned in the first post and get the changes, and I will write a spec à la PXML1.5, and then not trouble you any more (at least until there's something else that needs fixing )

 

[skeezix]

Anyone cluefull about unicode and languages?

I worked really hard on a bunch of insane unicode support a few years ago, but the new-baby sleep dep pretty much erased most of those memories and horrors

ie: iso8859-1 fis in char* no problem and manages a lot of languages, but obviously not unicode or every language. char* canhandle UTF-8 I imagine, but maybe libpnd needs to really talk UTF16 or be language-smart.

Thats a change I really expect coudl be an enormous amount of work.

But as in all things here, in the name of geting smething working _now_ I've ignored it. In a few weeks as things settle, maybe a big fork will be useful to attack that....

Anyone want to tackle researvhing the implications? I'm beat

jeff

 

[dflemstr]

Unicode: My suggestion is to stick with UTF-8, as it won't require any more changes in the code (aka, it fits into char*, so no wchar_t* required) than to set an encoding (+ Ångström probably uses UTF-8 everywhere already, since it's the worldwide standard).
There are no symbols in UTF-16 that can't be expressed in UTF-8.
However, you might get problems with string length, since some symbols use multiple chars/bytes and some don't, so you can't get the length of the (printed) string just by counting the chars.
If this is an issue, chose UTF-16.
Also, some devs that are stuck with windows machines will complain, since Notepad doesn't have unicode support at all (officially at least; there is a 'semi-hidden' UTF-8 save mode). Not much that can be done about that, though.

PLEASE do NOT use "iso8859-1", as it is an old standard and only officially used by Windows-PCs, and almost never in Linux (as you probably know; all the major distros uses UTF-8 for all languages). Also, it supports alot of symbols but not enough to satisfy all languages, which is a big - .

EDIT: clarified difference between symbol and char(=byte), hopefully.

 

[sindbad]

Also, some devs that are stuck with windows machines will complain, since Notepad doesn`t have unicode support at all (officially at least; there is a ``semi-hidden`` UTF-8 save mode). Not much that can be done about that, though.
There`s Notepad++ for stuff like that. Or if they just won't install better editors, Wordpad (yes, that thing) has better support for UTF-8.

 

[skeezix]

(And if you're curious the tortuous app I worked on years ago had to re-oriented the interface based on language, but then I ran into a dude who wanted (just to piss me off) Japanese and Yiddish at once, two back to back paragraphs. You know .. top down writing and then right to left, switching UIs as you went.

Life is hell sometimes (doubly so when youlet users define the spec..

jeff

 

[Cpasjuste]

Thanks for the help dflemstr

 

[White Flame]

So can anybody inform me as to why we're using something as ridiculously heavyweight as XML for fundamental tasks?

 

[Sphinxter]

Why not xml, it's fast,. easy and even a standard. You can even modify and shuffle your format without worrying about backwards compatibility. Anyone who wants to can join in with third party utilities can so with ease. If whatever you're using is too heavy try something lighter like libxml2.

 

[lulzfish]

"something as ridiculously heavyweight as XML"

heavyweight? :huh:

We're going to be talking about kilobyte-sized files here.
Maybe the menu programs are a few milliseconds slower because they have to look through XML tags instead of just some format made up specifically for the Pandora, but it gives you a lot of compatibility up front. It will be worth it for the standardization.

But seriously, heavweight? It's... text. It's not like we're using ODF here.

 

[Vimacs]

CODE

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10291 user 21 1 2900 488 360 S 0 0.0 0:00.06 pndnotifyd
10621 user 20 0 9928 6136 2044 S 0 0.6 0:02.00 matchbox-deskto


I guess 488k is bloated?

 

[skeezix]

Thats debug builds including all the symbols and such, too, I might add

We could've just used .desktop instead, in a lot of ways (since they hacked in multilingual support to the standards years ago.) But theres a lot of little hidden benefits -- pandora specific extensions for one, the ability to have multiple apps in a single PXML once we add that for another, the fact it can be self-checked and such because its XML and a much more consistent format than .desktop, the fact we've well defined some cool behaviour such as PXML user overrides, and so on. Its not a big heavyweight really, PXML is pretty small in its expended average and minimal form.

Anyway, sorry dflemstr, not had time to kerge you in; busy as hell week, and we've been hyper trying to get some things put together for the Olymptronica that EDs going to be presenting at; not sure if we'll get anything fancy in there, but we're trying to get the option there. (Why not hang ED out to demo stuff he's only seen for 10 seconds as we're hacking live, and dropping off anm SD to him at the show? woowoo )

I'll pull your PXML changes in early next week.

jeff

 

[White Flame]

I've done a lot of XML for customers, with configuration systems, data migration, file format translations, etc etc, and always find myself wondering why anybody would ever choose it.

But I guess the scope of this XML use is kept small so it won't bog down too much.

Are there any good XSD-aware XML editors for Linux? I've only used MS's stuff.

 

[dflemstr]

I actually wrote everything for my patch (schema, XML files, C-code, C++-code etc) win VIM. It has nothing more than syntax highlighting (or rather: I didn't set it up for anything else) but it served my purposes. I haven't done any validation during development though. This means that the XSD is manually created, and CAN contain, well not errors but some non-ideal solutions.

If you need a good XML editor, I'd recommend "xmlcopyeditor". It has a nice GUI for XML editing, XSD and DTD validation etc. Also supports XSL transforms, XPath evaluation etc. It's quite small and down-to-the-metal, no flimsy features that noone needs.
I haven't tried it a lot yet, but it seems to be enough for any XML-user.

@skeezix It doesn't really matter to me when you pull the changes in. This only makes it possible for me to iron out any bugs . I might even write a human-readable spec for the format this weekend, since the time is there.

Ah, and BTW, for anyone who's interested:
If you want to validate PXML, get the file "PXML_schema.xsd" from the repository, put it in the same folder as the PXML file, and add this to the PXML-tag:

<PXML id="youruniqueID" xmlns="http://openpandora.org/namespaces/PXML" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="PXML_schema.xsd"> .....