Release Pxml Fileformat V1.5

[skeezix]

Sorry, I'm babbling here and probably not making any sense.. long nights

- its not only my stuff in there, just most of it is (I like to have credit where credit is due)

- it certainly isn't perfect (the spec or the code); we only have so much time and resources The goal is to be _functional first_, with a decent overall design, and fit it up as we go (ie: so in general I try to hide crap and brute force and forth behind black box handles and functions, so they can be replaced without API changes.) As long as the overal design idea is good, we're cool

- things are ever chaing.. we've just now decided on a pnd file format change and some code there will get added and some redone. Woot

- a lot of this is just to get something in there, to lay out a future direction; sometimes direction is enough, so others can run with it, as long as there is a consistent ideal, but we'll get as much in as we can. ie: I'd rather avoid 50 devs creating their own functions to dim the backlight, when we can just include it in one place; by creating libpnd, I'm hoping people will submit patches for what should be common functions, and we'll hgave a few maintainers who ensure quality, and voila, makes everyones life easier. But some of it, like 'pnd file' support is awesome, and needs to be well defined day zero or doomed to fail .. so we have direction, we have the start of a framework, and so on

- a solid PXML spec woudl be nice; there _is_ such a thing as 'nearly XML' .. just like we all just do 'printf <foo>' and not create a whole XML schema and then have an XML api record it out. Its simple to parse basic XML and XML-like. Full XML support is hard.

- anyway, there is PXML spec, and theres the real implementations .. ie: BB Code is certainly way out, as far as I'd say; having every app work on some bbcode parsing etc would just suck. I'd rather it was real HTML if anything at all, but I'd rather it was just all plaintext in there myself. still, why not shoot for the stars?

EDs specs are pragmatic; they're not set in stone, they're 'living documents'. Make comments.. great make patches.. great

jeff

Don't try to put my pride in it We're trying to build something halfway well in goofy times with limited resources. We're not building things to support anyones ego. We're professionals here.

Sillyness!

jeff

 

[Polossatik]

This is indeed not moaning, this is a prime example of how open source gives added value, someone with time and knowledge about XML offering to to make this a better system. Most likely someone else will then add some more/better idea's etc..

thank you for doing this!

 

[Cpasjuste]

Thanks for the fast response. I guess the current codebase is at git://git.openpandora.org/pandora-libraries.git ?
Then I will make a patch, and you can decide wether it's worth it or not.

I did modify the PXML structure when i added tinyxml to libpnd.

In fact, i never worked with xml stuff before so i just did something easy to implement for me in libpnd. Like skeezix said, we have an hudge amount of work to do.

Of course, if you have the time to redefine the xml structure and add the parser for it in libpnd, your more than welcome !

Edit: yes it's the current codebase

 

[dflemstr]

Thanks for all your responses, now I know how the situation looks like.
I have as I said already looked at the code, made a new "sample" PXML file and started on a XSD for it, and modified the parser (not yet the writer) to accept the new format.

Step 1 will be to just use the existing pnd_pxml_t and pnd_pxml_handle, since the primary goal that I have when doing this is to change the file format, not the data stored in it. I really don't want to add more headache, if I can avoid it.

Step 2 will be (if possible) to overcome some of the bugs that the current pnd_pxml_t has. The structure only supports 4 languages at the moment (en, de, it, fr), and max 2 pictures, max 3 file associations, and no email field for the author (which, by the way, is in the current spec). Again, I will not change the existing functions (like pnd_pxml_get_description_en(...)) but will add support for an indefinite amount of languages (with something like pnd_pxml_get_description(..., const char *isoLang) on top of that) and maybe make higher level functions like "bool pnd_pxml_is_standalone(...)", to avoid saving the standalone field as a string in pnd_pxml_t.

When I have gotten somewhat further, I will make a post in the dev forum. I really don't want to fill a thread in the news section with techtalk

 

[rabidpoobear]

Thanks for all your responses, now I know how the situation looks like.
...snip...
When I have gotten somewhat further, I will make a post in the dev forum. I really don't want to fill a thread in the news section with techtalk

I'm glad you didn't get discouraged at the beginning of the thread. I have worked with ill-defined specs before and it is a HUGE pain. Just because libpnd is available doesn't mean everyone will be able to use it... what about people who want to write in HASKELL or Caml or FORTRAN or something? If their language already has an XML parser, it's MUCH easier for them to add support, if the format is valid XML, than if it is not XML compliant.

tags such as <pic1> <pic2> are absolutely not okay, I can say that right now. At the most, maybe a
<pic default="true">blah1.png</pic>
<pic>blah2.png</pic>
...etc
would be okay. I sincerely hope you have time to fix this, dflemstr, as I plan on doing a lot of Pandora development and I'd prefer if I can write my own PXML generating tools without having to reinvent an XML writer.

Remember, it's not about making it easy to write the files themselves (PXML files) it's about making it able to be used in every application XML can be used. And if you're going to break from the XML spec, you may as well not use anything resembling XML and instead go with an ini-style doc, those are much easier for the end user to write than PXML.

Also, please post a follow-up link to the Dev posting in this thread when you start on this, dflemstr. I'm interested to follow your progress.

 

[dflemstr]

Also, please post a follow-up link to the Dev posting in this thread when you start on this, dflemstr. I'm interested to follow your progress.

I've already started actually, as I said before
Status quo: I have the parser 100% complete. It can read the new format, and devs that use the current libpnd api don't have to change anything in their apps.
I have the standard complete "in my head", and 2 sample files, but not as a full XSD yet, I have started writing the XSD however.

What thus must still be done is the following:
- I'll finish the XSD. Should be easy.
- The current system has some limitations due to the internal design of libpnd. Look at Step 2 in my previous post to know what they are. I hope to be allowed to add some API functions and change some of the internal structure in libpnd to fix these problems.
- There are some problems with the format that can be changed. For example, the old "associations" system uses extensions ("*.bmp") instead of MIME-types("image/bmp"), which is a no-go for Linux. I hope to be able to communicate with the devs and try to fix this, because then it will become possible to open a file even if it has the wrong extension.
- I have to make a PXML writer for the new format. This should be easy, thugh; but I have to do this last, since I need to know beforehand if I'm allowed to do point 2 and point 3.

So, when I've finished the XSD, and made a very simple writer for the format (that still has the bugs mentioned in point 2), I will put up a GIT repo to pull from, the XSD, new documentation, example files, etc. in the dev forum. This should probably be possible until tomorrow (I don't promise anything).

Until then, please don't post too many posts that are directed at me, since I then feel like I have to answer them, which makes the development time longer

 

[PSyMastR]

May I make a suggestion?

<PXML version="VERSION_NUMBER">

That way, if the spec changes after the pandora is released, and there is a lot of stuff already supporting it, the pandora lib can support legacy and new versions without authors having to update their game/app that they wrote a while ago and don't plan to update it.

 

[dflemstr]

Good idea; however, one of the main reasons why you use XML in the first place, is that it's very backwards compatible. You can add tags in the future that are supported by newer menus etc, and the new files still work with the old menus. So, at the moment, there is really no reason why you would have to have a version attribute.

 

[skeezix]

Sure, feel free to modify the API (ie: I'm the owner, so I in turn will feel free to rename your functions or whatever

I havew always wanted to make a pnd_pxml_get_by_lang ( char *isolang, ... type routine; it makes me happy inside. I really dislike having 35 functions with a similar name.

(Someday we need to have unicode for all the internal strings, too, which hurts. I had started all this in strict C (not C++) to make easy portability and bindings to everything, but tinyxml got pulled in and its c++, so for things needing tinyxml some minimal C++ is in use. In truth, it should almost be full C++ for 'string' and tinyxml's unicode support, but its too late to make it all proper fancy OO Design (and that'd make it over complicated perhaps.. I've delivberately kept it very simple API so far.)

Maybe a fancy OO C++ wrapper on top?

(I had also intended to use expat earlier, but tinyxml seems faster and pretty simplified as well.)

Anyway, yes, feel free to submit to me a patch that modifies a few things, and I'll see about merging it in.

_Open_Pandora right?

As to structural changes, you really only need to pess with pnd_pxml.[ch] and pnd_tinyxml as I mentioned; perhaps the pnd_discovery code which copies bits from pxml to disco_t (since the API will change slightly as your comments), but like pndnotifyd and ./test/discotest will be unchanged since the changes shoudl be 'under the hood.'

Defiantely try to keep your changes to the PXML related though, as I'm working again on a bunch of changes through the codebase, so want to keep your merge easy

Once we launch the device, more info will nbe public (ie: dev wiki and such). Its complicated right now .. openikng up evberything is fun, but then leads to endless questions from people (which can be fine, but it eats an enormous amount of time.) So we're going by seat of pants.. seems the best way

jeff

Oh, while your'e at it..

Can you wrap the PXML internals with an ..

<app>
..
</app>

ie:

We'd like to build into the spec that multiple apps within a single PXML can be supported, but libpnd will for now only support a single one.

Spec to handle multiple, but code to handle one; we'll add multi support later.

(The reason for this is so a single .pnd or PXML.xml-app-dir can have multiple entry ponts, but only a single PXML.xml file is permitted.)

jeff

 

[dflemstr]

I had originally intended that this would be a small hack that would make a new format that was more XML compatible. But sure, I will look into 'cleaning up' the API a little and try to add an <app>-tag. I am currently stuck with the XSD and making the format compatible to the current internal structure, so I'll add that to the bottom of my TODO-list for now.

 

[skeezix]

Leave the <app> tag out then; I can add it later, its a trivial thing. (ie: to change the tinyxml handler, and nothing else. The libpnd side can be later.)
(and the spec and xsd of course.)

Cleaning up the API is not needed per se.. just where you're changing things for iso-lang, I think it best to ditch the hardcoded language functions. Duplication would be silly.

Don't sweat it -- I'll take care of it at merge.

Just change the tiny xml handler to your revised spec and call it a day.

jeff

 

[dflemstr]

Leave the <app> tag out then; I can add it later, its a trivial thing. (ie: to change the tinyxml handler, and nothing else. The libpnd side can be later.)
(and the spec and xsd of course.)

Cleaning up the API is not needed per se.. just where you're changing things for iso-lang, I think it best to ditch the hardcoded language functions. Duplication would be silly.

Don't sweat it -- I'll take care of it at merge.

Just change the tiny xml handler to your revised spec and call it a day.

jeff

The <app> tag is, as you say, a trivial thing to add. Just make pnd_pxml_fetch{,_buffer} return arrays and rewrite the parser a little

And, I really wouldn't mind reviewing some of the accessor functions for pnd_pxml_handle. For example, the same issue exists with categories, associations and pics as with the descriptions and titles: multiple functions instead of one that gives you an array. And why doesn't *_background etc return a bool already at that level? Would save some space, give better performance and ensure reliability.

My goal is to finish the new spec (it is already readable and can populate the pnd_pxml_t file completely) and THEN look at some of the accessor functions, because at the moment I have to use hacks to populate parts of the struct. Example:
CODE
//TODO: Fix pnd_pxml_t so that there can be more than 2 category 'trees' and more than 2 subcategories. Then this can be removed.
switch (i)
{
case 0: //first category counts as the main cat for now
app->main_category = pnd_pxml_get_attribute(pElem, PND_PXML_ATTRNAME_CATNAME);
break;

case 1: //...second as the alternative
app->altcategory = pnd_pxml_get_attribute(pElem, PND_PXML_ATTRNAME_CATNAME);
break;
} //code to handle subcategories follows, with similar hacks


The format currently supports more than 2 category trees. (What if something is a "Game", but belongs to "Internet" and "Graphics" aswell?)

So, it would actually ease my burden if I DID have to rewrite the api, as things would become alot easier for the parser.

 

[panik]

I just want to post a word of encouragement to the efforts of turning the PXML-specification into real, valid XML. This makes a lot of sense. Thanks!

Don't reply, just keep on patching!

 

[.Gogeta§§J4BR.]

just because you have worked a lot to come up with something doesn't mean it's as good as it can be.

you don't need to make the structure easy to read by eyes, it should be flawless to understand with a program.
one thing is "showing work" to humans, the other is properly making something consistent.

Besides the several good points people have made here (many of them were NOT yet-another-complain-over-our-precious-decisions), I guess there's no need to change everything as you already have apps to easily create/edit/use files inside this structure (whether people like it or not).

Objective complete: The pandora software developers now have a good and easy to use way to make their apps appear categorized, with a nice description and a fancy image in the Pandora.

That's good enough.

So the whole problem is solved and everyone should be happy with it

 

[Tor]

I just want to post a word of encouragement to the efforts of turning the PXML-specification into real, valid XML. This makes a lot of sense. Thanks!

+1
+1
+1
..

 

[dflemstr]

Please continue the discussion about the new format here :
www.gp32x.de/board/index.php?showtopic=46924