PXML specification

[sebt3]

@Tempel : I wasnt aware of a repos spec myself. So far I like what milkshake is doing.


Having something dedicated to PND would be way better than any solutions currently available. (for both users and devels)

 

[Klaue]

I think you misunderstood Klaue's suggestion. In this case, he's suggesting no change to the PXML, only a change to the repository specification. The idea is that, for each application listed in the JSON repo, you would add an extra field, "md5", with the value of the MD5 hash you calculated, preferably in hexadecimal. Then, my client downloads the JSON file, including all the MD5 hashes; when the client downloads a PND, it calculates the hash of the PND and compares it to the hash listed in the JSON. That way the client knows that the PND was downloaded properly without errors.

Not exactly, but close. I seem to have a stunning ability to not make myself clear


Yes, the hash calculation would be server side, but there's no need to do it client-side (except for checking the integrity of the downloaded file, but with todays internet.. when was the last time you had to redownload a file because your download failed? The last time that happened to me must be 1998 or so )


I'll try to make an example. Suppose you have a PND, called "You can't get ye flask", and you have 3 repositories


Those repositories have the following data of that PND:


repo1: id=klaue.youcantgetyeflask hash=d41d8cd98f00b204e9800998ecf8427e


repo2: id=klaue.youcantgetyeflask hash=dd1310245728cc97d7458bd562815d59


repo3: id=klaue.youcantgetyeflask hash=d41d8cd98f00b204e9800998ecf8427e


As you can see, the hash of repo2 differs from the rest. Now assume that repo1 was the one with the highest priority in the client (probably the official, if there will ever be one, or the largest of the existing ones). So what your program would do is, from the start:


1. Download all JSONs of all repositories.


2. Build a list of aviable PNDs


3. (Do other stuff, like checking which ones are allready installed)


4. The user searches for "You can't get ye flask"


5. You iterate trough your repositories, in descending priority order, until you find the game


6. You found it, in our example case, right in the first repository


7. Now you search the remaining repositories for the same ID - you'll find them in repo2 and repo3


8. You check the hashes, taking the hash of the highest priority repository (of the ones in which you found the file, in this case repo1) as the master


9. You'll notice that repo3 has the same hash, but repo2 has a different one, so you discard repo2 because it's probably a counterfeit entry or at least a repackaged one, but it's better to be save than sorry.


10. If your user wants to download the game, you'll download it either from repo1 or repo3, but NOT from repo2.


11. Because you have, in this case, 2 servers to download the file from, you can do some client side load balancing, for example if the user downloads another PND, you can parallelly (sp?) download that PND off repo1 and the youcantgetyeflask one from repo3. Or you can download the first part of the PND from repo1 and the second part, at the same time, from repo3 and then stick the parts together again.


The reason for that is that, in most cases, the clients download speed will be better than the servers upload speed, so the client will be faster if it downloads from multiple sources at the same time. It would also be lighter on the servers.


of course, you can do steps 5-9 inside step 2, but that doesn't matter


now clear what I meant? I don't really know how to say this any other way, sorry


EDIT: maybe to clarify: The version of the PND that is deemed the valid one is the one on the highest-priotitry repository. The reason for this is that people would probably configure the client to make the official repository (if there will be one) or the largest repository the highest priority. Now, If I made a PND, I would maybe upload it to various repositories, but I would sure as hgell upload it at least to the official or largest one.


If someone would build a malicious PND with the same ID as another one, he couldn't upload it to that repository because the valid version is allready on there (and of course no repository would allow a random user to overwrite an entry by another user), so he would have to upload it to a smaller repository.


With my proposed safety measure, using the hash of the highest priority repository would ensure that a bogus version on some of the smaller repositories wouldn't get through.


(Note that I assume that there will be multiple repositories in the future - the whole repository spec wouldn't make much sense if there weren't)

 

[milkshake]

yeah that makes sense - thats what tempel meant aswell when he was re-describing ur post.

 

[Tempel]

Yep, I understood all that Klaue, I was just saying that it could also be used for validation. And it's easier for me to code in download validation than it is for me to code client-side load balancing, so that's what I'll probably do first. But I've got the load-balancing in mind for the future.


@sebt3: The repo spec was created by dflemstr when he created The Box. It's on the wiki. Even though The Box isn't being used now, milkshake has made a similar PND-centric site that provides a repo conforming to the same spec. And I'm working on a downloader/updater that reads the spec, so it can be used with any site that conforms to it (milkshake's, or The Box, or anything else people make).

 

[milkshake]

so for now should I add my md5 file hash after the app.id in the JSON? what do you think?

 

[sebt3]

@sebt3: The repo spec was created by dflemstr when he created The Box. It's on the wiki.

Hoo, I didnt knew that.

so for now should I add my md5 file hash after the app.id in the JSON? what do you think?

I'm all for it. Get that in the spec too.

 

[Tempel]

It sounds like we're all in agreement, so I've added it to the spec. Two final questions before we really finalize it:


1) Should it be a mandatory or optional field? I've put it as mandatory simply because it's useful and should not be difficult for other repository-creators to add. But it's not entirely necessary, and having it as optional lowers the barrier ever so slightly to other sites adopting the standard.


2) Should it be encoded in hexadecimal? It could instead be encoded in base64 to save a little space, or encoded as an integer. But hex is maybe easiest?


Let me know what you think.

 

[milkshake]

I think it should be mandetory simply as its the smartest thing to do and is really simple to code anyway, in regards to format I don't mind as long as its simple to implament, I mean I already have it stored as md5 hash but before output if its simple to convert the hash into somthing else i can do it.


Or would i need to do this to The file itself and have another column in my table?

 

[Klaue]

I'd also make it mandatory - if you use it for load balancing, the whole thing would fall flat on its face if one of the servers wouldn't implement it.


I would maybe add a reason for it on the discussion page (e.g. copy paste my post above ), but since I do not have an user account and do not plan to get one.. oh well. The idea of a wiki where you need an account to change anything is against my taste anyway.

 

[milkshake]

I think we also need to include subcategory into the spec and the alpha beta release tag that was added into the PXML spec, will allow you to filter final from non final apps

 

[Tempel]

One thing at a time!


It sounds like we're pretty much in agreement on repo spec 1.1, so milkshake, please implement it whenever you're so inclined.

 

[milkshake]

great stuff, I will do it over the weekend.


in the mean time what do you think it will be benificial to have an optional subcatwegory object within categories? and what about the alpha,beta, release tag?

 

[Klaue]

in the mean time what do you think it will be benificial to have an optional subcatwegory object within categories?

The category is used for the placement in the pandora menu and conforms to the category names of the "Desktop Menu Specification" from freedesktop.org, which is also used in Gnome, KDE etc. I guess it's directly translated to that format and only in an array because that looks better.


Full Spec: http://standards.freedesktop.org/menu-spec/latest/


Categories: http://standards.freedesktop.org/menu-spec/latest/apa.html


As you can see in the examples of this, those entries are flat lists, you just list all that apply. Say your app was an action adventure, so you would want to put it in games and it's subcategories action and adventure, so you'd write "Game;ActionGame;AdventureGame"


or in the PXML/Server Spec


"categories": [


"Game",


"ActionGame",


"AdventureGame"


]


...scrap that. just before submitting, I took a look at the PXML spec to make sure I diddn't write bollocks and lo and behold, I did. They did use subcategories in it. That makes no sense to me because of the aforementioned stuff and because you could very well have more than just 2 menu levels, but oh well. I personally would leave it as a simple array, but when they did use subcategories in the PXML, for whatever reason, then.. well, use what you want, I guess.

 

[milkshake]

it just means we can have more than just a game category in a native app as game covers lots of stuff including emulators, so it would be beneificial if a subcat is supplied that we could take advantage off it just like the menu does on the desktop.

 

[Klaue]

it just means we can have more than just a game category in a native app as game covers lots of stuff including emulators, so it would be beneificial if a subcat is supplied that we could take advantage off it just like the menu does on the desktop.

As you can see in the examples of this, those entries are flat lists, you just list all that apply. Say your app was an action adventure, so you would want to put it in games and it's subcategories action and adventure, so you'd write "Game;ActionGame;AdventureGame"

 

[Yannick]

A changelog field would be nice.


I like to know what changed

 

[darfgarf]

A changelog field would be nice.


I like to know what changed

then people should get into the habit of putting things liek changelog and especially license in hte documentation file, at elast licenses are supposed to be visible to anyone who has a copy of the program, and if wrapped in a pnd most people would call that invisible

 

[milkshake]

A changelog field would be nice.


I like to know what changed

this could be added to the description if needed.

 

[Tempel]

it just means we can have more than just a game category in a native app as game covers lots of stuff including emulators, so it would be beneificial if a subcat is supplied that we could take advantage off it just like the menu does on the desktop.

I'll agree with Klaue on this for now. Just list all categories (including subcategories) right next to each other. That will still give enough information for a client to be able to sort applications intelligently. In fact, I would say that PXML's category/subcategory distinction is a mistake, and a flat list is how it should be (that's how it's used in .desktop files). So the current repo spec is fine, I think, as long as you're willing to include subcategories in the categories list.

 

[milkshake]

of course, not a problem.