Release Repo Spec Improvements

[bzar]

The basic problem in package removal information is that it's not AFAIK supported by the current data model. Of course I'm talking about the backend implementation details here and not the protocol, but that's because the backend implementation in this case has implications on what is feasible in the protocol. Letting the client know what packages have been removed since $TIME would require either:


Keeping a log of package removals instead just the current state


This solution is not pretty, because there exists a possibility that an application has been removed and reuploaded within the update interval, which would require all removal log items be checked every time.


Keeping all package entries even after removal with a state "removed"


So far the best solution I can think of. Has potential problems with remove/reupload, but those should be very unlikely. If an application is uploaded when the entry already exists in "removed" state, it should replace the old entry. Of course this makes listing queries one argument more complicated (not removed). The actual data pointed by removed entries would of course be deleted. Has a possibility of collecting deprecated entries, but without the actual data they should be small enough to not become a problem. "Update timestamp" field would be rrepurposed to "status changed timestamp" field to accomodate update queries.


Sending the packages the client currently has to the server and comparing that to available applications


Not nice for obvious reasons, because it sends the client's package information to the server, which should not need to be done. Unnecessary sharing of information, web traffic and generally an ugly query. Also if the client has packages that never were in the repo, this solution would show them as removed.


OK, given that, I'd propose a "state" field instead of repurposing the version field, with possible states of "new", "updated", "removed". This would require two timestamps per application (addition, last status change), but has the added benefit of notifying users of new applications.

 

[milkshake]

And I appreciate your opinion . I think the idea's slowly growing on me, but I still say that, whether using selectable fields or a second minimal profile, it's too soon to implement before seeing real need for either.


But as for the updates-only data, we still haven't settled on how to indicate package removal. Are there any thoughts?

I think we can get away with sending removals, if the user tries to download a file and it does not exist tell them and ask them if they would like to remove it from the local DB.

 

[bzar]

^ A reasonable alternative. Come to think of it, why would we need to notify removals in the first place? What user story does it cover?

 

[Tempel]

Yeah, that's definitely the simplest solution. So when a client tries to download a removed package it just gets a 404 or other error, then tells the user "oops, looks like this one's gone!". I figured it would be a little cleaner to inform of removals immediately, just so that the list of available packages doesn't get cluttered with removed ones. But it's probably rare enough for packages to be removed. And if I really want PNDstore to not have this problem, I'll just have it download the full package list once a week or month or so, thereby clearing out any missing ones.


And you all know how much I like the simple solutions .


So, to summarize, these are the changes to be made to the spec:

1. Each package can have an optional "filesize" field, showing the size in bytes of the PND.
   
2. Each package can have an optional "modified-time" field, showing the Unix time at which it was created/updated.
   
3. The repository information can have an optional "updates" field, containing a URL. This URL will contain "%time%" somewhere in it; clients can replace "%time%" with a past Unix time in order to show only packages that have changed since that time. This information will be delivered in a format identical to the standard, complete repository.
   
4. The "applications" array will be removed from the specification; all categories, previewpics, source links, and licences will have to be specified for the complete package.
   

If there are any more comments, please make them now! If there are no objections, I'll update the specification on the wiki later today, then implement it in PNDstore.

 

[milkshake]

Yeah, that's definitely the simplest solution. So when a client tries to download a removed package it just gets a 404 or other error, then tells the user "oops, looks like this one's gone!". I figured it would be a little cleaner to inform of removals immediately, just so that the list of available packages doesn't get cluttered with removed ones. But it's probably rare enough for packages to be removed. And if I really want PNDstore to not have this problem, I'll just have it download the full package list once a week or month or so, thereby clearing out any missing ones.


And you all know how much I like the simple solutions .


So, to summarize, these are the changes to be made to the spec:

1. Each package can have an optional "filesize" field, showing the size in bytes of the PND.
   
2. Each package can have an optional "modified-time" field, showing the Unix time at which it was created/updated.
   
3. The repository information can have an optional "updates" field, containing a URL. This URL will contain "%time%" somewhere in it; clients can replace "%time%" with a past Unix time in order to show only packages that have changed since that time. This information will be delivered in a format identical to the standard, complete repository.
   
4. The "applications" array will be removed from the specification; all categories, previewpics, source links, and licences will have to be specified for the complete package.
   

If there are any more comments, please make them now! If there are no objections, I'll update the specification on the wiki later today, then implement it in PNDstore.

Dont forget %time% needs to be a unix time stamp UTC (so no local time)


also in regards to loosing the application details:


what if someone is looking for a specific emulator or app? which is bundled into a multi app package? how will you provide the results without this additional info?


something to think about... I personally would not remove that info but if most people agree it is not needed then thats cool.


1 final point.


also in regards to the additional info that my site contains I belive this should be made available in jason format too, but only via additional request i.e. seperate JSON stream - you cant expect all usefull information to be in the app description.

 

[Cloudef]

Yeah, that's definitely the simplest solution. So when a client tries to download a removed package it just gets a 404 or other error, then tells the user "oops, looks like this one's gone!". I figured it would be a little cleaner to inform of removals immediately, just so that the list of available packages doesn't get cluttered with removed ones. But it's probably rare enough for packages to be removed. And if I really want PNDstore to not have this problem, I'll just have it download the full package list once a week or month or so, thereby clearing out any missing ones.


And you all know how much I like the simple solutions .


So, to summarize, these are the changes to be made to the spec:

1. Each package can have an optional "filesize" field, showing the size in bytes of the PND.
   
2. Each package can have an optional "modified-time" field, showing the Unix time at which it was created/updated.
   
3. The repository information can have an optional "updates" field, containing a URL. This URL will contain "%time%" somewhere in it; clients can replace "%time%" with a past Unix time in order to show only packages that have changed since that time. This information will be delivered in a format identical to the standard, complete repository.
   
4. The "applications" array will be removed from the specification; all categories, previewpics, source links, and licences will have to be specified for the complete package.
   

If there are any more comments, please make them now! If there are no objections, I'll update the specification on the wiki later today, then implement it in PNDstore.

These changes sound really good, I also agree on application array removal , however maybe there could be small 'provides' array with application ID's?


Eg. 'PND': 'mplayer' -> 'provides': 'mplayer', 'memcoder'

 

[Tempel]

I recently realized that, if PNDstore is going to be included in the firmware, I'll only really be able to update its backend between hotfixes. Improved frontends can be distributed as PNDs, but the backend will have to be very solid by the time HF6 ships. So whatever changes we make to the repo spec now will have to stick for a few months.


With that in mind, the applications array: Most of it is often redundant, so I really think it should be removed or replaced. I just did a quick scan of every package on the repo, and only 12 of them contained more than one application; with 210 packages available, that means less than 6% have multiple apps. For every package holding just one app, the application information is likely going to be entirely redundant. Of the packages with multiple apps, there's typically one main app with one or more supporting apps - configuration tools or level editors, for example (though Gnome Games and Tux4Kids are notable exceptions). Therefore, for a huge proportion of packages, the package information alone should be enough to find what you're looking for.


So, is it worth keeping application information just for the sake of those extremely rare packages? If so, can we do it in a more efficient way? A "provides" array isn't a bad idea, but realize that this isn't quite the same as pacman's or apt's idea of "provides", since applications aren't necessarily the same as packages (also, I would probably just stick to calling the array "applications" ). Furthermore, application IDs aren't intended to be read by humans, so a list of IDs wouldn't necessarily provide any useful information. To be useful, it would probably need to include titles and descriptions, but then we're just gradually moving back to the complete applications array and all its redundant data. I had originally envisioned having an applications URL for each package that would allow the complete application information for a package to be delivered only once the client needs that specific data. However, this would not be conducive to searching; you would need to already be looking at Tux4Kids before you'd be able to see Tux Paint, so it kinda defeats the purpose. So I'm still leaning towards complete removal, but I'd love to hear other suggestions.


EDIT: I should mention that both Gnome Games and Tux4Kids mention all contained applications within the package description. Therefore, it should be easy enough to find those packages even when you're searching for one of the contained apps.

 

[Caine]

I recently realized that, if PNDstore is going to be included in the firmware, I'll only really be able to update its backend between hotfixes. Improved frontends can be distributed as PNDs, but the backend will have to be very solid by the time HF6 ships. So whatever changes we make to the repo spec now will have to stick for a few months.

Either that or you have to add versioning support in the backend such that you explicitly specify which version of the protocols to utilize at the start of a connection.

 

[milkshake]

ok cool, let me know when its finalized so I can implement the needed changes to repo's json output

 

[Tempel]

milkshake: Aww, but I was hoping for an argument! But really, you, me, and Cloudef are the only ones this directly affects right now, so if you have any opinions, now's the time to share.


Caine: My concern isn't with breaking compatibility. It's that any repo improvements made after HF6 is released can't be taken advantage of in PNDstore until HF7+. Other clients, like Cloudef's, can be modified continuously if they're not in the firmware, so the repo spec could keep changing, but I want to make sure PNDstore has everything it needs before HF6 is released.

 

[milkshake]

milkshake: Aww, but I was hoping for an argument! But really, you, me, and Cloudef are the only ones this directly affects right now, so if you have any opinions, now's the time to share.


Caine: My concern isn't with breaking compatibility. It's that any repo improvements made after HF6 is released can't be taken advantage of in PNDstore until HF7+. Other clients, like Cloudef's, can be modified continuously if they're not in the firmware, so the repo spec could keep changing, but I want to make sure PNDstore has everything it needs before HF6 is released.

the only thing in addition to filsesize and added/updated date is an option to have "additional info" part of the site be pulled if the end user wants to read it.

 

[Caine]

Caine: My concern isn't with breaking compatibility. It's that any repo improvements made after HF6 is released can't be taken advantage of in PNDstore until HF7+. Other clients, like Cloudef's, can be modified continuously if they're not in the firmware, so the repo spec could keep changing, but I want to make sure PNDstore has everything it needs before HF6 is released.

Can't be taken advantage of by the builtin PNDstore, that part is true. There is nothing preventing you from packaging newer versions of PNDstore into a PND as long as you always have support for both the development version of the repo spec and the stable version included in the firmware.


Alternatively, you could have the firmware version of PNDstore auto-update itself (although it is a bit more risky).

 

[Tempel]

the only thing in addition to filsesize and added/updated date is an option to have "additional info" part of the site be pulled if the end user wants to read it.

My only concern with that is how it would make sense in general use of the repo spec. "Additional info" doesn't say much about what kind of information it holds. Maybe the spec could consider it an "extended description" or something? If so, should it allow for multiple languages like title and description do, or only one language?

There is nothing preventing you from packaging newer versions of PNDstore into a PND as long as you always have support for both the development version of the repo spec and the stable version included in the firmware.

For sure. But I'd rather avoid that if I can; by keeping the backend consistent, both the PNDstore GUI and the minimenu interface will use the same package lists; only one needs to request an update for the other to see it. Sending out a newer version in a PND could therefore cause confusion and irritation. It's certainly possible if necessary, but not ideal, so I'm doing my best to avoid it.

Alternatively, you could have the firmware version of PNDstore auto-update itself (although it is a bit more risky).

Oh please god no. It'll have to be included directly in the firmware, not encased as a PND. Auto-updates would be the worst. Unless, of course, ED gets the firmware opkg repository working. But I would not want to be the one to implement this in PNDstore.

 

[skeezix]

It'd be easy enough to check in the code, let ED run the 'build' to spit out the ipk, and then stick that intio a pnd as an installer..


Self updating python code is not too hard though, either.


jeff

 

[thatgui]

Would a "Update Reason" Field make any sense / be of any use ? So the user could easier decide wether upgrading is worth the while or not. Just a thought - crush me if you like

 

[milkshake]

*CRUSH*

 

[thatgui]

*CRUSH*

I demand a nicer crushing next time - I thought it was a good idea, because every time I see an update of a pnd in the repo I miss a "thats the things that have changed" text in the pnds details page.

 

[milkshake]

Tempel has mentioned we could have an "extended description" which would contain the info from the "additional info" part of my website, sometimes it has changelogs sometimes other stuff.

 

[Tempel]

Would a "Update Reason" Field make any sense / be of any use ? So the user could easier decide wether upgrading is worth the while or not.

To elaborate on what milkshake said, that would be troublesome just because that information can't be extracted directly from a PND; the uploader would need to enter this information manually, and it's not always easy to make people do that. But some people do use milkshake's additional info to show changelogs. So we'll let that additional info be included in the spec; if the uploader put a changelog in there, you'll be able to see it, but it's entirely up to the uploader to do so.


Also, I'll make all these changes on the wiki later today. For real this time.


EDIT: Okay, I lied. It's bedtime, and I haven't finished it yet, so I'll do it tomorrow.

 

[Tempel]

Version 3.0 has now been posted to the wiki. A summary of the changes made:

• Added optional "updates" URI, "info" text field, "size" integer, "modified-time" integer.
  
• Changed "md5" and "description" to be optional.
  
• Removed "applications" array; therefore moved "previewpics", "licenses", "source", and "categories" into package information.
  
• Reordered to put all mandatory fields first (note that field order need not be reflected in implementations).
  
• Changed example data to use URIs from milkshake's repo and data from Ivanovic's full PXML sample.
  

Let me know if there are any more last-minute changes you'd like to see. milkshake, you should be able to implement this whenever you want. I think the current version of PNDstore should be able to read the new version, but please keep the old version on hand in case something breaks.