Release Repo Spec Improvements

[milkshake]

Hi,


I want to put a few suggestions forward for the REPO spec.


First of all We need to trim some fat from the Bacon of the repo spec, make it leaner...


we dont need to pass the description for example untill the end user wants to find out more specfic details about the app, but at that point the client could reqiest details for a specific app VIA a simple API


I have already implamented getting "only the updated/added apps" since past visit with a simple API


repo.openpandora.org/includes/get_data.php?last_updated=<unix_time_stamp>


SO what I propose is not sending a description untill the end user actually wants to read that info then we could implament something like...


repo.openpandora.org/includes/get_data.php?app_id=<pkg_id>


or something to those lines, which will send all the additional info reuired i.e. screenshots, description etc


also I would like to be able to add a couple new "small" additions to the repo JSON.


1. the date the package was added/updated.


2. the filesize of the package.


This should make the clients quicker at getting the basic info and letting the user know of any updates etc,aswell as providing a convenient way of getting any additional info needed.


please discuss and let me know if these sound like sane improvements and or suggest more improvements.


thanks.

 

[mcobit]

All this sounds ver practical.


I agree, that it is useless to send al the info in the first query.


An appstoreapp should just get all the nifo about an app, when the user clicks on "Details" or something.


So, good ideas.


Edit: Well, the icon should be sent somehow...

 

[milkshake]

Edit: Well, the icon should be sent somehow...

I agree the icon URL should perhapps still be sent but then again it all depends on how the client frontend is laid out, it may not need an icon, but maybe for now i think it should still be kept.

 

[Cloudef]

EDIT: Actually, no.


It's better to let client download the full database snapshot, and do everything locally and only download PND's from server than spam the server with various API requests. The update merging with timestamp, already slims down the JSON quite a bit. Only the first DB grab takes time. (Heck you could even compress it?)

 

[bzar]

How about we let the client decide what to download? I propose an optional query argument "fields" with a comma separated list of fields that should be populated for each application in the response JSON. Default functionality without the fields argument would be to get all fields. Repo should respond with at least the desired fields (can give others as well, depending on implementation), or return an error if the field type is unknown.

 

[skeezix]

Its hard to say whats best, we dont' really have too many 'use cases' to go with yet.


Right now its all good, but repo only ahs 200 apps in it.


I'm thinking, once the repo hits 5000 apps, its many MB downlaod just to see whats gong on; thats not bad, but .. Pandora say 200K/s to repo, suddenly adding up.


So my thoguht is two modes..


- the slim dump -- everythign needed to determine app list and package/app versioning


USE: get list of all apps


USE: figure out which apps can be updated


USE: figure out which apps repos have, which local doesn't have, etc


-- the full dump -- the works!


USE: local repo management with subsequent gets later


-- per-app get full info


USE: given the slimp dump, maybe we want to get the full for a given app


But I'm just winging it here.


jeff

 

[Tempel]

I fully agree with the addition of package update date and filesize, as well as the only only-updates URL. These could all be optional additions to the spec; none of these fields would be required, so this could be repo version 2.1. As I suggested in an earlier private conversation, the only-updates URL could be included in the JSON file, with "%date%" or something similar to indicate where the timestamp goes. Just to clarify: when you say "unix time stamp", you mean the decimal integer rather than the YYYYMMDD format, right? I feel like it would make sense to have both that timestamp and the one listing package update time in the same format.


Trying to slim down the repository file would break backwards compatibility, so we'd have to call that version 3.0 (no big deal, since the repo is still the only place to implement it). I think that much of what's included should stay; if browsing through available packages, having titles, descriptions, ratings, and categories doesn't take up much space, and makes it easy to scroll through and quickly get the gist of a package. But I do think that the applications array could be removed from each package listing. Much of its information is redundant, and I expect that any extra information in it is rarely needed. I'm thinking we could do away with it altogether, but perhaps it would instead be better to just replace it with a URL that can be used to obtain that information.


Also, I think skeezix's suggestion of a super-slim dump isn't a bad idea, but probably isn't necessary yet. Something that just contains package ID, version, download URL, and MD5 would be enough to act as a quick updater. But I think the only-updates URL is probably enough to make updates quick and easy.

 

[EvilDragon]

Are screenshots available by request as well?

 

[Tempel]

Yes. Each listed package allows for a list of URLs to screenshots; icons work the same. After obtaining the repository file, this allows screenshots and icons to be downloaded as needed by the client. I haven't yet implement this in PNDstore, but it's on my big todo list.

 

[milkshake]

@tempel im reffering to the time in seconds since Unix Epoch on January 1st, 1970.


for example right now it is "2011/05/19 22:17:00" when converted to unix time stamp its now "1305843420"


yes I think we should add this optional updates/new only to the spec aswell as filesize and date the package was added/updated


the benefits of those that use the get updates/new apps only means that once they have downloaded the database once, from then on in they only have to get updates which would be really small and that means faster download, less bandwidth needed


maybe the addition on an optional slimline version for clients that only need update information for apps.

 

[EvilDragon]

I would love to include PNDStore into the next HF. Tempel, do you think we could work together on creating a recipe so we can include it into the OS?

 

[Tempel]

milkshake: Yep, that's what I thought you meant. I think we're mostly in agreement, so the spec could be changed soon. But we still haven't figured out how package removal should be handled in the updates-only list. It should be simple enough to just do:

{

    "id": "<package_id>",

    "version": "removed"

}

or something similar, but will you be able to implement that? For that matter, might we be able to get away without implementing that? Also, should we remove application information from the spec right away, or leave that until a later update?


But note that, no matter what changes we make, you'll have to let an updated PNDstore propagate to users before implementing the repo spec yourself, or PNDstore might choke on the new spec.


EvilDragon: I'd be happy to help with that. I know very little about OE recipes, but I'll certainly help however I can. Would you rather install the files directly, or include it in a PND? Since I have your ear, I'll also add that fixing the libstdc++ and ctypes bugs would help , though they can, of course, be worked around.

 

[bzar]

Oh, I forgot to mention I fully agree with adding the upload timestamp. Being able to request only changed applications should really speed up update checks. I concur with using a unix timestamp, but would like to add that the timezone should be defined. I propose UTC.

 

[milkshake]

UTC is universal so yes, i was using UTC anyway i just forgot to mention it.


I think we can get away with sending removals, if the user tries to download a file and it does not exist tell them and ask them if they would like to remove it from the local DB.

 

[bzar]

I'm not doing a client so I'm just thinking out loud here, but what is the benefit of having two JSON "profiles" (full and trimmed) instead of freely selectable set of fields? However you define the trimmed profile there will always be cases where the application will want "that one field" from the full version and thus can't use the trimmed version, resulting in more traffic. Additionally, sometimes it may be enough to get even less data than the trimmed version provides. In both cases the resulting traffic and server processing (unless some kind of caching is implemented to provide a ready answer without generating one from the database) can be minimized by using customizable queries instead of profiles.


Disclaimer: I'm definitely not trying to get this done "my way". Just genuinely interested in the rationale and technical aspects on which the decision is based.

 

[milkshake]

I'm not doing a client so I'm just thinking out loud here, but what is the benefit of having two JSON "profiles" (full and trimmed) instead of freely selectable set of fields? However you define the trimmed profile there will always be cases where the application will want "that one field" from the full version and thus can't use the trimmed version, resulting in more traffic. Additionally, sometimes it may be enough to get even less data than the trimmed version provides. In both cases the resulting traffic and server processing (unless some kind of caching is implemented to provide a ready answer without generating one from the database) can be minimized by using customizable queries instead of profiles.


Disclaimer: I'm definitely not trying to get this done "my way". Just genuinely interested in the rationale and technical aspects on which the decision is based.

I like your way the best to be fair, tempel? would this be able to make it to the 2.1 spec perhaps?

 

[EvilDragon]

EvilDragon: I'd be happy to help with that. I know very little about OE recipes, but I'll certainly help however I can. Would you rather install the files directly, or include it in a PND? Since I have your ear, I'll also add that fixing the libstdc++ and ctypes bugs would help , though they can, of course, be worked around.

Cool. Well, I can take a look at those, but I'm not sure I can fix it with the current version of the OE tree we're using, should be fixed with a big new release.


Maybe I'll do some hacks.


I guess it would make sense installing the files directly. If other frontends can use command line options for it, then a PND doesn't make much sense.


If you want, I can setup an account for the GIT so you can put your latest versions directly there.

 

[Tempel]

B-ZaR: It's certainly an interesting idea, but it seems like an unnecessary degree of complexity. That's really my only objection, that implementing it repo-side, and making good use of it client-side, would be rather complicated without, I think, providing too much benefit. I've already said that I'm in favour of keeping most data in the spec (except for the applications list); most of the data is, I think, generally useful, and the slight decrease in filesize that would come from dropping some fields probably isn't worth the extra complexity. So I'm still in favour of a full repo, and maybe a super minimal one to complement it. Of course, I don't know all the use-cases that users and devs can imagine, so maybe I'll have to change my mind, but that's what I think for now.


EvilDragon: I thought command-line options could be given to PNDs through direct invocations of pnd_run.sh. Or is the issue that the calling program can't see the output? It's my understanding that minimenu can show only PNDs, not in-built programs, so I thought PND would be better. Or I guess a simple wrapper PND could be made to launch it. Regardless, if it's directly installed, the ctypes bug will need fixing, since the current workaround just includes the missing file in the PND. My work is currently on Github, so it can be brought into git.openpandora.org whenever convenient.

 

[bzar]

B-ZaR: It's certainly an interesting idea, but it seems like an unnecessary degree of complexity. That's really my only objection, that implementing it repo-side, and making good use of it client-side, would be rather complicated without, I think, providing too much benefit. I've already said that I'm in favour of keeping most data in the spec (except for the applications list); most of the data is, I think, generally useful, and the slight decrease in filesize that would come from dropping some fields probably isn't worth the extra complexity. So I'm still in favour of a full repo, and maybe a super minimal one to complement it. Of course, I don't know all the use-cases that users and devs can imagine, so maybe I'll have to change my mind, but that's what I think for now.

Fair enough. If it adds too much complexity compared to the "profile" based solution I can understand your stance.


I'm not in the position to give a statement on the increase in implementation complexity, as I'm not implementing a server interface (or even planning to), but I would have guessed it to be not much more than the second profile would add. Of course this depends on implementation details, but both of them boil down to adding conditional blocks that add data to the response only when certain condition is filled (full profile, or field mentioned in "fields").


Specification wise I can understand your position too. It's less complex to define two profiles than a set of fields that can be mixed and matched. I don't consider this too big of an increase either, but then again I'm not the one doing all the work


Any way you'll end up doing this, I'll respect your decision. Just voicing out my opinion

 

[Tempel]

And I appreciate your opinion . I think the idea's slowly growing on me, but I still say that, whether using selectable fields or a second minimal profile, it's too soon to implement before seeing real need for either.


But as for the updates-only data, we still haven't settled on how to indicate package removal. Are there any thoughts?