milkshake
Advanced Member
@rockthesmurf I dont suppose you want to have a go at parsing the file archive for the download links? 
(dont you just hate cheeky requests)
(dont you just hate cheeky requests)
Pandora pandora repo/website/native app-manager development
- Thread starter milkshake
- Start date
Tempel
Active Member
Well milkshake, it sounds like you're already contemplating the next version of the repo spec. I don't think we should be making any changes quite yet; we probably want to wait to see where problems appear first. And even if we do decide on changes, don't go implementing them until we've formalized them.
Having said that, we can certainly start brainstorming!
A datestamp could be useful to save on bandwidth and processing. I imagine this would be something like a "last-updated" field within the "repository" field. Problem is that I could use this to quickly determine if there have been no updates, but I can't think of how I would robustly implement something that gets only as many bytes as necessary and then parses it without having the full JSON page there. I'm sure it could be done, but I'm not sure all the hacking would be worthwhile; after all, the repository page shouldn't be all that large (yours is only 68kB so far). Then again, I'm not very well versed in web technologies, so maybe this would be easier than I'm making it out to be. As an alternative, isn't there some way for a server to say "this page hasn't changed since you last requested it"?
App ratings could be good; my only concern would be that different sites would implement different ways of rating software. Perhaps the spec could include an optional "rating" field for each app that's given as a percentage; then, any spec-implementing site would just have to translate their own rating system into a percentage, and any client could display this percentage however it wants. Now, when comparing with traditional package managers, a rating seems a little superfluous; however, when comparing to commercial app stores, it doesn't look like a bad idea.
Comments trouble me, simply because how many would be included? how big would the repo page become? Perhaps comments could be handled like icons: if available, a URL to download them (also in JSON form?) would be included for each app. Again, though, this seems superfluous for a package manager, and possibly even for the client-side of an app store. Perhaps, instead of including comments, each app could include a link to its human-viewable page on your site; if a user wants to see more about an app than is available in the repo, the client could launch a web browser and open that page, whereupon comments (and any other extras, like screenshots) would be viewable.
I wrote too much. Keep up the good work, and I'll get that client caught up to yousoon eventually.
Having said that, we can certainly start brainstorming!
A datestamp could be useful to save on bandwidth and processing. I imagine this would be something like a "last-updated" field within the "repository" field. Problem is that I could use this to quickly determine if there have been no updates, but I can't think of how I would robustly implement something that gets only as many bytes as necessary and then parses it without having the full JSON page there. I'm sure it could be done, but I'm not sure all the hacking would be worthwhile; after all, the repository page shouldn't be all that large (yours is only 68kB so far). Then again, I'm not very well versed in web technologies, so maybe this would be easier than I'm making it out to be. As an alternative, isn't there some way for a server to say "this page hasn't changed since you last requested it"?
App ratings could be good; my only concern would be that different sites would implement different ways of rating software. Perhaps the spec could include an optional "rating" field for each app that's given as a percentage; then, any spec-implementing site would just have to translate their own rating system into a percentage, and any client could display this percentage however it wants. Now, when comparing with traditional package managers, a rating seems a little superfluous; however, when comparing to commercial app stores, it doesn't look like a bad idea.
Comments trouble me, simply because how many would be included? how big would the repo page become? Perhaps comments could be handled like icons: if available, a URL to download them (also in JSON form?) would be included for each app. Again, though, this seems superfluous for a package manager, and possibly even for the client-side of an app store. Perhaps, instead of including comments, each app could include a link to its human-viewable page on your site; if a user wants to see more about an app than is available in the repo, the client could launch a web browser and open that page, whereupon comments (and any other extras, like screenshots) would be viewable.
I wrote too much. Keep up the good work, and I'll get that client caught up to you
Rockthesmurf
Advanced Member
milkshake said:@Rockthesmurf I dont suppose you want to have a go at parsing the file archive for the download links?
Interestingly, aside from being very busy, I have been watching this thread, and realized I hadn't contributed in a while, and scraping the file archive was the next thing I thought I could do. So, as of yet, nope I haven't done anything, but it is the next thing I was going to look at (unless anyone else gets in there first).
It feels like we (you) are making good progress though, which is great!
Steve
Last edited by a moderator:
milkshake
Advanced Member
UPDATE:
ok i have now the beggings of a very basic (mish mash / unstyled) functioning download/upload website
its obviously not complete but functional.
upload your apps here
browse apps here - click on an apps title to get more details plus the option to download the application.
the very very basics are now complete
things to do next:
ok i have now the beggings of a very basic (mish mash / unstyled) functioning download/upload website
its obviously not complete but functional.
upload your apps here
browse apps here - click on an apps title to get more details plus the option to download the application.
the very very basics are now complete
things to do next:
- add registration page for people who wish to upload an application
- add comments / app rating / report app utilities
- add downloadlist for logged in members (kinda like a shopping basket) - this will save a list of apps just for the current session you wish to download, then you can download them at your luiesure
- give website a proper theme
- add more details from PXML i.e. different languages, authors/apps website
- the ability to allow people up update an app but before it gets added an email gets sent to the original app uploader/author to confirm its ok and everything is correct
- the ability to edit certain things before submitting to database i.e. description
- the ability to get apps that are in the file archive automatically
- option to add screenshots to existing applications
- + others ideas that I may have or have had but cant think of right now.
milkshake
Advanced Member
UPDATE:
fixed a few things.
added more details to the app details pages including download counter.
added filter to top of apps list.
added extra checks in the pxml data stripper for categories so they match the standard.
added some extra db tables and columns in preparation of additional site updates.
that's it for tonight, doesn't seem alot but i spend a good few hours on it, but im plodding along nicly
again details are as such...
upload your apps here
browse apps here - click on an apps title to get more details plus the option to download the application.
fixed a few things.
added more details to the app details pages including download counter.
added filter to top of apps list.
added extra checks in the pxml data stripper for categories so they match the standard.
added some extra db tables and columns in preparation of additional site updates.
that's it for tonight, doesn't seem alot but i spend a good few hours on it, but im plodding along nicly
again details are as such...
upload your apps here
browse apps here - click on an apps title to get more details plus the option to download the application.
sebt3
homebrew player (P. & C.)
definatly need more screenies
milkshake
Advanced Member
how do u mean? screen-shots are not part of the repo standard (yet! i hope they will be) also its impossible to get them from the pnd because for speed purposes the screen-shots that could be in the pnd are not appended to the pnd, and what i found out in the very beginning was my ISP doesnt have the required version of squashfs-tools installed in server and its not available for him to install on the version of centOS they are using
sebt3
homebrew player (P. & C.)
I was more thinking about a way to edit these pages to add screenies.milkshake said:how do u mean? screen-shots are not part of the repo standard (yet! i hope they will be) also its impossible to get them from the pnd because for speed purposes the screen-shots that could be in the pnd are not appended to the pnd, and what i found out in the very beginning was my ISP doesnt have the required version of squashfs-tools installed in server and its not available for him to install on the version of centOS they are using
Screenshot are requiered when it come to choose software.
Last edited by a moderator:
milkshake
Advanced Member
@tempel
if we cant change the repo spec (wich I highly doubt) is there anyway to implament a browser cach controll maybe implamenting webkit? then we could use the etag header to controll when to update the local cache?
probably a bit far fetched really ... just a thought
im just thinking about site design at the moment, i hate design lol.
if we cant change the repo spec (wich I highly doubt) is there anyway to implament a browser cach controll maybe implamenting webkit? then we could use the etag header to controll when to update the local cache?
probably a bit far fetched really ... just a thought
im just thinking about site design at the moment, i hate design lol.
Tempel
Active Member
Ahh, yes! That's what I was thinking when I asked 'isn't there some way for a server to say "this page hasn't changed since you last requested it"?', but I didn't know any name for it. It's actually not far-fetched at all: I'm now reading about how to do it with Python's urllib, and it looks like it won't be more than a few lines of code.
See, I like this approach, because it a part of the HTTP standard to do exactly what it's supposed to do. On top of that, it doesn't stop the old naive way from working. As long as you can modify your code to include last-modified or etag (or both) in the headers, I really think this is the way to go.
Also yeah, I'm dreading when I have to start putting together the GUI for my stuff . You might just want to ask for lots of feedback, possibly even in General Talk.
EDIT: I see you already have a strong etag in your header. Is this accurate? I.E. does the etag change every time the repo changes? If so, you don't have any extra work to do on this; I can take care of the rest
See, I like this approach, because it a part of the HTTP standard to do exactly what it's supposed to do. On top of that, it doesn't stop the old naive way from working. As long as you can modify your code to include last-modified or etag (or both) in the headers, I really think this is the way to go.
Also yeah, I'm dreading when I have to start putting together the GUI for my stuff
. You might just want to ask for lots of feedback, possibly even in General Talk.EDIT: I see you already have a strong etag in your header. Is this accurate? I.E. does the etag change every time the repo changes? If so, you don't have any extra work to do on this; I can take care of the rest
milkshake
Advanced Member
@tempel
yes the etag is UTC timestamp what is taken from the db of the last added/updated app its already inplemented after I wrote that post so that it would also cache the jSON locally if and when I do some ajax.
Yes the etag changes when the db changes test it if you wish add an app then refresh ur request. My PHP checks ur cached etag against the lastest update if its the same sends the header to say data has not changed if it doesn't match it sends the data again.
edit:
at the moment the jSON is generated based on the order in which they were added to the database not taking updated apps into consideration.
would you like me to generate the jSON based on last updated apps so that the list is on order based on last updated apps?
would that make it easier at all?
yes the etag is UTC timestamp what is taken from the db of the last added/updated app its already inplemented after I wrote that post so that it would also cache the jSON locally if and when I do some ajax.
Yes the etag changes when the db changes
test it if you wish add an app then refresh ur request. My PHP checks ur cached etag against the lastest update if its the same sends the header to say data has not changed if it doesn't match it sends the data again.edit:
at the moment the jSON is generated based on the order in which they were added to the database not taking updated apps into consideration.
would you like me to generate the jSON based on last updated apps so that the list is on order based on last updated apps?
would that make it easier at all?
Tempel
Active Member
Hmm... I could imagine using the order of applications list in the JSON to speed up parsing, but I don't think it's worthwhile. So no, I'd say don't worry about the order. If eventually your repo gets so huge that my software slows down too much, then we can talk about it; for now, it's fine as is
And awesome about the etag (although if it's just UTC of the last update, it might make more sense to use the last-modified header entry, but it really doesn't matter either way). I'll implement support for it sometime soon, though I'll be away this weekend. Still: soon.
And awesome about the etag (although if it's just UTC of the last update, it might make more sense to use the last-modified header entry, but it really doesn't matter either way). I'll implement support for it sometime soon, though I'll be away this weekend. Still: soon.
milkshake
Advanced Member
I'm added the change to gen the jSON by latest updated apps now instead anyway even if it wont make a difference at the moment, I can change it from etag to last updated if you like although I don't think it will make too much difference.
Tempel
Active Member
ETag versus last-modified, it really doesn't make a difference. I just thought it would make more sense conceptually. Ideally, having both would probably be nice, but it's certainly not necessary. How it is now will work just fine for me.
milkshake
Advanced Member
Excellent, have a good weekend.
milkshake
Advanced Member
UPDATE:
just to keep people in the loop I havent stopped working on it, im currently working on the registration system, im doing it using jquery validating the fields using this and also validating again server side before submission.
Im starting with the ajax method first once I have it all working ill work on just a standard non fancy pants method of logging in (although id prefer everyone to use the fancypants way ).
just to keep people in the loop I havent stopped working on it, im currently working on the registration system, im doing it using jquery validating the fields using this and also validating again server side before submission.
Im starting with the ajax method first once I have it all working ill work on just a standard non fancy pants method of logging in (although id prefer everyone to use the fancypants way
).
milkshake
Advanced Member
OOPS - double post sorry.
UPDATE:
Just to keep people in the loop I havent stopped working on it, im currently working on the user registration system.
I want my site to be fancy and use jQuery for animation and ajax etc.
I'm doing validation both client side using jQuery and also server side with php before final submission.
I'm starting with the fancypants side first, once I have it all working, ill work on just a standard method of logging in (just PHP no ajax, although id prefer everyone to want to use the fancypants way ).
UPDATE:
Just to keep people in the loop I havent stopped working on it, im currently working on the user registration system.
I want my site to be fancy and use jQuery for animation and ajax etc.
I'm doing validation both client side using jQuery and also server side with php before final submission.
I'm starting with the fancypants side first, once I have it all working, ill work on just a standard method of logging in (just PHP no ajax, although id prefer everyone to want to use the fancypants way
).
milkshake
Advanced Member
UPDATE:
the fancy ajax version of my sign-in, registration, validation is reaching the stage where I would like some people to test it for me, preferably people who have been interested in the project so far and have posted.
there is no NON-JAVASCRIPT version ready yet, ill get to that once the fancy bit is done.
@tempel, @rockthesmurf, @sebt3, @foxblock @wizard, @Klaue @B-ZaR etc
if you want to try this section for me when I'm ready for someone to test it let me know.
the fancy ajax version of my sign-in, registration, validation is reaching the stage where I would like some people to test it for me, preferably people who have been interested in the project so far and have posted.
there is no NON-JAVASCRIPT version ready yet, ill get to that once the fancy bit is done.
@tempel, @rockthesmurf, @sebt3, @foxblock @wizard, @Klaue @B-ZaR etc
if you want to try this section for me when I'm ready for someone to test it let me know.
Tempel
Active Member
I'd be happy to try it out, though I wouldn't be able to offer much in the way of real testing (limited web dev knowledge again).
Also, I just tried to upload another PND (from here), and was told "PXML data missing: the author-website you entered is invalid!" In the PXML's author field, I have 'website="http://randy.heydon.selfip.net/Programs/The Lonely Tower/"'. What's wrong with that? Also, the error page ends with "...Done!", but it hasn't showed up in the program list. I don't know if an app should be rejected for a bad URL, since it doesn't affect how it runs; maybe a "here's an error! Upload anyway?" would work?
Also, I just tried to upload another PND (from here), and was told "PXML data missing: the author-website you entered is invalid!" In the PXML's author field, I have 'website="http://randy.heydon.selfip.net/Programs/The Lonely Tower/"'. What's wrong with that? Also, the error page ends with "...Done!", but it hasn't showed up in the program list. I don't know if an app should be rejected for a bad URL, since it doesn't affect how it runs; maybe a "here's an error! Upload anyway?" would work?
milkshake
Advanced Member
UPDATE:
ok sent a few people URL to my test site.
just a reminder the test site isnt really designed with the pandora in mind , the repo side of my site will be for that with native access.
my website is ment to be a desktop site so people can upload, download from a pc and manually add to pandora.
although it should work on the latest chromium and firefox on the pandora it wont look as nice. (at the moment).
ok sent a few people URL to my test site.
just a reminder the test site isnt really designed with the pandora in mind , the repo side of my site will be for that with native access.
my website is ment to be a desktop site so people can upload, download from a pc and manually add to pandora.
although it should work on the latest chromium and firefox on the pandora it wont look as nice. (at the moment).
