looking for help on project

[milkshake]

we can have a seperate entry to declaire the software alpha, beta or final that could help in the meantime no?

Therefore the best way imo to solve this is to add some kind of flags to the version tag like (with pre-defined values).

<version major='number' minor='number' release=number build='number' flags='beta,alpha,pre-release,gold,whatever' />

just what i was thining i have already add the slot in my database application table just in case (not now when i first set it up)

 

[Klaue]

when the data is in the array there is no escape, only when I call json_encode() do the escaped characters appear, i belive this is how its meant to be

I researched a bit more. While it is not illegal to encode/escape the forward slashes, it is also not needed. This was fixed in PHP development as of 09/16/2010 (see bug #49366) and should therefore probably be included in the newest PHP release, 5.3.5 (which came out this month) or even in the one before, 5.3.4 (according to wikipedia, that one came out in december). If you have a current version, try calling the function like this: json_encode($yourvar, JSON_UNESCAPED_SLASHES)


The encoded forward slashes add in your case a minimum of 5 bytes per pnd, probably more (because if there is an icon, it is also an url, +5 bytes, and some pnd use an url for the vendor). so let's say it adds about 8 bytes per PND.


A simple search over your current repository JSON revealed that you currently have 1299 bytes = 1.3 KB unneeded overhead because of this.


Not nearly dramatic, but I personally would fix it anyway


Also, other JSON implementations might not expect escaped forward slashes and therefore not be able to read your JSON (thios is just a guess, I diddn't read the spec to see if that was a required feature)

plus using double quotes when reading back the string would process the backslashs and not display them anyway only single quotes would keep them there.

I'm not sure if I understand what you want to say, but if you read json from a file (or from a website), something like "name":"val\/e" would be exactly like this in your string variable (as if you made a String s = "\"name\":\"val\\/e\""; ) without any character being eaten. The escapes would only be resolved when you put it trough an json decoder - if that decoder could handle escaped forward slashes, that is.

 

[milkshake]

I'm not sure if I understand what you want to say, but if you read json from a file (or from a website), something like "name":"val\/e" would be exactly like this in your string variable (as if you made a String s = "\"name\":\"val\\/e\""; ) without any character being eaten. The escapes would only be resolved when you put it trough an json decoder - if that decoder could handle escaped forward slashes, that is.

read the sections single quotes and double quotes http://uk.php.net/types.string that will explain what I mean, im pretty sure it works the same wasy in C++


single quotes

echo 'test\n\t\test';

output: 

test\n\t\test

double quotes

echo "test\n\t\test";

output: 

test

     test

so as far as im aware although I may be wrong if somone echos out the json data using double quotes instead of single quotes the escape characters shouldnt show.

 

[Klaue]

if somone echos out the json data using double quotes instead of single quotes the escape characters shouldnt show.

I'm talking about json escaping/parsing, not php string escaping/parsing.


put \t\t in a file and read that in a var in php. then output it, firstly by


echo $var


and then by


echo "$var"


both will net you a "\t\t" in the output, not two tabs. (echoing '$var' would AFAIR get you $var as text)


(the reason is that, in a file, a tab is just a normal character, 0x09 in ASCII, as is the backslash. the converting from \t to char(0x09) is not done when reading external sources, why would it)


by the way, single quotes are for characters in C++

 

[milkshake]

your example echo "$var"; if $var = '\n\n'; im pretty sure you would end up with 2 new lines not \n\n unless the string was \\n\\n to escape it, again i may be wrong cos im not in a place I can test this.


if you read \t\t from a text file that is different because your not just reading a string but the text encoding aswell thats the only reason it works that way when reading from a file.


in regards to json_encode, I will try your suggestion.

json_encode($yourvar, JSON_UNESCAPED_SLASHES)

by the the way my php version is 5.2.17 so not the most recent version but its a shared server so I doubt ill be able to get it upgraded.


edit: the modifyer JSON_UNESCAPED_SLASHES does not exist in my version of php so there is no way around this for me unfortunatley unless i create my json manually wich would be a bit of messing around.

 

[Klaue]

your example echo "$var"; if $var = '\n\n'; im pretty sure you would end up with 2 new lines not \n\n unless the string was \\n\\n to escape it, again i may be wrong cos im not in a place I can test this.

It's just another way of storing it. PHP's strings are probably just arrays of characters like in many other languages. a "\n\n" would therefore be an array of two newline characters. a '\n\n' or a \n\n read from an external source (like a file) would just be an array of four characters, \, n, \ and n again. When the stuff is in a string, that escape-parsing allready went through, so $var='\n\n';echo "$var"; would just output the 4 characters of its internal array, therefore \n\n, not two newlines.


It would really, really surprise me if php handled this in any other way.


if it weren't like this, wouldn't the following fail:


$var = '"';


echo "$var"


because it would "translate" to


echo """;


?


Wouldn't every read of a text file which contains " or ' fail? Or any other text source like the text which I'm writing and the boards php has to parse?

edit: the modifyer JSON_UNESCAPED_SLASHES does not exist in my version of php so there is no way around this for me unfortunatley unless i create my json manually wich would be a bit of messing around.

oh, there's an easy workaround:


str_replace("\\/", "/", $jsonEncoded)

 

[milkshake]

str_replace("\\/", "/", $jsonEncoded) doh!! :lol: of course.


in regards to

$var = '"';

echo "$var"

//because it would "translate" to

echo """;

I will look into this tonight. just for fun.

 

[Tempel]

Regarding string versions: I wouldn't say a full string version number makes comparisons impossible - after all, many standard package managers on Linux handle it just fine. The key is an agreed-upon standard to make those comparisons consistent, and we could just crib Debian's standard if we wanted. The problem, though, is that package databases tend to have a few hand-picked maintainers, whereas we have just anybody making PNDs; it might not be easy to get everyone to follow the rules. So, I'm starting to like the "flags" concept because it sidesteps this problem and allows for backwards compatibility with existing PNDs. Regardless, we've got to get this implemented into libpnd if we want it to happen.


Regarding escape slashes: All I know is that Python's JSON parser seems to handle URIs correctly whether the slashes are escaped or not. It works as-is, but presumably should continue to work if all the backslashes were removed.


Regarding datestamps: Something like your "&dt" suggestion would work. My only concern is whether that could be included in the repo spec. I don't see a reason why not, but as I said, I'm not overly familiar with web technologies. Maybe the repo file could include a datetime entry that gives the url to the last-updated time. Oh, and UTC would certainly be a must.

 

[milkshake]

in regards to this:

been thinking even more about it and if you wanted to collect all the json you could just call the page directly without any aditional elements i.e.
op.liquidfists.com/includes/getData.php


when you receive the json you could log the UTC datetime locally then the next time you visit you could call.


op.liquidfists.com/includes/getData.php?&dt=UTCdatetimestampand


my site would then only send any additional or updated entries via JSON?


this sounds like the best way what do you think?

do you think that would be the best way?


should we be looking at trying to implement this? I mean officially the repo spec is unofficial anyway as its what dflmstr wrote. but its sound.


would we have to speak with anyone to get my idea implemented into the spec and if so its not really part of the repo file its just a UTC timestamp that the client provides the site then we give you the updates kinda thing like a little handshake so to speak.

 

[milkshake]

UPDATE:


ok i have now the beggings of a very basic (mish mash / unstyled) functioning download/upload website


its obviously not complete but functional.


upload your apps here


browse apps here - click on an apps title to get more details plus the option to download the application.


the very very basics are now complete


things to do next:

• add registration page for people who wish to upload an application
  
• add comments / app rating / report app utilities
  
• add downloadlist for logged in members (kinda like a shopping basket) - this will save a list of apps just for the current session you wish to download, then you can download them at your luiesure
  
• give website a proper theme
  
• add more details from PXML i.e. different languages, authors/apps website
  
• the ability to allow people up update an app but before it gets added an email gets sent to the original app uploader/author to confirm its ok and everything is correct
  
• the ability to edit certain things before submitting to database i.e. description
  
• the ability to get apps that are in the file archive automatically
  
• option to add screenshots to existing applications
  
• + others ideas that I may have or have had but cant think of right now.
  

 

[Tempel]

been thinking even more about it and if you wanted to collect all the json you could just call the page directly without any aditional elements i.e.


op.liquidfists.com/includes/getData.php


when you receive the json you could log the UTC datetime locally then the next time you visit you could call.


op.liquidfists.com/includes/getData.php?&dt=UTCdatetimestampand


my site would then only send any additional or updated entries via JSON?


this sounds like the best way what do you think?

Ahh, I don't know! That certainly sounds good, but so did other options. I'm reluctant to make any decision now simply because we haven't yet seen any real problems that need fixing. And any changes we make could introduce other problems unexpectedly. This one bothers me only because it means two different styles of parsing (and how would you indicate apps being removed?), but it would be the most bandwidth-efficient.

ok i have now the beggings of a very basic (mish mash / unstyled) functioning download/upload website

Crap, I better hurry up!

 

[Klaue]

I see you have fixed the wrong language infos and removed the additional backslashes. nice work.


You got me to where I'm thinking about making a client like Synaptic (of coure not quite that feature rich) for it.. Of course, I'll wait to see how Tempel's turns out.


By the way, for what exactly would the date parameter you two talk about be? If it was just for getting the newest pnds, shouldn't that be done on the client site? Because such a date parameter is not in the spec, as far as I can see, so any client implementing that would have to implement a special case just for your repository that wouldn't work on others.. (I don't think ressources would be an issue here.. For one of my projects, I build a JSON with about four times more data per entry than in the repos and an index of over 6000 items is built in less than a second server side (PHP). JSON is just that awesome)


(Just so you don't think I'm always bitching, it is just meant as a question )

 

[milkshake]

Ahh, I don't know! That certainly sounds good, but so did other options. I'm reluctant to make any decision now simply because we haven't yet seen any real problems that need fixing. And any changes we make could introduce other problems unexpectedly. This one bothers me only because it means two different styles of parsing (and how would you indicate apps being removed?), but it would be the most bandwidth-efficient.

hmmm im not sure how I would indicate removed items if I or you or anyone has any ideas (that doesnt involve any aditional,aditional change to the spec) please post them

I see you have fixed the wrong language infos and removed the additional backslashes. nice work.


You got me to where I'm thinking about making a client like Synaptic (of coure not quite that feature rich) for it.. Of course, I'll wait to see how Tempel's turns out.

indeed i have i forgot to put that in my update post last night.


well maybe a choice in clients wouldnt necessarily be a bad thing? but yes Im excited about tempels client.

By the way, for what exactly would the date parameter you two talk about be? If it was just for getting the newest pnds, shouldn't that be done on the client site? Because such a date parameter is not in the spec, as far as I can see, so any client implementing that would have to implement a special case just for your repository that wouldn't work on others..

this is in regards to what information we send in the JSON when checking for changes i.e. new apps added or an ap being updated or removed.


we were just thinking how we could reduce bandwidth and increase overall efficiency but of course it would require a change in spec (which at this moment because the spec isnt really official it could be added). but of course I wouldnt implement anything unless its in the spec, we were just thinking about it, as if there are multiple sources for the clients if it had to download large json from each it might take a while (cydia does this and it takes ages).


EDIT:


Just a thought, dont you think the PXML should include an optional section for licence? i.e. BSD, GNU GPL etc and source code uri?


it would then be easier to know what licence the app falls under and also force people to accept the licence before downloading.


I know i could implament this on my site but I would have to ask the uploader if he wanted to use a licence and then to choose one upon upload but it would be much simplar if it was included in the PXML spec.

 

[Klaue]

we were just thinking how we could reduce bandwidth and increase overall efficiency but of course it would require a change in spec (which at this moment because the spec isnt really official it could be added). but of course I wouldnt implement anything unless its in the spec, we were just thinking about it, as if there are multiple sources for the clients if it had to download large json from each it might take a while (cydia does this and it takes ages).

Well, you have currently 114 PNDs in your JSON and the size is just 52161 Bytes. That means about 0.4 KB per PND. Even with 10'000 PNDs, that would only be 3.9 MB, which is really small with todays internet speeds.


Also, in addition, you could send the data gzipped, which works very well with text data. Your current JSON would deflate to about 10 KB (using another archiver, I have no gzip ready here), which means about 90 bytes per PND. 10'000 PNDs would therefore equal to 0.9 MB. It's doubtfull if there will ever be that many PNDs anyway.


gzipped content should allready be understood by any web clients (and probably all webservers support it), so there would maybe not even be a need to change the spec.

Just a thought, dont you think the PXML should include an optional section for licence? i.e. BSD, GNU GPL etc

I think this is more something that the PND builder should display on the first start, but I personally never cared much about licences anyway (of course, there are exceptions, but that would go too far here - let's just say as long as it's an OS licence and I do not plan to work on the code, I don't care which specific one it is, be it the Gnu public licence, the do what the fuck you want licence (yes, it's real) or any other)

 

[milkshake]

Also, in addition, you could send the data gzipped, which works very well with text data.

interesting would that mean a change in the spec though as all clients would need to complament gzip decompression.

 

[milkshake]

UPDATE:


fixed a few things.


added more details to the app details pages including download counter.


added filter to top of apps list.


added extra checks in the pxml data stripper for categories so they match the standard.


added some extra db tables and columns in preparation of additional site updates.


that's it for tonight, doesn't seem alot but i spend a good few hours on it, but im plodding along nicly


again details are as such...


upload your apps here


browse apps here - click on an apps title to get more details plus the option to download the application.

 

[milkshake]

UPDATE:


Just to keep people in the loop I havent stopped working on it, im currently working on the user registration system.


I want my site to be fancy and use jQuery for animation and ajax etc.


I'm doing validation both client side using jQuery and also server side with php before final submission.


I'm starting with the fancypants side first, once I have it all working, ill work on just a standard method of logging in (just PHP no ajax, although id prefer everyone to want to use the fancypants way ).

 

[Klaue]

UPDATE:


Just to keep people in the loop I havent stopped working on it, im currently working on the user registration system.


I want my site to be fancy and use jQuery for animation and ajax etc.


I'm doing validation both client side using jQuery and also server side with php before final submission.


I'm starting with the fancypants side first, once I have it all working, ill work on just a standard method of logging in (just PHP no ajax, although id prefer everyone to want to use the fancypants way ).

please only use JS for eye candy and not basic usage. text browser- and noscript-users will thank you.

 

[milkshake]

please only use JS for eye candy and not basic usage. text browser- and noscript-users will thank you.

im hopeing that there are not many noscript-users out there but I will make the website to work fully functional both with JS eye candy and without so it will work if JS is disabled but of course it will require the page to change/refresh alot (pagination) etc which the JS stuff will not require.

 

[bzar]

^ Just to comment: if not using AJAX for basic functionality makes the site less usable, how will the non-textbrowser/noscript people feel? I'm guessing there's more of them (us) . Of course if it makes no difference if AJAX is used or not, it's of course better to err on the side of caution


EDIT: I was of course replying to Klaue, but milkshake went and ninja'd