Pandora pandora repo/website/native app-manager development

[Tempel]

@Tempel once I have a database of files ill create the JSON repo script so you can test your native client

I've been testing with mock repo files based on the spec, so no need to rush. As long as you output a repo that conforms to spec and can be parsed by Python's JSON library, there'll be no problems on that front.

There will be problems with me getting the rest of the client done, just because I'm bad at managing my time. But it'll happen.

 

[milkshake]

@tempel don't worry I trust it will get done and the end result will be excellent

@rockthesmurf the only way I can 100% check that I don't already have the file from the app store is to compare md5 hash's (when I first download them I get the md5 hash and save it to my db along with the app data).
Its almost impossible to have 2 files with the same hash, the main problem with this is that I cant get the md5 hash from the file remotely so I have to download the file every time then discard the file if I have an md5 hash match.

I did use the name of the app from the download link initially to try and verify if I already had that app saved however I released after a while that when clicking (in a browser on the pc) the direct download links I parsed from the appstore the file it sends doesn't always have the same filename e.g. if the file was updated the link remains the same but the file it sends has a different name i.e.

http://apps.openpandora.org/cgi-bin/dl.pl?mame4all-beta4.pnd

expecting
mame4all-beta4.pnd

actually sends
mame4all-beta7.pnd

what makes this worse is when using the copy() command with the direct download link the file filename it will save as on my server is mame4all-beta4.pnd.

This is very frustrating as I cant now trust this form of verification and all other methods lead to having to download the file anyway sooo...

md5 is the way forward, it just means raping craigs server every time I run the pull script.

any ideas that I could use that would mean not transferring the file to my server first?

 

[PokeParadox]

I don't know php... but can't you check the filesize without downloading the file...

Check the filesize in bytes, compare to the filesize that you have locally... if they differ, THEN do the download and md5 check.

 

[milkshake]

i tried getting the file size remotely but the server doesn't send the file size in the header, other websites do but the appstore doesn't

 

[slaeshjag]

i tried getting the file size remotely but the server doesn't send the file size in the header, other websites do but the appstore doesn't

The server that the appstore is running on is really silly, ignores pretty much every special request I tried. As a placeholder, would a hash of the first say, 32kB do as a identifier? Any change should change *something* in the headers, so I guess that would sort of work? Something like this: http://pastebin.com/DUQjjDQ9

 

[WizardStan]

Both ISO and squashfs have size information somewhere near the front of their files, so yeah, if something changed, it's pretty much guaranteed to change something within the first 32K.

 

[Rockthesmurf]

Well, one way to look at it is: if you end up with a site that works better than the existing ones, then over time everyone will move over to it. If in the mean time we have to run a script once a day that syncs up with the existing app stores, I guess it isn't the end of the world - if it becomes a problem we could reduce the frequency to once every two days or once a week. To be honest with you though, if it becomes a problem for the official store that we are pulling large amounts of content somewhat regularly, it'd be nice if they simply provided us a better way to query what is currently on the app store in a more convenient way, e.g. a script that just returns us a list of all appids, version numbers and download links, as a combination of the appid and version number should be a unique identifier.

Steve

 

[milkshake]

Ok just a little progress report, all is working, my errors are now nicly written to the screen so that its easier read as it checks each file.

So for the moment I check that the md5 file has doesn't match then add to the db...

...sounds good right? well it is but if the file has been updated we will receive a different md5 hash and because of this we "potentially" will end up with the same app twice. (old and a new version).

so to try and counter this im going to copy the file to my server, pull the xml file, get the id attribute from the applications xml element (this should be unique to each app) if we have a match then we check the version number.

If the app we are checking has a higher version number then we know we can replace the old version, if not we discard it and move on to the next app.

I would like to know however if the id attribute of applications xml element changes if someone updates the app? I hope not or the above doesn't work.
As far as I am aware it should be different for each app but the same if the app is just updated. (scary though: I have realised I'm relaying alot on the PXML to be written correctly if not then im foobarred)

@slaeshjag
Im going to give your method a report back.

@rockthesmurf
I couldn't agree more, but if they did that they might as well implement the repo spec and then I wouldn't need to go to all this trouble lol but I highly doubt craigx will do that any time soon, well If all goes well he may just adopt my site and ill give him the php and myql and they can host it?
who knows.

I'm really loving working on the project so far, the only downfall is I'm rubbish at error handling and debugging lol.
I have probably spent 100+ 50+ hours so far of my time on this - but b4 this project I didn't do any php work for over a year so I have had to re-learn alot.

 

[milkshake]

@slaeshjag
ok your method worked although only slightly modified...

$data = file_get_contents($this->downloadURI.$appFileName.".pnd", FALSE, NULL, 0, 102400);
	if ($data !== false) {
		$this->smallMd = md5($data);
	}else{
		echo "false";
	}

for some reason when the data to read was set to 32kB some apps were reporting duplicates with a generated hash match based on those first 32kB so increased have now increased the the amount to read the first 100kB and now its seems to be fine.

phew that should hopefully solve the bandwidth issues for apps.openpandora.org every time i run the script.

If you want to run my script and watch it work click here.

 

[milkshake]

ok I have now implamented the id/version checker for apps that have the same id it goes onto check the version number, this is how the code looks..

	public function doubleCheck($idCheck, $idDb, $versionCheck, $versionDb){
		//once we know that the md5 hash doesnt match another in a repo we need to
		//now check the "<application id=" and the app version incase this is		
		//an app that we have but need to update
			
		//$idDb contains both the id and the id_app in an array from the db 		
		foreach($idDb as $value){			
			if(in_array($idCheck,$value)){
				$id_apps = $value['id_apps'];
				$idMatch = true;
			}	
		}
		unset($value);
		
		//if match get version number
		if($idMatch===true){
			foreach($versionDb as $value){
				if(in_array($id_apps,$value)){
					$match['major'] = $value['major'];
					$match['minor'] = $value['minor'];
					$match['release'] = $value['release'];
					$match['build'] = $value['build'];
				}
			}
			unset($value);

			//check which version is newer if this apps version is lower then return error
			foreach($versionCheck as $key=>$value){
				if($versionCheck[$key]<$match[$key]){
					$this->_errorList = "this version is older then the one on the server";
					break;
				}
			}
		}
	}

is this the best way to check version numbers do you think?

 

[milkshake]

UPDATE:
I have taken the id check further then the above code so that if it is a newer version it updates the PXML data in the database to the new PXML data and also adds any new locales that it may find, it doesn't delete existing locales though. but that can only be a good thing right?

anyway you can run the script if you like and see if there are any apps that are older that what I have in the Database already

run script

 

[milkshake]

QUICK-UPDATE:
ok I think Iv completed the file checking side of things, I have even implemented the update of a file if we have it but there is a newer version that has been parsed from the appstore or uploaded.

looking good

also im going to try and implament the option for people to specify a URL to get the pnd file from instead of uploading, saves them having to wait to upload if they have slow upload speed as it should be quicker to sideload also saves peoples bandwidth for those that have a limit.

 

[milkshake]

UPDATE:

I have added the ability to side-load a pnd from another website or upload from your own pc please give it a try and give some feedback.

of course still plenty more work to do but it finally feels like im getting there.
all new pnd's successfully added also get added to my database which is cool.

it also tells you if there is anything wrong with the file.

please try it.

click here

 

[sebt3]

UPDATE:

I have added the ability to side-load a pnd from another website or upload from your own pc please give it a try and give some feedback.

of course still plenty more work to do but it finally feels like im getting there.
all new pnd's successfully added also get added to my database which is cool.

it also tells you if there is anything wrong with the file.

please try it.

click here

I successfully uploaded 2 files there.
I'll need to be more carfull when I fill these fields now (I still fill these manually, the technicals one are generated ;P)
I mean, you're getting there obviously

I now want to see the rest of the UI (how will I manage my pnds in there, how will be the pnds displayed there etc...)

 

[milkshake]

I know, I guess displaying the data is easier then adding and verifying it so im happy that i feels like im making progress.

lol yes please be careful when writing ur PXML files sebt3 what I also plan to do is write an online a PXML generator so you just fill in all the form fields and my site will supply you with a valid PXML.xml file, though ill do that when the rest of the site is complete

I see you added minitube, and audacious thanks for testing

anyone else who has a go, if you find any errors let me know, for now the only real errors should come from invalid URL's if you try to side-load an app.

 

[Tempel]

Well, I got some ugly-looking error by clicking "Submit file" without having selected a file. I also managed to upload BubbMan2 quite successfully.

 

[milkshake]

UPDATE:

ok 1 tick off my list I posted earlier.

my JSON repo list is here.

this is follows the current repo-spec perfectly

so tempel if you wanna test the output with what you have done so far to check it works?
also when it comes to downloading the app the way I will do it is to force download using headers to change the name so that file the browser recieves is different to the pnd stored on the server.

although I haven't implemented this yet

$file = "./directory/sd76fd766fd.pnd":

// It will be called name according to id and version number
$appName = $data['id'].$data['version'].".pnd";

if(file(exists($file)){
    header('Content-Description: File Transfer');
    header('Content-type: text/plain');
    header('Content-Disposition: attachment; filename="$appName"');
    header('Content-Transfer-Encoding: binary');
    header('Cache-Control: must-revalidate, post-check=0, pre-check=0')
    header('Content-Length: ' . filesize($file));
    ob_clean();
    flush();
    readfile($file);
    exit();
}

well it turns out my domain name expired I though it had another year, so I renewed it and it should flood back through all the DNS servers shortly, on my computer it now works at least .

if you want to test the sideload/upload form its here

ok so now I have my JSON yay.

edit:
I will have separate arrays if you want me to provide any app ratings? or subcategories? or comments etc. etc.

 

[milkshake]

UPDATE:

we have a basic table with the apps from my site - not the final design this is just a test.
view here

 

[Tempel]

It works perfectly, good job milkshake!

Well actually, there's one issue: in your repository information, you have version set to 0.5, when it should be 1.0. Remember, that version number specifies what repository specification version you're using, not the version of your backend software. My code checks this number to make sure that it's a version that it knows how to handle. Of course, version 1.0 is the only version that exists so far, but if it gets updated in the future, or someone else makes their own changes for their own site, then knowing what version it is will be important for the client-side code.

 

[milkshake]

Ok I have amended this to 1.0

in regards to app ratings and comments etc how would you want me to handle them in a separate json array? or would you prefer not to have the extras?

edit: I was also thinking we need a way of checking if any apps have been updated i.e. datestamp maybe? (that you can compare with if you end up caching the json locally).