Pandora pandora repo/website/native app-manager development


I tried uploading my work so far to my web host:

http://www.gamesdev.com/pandora/

Other then creating an MySQL database and a MySQL user, everything else seemed to kick in automatically, which is good.

I just put a bunch of random .pnd's into the upload directory and everything else is done automatically.

I think next I might take a look into this idea of pulling the pnd's currently on the official AppStore into this test site.

Steve
 
Cool I applaud you on picking up PHP so so quickly!! :D So how could you create a PHP script to get all pnd's from the app store wouldn't u have to parse the html for the links? I dont think the app store uses a database its all done in pearl I think.

I think it will be a while yet till my site is complete there is so much I wanna do but only have time to code small bits at a time. Even then I get side tracked improving existing code lol. Don't suppose that ever happens to you guys?

Appart from that I'm glad its finally going to happen ... A native app/manager wich pools its info from a repo :) I love it when a plan comes together and especially when its because of the community! :D
 
@Rockthesmurf I looked through your code, where do you check for multiple applications per .pnd file? if you look at the your page http://www.gamesdev.com/pandora/ your output for DEGA outputs all the title and description results for the second declared <application tag in the pxml file.

your code..

Code:
$applications = $item->getElementsByTagName("application");
                    foreach ($applications as $application)
                    {

loops through each xml element named application..
As a result if there is more than 1 element in the PXMl named <application any data pulled from nested elements of the fist <application will be overwriten by the data retrieved from the second declared <application.
If you check, your results for the dega pnd file show all the data from the second <application.

if you check my previouse example you will see how I solved it just read the //comments

apart from that you are flippin amazing at php (much better than me) you have even wrote classes already, although I would keep them in a seperate php file and just use include();
 
Last edited by a moderator:
Im going to sleep now :p I just converted a couple of my pndtools functions to classes for ease of use :) and makes it easier to update at a later date too.
 
Okay, did a little test pull of the official AppStore:

http://www.gamesdev.com/pandora/

You can note it currently moans about a couple of PXML's it doesn't understand.

Milkshake, I think we should team up on this now, I've pretty much learnt what I wanted to learn now. I would like to share my pull script with you that pulls the latest from the official AppStore, providing you are willing to stick at it, I think we should be able to come up with something pretty cool here!

Steve
 
great to hear ill set you up a ftp login just PM me with what username and password you would like and ill set it up for you.

yeah I agree I think this will be something better than cool.

I have lots of other neat ideas too like:
online PXML.xml file generator.
we could even create an online tool that allows developers to upload a zip file containing there app files inc icon and pxml of course and with that the site could generates a valid pnd file for download. (although that would require squashfs-tools 4.0 or above to be installed on the server which my isp doesn't have, the latest version they have is 3.0 but its an idea in case we can get a better host :) )

I'm really excited about the possibilities.
 
ok just a quick update.

ok in terms of progress I haven't really gotten any further but I have spent a good many hours converting the old functions into classes for uploading pnd's, pulling xml and icon files from pnd's and for stripping needed info from the xml file whilst checking for erros/missing information.

so I have 3 very well commented classes for you to check rock :) probably overly commented I just want to make sure you fully understand my madness lol
and I re wrote my upload script again from scratch using the new classes instead of using the old functions.

in regards to the database side of things I have a class from an old project which will work fine though I might tinker with it.

it already has a set of methods you can call to check for certain datatypes etc before submitting to the database.
I have all my mysql tables set up ready also just haven't connected any code and database up yet.

also I was thinking we can give the uploader the chance to edit the description, change/add titles/locales etc before finally submitting it (this wont affect the pxml's within the pnd's but it could make for more detailed pnd information available from our server when browsing pnd files.

thats it for now, Ill try to keep this topic updated with my progress when I make any.
 
I'm not really sure how well it'd work if we both are attempting to access the FTP and make changes, we could easily end up treading on the toes of each other. I have my site up for testing my ideas out, and I can just share up anything I do with you, and then you can pick up any bits of code that you think are useful, and ditch the rest. Initially it would be nice to get something up and running on your web host that has all the .pnd files from the official app store, stores the data about them in a MySQL database, and provides an webpage that lists all the .pnd files along with their icons. That would bring it inline with what I have done so far. You have highlighted an issue with my handling of multiple applications per .pnd, but presumably your version does not have this issue, so that's good. On top of that, there are a few .pnd files that my script does not parse (I believe the .pnd's are not well formed) but it'd be nice to get to the bottom of this, so I'll have to let you know which .pnd files are causing the problems so you can investigate if you have similar problems. Also, there are one or two .pnd's that currently have got through to my MySql database but have subtle problems (again will probably be due to bad pxml data, but it will be nice to confirm) I will make a list of these when I get a chance (hopefully tonight).

It would also be nice if there was an 'admin' area of your website, where we can put scripts like the 'pull script' I've put together that will pull everything from the Official AppStore. Currently it pulls every single .pnd file, but it'd be trivial to give it the option to only pull .pnd files that it hasn't already pulled (I'm not sure if it'll be possible to query the remote file size, that way we could potentially only pull new files, or files that have changed size, to try and get any new versions that are uploaded with the same name). Certainly with my database class, there is a query method where you can query an existing appid, it'd be a tiny chance to be able to also query a .pnd filename, to check for the existence of a file before pulling - hopefully this is something you'll be able to add at your end easily enough.

What do you think is a realistic ETA for getting your site up and running well enough that it can be used by whoever wants to, for their .pnd downloading needs?

Steve
 
Realistically it depends on how much I get to work on it, at the moment its only backend logic we are working on but there is also the front end of the site which I want to use Ajax and jQuery to make very simple to use and look nice. I'd say realistically maybe a month maybe longer I couldn't say but I could prob get a more basic site done sooner.

Also in regards to stepping on toes and whatnot u could copy files from my server and put them on urs to edit then if you make any changes you could place then in an update/change folder in my server?
 
If its something really basic maybe only a couple of days :) I'll do my best in the mean time pm me with what you want as ur username and password and I'll set the ftp up so u can have a look at my classes if you think anything you would change or missing let me know it could help u implement other ideas.
 
milkshake said:
Also in regards to stepping on toes and whatnot u could copy files from my server and put them on urs to edit then if you make any changes you could place then in an update/change folder in my server?
This is what source control management is all about. You'd likely be best off using a git server; github.com is pretty good (if you don't mind making your source code public).
 
Last edited by a moderator:
@Rockthesmurf if you want to share the pull script I can start working on getting all the pnd's across to the server? and working with the database class.

edit: just for learning purposes I started my own basic html parser for the app store while I wait to see ur code :p

Here is the code...

Code:
$category = array('Emulators','Games','Applications','Other');
$i=0;
foreach($category as $value){
	$url = "http://apps.openpandora.org/cgi-bin/viewarea.pl?".$value;
	
	$code = file_get_contents($url);
	$doc = new DOMDocument();
	@$doc->loadHTML($code);
	
	//search for li elements as they contain the name of the app
	//stored within the onclick attribute - populate $link array
	foreach ($doc->getElementsByTagName('li') as $element) {
		if ($element->hasAttribute('onclick')) {
			$link[] = $element->getAttribute('onclick');
		}
	}
	
	$cat = array('Emulator','Game','Application','Other');
	//for app name store within link array 
	foreach($link as $val){
		
		//regex magic to get just the app name
		preg_match("/$cat[$i]\/([^.]+)/",$val,$match[]);
	}
	$i++;
	
	//unset variables to be on safe side
	unset($link); unset($val); unset($code); unset($doc);
}

echo "there are ".sizeof($match)." apps in the app store, the link for each one is here.<br/><br/>";
foreach($match as $app){
		echo "<b>$app[1]</b><br/>";
		echo "<a href='http://apps.openpandora.org/cgi-bin/dl.pl?$app[1].pnd' alt='>http://apps.openpandora.org/cgi-bin/dl.pl?$app[1].pnd</a><br/><br/>";	
}
//unset variables to be on safe side
unset($match); unset($value);

it gets the download uri's for each app in the app store.

the output is here.
 
Last edited by a moderator:
Sure, be aware this was just something I quickly knocked up just to test in theory if it is possible to do what we want:

Code:
<html>
 <head>
  <title>Pandora App Store</title>
 </head>
 <body>

<?php

// Setup various directory locations.
define( "kRootDir",			dirname( __FILE__ ) );
define( "kUploadDir",		kRootDir . "/upload" );
define( "kUploadingDir",	kRootDir . "/uploading" );
define( "kDownloadDir",		kRootDir . "/download" );
define( "kIconDir",			kRootDir . "/icon" );


function GrabFromAppStore( $url )
{
	$xmlDoc = new DOMDocument( );
	libxml_use_internal_errors( true );
	$xmlDoc->loadHTMLFile( $url ) or die( "Failed to loadHTMLFile" );
	
	$htmls = $xmlDoc->getElementsByTagName( "html" );
	foreach ( $htmls as $html )
	{
		$bodys = $html->getElementsByTagName( "body" );
		foreach ( $bodys as $body )
		{
			$divs = $body->getElementsByTagName( "div" );
			foreach ( $divs as $div )
			{
				$class = $div->getAttribute( "class" );
				if ( $class == "itemlist" )
				{
					$lis = $div->getElementsByTagName( "li" );
					foreach ( $lis as $li )
					{
						$onClick = $li->getAttribute( "onclick" );
						$toStrip = "location.href='..";
						if ( strpos( $onClick, $toStrip ) === 0 )
						{
							$url = substr( $onClick, strlen( $toStrip ), strlen( $onClick ) - strlen( $toStrip ) - 2 );
							echo '<p>' . 'http://apps.open-pandora.org' . $url . '</p>';
	
							$appPage = new DOMDocument( );
							$appPage->loadHTMLFile( 'http://apps.open-pandora.org' . $url );
							$appDivs = $appPage->getElementsByTagName( "div" );
							foreach ( $appDivs as $appDiv )
							{
								$appId = $appDiv->getAttribute( "id" );
								if ( $appId === "appinfo" )
								{
									$appInfoDivs = $appDiv->getElementsByTagName( "div" );
									foreach ( $appInfoDivs as $appInfoDiv )
									{
										$appClass = $appInfoDiv->getAttribute( "class" );
										if ( $appClass === "install" )
										{
											$aTags = $appInfoDiv->getElementsByTagName( "a" );
											foreach ( $aTags as $aTag )
											{
												$href = $aTag->getAttribute( "href" );
	
												if ( $href && strlen( $href ) > 0 )
												{
													$href = 'http://apps.open-pandora.org/cgi-bin/' . $href;
	
													$questionMarkPos = strpos( $href, '?' );
													$andPos = strpos( $href, '&' );
													if ( $questionMarkPos >= 0 && $andPos >= 0 )
													{
														$questionMarkPos += 1;
														$filename = trim( substr( $href, $questionMarkPos, $andPos - $questionMarkPos ) );
	
														echo '<p>filename : ' . htmlspecialchars( $filename ) . '</p>';
														echo '<p>href : ' . htmlspecialchars( $href ) . '</p>';
														$data = file_get_contents( $href );
	
														echo '<p>Data : ' . htmlspecialchars( substr( $data, 0, 100 ) ) . '</p>';
	
														$toFind = '<META HTTP-EQUIV="Refresh" CONTENT="';
														$ix = strpos( $data, $toFind );
														if ( $ix >= 0 )
														{
															$endQuote = strpos( $data, '"', $ix + strlen( $toFind ) );
															$text = substr( $data, $ix + strlen( $toFind ), $endQuote - ( $ix + strlen( $toFind ) ) );
															$parts = split( ';', $text );
															foreach ( $parts as $part )
															{
																echo '<p>Part : ' . $part . '</p>';
	
																if ( strpos( $part, 'URL=' ) === 0 )
																{
																	$newUrl = substr( $part, strlen( 'URL=' ) );
																	while ( $newUrl[ strlen( $newUrl ) - 1 ] == '_' )
																	{
																		$newUrl = substr( $newUrl, 0, strlen( $newUrl ) - 1 );
																	}
																	echo '<p>New Url : ' . $newUrl . '</p>';
																	$src = 'http://apps.open-pandora.org/cgi-bin/' . $newUrl;
																	$dst = kUploadingDir . "/" . basename( $filename );
																	echo '<p>src : ' . $src . '</p>';
																	echo '<p>dst : ' . $dst . '</p>';
	
																	if( !
																		copy(
																			$src,
																			$dst
																		)
																	)
																	{
																		echo("failed to copy file");
																	}
																}
															}
	
														}
													}
												}
											}
										}
									}
								}
							}
						}
					}
	
					// Done here.
					break;
				}
			}
		}
	}
}

GrabFromAppStore( 'http://apps.open-pandora.org/cgi-bin/viewarea.pl?Emulators' );
GrabFromAppStore( 'http://apps.open-pandora.org/cgi-bin/viewarea.pl?Games' );
GrabFromAppStore( 'http://apps.open-pandora.org/cgi-bin/viewarea.pl?Applications' );
GrabFromAppStore( 'http://apps.open-pandora.org/cgi-bin/viewarea.pl?Other' );


?>

 </body>
</html>

Before we hammer open-pandora.org we transfer requests, it'd be worth adding in the couple of lines required to only pull files that you don't already have, otherwise every time you run the script it'll transfer the entire repository across.

There are a couple of variables near the top you might need to change, hopefully you'll be able to tweak it around to get it up and running.

It currently prints out random debug rubbish as it goes, as I say, it was just testing as I go along, it'd want neatening up a bit before production use! I just have been too busy tonight to do anything else.

Steve
 
ok this is my database class I havent tested it yet but it looks good to me :p
obviously I have a seperate validation class to validate data before sending it to be added to the database.

Code:
	///////////////////////////////////////////////////////////////////////////////////////
	///////////////////CHECKLIST OF UPLOAD PHP CLASS :) ////////////////////////////////////
	//
	// connect();		-private-	connects to database and selects appropriate schema
	// addRecord();		-public-	adds record to table specified
	// updateRecord();	-public-	updates record
	// deleteRecord();	-public-	deletes record
	// getRecord();		-public-	gets records and returns data as array
	// getRecordWhere();-public-	gets records with option to specify filters and returns 
	//								data as array
	// close();			-public-	close connection to database
	//
	///////////////////////////////////////////////////////////////////////////////////////
	///////////////////////////////////////////////////////////////////////////////////////
	
class dB {
	
	private $link;
	private $server='localhost';
	private $username='******';
	private $password='******';
	private $database='******';
	
	function __construct(){
		$this->connect();
	}
	
	//connect to mysql and select right database
	private function connect(){
		$this->link = mysql_connect($this->server, $this->username, $this->password)or die("failed to connect to database: ".mysql_error());
		mysql_select_db($this->database, $this->link)or die("failed to select database: ".mysql_error());
	}
	
	//add new record
	public function addRecord($table_name,$columns,$values){
		$columns = implode(',',$columns);
		$values = "'".implode("','",$values)."'";
		mysql_query("INSERT INTO $table_name ($columns) VALUES ($values)")or die("failed to insert data: ".mysql_error());
	}
	
	//update existing record
	public function updateRecord($table_name,$columns,$values,$appId){

		if(!is_array($columns)){
			$columns = array($columns);
		}
		if(!is_array($values)){
			$values = array($values);
		}
		$upDate = "";
		foreach($columns as $key=>$value){
			$upDate .= " $value = '$values[$key]'";
		}
		$upDate = ltrim($upDate," ");
		mysql_query("UPDATE $table_name SET $upDate WHERE id='$appId'")or die("failed to insert data: ".mysql_error());
	}
	
	//delete existing record
	public function deleteRecord($table_name, $appId){
		mysql_query("DELETE FROM $table_name WHERE id='$appId'")or die("failed to delete data: ".mysql_error());
	}

	//get records from table can also set order by and also limit
	public function getRecord($table_name, $columns, $orderby, $limit){
		
		$finalfilter = rtrim($finalfilter, " AND ");
		$query = "SELECT $columns FROM $table_name";
		
		if(isset($orderby)){
			$query .= " ORDER BY $orderby";
		}
		if(isset($limit)){
			$query .= " LIMIT $limit";
		}
		
		$result = mysql_query($query)or die("failed to get records: ".mysql_error());
		while($row = mysql_fetch_array($result)){
		  $array[] = $row;
		}
		return $array;
	}
	
	//get records from table using where option can also set order by and also limit
	public function getRecordWhere($table_name, $columns, $filter, $orderby, $limit){		
		
		foreach($filter as $value){
			$finalfilter = "$value[0]='$value[1]' AND ";
		}
		
		$finalfilter = rtrim($finalfilter, " AND ");
		$query = "SELECT $columns FROM $table_name WHERE $finalfilter";
		
		if(isset($orderby)){
			$query .= " ORDER BY $orderby";
		}
		if(isset($limit)){
			$query .= " LIMIT $limit";
		}
		
		$result = mysql_query($query)or die("failed to get records: ".mysql_error());
		while($row = mysql_fetch_array($result)){
		  $array[] = $row;
		}
		return $array;
	}

	//close connection to database when done
	public function close(){
		mysql_close($this->link)or die("failed to clode db: ".mysql_error());
	}
}
 
I was recommended to use mysql_real_escape_string when adding stuff into a database. I'm just a beginner so I don't know how good the advice is, but you may at least want to consider it, in case it is of use to your class - which is looking nifty!

Code:
        // Add main info.
        $query =    'INSERT INTO ' . $this->Sql[quote=" $this->m_Database "] . '.' . $this->Sql[quote=" $this->m_MainTable "] .
                    " (`appid`, `version`, `download_name`, `author_name`, `author_website`, `author_email`) VALUES (" .
                    "'" . mysql_real_escape_string( $pndInfo->AppId	)			. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->Version	)		. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->DownloadName	)	. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->Author->Name	)	. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->Author->Website )	. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->Author->Email	)	. "'" .
                    ")";
[/quote]Steve
 
Last edited by a moderator:
Rockthesmurf said:
I was recommended to use mysql_real_escape_string when adding stuff into a database. I'm just a beginner so I don't know how good the advice is, but you may at least want to consider it, in case it is of use to your class - which is looking nifty!

Code:
        // Add main info.
        $query =    'INSERT INTO ' . $this->Sql[quote=" $this->m_Database "] . '.' . $this->Sql[quote=" $this->m_MainTable "] .
                    " (`appid`, `version`, `download_name`, `author_name`, `author_website`, `author_email`) VALUES (" .
                    "'" . mysql_real_escape_string( $pndInfo->AppId	)			. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->Version	)		. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->DownloadName	)	. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->Author->Name	)	. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->Author->Website )	. "', " .
                    "'" . mysql_real_escape_string( $pndInfo->Author->Email	)	. "'" .
                    ")";
Steve
[/quote]

I was going to do it as part of the validation but i think ill do it on the db class. :)
 
Last edited by a moderator:
ok I updated the class with ur suggestion...

Code:
	///////////////////////////////////////////////////////////////////////////////////////
	///////////////////CHECKLIST OF UPLOAD PHP CLASS :) ////////////////////////////////////
	//
	// connect();		-private-	connects to database and selects appropriate schema
	// addRecord();		-public-	adds record to table specified
	// updateRecord();	-public-	updates record
	// deleteRecord();	-public-	deletes record
	// getRecord();		-public-	gets records and returns data as array
	// getRecordWhere();	-public-	gets records with option to specify filters and returns 
	//					data as array
	// escape_string();	-private-	mysql_real_escape_string our data before submitting
	// close();		-public-	close connection to database
	//
	///////////////////////////////////////////////////////////////////////////////////////
	///////////////////////////////////////////////////////////////////////////////////////
	
class dB {
	
	private $link;
	private $server='localhost';
	private $username='******';
	private $password='******';
	private $database='******';
	
	function __construct(){
		$this->connect();
	}
	
	//connect to mysql and select right database
	private function connect(){
		$this->link = mysql_connect($this->server, $this->username, $this->password)or die("failed to connect to database: ".mysql_error());
		mysql_select_db($this->database, $this->link)or die("failed to select database: ".mysql_error());
	}
	
	//add new record
	public function addRecord($table_name,$columns,$values){
		$columns = implode(',',$columns);
		$values = $this->escape_string($values);//escape our data :) 
		$values = "'".implode("','",$values)."'";
		mysql_query("INSERT INTO $table_name ($columns) VALUES ($values)")or die("failed to insert data: ".mysql_error());
	}
	
	//update existing record
	public function updateRecord($table_name,$columns,$values,$appId){
		
		if(!is_array($columns)){
			$columns = array($columns);
		}
		if(!is_array($values)){
			$values = array($values);
		}
		$values = $this->escape_string($values);//escape our data :) 
		$upDate = "";
		foreach($columns as $key=>$value){
			$upDate .= " $value = '$values[$key]'";
		}
		$upDate = ltrim($upDate," ");
		mysql_query("UPDATE $table_name SET $upDate WHERE id='$appId'")or die("failed to insert data: ".mysql_error());
	}
	
	//delete existing record
	public function deleteRecord($table_name, $appId){
		mysql_query("DELETE FROM $table_name WHERE id='$appId'")or die("failed to delete data: ".mysql_error());
	}

	//get records from table can also set order by and also limit
	public function getRecord($table_name, $columns, $orderby, $limit){
		
		$finalfilter = rtrim($finalfilter, " AND ");
		$query = "SELECT $columns FROM $table_name";
		
		if(isset($orderby)){
			$query .= " ORDER BY $orderby";
		}
		if(isset($limit)){
			$query .= " LIMIT $limit";
		}
		
		$result = mysql_query($query)or die("failed to get records: ".mysql_error());
		while($row = mysql_fetch_array($result)){
		  $array[] = $row;
		}
		return $array;
	}
	
	//get records from table using where option can also set order by and also limit
	public function getRecordWhere($table_name, $columns, $filter, $orderby, $limit){		
		
		foreach($filter as $value){
			$finalfilter = "$value[0]='$value[1]' AND ";
		}
		
		$finalfilter = rtrim($finalfilter, " AND ");
		$query = "SELECT $columns FROM $table_name WHERE $finalfilter";
		
		if(isset($orderby)){
			$query .= " ORDER BY $orderby";
		}
		if(isset($limit)){
			$query .= " LIMIT $limit";
		}
		
		$result = mysql_query($query)or die("failed to get records: ".mysql_error());
		while($row = mysql_fetch_array($result)){
		  $array[] = $row;
		}
		return $array;
	}

	//mysql_real_escape_string data before submitting to server
	private function escape_string($data){
			
		//incase we run script on a server where magic quotes is enabled...
		if(get_magic_quotes_gpc() == 1){
				
			//is $data an array?
			if(is_array($data)){
				foreach($data as $value){
					$stSlashDataArray[] = stripslashes($value);
				}
				unset($value);
				foreach($stSlashDataArray as $value){
					$eacapedDataArray[] = mysql_real_escape_string($value);
				}
				return $eacapedDataArray;
			}else{ //if not an array
				$data = stripslashes($data);
				return mysql_real_escape_string($data);
			}
				
		}else{//if magic quotes is disabled...
			
			//is $data an array?
			if(is_array($data)){
				foreach($data as $value){
					$eacapedDataArray[] = mysql_real_escape_string($value);
				}
				return $eacapedDataArray;
			}else{ //if not an array
				return mysql_real_escape_string($data);
			}
		}
	}

	//close connection to database when done
	public function close(){
		mysql_close($this->link)or die("failed to clode db: ".mysql_error());
	}
 
UPDATE: Myscript is running :) my server is filling up with appstore apps and my database is populating automatically, checking for errors on the way.

Ill let you know when its complete if all went well.

@Tempel once I have a database of files ill create the JSON repo script so you can test your native client :p

happy times :)
 
The script works but there were issues with some of the files im just gonna improve my error reporting so I know what the problems are incase my checks arnt working properly..
 
Back
Top