I have some code for pasrsing the appstore download links if you need anything from that i can find it from my old code.
Don't worry, I got plenty of experience with malformed PXMLs while writing PNDstore
Parsing them will never be perfect, but it should be enough to give the right idea to end users.I think I've figured out how to crawl through the download links, but your code could be useful. Post it up wherever, and I'll look over it. Assuming people think this will actually be useful.
I wish taht more Progamms follow these basic standards, would really help the Pandora Useres and it's a important addition to the usability of the Pandora IMHO.problem with that is not all the apps in the appstore follow the PXML correctly so for alot of them you will be unable to provide full detail, and also if you thinking parse the info from the app pages within the app store ... well even the version details are messed up, but maybe you can do it
Large parts of the spec have been changed since the beginning of the PND system, older PNDs dont follow these specs : they were created before the changesI wish taht more Progamms follow these basic standards, would really help the Pandora Useres and it's a important addition to the usability of the Pandora IMHO.problem with that is not all the apps in the appstore follow the PXML correctly so for alot of them you will be unable to provide full detail, and also if you thinking parse the info from the app pages within the app store ... well even the version details are messed up, but maybe you can do it
Is Some kind of "auto-correct-PND-Files-tool" possible?
<?PHP
//Basic app store URL parser class by Jake (aka milkshake)
class parse_app_store{
public $links_array;
private $category = array('Emulators','Games','Applications','Other');
private $cat = array('Emulator','Game','Application','Other');
private $downloadURI = "http://apps.openpandora.org/cgi-bin/dl.pl?";
private $_error_list;
function __construct(){
$this->reset_error_list();
$this->parse_links();
$this->links_check();
}
//returns array of all pnd filenames on the appstore website
public function parse_links(){
$i=0;
foreach($this->category as $value){
$url = "http://apps.openpandora.org/cgi-bin/viewarea.pl?".$value;
$code = file_get_contents($url);
$doc = new DOMDocument();
@$doc->loadHTML($code);
//search for li elements as they contain the name of the app
//stored within the onclick attribute - populate $link array
foreach ($doc->getElementsByTagName('li') as $element) {
if ($element->hasAttribute('onclick')) {
$link[] = $element->getAttribute('onclick');
}
}
//for app name store within link array
foreach($link as $val){
$cat = array('Emulator','Game','Application','Other');
//regex magic to get just the app name - \/([^.]+)/ -
preg_match("/".$this->cat[$i]."\/([^\&]+)/",$val,$match[]);
}
$i++;
//unset variables to be on safe side
unset($link,$val,$code,$doc);
}
//for each match create new array called $links_array with name of app
//as array-key and URL as array-value.
foreach($match as $app){
$name = substr($app[1], 0, -6);
$this->links_array[$name] = $this->downloadURI.$name.".inf";
}
//unset variables to be on safe side
unset($match,$value);
}
//simple, check to make sure we have links in array
private function links_check(){
if(empty($this->links_array) && !is_array($this->links_array)){
$this->_error_list[]='no links found';
}
}
//check whether any errors have occurred
//returns Boolean
public function is_error(){
if (sizeof($this->_error_list) > 0){
return true;
}else{
return false;
}
}
//return the current list of errors
public function get_error_list(){
return $this->_error_list;
}
//reset the error list
public function reset_error_list(){
$this->_error_list = array();
}
}
?><?PHP
include('class/parse_store_test.php');
//instantiate our object
$p = new parse_app_store();
//check for errors
$err = $p->is_error();
if($err==true){
$err_array = $p->get_error_list();
foreach($err_array as $error){
echo $error."<br/>";
}
exit();
}
//if no erros get array
$url_array = $p->links_array;
echo count($url_array)." links parsed<p>";
//to test we have our downloads links lets generate a list.
foreach($url_array as $k=>$v){
echo "<a href='$v'>$k</a><br/>";
}
?>#!/usr/bin/env python2
import os, urllib, json
from lxml import html, etree
download_uri = 'http://apps.openpandora.org/cgi-bin/dl.pl?%s.pnd'
def download_apps(storage_dir):
categories = ('Emulators', 'Games', 'Applications', 'Other')
directory_uri = 'http://apps.openpandora.org/cgi-bin/viewarea.pl?%s'
destination_path = os.path.join(storage_dir, '%s.pnd')
for cat in categories:
page = html.parse(directory_uri % cat)
for i in page.xpath("//div[@class='itemlist']/ul/li/@onclick"):
name = i.rsplit('/',1)[1].rsplit('.',1)[0]
print 'Getting', name
urllib.urlretrieve(download_uri % name, destination_path % name)
def generate_json(storage_dir, output):
repo_data = {'repository':{'name':'Pandora Apps', 'version': 3.0},
'packages':[]}
pxml_start_key = '<PXML'; pxml_end_key = '</PXML>'
# Bytes of PND to check at a time, so we don't have to load the whole thing
# into memory at once.
window = 4096
# But maintain some overlap between each window to ensure the '<PXML'
# doesn't get cut off at the edge of one.
seek_jump = -window + len(pxml_start_key)*2
for fname in (i for i in os.listdir(storage_dir) if i.endswith('.pnd')):
print 'Parsing', fname
fpath = os.path.join(storage_dir, fname)
f = open(fpath, 'rb')
f.seek(-window, os.SEEK_END)
# Seek backwards to start of PXML.
pxml_start = -1
while pxml_start == -1:
text = f.read(window)
f.seek(seek_jump-window, os.SEEK_CUR)
pxml_start = text.rfind(pxml_start_key)
f.seek(pxml_start-seek_jump, os.SEEK_CUR)
# Read remainder of file, then remove everything after end of PXML.
pxml = f.read().split(pxml_end_key, 1)[0] + pxml_end_key
# Strip out the stupid goddamn namespace stuff that I hate.
pxml = '<PXML>' + pxml.split('>',1)[1]
# Parse PXML through lxml.etree.
parser = etree.XMLParser(recover=True, remove_comments=True)
root = etree.fromstring(pxml, parser)
# Extract relevant data.
package = {}
package['id'] = root.xpath('/PXML/*/@id')[0]
package['uri'] = download_uri % fname[:-4]
package['version'] = {'major':'0', 'minor':'0', 'release':'0',
'build':'0', 'type':'release'}
try:
version = root.xpath('/PXML/*/version')[0]
package['version']['major'] = version.xpath('@major')[0]
package['version']['minor'] = version.xpath('@minor')[0]
package['version']['release'] = version.xpath('@release')[0]
package['version']['build'] = version.xpath('@build')[0]
package['version']['type'] = version.xpath('@type')[0]
except IndexError: pass
titles = {}
for title in root.xpath('''/PXML/package/titles/title | /PXML/package/title
| /PXML/application[1]/titles/title | /PXML/application[1]/title'''):
lang = title.get('lang')
if lang is not None and lang not in titles:
titles[lang] = title.text
descriptions = {}
for desc in root.xpath('''/PXML/package/descriptions/description |
/PXML/package/description | /PXML/application[1]/descriptions/description
| /PXML/application[1]/description'''):
lang = desc.get('lang')
if lang is not None and lang not in descriptions:
descriptions[lang] = desc.text
package['localizations'] = {}
for lang in titles.iterkeys():
l = {}
try:
l['title'] = titles[lang]
l['description'] = descriptions[lang]
except IndexError: pass
package['localizations'][lang] = l
package['size'] = os.path.getsize(fpath)
# TODO: Calculate MD5.
try:
author = root.xpath('/PXML/*/author')[0]
package['author'] = {}
for i in ('name', 'website', 'email'):
j = author.get(i)
if j is not None:
package['author'][i] = j
except IndexError: pass
# TODO: Icons, previewpics from apps.o.o.
package['categories'] = list(set(root.xpath('/PXML/application/categories//@name')))
# Add package to list.
repo_data['packages'].append(package)
json.dump(repo_data, open(output, 'w'))
if __name__ == '__main__':
storage = '/media/disk/pnd_storage'
outfile = '/home/randy/www/extra/repo.json'
download_apps(storage)
generate_json(storage, outfile)I think it would be fair that you 2 handle this, so you'll know the pain you've put on us, the devs, with all your changes to the specs ! (joking)
"repositories": [
"http://repo.openpandora.org/includes/get_data.php"
,"http://randy.heydon.selfip.net/extra/repo.json"
]Missing data was basically handled by hoping for the best and falling back on defaults if need be. For example, at least one PXML had no version number, so it defaults to 0.0.0.0. And a couple didn't parse titles properly, but I haven't come up with a clever solution for that yet. I also don't have icons or screenshots, though I could eventually just provide the URLs to the ones on the appstore.
Yes, one more reason to update all the old Files, I would like to see the entire Pandora PND Archive from ED AND Craigs App Store AND the rest of the missing PNDs, found often in the Wiki list in the Repo.Large parts of the spec have been changed since the beginning of the PND system, older PNDs dont follow these specs : they were created before the changes
This really would be an improvement, you can't expect from the average User to seach 4 sources for new Pandora Software, this is Madness.
Yes, one more reason to update all the old Files, I would like to see the entire Pandora PND Archive from ED AND Craigs App Store AND the rest of the missing PNDs, found often in the Wiki list in the Repo.
And I'm not even count the Forums to the list. ^^""
<li onclick="location.href='../cgi-bin/viewapp.pl?/Application/rockbox.inf';" style="cursor:pointer;">
//rockbox.inf is what is contained within the redirect URL
<META HTTP-EQUIV="Refresh" CONTENT="3;URL=dl.pl?rockbox.inf">
