The Pnd Package Manager

[polyroy]

I would hate to have to use the File Archive to get software.

The way I see this PND "Package Manager" is a very cool way (on the Pandora or not) to get software for the Pandora.

I can easily see the points to some of the arguments but some of them are "linux user" arguments and seem to be semantic battles, etc. If the file archive can't sync with anything (this or any other project like it) easily, then it seems to me that it needs to be replaced. Someone came up with something user friendly and I think we should embrace it. If we need also to come up with a new File Archive, I think that's what needs to be done. If projects like this get shot down because they're not linux enough, then the Pandora will be just as nichey as linux itself. I would rather have an easily browsable online database like this than a crappy Package Manager in the classic sense any day.

You know, there is something to be said for centralizing everything and making it as simple as possible. I don't think that is a "Linux" argument per se. You say you'd hate to have to use the File Archive to get software? The PND Package Manager (could you reconsider the name? PND Package Repository makes so much more sense) is still a website (but a good one at that), and your way of interfacing would be comparable to the way it's going now with the File Archive. Unless a localized client is developed which does the following:

- Retrieve complete list of PND packages and metadata and store this on the Pandora (including screenshots, like goPlay).
- Have a list of "sources", locations of servers which have the packages (possibly just the master one).
- Track which packages are installed and match them to a specific SD-card (can we ID SD-cards uniquely? possibly store an identifier on it?) so we can easily manage things.
- Offer the ability to flag packages for download, and download them upon request. Think of going through the list and flagging everything that is even remotely of interest to you, just so you can check it out, and doing the actual waiting (for download) afterwards or in the background so you are not interrupted.
- Notice whenever newer versions of the packages you have are available, and suggest to download them the next time you have inserted the SD card associated with that package.

Preferably this kind of access can be partially implemented in libpnd. It seems suitable for it at least.

 

[Poem58]

So if I understand this correctly:

I have a SNES emu installed on my Pandora let's say SNES ver 1.10
and a Homerew app let's say BREW ver 1.55

The devs release SNES ver 2.0
and BREW ver 1.75

Using the Pandora file archive I would have to browse each section and manually find each update, check to see what versions I had then download each to upgrade.

Using your manager, I would log in and it would tell me that I have 2 apps that have updates and prompt me to install?

If so then I would highly support this as I found it time consuming to check every piece of homebrew for the PSP for updates.
Seeing as how the storage and capabilities of the Pandora will allow for many types of software ranging many categories, you could easily spend an hour or more each time searching for updates to many programs. Any Idea to streamline and make updates a quick couple minute check any given day are welcome. Imagine as much as we F5 on these boards if we had like 20 categories of messages to search around in to find new information.

If this is really what this is suggesting, I cannot see a reason NOT to do it. Or why if it's as functional as it sounds, The Pandora team would not want it as the main file update system.

If I understood, someone made it sound as if not having just a file archive for a new Pandora owner to browse might hurt future sales, wouldn't they be able to browse this just the same?

the only thing I don't yet understand it this Open ID thing for login, I have never heard of Open ID but as long as it is a simple sign up that shouldn't be a problem.

If I have this right all I can say is great work dflemstr!

Edit: As above I would agree maybe a name change is in order. Having never really used linux "Package Manager" meant nothing to me (which is why I read this thread) maybe "Pandora Software Manager" or "Pandora Software Updater" or something might be better?

 

[jlepley]

I for one like the idea and don't understand the confusion. But let me make sure I've got this straight.

It's basically an application that will run on a server. Users will upload their PND files straight from a web browser onto a web site that acts as a liaison between the user and the application. The application will then read the PND file, extract the description and screenshots, and generate a web page from which users will be able to learn about and download the application. Meanwhile, a daemon will run on the Pandora, check the applications on the internal memory and SD cards against the PND applications that have new updates, and prompt the user to download the available updates.

Do I pretty much have that right?

 

[dflemstr]

Oh, wasnt .pnd just uncompressed .iso with a footer XML, so totally uncompressed (?).
The repository should allow access to the pnds in some highly compressed form (like LZMA or bz2),
that would be decompressed on-the-fly (as it comes in) by the on pandora software.
We will be using slow links anyways.

This, you'd need to discuss with Cpasjuste or skeezix, as these are the people responsible for the actual PND format.

File Archive: You specify a file to upload, give it a description, give it a screenshot, metadata, version information, etc, etc, and THEN you upload it.
PND Manager: You upload the PND file, the PND Manager automatically opens and organizes the file, and you are done. All screenshots etc are loaded automatically.
These two features require completely different backends, and since the File Archive doesn't support post-processing of uploaded files, I can't really do anything with it.

And what about "PND Uploader"? Application that upload PND file to File Archive and fill those description, screenshots etc for user. So not new File Archive backend, but new frontend (client application). This application can even check File Archive for new version of PND.

edit: bb code correction

It would be possible, but since the File Archive is closed-source it would be difficult. If someone wants to do it, then please do it; I don't think that I will

So if I understand this correctly:

I have a SNES emu installed on my Pandora let's say SNES ver 1.10
and a Homerew app let's say BREW ver 1.55

The devs release SNES ver 2.0
and BREW ver 1.75

Using the Pandora file archive I would have to browse each section and manually find each update, check to see what versions I had then download each to upgrade.

Using your manager, I would log in and it would tell me that I have 2 apps that have updates and prompt me to install?

First, remember that the system isn't fully finished yet, so what I'm going to tell you now is how I'm *hoping* that it will be.
Basically, you are going to have an application installed on your Pandora (NOT the PND Manager) that downloads a file from the PND Manager every day. This file would tell the system if there is a new update for your application, and if there is one, it would be downloaded automatically. So basically, you can just download a PND package, put it in a folder (lets say, a "PND Packages" folder on your SD card) and then that PND package would be updated automatically while it is on your SD card. Then, if you remove the PND file, it would stop getting updates (as if you had "uninstalled" the PND)

Edit: As above I would agree maybe a name change is in order. Having never really used linux "Package Manager" meant nothing to me (which is why I read this thread) maybe "Pandora Software Manager" or "Pandora Software Updater" or something might be better?

Yes, a new name would be good. I would really like to have PND in the name, but if enough people suggest another name, we'll go with that.

Oh, and I'll put up a todo-list in my first post with the suggestions that I have gotten so far.

 

[polyroy]

Using the Pandora file archive I would have to browse each section and manually find each update, check to see what versions I had then download each to upgrade.

Using your manager, I would log in and it would tell me that I have 2 apps that have updates and prompt me to install?

Essentially, yes, that would be possible. But it requires the PND Package Manager to have information on your system. I do not see a need to log in with the service other than for developers submitting new PND files, because then it would not be true REST. It would be much nicer if your own Pandora were to track what was installed, since you can also identify which PND files are removed from SD cards (which is not possible with the online-only version).

If this is really what this is suggesting, I cannot see a reason NOT to do it. Or why if it's as functional as it sounds, The Pandora team would not want it as the main file update system.

For developers that need packages from Angstrom for which no PNDs exist, another option is needed: IPKG. Also no PND's can be made which integrate closely with the OS without some difficulties. I don't think the PND functionality can be integrated using IPKG alone, so we'll have to settle for two package managers. One for internal hackish OS-modification stuff, and one for the general public.

If I understood, someone made it sound as if not having just a file archive for a new Pandora owner to browse might hurt future sales, wouldn't they be able to browse this just the same?

But you'd not have the established community of the File Archive admiring all the Pandora packages, so you won't easily convince them to buy a Pandora if they don't notice what is available.

Edit: As above I would agree maybe a name change is in order. Having never really used linux "Package Manager" meant nothing to me (which is why I read this thread) maybe "Pandora Software Manager" or "Pandora Software Updater" or something might be better?

The local client should be called a Manager, the service itself should be called a Repository. Recognizing that "package" might be confusing, so I opt for PND Software Repository for everything on the net, and PND Software Manager for the app on the Pandora itself.

 

[dflemstr]

I for one like the idea and don't understand the confusion. But let me make sure I've got this straight.

It's basically an application that will run on a server. Users will upload their PND files straight from a web browser onto a web site that acts as a liaison between the user and the application. The application will then read the PND file, extract the description and screenshots, and generate a web page from which users will be able to learn about and download the application. Meanwhile, a daemon will run on the Pandora, check the applications on the internal memory and SD cards against the PND applications that have new updates, and prompt the user to download the available updates.

Do I pretty much have that right?

Exactly right.
Except that:

1. It will be possible to use a local application to browse packages, too; you don't need to visit the web page
2. The updates *can* happen completely automatically; you won't have to care about them

 

[emil10001]

My only further sentiment on this is to reiterate that you should do everything possible to make this work with the official File Archive, even if it means convincing Ed to use some other, more open software for it. (Even though many closed-source platforms provide an API, and I would be that this one does too.) The reason for this is that if you push ahead and say that people should just accept your way of doing things because it's better, it's going to piss people off, annoy users, and, if things get really ugly, possibly fracture the community. I get that the File Archive is not currently being used to house PND files, but my bet is that it will, as soon as those units start shipping. What you need to understand is that going down this road is really only going to cause user frustration at needing to search in multiple places. On top of that, it opens the door for somebody else to come in and create yet another package manager because the one that you made doesn't work with the File Archive, further fracturing the userbase.

I get that you want to be innovative, and it is a really cool idea, but if you don't do it right, you're just going to end up pissing people off and making things difficult for the end users. I don't mean to be harsh, but I think that compatibility with the official archive should be a priority for you, because it will benefit both the end user (ease of use, one interface to go to for everything), and you (people will use your utility because it does what they want).

 

[dflemstr]

My only further sentiment on this is to reiterate that you should do everything possible to make this work with the official File Archive, even if it means convincing Ed to use some other, more open software for it. (Even though many closed-source platforms provide an API, and I would be that this one does too.) The reason for this is that if you push ahead and say that people should just accept your way of doing things because it's better, it's going to piss people off, annoy users, and, if things get really ugly, possibly fracture the community. I get that the File Archive is not currently being used to house PND files, but my bet is that it will, as soon as those units start shipping. What you need to understand is that going down this road is really only going to cause user frustration at needing to search in multiple places. On top of that, it opens the door for somebody else to come in and create yet another package manager because the one that you made doesn't work with the File Archive, further fracturing the userbase.

I get that you want to be innovative, and it is a really cool idea, but if you don't do it right, you're just going to end up pissing people off and making things difficult for the end users. I don't mean to be harsh, but I think that compatibility with the official archive should be a priority for you, because it will benefit both the end user (ease of use, one interface to go to for everything), and you (people will use your utility because it does what they want).

I see your point, and I'll see what I can do.
I have added it to the TODO list (or I will, after posting this ).
The best solution would be to simply scan the File Archive for PND files, and download them to the PND Manager every so often. There are a few obstacles that would need to be countered:

1. If someone uploads an invalid PND file (let's say that someone takes an .exe file and just renames it to .pnd) in the File Archive, what should happen? The PND Manager validates all PND packages uploaded through it, and this validation would have to happen to the file archive too.
2. What should we do "in the other direction"? Should we sync all the packages from the PND Manager back to the File Archive? If so, how?
3. What about owners? Who should "own" packages that come from the File Archive (aka who should have the rights to delete them)? We cannot rely on the fact that people will have accounts in both systems, it won't work.

 

[emil10001]

I see your point, and I'll see what I can do.
I have added it to the TODO list (or I will, after posting this ).
The best solution would be to simply scan the File Archive for PND files, and download them to the PND Manager every so often. There are a few obstacles that would need to be countered:

1. If someone uploads an invalid PND file (let's say that someone takes an .exe file and just renames it to .pnd) in the File Archive, what should happen? The PND Manager validates all PND packages uploaded through it, and this validation would have to happen to the file archive too.
2. What should we do "in the other direction"? Should we sync all the packages from the PND Manager back to the File Archive? If so, how?
3. What about owners? Who should "own" packages that come from the File Archive (aka who should have the rights to delete them)? We cannot rely on the fact that people will have accounts in both systems, it won't work.

Thank you. I appreciate you adding that to your TODO list.

Suggestions to your questions:
1. Perhaps allow an entry to be created, but just write a standard error message for the description, and disable it from being downloaded. Also, perhaps some sort of automated e-mail sent to Ed about it?
2. I wouldn't bother with this part, if you have everything available from the File Archive in your collection, there would be no need to push things back over there (unless there was some simple, automated way of doing that).
3. Good question, don't have a good answer for this one. My best suggestion here would be to possibly see if Ed's archive can produce a changelog, and export that to you on some regular schedule. That way, if the changelog notes a file has been deleted, your system could automagically delete the file there as well.
EDIT: In addition, you might want to use moderators to comb over this sort of stuff, so that users can report bad files, and the mods would be able to delete, if the owner didn't take action, or if the files were pulled automatically. I would guess that if you got this system up and running you could probably find a couple of people to help moderate it.

I really hope that you get your system off the ground and have some way to pull stuff over from the Archive. Because, from the looks of it, if one had to use a web-based interface for downloading files, I'd much rather use yours.

 

[dflemstr]

1. Perhaps allow an entry to be created, but just write a standard error message for the description, and disable it from being downloaded. Also, perhaps some sort of automated e-mail sent to Ed about it?

Well, packages are immutable (meaning that the cannot be changed after they have entered the PND Manager system - you can create packages and delete them, never anything in between. So having incomplete packages in the system would make things very complicated indeed. It might be possible - I'll look into it. One sustainable solution would otherwise of course be to just validate all incoming packages from the Archive and just skip those that don't validate.

3. Good question, don't have a good answer for this one. My best suggestion here would be to possibly see if Ed's archive can produce a changelog, and export that to you on some regular schedule. That way, if the changelog notes a file has been deleted, your system could automagically delete the file there as well.

We cannot count on anything coming from the (closed-source) file archive system, but I'll see if something is possible. Another solution would be that there simply is a special user - "filearchive" - that just owns all the packages from the file archive. This would be a dummy user, meaning that it wouldn't be possible to log in as that user, but packages could still be deleted because...

EDIT: In addition, you might want to use moderators to comb over this sort of stuff, so that users can report bad files, and the mods would be able to delete, if the owner didn't take action, or if the files were pulled automatically. I would guess that if you got this system up and running you could probably find a couple of people to help moderate it.

... there already is a moderator system in place in the PND manager system
The site administrator can give users "super user" privileges, and super users can remove any package, among other things.

We'll have to think some more about this; it isn't really necessary to find a solution soon, we still have 2 months™ until the Pandora is released, right?

 

[polyroy]

I just figured that if we were to produce a local client, we should really integrate it with the existing apps. The app launcher we have now could be extended to browse via the PND Software Repository (I like that name). Pretty much the complete PND Software Repository API could be implemented in libpnd for querying, and we'd have a great thing going. You'd have a comparable experience as those famous app stores... only better in the sense that it actually manages software on removable devices.

Imagine the comfort of having everything integrated; meant the way it should be

 

[dflemstr]

I just figured that if we were to produce a local client, we should really integrate it with the existing apps. The app launcher we have now could be extended to browse via the PND Software Repository (I like that name). Pretty much the complete PND Software Repository API could be implemented in libpnd for querying, and we'd have a great thing going. You'd have a comparable experience as those famous app stores... only better in the sense that it actually manages software on removable devices.

Imagine the comfort of having everything integrated; meant the way it should be

Well, I am currently not caring about the client side if things (I will at a later time but not now) so if anyone wants to coordinate/create something, please, go ahead!
Or we could just use Cpasjuste/skeezix what they think and if something like this could be done.
The API isn't really set in stone either (I haven't written the spec yet) so if there's something you'd need, just tell me. The only way to currently see what the API provides is to test the application; I'll have to write some documentation soon.
Anyways, I'll probably sign off now; I've been awake since 2009-06-30 20:00, and it's now 20:41 where I'm at. Happy discussing.

 

[emil10001]

1. Perhaps allow an entry to be created, but just write a standard error message for the description, and disable it from being downloaded. Also, perhaps some sort of automated e-mail sent to Ed about it?

Well, packages are immutable (meaning that the cannot be changed after they have entered the PND Manager system - you can create packages and delete them, never anything in between. So having incomplete packages in the system would make things very complicated indeed. It might be possible - I'll look into it. One sustainable solution would otherwise of course be to just validate all incoming packages from the Archive and just skip those that don't validate.

Perhaps you could just disabe the download link and leave the pnd file in there, with something in the description about the file being corrupted. You probably could have some error handling in your import function that would take care of this action before inserting it into the database. That way, you wouldn't need to worry about it breaking any of your rules.

3. Good question, don't have a good answer for this one. My best suggestion here would be to possibly see if Ed's archive can produce a changelog, and export that to you on some regular schedule. That way, if the changelog notes a file has been deleted, your system could automagically delete the file there as well.

We cannot count on anything coming from the (closed-source) file archive system, but I'll see if something is possible. Another solution would be that there simply is a special user - "filearchive" - that just owns all the packages from the file archive. This would be a dummy user, meaning that it wouldn't be possible to log in as that user, but packages could still be deleted because...
... there already is a moderator system in place in the PND manager system
The site administrator can give users "super user" privileges, and super users can remove any package, among other things.

That's pretty much exactly what I had in mind.

 

[Pseudonym]

Ok, lets say that the local application downloads a file(hopefully temporary)and checks to see if there are any updates.
1.How will it know what apps are installed, a history on the 512MB NAND, or on an SD card,or on the server?
2a.If it's on an SD card, what if that SD cards not in?
2b.If on a server/NAND/SD, what if the SD card with the App is not inserted?(ask for you to insert it/do a check?)
3.What if you're on a non-Pandora venue(coughcomputercough)and want to update a file? In other words, will it edit a PND package, replace and re-link a PND package, or something else, and will that be able to happen with a local file, or only an online file.
4.(I think this was stated before, but I didn't find an answer)will the local app be able to read a database file that holds all current apps?

Either way, I plan to use this as a primary source for software, though I would like to have a Debian-esque (and I think RPM linuxes too)package management app that can connect to multiple servers (such as this one). Then hopefully one of the File archive people (most notably Ed) might modify the server to be able to connect to this system as well.

 

[polyroy]

Ok, lets say that the local application downloads a file(hopefully temporary)and checks to see if there are any updates.
1.How will it know what apps are installed, a history on the 512MB NAND, or on an SD card,or on the server?
2a.If it's on an SD card, what if that SD cards not in?
2b.If on a server/NAND/SD, what if the SD card with the App is not inserted?(ask for you to insert it/do a check?)
3.What if you're on a non-Pandora venue(coughcomputercough)and want to update a file? In other words, will it edit a PND package, replace and re-link a PND package, or something else, and will that be able to happen with a local file, or only an online file.
4.(I think this was stated before, but I didn't find an answer)will the local app be able to read a database file that holds all current apps?

Either way, I plan to use this as a primary source for software, though I would like to have a Debian-esque (and I think RPM linuxes too)package management app that can connect to multiple servers (such as this one). Then hopefully one of the File archive people (most notably Ed) might modify the server to be able to connect to this system as well.

I'd guess the best way for 1) is both; Every SD-card would have it's own index for the apps on that card (would have to be scanned of course for manual additions, but to speed up the initial list, this would be more practical), for when it is inserted in someone elses Pandora (they need the applist anyway), and the Pandora itself will need an index of those files so that it knows that you've already installed such an app on SD card number 214

The entire metadata cache should not be kept in that 512MB, since it could get pretty big with all the screenshots and stuff. This should be user-selectable. You're probably going to end up with a single SD card dedicated to configuration files and such (primary extension of the 512MB) anyway, so it might be stored there.

For 2a) I guess you'd lose the information associated with that SD card, it would simply allow all basic functionality, but not know about all the packages you have.

2b) Asking to insert another card would be an option, but also ask to download directly instead so that you can install common apps on more than one SD card. The configuration those apps create should be stored either on the internal flash or on a primary SD card which you have inserted nearly always (as a form of home directory). Note that with all those SD cards storing random data, we'd really need an universal system for resource management. Preferably not just for the apps, but also for all other kinds of files. No idea how to do this properly yet.

3) If the pandora scans the card for updates, it would make sense to detect the PND archives you put on it, and mark them as "installed" in the system and update all references. The PND will always have an incremented version number (if you don't do that, you are messing things up deliberately).

4) Current as in "what is stored on the archive"? Well, you'd need a metadata cache for that, and if you want it complete, it'll need quite some space for the screenshots and stuff. The availability depends on where you store it (on the SD card or internally) and if that medium is available or not.

A proposal for such resources:

• Always stored locally:
  • List of SD cards with their resource content, you can manipulate this list (say exclude SD card x from indexing unless when it is inserted)
  • List of chan]ges to make to those resources (e.g. decide to remove/install app x from SD card y when it is inserted)
• Optionally stored locally:
  • Cache of (partial) resource metadata from the used repositories
• Optionally stored on every SD card:
  • Identical or more complete cache of (partial) resource metadata from the used repositories than is stored locally

This would make some data redundantly available, but it would reduce quite some hassles. It might also be a good idea to restrict metadata retrieval to certain categories of resources, so you don't get every C64/Amiga PD file in the world in that cache.

And lastly, if I were to have a large ROM or other personal file library, too big for the Pandora, could I set up a repository (every file auto-wrapped in a PND package) as a source for those files for private use? This is very much worth looking into, and then a PND file would essentially become a general purpose resource.

 

[Kramy]

Wow... this thread sure is packed with endless explanations.

I'll toss my 2 cents into the mix.

1) Because of the huge difference between PND files and regular packages, a PND manager makes sense. PND apps will likely make no changes to the underlying OS, which is quite different from the purpose of apt-get or Synaptic.

2) Java server backend. That's win. I fully believe you when you say it'll be blazing fast. I've messed around with Servlets before, and found them to be waaaaaaaay faster than anything .cgi or .php. (a's added for emphasis)

3) As long as we get a local app that can connect to the PND repo, I see no reason to use the file archive. It's terribly clunky. We definitely need something like it to locate software, but so far your PND repo looks superior.

4) For a local program, it'd be nice to be able to pick where stuff gets saved. I like to sort programs into subdirs - Favourite Games, TDs, etc.; In other words, a local client would have to be able to scan multiple(all) directories on an SD card to locate the "installed" apps. Most repository software seems to only look for stuff in specific places, so keep it in mind. Another must-have feature that a lot of software forgets about is "Move File". I can't count the number of torrent clients I've tried that can't move files, and can't seed a file after it's been moved. And yet I can tell you 100% of them were linux torrent clients, which leads me to believe independent hobby developers tend to forget stuff. My point? A local program is going to need more polishing and work than your web interface, and definitely needs more than one person working on it.

5) I neglected to read the last few super-long posts in this thread, so if I repeated something... oh well. (I dutifully read all of page 1 - 40 posts)

 

[gruso]

I'd really like to see this implemented, but if it's going to happen the server side would need to be sorted out very soon, before the official file archive starts to fill up. Hypothetically, if ED agreed to set up a new Pandora archive, what would his options be? What solution would offer the least time & cost to implement?

 

[Lorkile]

I have to suppose I would use it, as anything that makes it easier to download and install things on an unfamiliar OS will be a major boon for me. That, or I misunderstand and likely won't be back here long enough that I've forgotten about this post.

 

[dflemstr]

OK, some random replies to everyone:

1. The PND "Metadata" file, aka the repository file, only contains text information about packages (description, creation date, version, etc). Screenshots should NOT be downloaded unless absolutely necessary; when you download the repository metadata you'll only get a link to the screenshot. I want to reduce traffic as much as possible
2. You don't have to create an index at all for installed packages, IF you just:
   1. Create a file that contains all the PATHs to the folders where packages are dropped.
   2. Maintain the strict naming scheme. Currently, packages have the name <name>-w.x.y.z.pnd when they are downloaded. If we tell people that they shouldn't rename files, we can simply organize them by looking at their names.
   The thing with an index is that an index doesn't allow for PND files being moved (remember, they aren't "installed", they are just dropped in a folder somewhere, and the system loads their contents on demand.
3. If the medium where the PND is isn't inserted, we simply pretend that the PND doesn't exist (because it practically doesn't). Simple

Y'all, just forget the concept of "installation"; PND packages don't get installed: they are atomic. So we don't have to keep track of installed applications, we can simply specify some folders that should be scanned, and the found PND files are searched for in the repository index. If an update is found, we remove the old PND and replace it with a new PND.

 

[lulzfish]

I imagine PND files will be a lot like the Portable Applications that are available for some Windows programs.