The Pnd Package Manager

Is this something that you would use?

  • Yes, definitely.

    Votes: 84 77.8%
  • Well, I don't know; I'd really need to have the feature that I'm going to specify below.

    Votes: 7 6.5%
  • Nah, the File Archive is enough for my needs.

    Votes: 10 9.3%
  • Yeah, and I'd help improving it.

    Votes: 7 6.5%

  • Total voters
    108

dflemstr

It's a ball.
Joined
Jul 31, 2008
Messages
2,514
Location
Stockholm, Sweden
Website
Visit site
Hello everyone,
about a month ago I started thinking about how the whole user experience is going to be like on the Pandora when it gets released. As you all know, we're getting an Ångström-based (yes, å is a vowel contrary to popular belief) distribution that has the ability to give us a very well-polished and integrated experience on the Pandora, so I didn't think that this is going to become a problem; the software we get on the Pandora couldn't be better under the circumstances.

One thing I saw as problematic, however, was how we would get software onto the Pandora. We'll have specialized packages, PND-files, that make installing applications really easy: you just download them from the Internet, drop them in a folder on the Pandora and BAM, the package is installed. Sounds easy, right? Well, I didn't think it did. I thought that it could be made even easier.

The first thing that in my opinion was too complicated was the acquiring of packages: The plan was that we would upload packages to the Pandora File Archive, and then download them from there. The problem with this is that the file archive seems to be REALLY old, isn't easy to navigate, isn't specialized for the handling of PND packages, and just doesn't behave intuitively.

The second problem that I saw was that with the proposed system, it would be impossible to auto-update packages for the Pandora. You would manually have to search for a new version of your package every so often in the Pandora File Archive and then download it manually just to get new features. I thought that it should be possible for this to happen automatically.

The third problem ... well, I'll cover that some other time.

So, I tried to find a solution to these problems, and finally created one that in my opinion at least is a step in the right direction:
The PND Package Manager

So, what is the PND Manager?

The PND Manager is a drop-in replacement for any other online PND package management system (I don't know if there are any in existence, but yeah). It is an application that will run on a web server - just like the Pandora File Archive - and manage and store PND files for you.

You can very easily upload packages to it, browse packages with it, and download packages from it.

It is NOT meant to replace the Pandora File archive, it just manages PND files (including the uploading/downloading of them); you can still upload manuals/image galleries or other types of files to the File Archive.

Here's a list of all the features that make it unique:

Really easy to use
  • The navigation is really intuitive; if you think that a link does something, that is usually what it does, too.
  • The interface is optimized for the Pandora screen.
  • I have used jQuery and AJAX to make the whole application feel like a local application.
  • The login system uses OpenID, so you don't even have to create a new account to upload packages!
  • You don't have to write things like descriptions, or create screenshots of what you upload; the PND Manager automatically analyzes your PND package and fills in all the fields for you.

A modern system
  • The PND Manager builds on cutting-edge technologies:
    • The framework I use (Lift) is at the time of writing only 3 hours old
    • Well, OpenID is kinda cutting-edge...
    • I use the Scala programming language.
  • The theme is very modular, and the design is very easily changed if a designer wants to change it. I have made the current theme, but it isn't the best theme ever; I would gladly accept volunteers that could make better themes.
  • New features can be added easily on demand. Tell me what you want, or write me a patch, and it'll be added!

Extensibility
  • The PND Manager exposes a REST-API that makes it possible to write applications for the Pandora that simplify package management, such as tools similar to "apt-get" or openSUSE's one-click install, package updaters, or local package browsers similar to the iPhone AppStore.
  • The PND Manager core is really loosely coupled; if someone who can code wants to replace a part of it or add something, it's possible.
  • The PND Package manager is open source, under the Apache 2.0 (compatible with GPLv3) license.

Some extras
  • It is completely translatable; as you can see from the screenshots, I've translated it into Swedish, and anyone can add translations for it. And if the PND package that you upload contains translations, the PND Manager will use those, too.
  • It is really, really fast. I haven't finished optimizing it yet, but so far, it's by far faster than the Pandora File Archive (And I don't mean network latency here)
  • It's easy to administrate. See the wiki page for administration for more information.
  • Technical details:
    • The application is clusterable
    • It doesn't need access to a file system; it only needs a database to work with.
    • The app is compatible with almost every SQL database known to man, since it uses JDBC.

How it works

Step 1: Log in with an OpenID


Step 2: Upload a package


Step 3: Enjoy your package!


Step 4: (And let others download it)
(Hmm, this screen shot was deleted by ImageShack; oh well)

Trying it out
(yes, you heard me right)

If you want to try out the application and see how it feels like, go to the instructions page on the wiki. Oh, and there you'll also find information about deploying, customizing and contributing to the application.

The quick guide for Ubuntu if you want to test the app:
Code:
sudo aptitude install git maven2 openjdk-6-jdk
git clone git://github.com/dflemstr/pndmanager.git
cd pndmanager
mvn jetty:run
...then wait until the Console says (it could take quite a while, because maven needs to download lots and lots of dependencies):
Code:
[INFO] Started Jetty Server
[INFO] Starting scanner at interval of 5 seconds.
...and go to http://localhost:8080/

Note that when you try out the application, it will start a server on your computer, and use a local database for packages, so it MIGHT require a lot of ram. It will also delete almost all of its traces after you stop the test server, so don't worry that it'll eat your hard drive & hamster. Oh, and you probably don't have PND packages to test the application with; if this is the case, use the 3 packages that are included in the repository.

If you're into that kind of thing, here's the source code. You'll need it when trying out the app anyways, but in case you are interested in browsing it online, there's the link.

I am really interested in hearing what you think about it, and if there's anything I can improve before release. I had originally planned on only using this application myself, but since people seemed interested in something like this, I thought that I'd ask if this would be something that people would use. So just throw it at me!

What will be done in the future?

So, I've read through your responses, and here are the changes that you have suggested so far:
  • Implement caches to make the app faster: [Done]
  • Create a FTP frontend (meaning: expose the whole data structure of the PND Manager to a file system) for the PND manager so that it becomes easier to browse packages [WIP]
  • Add download counts to packages: [Done]
  • Create a local application browser on the Pandora itself so that you don't have to visit the web page to download apps: [I won't do this; volunteers?]
  • Add an auto-update program for the Pandora that automatically checks for updates for "installed" PND packages: [Same as above]
  • Change the name of the application to something else. ("PND repository", "Package browser", "Pandora package manager"?)
  • Scan/spider the File Archive for PND packages every so often, and load the found PND files into the manager automatically [WIP]
 
Having looked at the screenshots and having read your many posts about it, it seems like you've put a lot of thought into making the design of your PND manager extensible, efficient, and easily usable by people both on "full-size" computers and on the Pandora. If you put up a public server where you have a few things "uploaded" and people can test it out in a read-only manner, you'd probably get a lot of input as to how it can be improved (and if people would like to use it). I'd love to try it out and give feedback, but setting up an application server and going through with all of the installation/compilation/configuration steps just to do that would be a bit more work than I'd like to put in just to test it out.

Looks great though, and I'm really looking forward to trying it. If it's as good as you make it seem, hopefully it'll be officially adopted and used as the Official Pandora Archives. :D
 
Vorporeal said:
Having looked at the screenshots and having read your many posts about it, it seems like you've put a lot of thought into making the design of your PND manager extensible, efficient, and easily usable by people both on "full-size" computers and on the Pandora. If you put up a public server where you have a few things "uploaded" and people can test it out in a read-only manner, you'd probably get a lot of input as to how it can be improved (and if people would like to use it). I'd love to try it out and give feedback, but setting up an application server and going through with all of the installation/compilation/configuration steps just to do that would be a bit more work than I'd like to put in just to test it out.

Looks great though, and I'm really looking forward to trying it. If it's as good as you make it seem, hopefully it'll be officially adopted and used as the Official Pandora Archives. :D
Hey, installing the server takes 4 commands on any Linux distribution, and there's nothing to set up!
But yeah, I should probably find a test host somewhere.
 
Last edited by a moderator:
dflemstr said:
Vorporeal said:
Having looked at the screenshots and having read your many posts about it, it seems like you've put a lot of thought into making the design of your PND manager extensible, efficient, and easily usable by people both on "full-size" computers and on the Pandora. If you put up a public server where you have a few things "uploaded" and people can test it out in a read-only manner, you'd probably get a lot of input as to how it can be improved (and if people would like to use it). I'd love to try it out and give feedback, but setting up an application server and going through with all of the installation/compilation/configuration steps just to do that would be a bit more work than I'd like to put in just to test it out.

Looks great though, and I'm really looking forward to trying it. If it's as good as you make it seem, hopefully it'll be officially adopted and used as the Official Pandora Archives. :D
Hey, installing the server takes 4 commands on any Linux distribution, and there's nothing to set up!
But yeah, I should probably find a test host somewhere.

From what I could tell (I did read through the installation guide), the 4 commands will give you the .war file, which you then need to have an application server to actually use. It's the setting up the application server which would be a pain. (I'm assuming it would be a pain - trying to set up Apache for a simple Trac server/SVN was a pain - I can't imagine Tomcat being loads better.)
 
Last edited by a moderator:
Vorporeal said:
dflemstr said:
Vorporeal said:
Having looked at the screenshots and having read your many posts about it, it seems like you've put a lot of thought into making the design of your PND manager extensible, efficient, and easily usable by people both on "full-size" computers and on the Pandora. If you put up a public server where you have a few things "uploaded" and people can test it out in a read-only manner, you'd probably get a lot of input as to how it can be improved (and if people would like to use it). I'd love to try it out and give feedback, but setting up an application server and going through with all of the installation/compilation/configuration steps just to do that would be a bit more work than I'd like to put in just to test it out.

Looks great though, and I'm really looking forward to trying it. If it's as good as you make it seem, hopefully it'll be officially adopted and used as the Official Pandora Archives. :D
Hey, installing the server takes 4 commands on any Linux distribution, and there's nothing to set up!
But yeah, I should probably find a test host somewhere.

From what I could tell (I did read through the installation guide), the 4 commands will give you the .war file, which you then need to have an application server to actually use. It's the setting up the application server which would be a pain. (I'm assuming it would be a pain - trying to set up Apache for a simple Trac server/SVN was a pain - I can't imagine Tomcat being loads better.)
You're right, but the 4th/5th command (depending on the path you take) gives you a working JETTY server that "just works"; no configuration needed. That server will serve the WAR file for you. Read the last section.

EDIT: I updated the guide to make more sense, kind of.
 
Last edited by a moderator:
I just had a lok at the screenshots for your application.
Looks good, but it also looks like it needs to be able to connect to a central server to work, and I don't have a wifi connection in my house.
So the Pandora file archive will still be where I'll get my files from.
 
I just had a lok at the screenshots for your application.
Looks good, but it also looks like it needs to be able to connect to a central server to work, and I don't have a wifi connection in my house.
So the Pandora file archive will still be where I'll get my files from.
 
White Demon said:
I just had a lok at the screenshots for your application.
Looks good, but it also looks like it needs to be able to connect to a central server to work, and I don't have a wifi connection in my house.
So the Pandora file archive will still be where I'll get my files from.
The application IS a central server to get files from :p (it runs on the web, like the Pandora File Archive), and it works on your normal computer too, just like the File Archive.
And yes, you still need an Internet connection to download packages, obviously. Someone else can create a local application (like the iPhone AppStore or anything) if they want to; this is what runs on the web and makes such things possible.
 
Last edited by a moderator:
I just had a lok at the screenshots for your application.
Looks good, but it also looks like it needs to be able to connect to a central server to work, and I don't have a wifi connection in my house.
So the Pandora file archive will still be where I'll get my files from.
 
My assumption was that the Pandora File Archive will still be where the files live. This is just a streamlined way to manage them. Dflemstr, do I assume correctly, or am I making an ass?
 
Gruso said:
My assumption is that the Pandora File Archive will still be where the files live. This is just a streamlined way to manage them. Dflemstr, do I assume correctly, or am I making an ass?
You are indeed making an ass :p
The PND Manager stores the files in its own database, because it needs the extra speed. It is, as I said, a replacement for the File Archive, at least for uploading PND files. I'll edit my first post to clarify this; thanks for your questions!
 
Last edited by a moderator:
Saw that coming. :D This is the line in your first post that confused me:

"It is NOT meant to replace the Pandora File archive; just the PND part of it."

So you might need to clarify where the files will live. If devs are uploading all their work to the File Archive, does that mean all this stuff has to be replicated somwhere else? Would it not be possible to have this working with the File Archive - even if it meant that ED had to make changes at his end?

I really like the look of this and I'd love to use it. I'm just kinda not getting it.
 
Gruso said:
So you might need to clarify where the files will live. If devs are uploading all their work to the File Archive, does that mean all this stuff has to be replicated somwhere else? Would it not be possible to have this working with the File Archive - even if it meant that ED had to make changes at his end?

I really like the look of this and I'd love to use it. I'm just kinda not getting it.
I am trying to make the post clearer by the minute ;)
So here are some answers:
The file archive is really a pain in the ass to work with (believe me, I've tried, and even asked the people responsible). It uses a cgi script to do all the heavy lifting, and the database and file repository is really not accessible form the outside.

And this is the main reason why you can't integrate the two:
File Archive: You specify a file to upload, give it a description, give it a screenshot, metadata, version information, etc, etc, and THEN you upload it.
PND Manager: You upload the PND file, the PND Manager automatically opens and organizes the file, and you are done. All screenshots etc are loaded automatically.
These two features require completely different backends, and since the File Archive doesn't support post-processing of uploaded files, I can't really do anything with it.

Analogy: Imagine that you are inventing E-Mail. What would be the most sensible thing to do: Sending diskettes by normal mail, or sending messages over the Internet? It's almost the same situation here: Nothing really improves if I use the File Archive as the database; we just get the same features with a different UI, meaning no auto-updates, no streamlined management process, no automatic metadata loading, no repository features.
 
Last edited by a moderator:
Update: It is currently not possible to compile "pndmanager", because the http://maven.reucon.com/public/ dependency repository is offline. So, it won't be possible to test the application right now.
 
I see a few problems with this. First, is the problem that Gruso pointed out, the fact that you'd now be having devs upload to another place. This would inevitably become a situation where people wouldn't be able to use one or the other site, but would need to go to both to look for packages, since most devs would load to both sites, some devs would upload to the file archive, and others would upload to your site. I would agree with Gruso and say that it would be worth it to work something out with Ed to get both systems working off of the same information, automatically.

The next issue is that you're using a lot of traffic to accomplish something that should be able to be done mostly offline. Most (if not all) modern package managers will have a local database with a list of applications, descriptions, dependencies, etc. so that you can browse the archive that way. This is much faster for users to browse through; it is especially important on portable devices where some of us might be using modems for data. You would need a standalone app for this.

Third, you don't seem to have a way to download multiple packages. Again, a feature of most modern package managers. This is fairly important, again especially on a portable system, because it makes installing this stuff much faster and easier for the user.

What I would really like to see is a local client that can be used offline, which connects to the main file archive (to update its database and download files), and allows multiple files to be downloaded simultaneously. I would use your project (which doesn't seem to be a package manager, but a web based repository) over the file archive if you made it work with the files in the archive, because it looks really nice. Otherwise, there's no point in asking, because we (as the users) will need to use both to find what we're looking for.
 
emil10001 said:
I see a few problems with this. First, is the problem that Gruso pointed out, the fact that you'd now be having devs upload to another place. This would inevitably become a situation where people wouldn't be able to use one or the other site, but would need to go to both to look for packages, since most devs would load to both sites, some devs would upload to the file archive, and others would upload to your site. I would agree with Gruso and say that it would be worth it to work something out with Ed to get both systems working off of the same information, automatically.
It would be possible to install a spider that would synchronize the two databases, but this would eat a lot of CPU. There's really no other solution, because the File Archive software is closed source, afaik. However, I see absolutely no reason why people would upload PND files to the file archive if this would become something official. I mean, is there a dev that would CHOOSE not to make his/her app available in a package browser directly on the Pandora, get auto-updates, etc? Name me the advantages of uploading a PND in the File Archive instead of the manager, and I'll see to fixing the issues with the manager. There just shouldn't be a need to ever upload PND packages to the File Archive.
emil10001 said:
The next issue is that you're using a lot of traffic to accomplish something that should be able to be done mostly offline. Most (if not all) modern package managers will have a local database with a list of applications, descriptions, dependencies, etc. so that you can browse the archive that way. This is much faster for users to browse through; it is especially important on portable devices where some of us might be using modems for data. You would need a standalone app for this.
This is just the HTML browser, remember? I have an exposed API that enables packages to work with the database locally. For example, an application can download the (dynamically generated) file http://*host*/api/repository.xml, which contains metadata with all the information you need. So it would of course be possible to browse the database offline, but you wouldn't be able to install anything, obviously.
emil10001 said:
Third, you don't seem to have a way to download multiple packages. Again, a feature of most modern package managers. This is fairly important, again especially on a portable system, because it makes installing this stuff much faster and easier for the user.
Again, this isn't a package manager in the sense of apt-get; that still has to be implemented. This is more like http://appnr.com/ for Ubuntu, but my app is also a repository, something which appnr of course is not. So when someone makes a frontend to the manager (aka a local application like apt-get/yum/pacman/whatever) that frontend can have whichever features you choose. The API allows for a lot of things.
emil10001 said:
What I would really like to see is a local client that can be used offline, which connects to the main file archive (to update its database and download files), and allows multiple files to be downloaded simultaneously. I would use your project (which doesn't seem to be a package manager, but a web based repository) over the file archive if you made it work with the files in the archive, because it looks really nice. Otherwise, there's no point in asking, because we (as the users) will need to use both to find what we're looking for.
Yes, this can be done. I hope to start working on it soon; however, I would prefer if someone who is used to building such tools would do it, since I probably would create suboptimal solutions. I plan on documenting the REST API of the PND Manager soon, so that it becomes easy for devs to work with it.

BTW: I call the application PND Package Manager because it manages PND Packages, nothing else. The name doesn't imply that it should install and upgrade packages for you; that will be handled elsewhere. I hope to create a complete suite, however, that makes up the "PND Package Management platform", so that there won't be 100's of tools that all do the same job, but instead an integrated set of tools that can be used everywhere.
 
Last edited by a moderator:
Ok, I'm catching on but I'm still not there. Sorry for the dumb questions. ^_^

If this system can't make use of the official archive, then where will the apps be hosted?
 
Gruso said:
Ok, I'm catching on but I'm still not there. Sorry for the dumb questions. ^_^

If this system can't make use of the official archive, then where will the apps be hosted?

OK, so I'll make an overview over the system to clarify a few things (I like lengthy answers).
Yes, I know that a lesser explanation would suffice for you, Gruso, but this is in case other people start asking questions too.

This is a description of the PND Package Manager from the ground up - every little detail :p .

First of all, the PND Manager is a web application that lives on a Java JSP/Servlet server - aka a machine standing in some corridor somewhere. I create a WAR file (Web Application aRchive) containing all of the PND Manager application files (HTML files, CSS, JS, Scala code classes etc) and give it to a JSP/Servlet server, and this server extracts the archive and starts a number of processes/threads (usually about 500; the number adjusts dynamically) that all run the application. So basically, the PND Manager is like a UNIX daemon - an application that runs all the time - it just runs massively parallel so as to utilize a server's multi-core processor better.

Now, my application, when started, will listen to a port - by default port 8080, but this can be configured - for incoming HTTP (aka Internet) requests. This is the primary interface for the application. It receives HTTP requests and responds to them. Yeah. Basically. (It seems so pointless when you look at it from this POV :( )

So far so good. Now, there are three primary "HTTP interfaces" for the application.

The first is the browser interface; it's what you have seen in the screenshots: the user's interface to the application. It enables users to send web page requests to the application, and it will give back user friendly HTML. This includes the whole user handling system and all of the nice UI animations and what have you. The human interface, so to speak.

The second interface is the file interface. You send a request to the application (for example with the URL http://the.server.address/packages/mypackage-1.2.3.4.pnd or http://.../screenshots/1.png) and you'll get back various files. You also send requests containing file data to "upload" a file. This interface is used both by the human and API interfaces.

The third interface is the API interface. Applications can download for example the http://.../api/repository.xml file, and they'll get information about the stored packages in XML format. It's what computers would use to communicate with the application.

Now, when the PND Manager gets HTTP requests, it obviously manages them in various ways. It has to render the nice tables that you get when browsing, and collect data about packages etc. Most of this is handled exactly like an app would handle things on your computer: everything is computed using a CPU, and I cache many things in RAM.

I now come back to answering your question, Gruso:
The PND manager obviously needs to store all of its data (uploaded PND files, user data, statistics and metadata) somewhere in a persistent way. This can't be done in RAM, because RAM is finite and if the server would go down, we would lose all of the data. So, when the PND Manager starts, it asks for a data management system, aka a SQL relational database. This can be anything (I don't care, it's up to the administrator to provide a system for me) but most often, it's a MySQL-based database system. So, the PND manager uses its own system to store all of its files, and this system is a simple database. If the admin choses to store the data on the server, so be it, or if he choses to create a dedicated database server, he can do that too. My system is so flexible that you wouldn't believe it ;) . For example, when you test my application as described in my first post, the data management system automatically creates a database in RAM and uses it while the testing session goes on (and the database is of course lost after the application quits, because RAM is volatile and data stored in it isn't persistent).

TL;DR People tend to think that you'd need a "file storage site" somewhere, but this isn't the case. The PND Manager *is* the file storage site. It manages the data on its own, and doesn't need any external file management systems to work.

I hope you all gained some knowledge from this ;) .

Yeah, I suck at explaining things, don't I
 
Last edited by a moderator:
Correct me if I am wrong but does this mean that the uploader of the PND package also hosts the file? This means when the uploader turns his/her computer/pandora whichever it is uploaded from off the file is 'gone'

In that case Id rather see a central server like the file archive.

(turning a computer off might be blasphemous to some persons but a lot of us do turn it off once in a while)

and excuse all my noobness
 
Back
Top