The Pnd Package Manager

[javaJake]

maybe I'm old school, but I prefer to keep relational data in a relational DB, and big giant zip files out of it, with references from the DB to the files (ie: name, whatever.)

This practice is (still?) discouraged heavily by MySQL users, so I assume it's universally bad.[/quote]

I agree with others here concerning resource usage. One of the features that most software developers, even experienced ones, ignore the most is keeping it lean. It's one of those things where you don't realize how nice it is until you have it, and then you can't live without it. (Also: want not waste not, or something like that. )

 

[Kramy]

- if you're talking about pnd downloads to a pandora/some-device, then streaming is the way to go; it would be insanity to pull the files into memory while serving them out. *madness* I say, madness! You'll be wanting to pull in reasonably sized chunks .. madness to keep multiple MB/GB in RAM while sending down to someones slow-ass dial modem for hours on end. Make a Stream object pair (In on server, Out on client), and they can just be dummies at first, who does whole file in RAM sillyness. But as time permits, down the road, you can make them have smarts and manage intelligently.

I absolutely agree. I was recently looking for cheap webhosts that wouldn't squirm about me throwing up a 60 minute video. The deciding factor? Most webservers cache entire files when they send them out, and thus are configured to only allow 10MB files. Some webservers are set up to only cache a megabyte or two at a time, which works much better for a huge HD 400MB vid. Even better yet, a few can read the metadata in mp4 files to allow seeking.

Never assume that a session or transaction will finish instantly. Actually, assume that it'll take an hour, just to be safe.

I'm inclined to throw all the files on an FTP server just to keep the repository as light as possible. That would also allow the repository to remain up while the FTP server is overloaded, etc.

 

[dflemstr]

- you're not actually putting .pnd files into a database are you? (That sort of thing always makes my teeth itch; maybe I'm old school, but I prefer to keep relational data in a relational DB, and big giant zip files out of it, with references from the DB to the files (ie: name, whatever.) No reason to keep the entire repository as one big file (which introduces problems of its own, when trying to move the repo via flash drive, or onto a hard drive with a file system with 2GB file size limits, etc and so on. It also means ytou have one big file which is hard to work with, instead of many small files more easily split across things.) And most importantly, 1) DBs can go corrupted, and 2) maybe someone wants to keep the repo, and then later ditch the software for another repo software, and still have all the pnd files. Its never wise to suck agnostic data into a specific apps domain.

I most certainly am
I must say that I've probably gotten used to high-performance environments where extreme speed is the absolute goal, and thus haven't thought about the fact that people might want to run the app on less capable servers. You must know, however, that it most of the time is much easier to split up a database in multiple "files" or make backups of it etc, than it is to do so with a file system. Sharding is easy peasy and can be done across hosts transparently, and database backups, snapshots etc can be done in milliseconds with MySQL, for instance.

Of course, I never intended the database storage option to be the only one, especially after people wanted to grant access to file archive PNDs transparently in the PND Manager, so I'm currently loosening up the core a bit and allowing for PND files to be stored *anywhere*. You'll see what this means when I'm done

- if you're talking about pnd downloads to a pandora/some-device, then streaming is the way to go; it would be insanity to pull the files into memory while serving them out. *madness* I say, madness! You'll be wanting to pull in reasonably sized chunks .. madness to keep multiple MB/GB in RAM while sending down to someones slow-ass dial modem for hours on end. Make a Stream object pair (In on server, Out on client), and they can just be dummies at first, who does whole file in RAM sillyness. But as time permits, down the road, you can make them have smarts and manage intelligently.

My reason for doing it this way originally was that I thought that many people would want to download the same file simultaneously (and that this would be the rule, not the exception; there will be some really popular apps for the Pandora and some small, unpopular ones that'll be downloaded only once in a while) so it would make sense to share memory between these downloads. But with the new storage system, this will change, of course.

@ Kramy Yes, I guess that one of the first storage engines I'll make available is one with external links (FTP, HTTP, HTTPS) so that files can be uploaded anywhere, even on the Pandora File Archive ( ) or an FTP server somewhere. This will mean that I cannot depend on files being there any longer after the URL has been specified once (since the file might be deleted on the external server) but I'll put up some kind of system to handle this gracefully; the only option will probably be to just remove the PND file metadata entry from the PND Manager entirely.

Oh, and since we're using *ISO* with data appended *to the end of the file* here, an uploaded PND will have to be loaded completely into memory at least once, no matter what we do.

 

[polyroy]

Oh, and since we're using *ISO* with data appended *to the end of the file* here, an uploaded PND will have to be loaded completely into memory at least once, no matter what we do.

You still don't need to do that. Just have the chunks in memory you are actually using. Mostly, this will be just the end (PXML) the directory tables and some random midsections which happen to contain data you need. ISO is designed to be accessed this way, you don't suppose a DVD drive reads the complete DVD before it starts to play?

 

[urjaman]

Oh, and since we're using *ISO* with data appended *to the end of the file* here, an uploaded PND will have to be loaded completely into memory at least once, no matter what we do.

Ugh, oh no, just stream the file to disk, then fopen() it and then fseek() to the end of it, etc?

This your database stuff sounds way too limited now.
When multiple downloads to the same file occur, linux disk cache will automatically take care of not reading it from the disk many times, if ram permits.

 

[dflemstr]

Oh, and since we're using *ISO* with data appended *to the end of the file* here, an uploaded PND will have to be loaded completely into memory at least once, no matter what we do.

Ugh, oh no, just stream the file to disk, then fopen() it and then fseek() to the end of it, etc?

This your database stuff sounds way too limited now.
When multiple downloads to the same file occur, linux disk cache will automatically take care of not reading it from the disk many times, if ram permits.

Temporary files, eh? Might be a solution, but isn't really preferred on a server. I could read directly from a stream (so that I read the meta data *directly* from the FTP server or wherever the file resides) which would be a lot faster if this is something I should focus on. This is pretty low priority ATM though (premature optimization is the root of all evil and all that)

 

[urjaman]

Temporary files, eh? Might be a solution, but isn't really preferred on a server.

My purpose was to say that if you stored the .pnd's on disk as files instead of in the db, you could just read and seek in them arbitarily.

 

[dflemstr]

Temporary files, eh? Might be a solution, but isn't really preferred on a server.

My purpose was to say that if you stored the .pnd's on disk as files instead of in the db, you could just read and seek in them arbitarily.

Indeed I could, but I want to make things more abstract than that as stated above, so I can't rely on actual files being present, but must instead prepare for just streams being present.

Because my current plan is to have multiple "storage back ends", so that when creating a new package, you'll be able to specify a URL where the PND file is, and if the user uses that option instead of uploading a PND, I'll never even store the PND file locally at all (Except when parsing the meta data), just use the URL instead.

Or should I really just change the system to use the file system instead of the database for the files, and skip all the rest? I'm really at a loss here, don't know which features people really want

 

[urjaman]

Or should I really just change the system to use the file system instead of the database for the files, and skip all the rest? I'm really at a loss here, don't know which features people really want

I'm not sure if i want anything, i'm not really into running the server so it doesnt matter that much for me.
But anyways, what i would do is to have a system that can identify a file as being anywhere, but with this logic:
1. We need pnd X
2. If pnd X is remote, download pnd X to a temporary file
3. Access it as a file in the FS (temporary or not)
4. The system could cache the downloaded files, if wanted/possible/whatever. Or just get rid of them.

 

[dflemstr]

Or should I really just change the system to use the file system instead of the database for the files, and skip all the rest? I'm really at a loss here, don't know which features people really want

I'm not sure if i want anything, i'm not really into running the server so it doesnt matter that much for me.
But anyways, what i would do is to have a system that can identify a file as being anywhere, but with this logic:
1. We need pnd X
2. If pnd X is remote, download pnd X to a temporary file
3. Access it as a file in the FS (temporary or not)
4. The system could cache the downloaded files, if wanted/possible/whatever. Or just get rid of them.

This means local storage for all files. Which in turn means that it doesn't make sense to have different storage locations; just having local (file system based) storage should be enough.
I'll see what I can do; having a local storage system would also have the advantage that you can upload files from other sources than the HTML front end (for example, you could have a local Pandora application from which you could upload packages too, not just download them, or you could auto import from the File Archive)

 

[skeezix]

Learn to lvoe the disk cache and memory cache; dont' fight or try to reimplement them, you'll lose

Keep pnd files on disk, and fetch bits as you need; the OS will optimize that stuff

jeff

 

[dflemstr]

Learn to lvoe the disk cache and memory cache; dont' fight or try to reimplement them, you'll lose

Keep pnd files on disk, and fetch bits as you need; the OS will optimize that stuff

jeff

From experience, I can tell you that Windows servers don't, but then again, who in this community would ever host anything on a Windows server?

When I started working on this project, I had one goal to make the app completely platform-independent; this would include allowing the app to run on a server with read-only hard drives. I'm now scrapping that part, but maybe it's worth it.

 

[skeezix]

I would say that 'no one' running a file repo has a disk space issue. (At worst they may have partition issues, so may have to split files across multiple non-unioned filesystems.)

It is of course possible someone might be running on a free public host with no disk beyond basic stuff, and a local DB of the 'right types', but I think that is the edge case, not the norm.

In practice, you rather only want a couple, and you can basicly assume ED will give you hosting space. So strikes me you might be optimizing against the wrong case, but at the same time, trying to support a pure approach is always cool, since it onyl opens up other options.

I'm just really against sticking thousands of poswsibly large files into a DB; it just leads to misery Even an in-DB should be able to get you 'chunks' from middle of filesw I'd think (even avoiding 'substring' hacks), but keepign themout of the DB lets you do all the usual stream approaches; heck, you could just use http file transfers, scp's, etc, as well suddenly.

Anyway, sorry, I'm not really following your thread too much so I'm not really on top of it to make recommendations; these are just 'off the cuff' remarks, so feel free to ignore

jeff

 

[White Demon]

I haven't read this whole thread (I did read the first 2-3 pages), so I may be going totally off track here, but I thought I'd give my 2 cents about this.

It appears that the purpose of this project is to totally replace the file archive and have all the files/packages/whatever you want to call them stored in one single place and access them trough a single application on the Pandora. While I agree that storing everything in a single place is a good idea, only being able to access them only through a special application is not.

As I said in a short post on the first page of this thread, my home internet is NOT wifi, and since there is no ethernet plug in the Pandora I will not be able to get my P to connect to the net. Therefore, if the only way to access the repository is through the use of an app on the P, then not only will the repository be useless to me, but it will also render my Pandora useless, as I will not be able to download any apps for it.

There has also been talk of the app being able to run on PC, but all the talk has been linux based. If you want to allow people to also access the repository through a PC, then at least a Windoze version of the app will have to be compiled and released too for those people who buy a P that want to access the repository from their PC and are running Windoze. (maybe a Mac version of the app too?)

I do really think though that the repository should be able to be accessed through a standard web browser in addition to whatever app is decided, so that anyone can access the archive, no matter where they are, and from any computer, be it Linux/Windoze/Mac/Amiga/whatever. What about people who access it from work or a net cafe or some other place where they are not able to install an app on the computer they are using?

This new repository should be able to be accessed not only from the P, but from any internet able computer running any OS. If this is not the case, then I don't see any point in replacing the file archive already in place here.

 

[polyroy]

I haven't read this whole thread (I did read the first 2-3 pages), so I may be going totally off track here, but I thought I'd give my 2 cents about this.

[...]

This new repository should be able to be accessed not only from the P, but from any internet able computer running any OS. If this is not the case, then I don't see any point in replacing the file archive already in place here.

If you'd have demoed the app (first post) you'd know that an universal web frontend is the primary interface. But since a repository alone is less useful than a full blown package manager (requiring it to be run from the pandora itself), a localized app integrated with the Pandora would be very useful. Even though this thread is not about discussing said app, it is about providing the ability for such an app to access the repository easily.

 

[Nation.A.List]

my home internet is NOT wifi, and since there is no ethernet plug in the Pandora I will not be able to get my P to connect to the net.

USB to ethernet adaptor ???

 

[dflemstr]

I haven't read this whole thread (I did read the first 2-3 pages), so I may be going totally off track here, but I thought I'd give my 2 cents about this.

[...]

This new repository should be able to be accessed not only from the P, but from any internet able computer running any OS. If this is not the case, then I don't see any point in replacing the file archive already in place here.

If you'd have demoed the app (first post) you'd know that an universal web frontend is the primary interface. But since a repository alone is less useful than a full blown package manager (requiring it to be run from the pandora itself), a localized app integrated with the Pandora would be very useful. Even though this thread is not about discussing said app, it is about providing the ability for such an app to access the repository easily.

What he means is the following (in case some of you don't get it):

1. The app will be accessible by any standard browser, just like the file archive. It is a WEB application.
2. The app is not something you have running on your own computer; we yet have to create something like that.
3. We very much care about platform abstraction here, so the app will really be usable from anywhere (heck, I can even write a WAP frontend if you want!)

So, the PND Manager can already be used in every situation that the File Archive could have been used; we're just trying to make it even more accessible.

(Or not ATM; have had much to do lately so I haven't been working on it)

 

[White Demon]

@Polyroy: I remember dflemstr saying something in the original first post about compiling something to run under linux. Since I only have Windoze installed, I couldn't demo it. Hence what I wrote in my post above about concerns over how it could be accessed.

@dflemstr: That's all I need to know, I'm all aboard now. Thanks!

 

[polyroy]

@polyroy: I remember dflemstr saying something in the original first post about compiling something to run under linux. Since I only have Windoze installed, I couldn't demo it. Hence what I wrote in my post above about concerns over how it could be accessed.

Oh, well, that is if you are actually planning on running your own repository, dflemstr is running a demo server with the app himself, which you can access using just about any browser I believe.

Edit: the demo server, as mentioned elsewhere in this topic, appears to be offline...

Also, there is still the question of a name change for this project, anyone willing to debate this issue now that the big discussions are out of the way?

 

[dflemstr]

@polyroy: I remember dflemstr saying something in the original first post about compiling something to run under linux. Since I only have Windoze installed, I couldn't demo it. Hence what I wrote in my post above about concerns over how it could be accessed.

Oh, well, that is if you are actually planning on running your own repository, dflemstr is running a demo server with the app himself, which you can access using just about any browser I believe.

Edit: the demo server, as mentioned elsewhere in this topic, appears to be offline...

Also, there is still the question of a name change for this project, anyone willing to debate this issue now that the big discussions are out of the way?

The app can be compiled and run on Windows, Mac OS, Solaris, Linux, BeOS/Haiku, FreeBSD and some other smaller platforms. Oh, and if you have compiled the app on one platform it works on all other platforms without modification I have taken great care to make that possible; there's no platform-specific code anywhere in the project.

I have a forum running for the SpaceRTS project, and it's on the only server I own so I had to take the forum down while hosting the demo site (Which some of the forum members didn't like at all). Now the forum is up again, so I'll either have to find some other hosting space for the demo app or figure out how the Apache "mod_proxy" needs to be configured to be able to run the sites in tandem (Because Apache Web Server and Apache Tomcat don't play well together). Oh well, this isn't really hugely important ATM; I need to find time to do everything I've promised to do (the first post in this thread has a todo list that I actively update) before exposing the app to the public in order to gain even more constructive criticism

If you read the todo list, it also contains the name change issue, so please come with suggestions!