The Pnd Package Manager

[dflemstr]

About the ISO format for PND's: There are some libraries (one being libMirage) which can query (read files from) ISO files without the expense of OS-dependent mounting, so it should be possible to put the pxml INSIDE the ISO file to circumvent at least the issues of format ambiguity

Sorry, I'm in a server environment here, remember? This means:

1. No native dependencies
2. Everything has to be runnable in parallel (aka thread-safely) without creating the need to spawn multiple processes
3. Ideally, I shouldn't have to give over control to another library that might mutate my data; in some cases, my data needs to be immutable since I try to reduce the number of memcopies wherever possible and don't create unnecessary copies of stuff.

EDIT: I'll probably just write a custom loader, no probs

 

[polyroy]

Seems nice, one of the things I didn't comprehend the first time too But that won't be useful without having fixed locations of executables since with package-dependent mount names we'd get nowhere. Say, could we add "provides" to pxml in the way that when such a PND (say vlc) is mounted, it will create a symlink from (among others) /usr/bin/vlc to the appropriate files inside the mount? Then a PND would really have the capability to "extend" the system's firmware too.

Thinking again, this would cause frequent writes to the NAND Flash, every time we'd insert or remove an SD card for however many files the present PND's would want to make. To prevent this we'd have to consider RAM filesystems, does anyone know how Live CD's do this (mostly permanent from the medium with modifications in memory?) Perhaps we could make some of the writes persistent, and add those we get from PND's via special syscalls as temporary files (the symlinks). You'd most probably need to rewrite the FS driver too...

EDIT: I'll probably just write a custom loader, no probs

You have very specific needs, I was about to suggest that

 

[dflemstr]

Thinking again, this would cause frequent writes to the NAND Flash, every time we'd insert or remove an SD card for however many files the present PND's would want to make. To prevent this we'd have to consider RAM filesystems, does anyone know how Live CD's do this (mostly permanent from the medium with modifications in memory?) Perhaps we could make some of the writes persistent, and add those we get from PND's via special syscalls as temporary files (the symlinks). You'd most probably need to rewrite the FS driver too...

Oh right, forgot to respond to this. I thought that this was already possible with PNDs? The whole overlays thing should be working already; look at the source code to see how it's done.
*cough*UnionFS*cough*

BTW, anyone aware of a ISO9660 specification that actually is a bit more down-to-the-metal and usable? (I only find specs that use up 20 pages explaining what a freaking byte is, just to make sure that all of the library implementors are on the same page......

 

[polyroy]

*cough*UnionFS*cough*

Oh, right. For some reason I missed that. Thanks, ignore the previous post then

BTW, anyone aware of a ISO9660 specification that actually is a bit more down-to-the-metal and usable? (I only find specs that use up 20 pages explaining what a freaking byte is, just to make sure that all of the library implementors are on the same page......

I laughed at that too Well, this page (allthough the title is pretty off) summarizes things nicely I guess. Also, you might want to take the libMirage code as a reference, if this doesn't conflict with your intentions of using the shortest possible way of writing the code perhaps

Edit: don't forget Joliet (allthough I hope that the patents stay cool here, as opposed to what happened to FAT32.) And forgive them their use of Comic Sans MS

 

[skeezix]

About the ISO format for PND's: There are some libraries (one being libMirage) which can query (read files from) ISO files without the expense of OS-dependent mounting, so it should be possible to put the pxml INSIDE the ISO file to circumvent at least the issues of format ambiguity

To be precise, we expect the PXML.xml file to be both inside the iso and appended.

ie: A typical build dir will include the PXML; the pnd-build.sh (or whatever) will make the .pnd file by iso'ing up the dir, and then appending the PXML.xml (found within the dir.) If the iso is then unpacked by someone, the appended PXML is lost.. but is still found in the resulting dir.

Thus the PXML is found within the iso, but is _handily_ appended, to make it faster to locate.

(If the PXML is not appended, the reverse scan may eventually find it, though it could be slow. I have at various times opted to limit how far back the treverse scan will go, or let it go all the way. I don't recall what I've got in the code right now, but it sure sounds like something I should make as a param in the API doesn't it? Anyway..)

jeff

 

[skeezix]

Some good ideas here, but let us remember -- it is better to start small and go successfull, then start over complex and trip

So I'm (like I can speak for anyone else not going to define PGP signatures or the like right now, and leave a lot of things open. We can nail it down later.

(The code is full of enough holes, as is Offer still open.. if anyone has time to help with testing or filling in holes, let me know

jeff

 

[dflemstr]

Just for reference: I need the ISO parser because I want to locate screenshots and the like in the ISO file, so that's the reason why I'm making one.

 

[axenzoku]

As long as .pnd packages were associated to open with a "package installer" app, then it would be like windows...
You download the ".pnd" installer, double click and it would install the application...
Now you could make the app save it's repository location, version info, and local installation directory to some kind of "universal install log" file that an auto-updater could check against for "modified file" type of updates.
Then just download, extract, install the new version with different methods like merge or complete replacement...
Just an idea...
Then someone would only have to run the auto updater once to check for newer versions, then download and install them...
Then turn off the updater.
Might end up being less work than this whole server, web interface, automatically taking screenshots and organizing the archives thing...
Without having multiple repositories dedicated to pandora apps, a simple auto-updater would be much easier, but it would have to be combined with a package installer so they could work together... Then you still get props for doing something awesome

 

[dflemstr]

As long as .pnd packages were associated to open with a "package installer" app, then it would be like windows...
You download the ".pnd" installer, double click and it would install the application...
Now you could make the app save it's repository location, version info, and local installation directory to some kind of "universal install log" file that an auto-updater could check against for "modified file" type of updates.
Then just download, extract, install the new version with different methods like merge or complete replacement...
Just an idea...
Then someone would only have to run the auto updater once to check for newer versions, then download and install them...
Then turn off the updater.
Might end up being less work than this whole server, web interface, automatically taking screenshots and organizing the archives thing...
Without having multiple repositories dedicated to pandora apps, a simple auto-updater would be much easier, but it would have to be combined with a package installer so they could work together... Then you still get props for doing something awesome

This is a nice summary of what we've been discussing on the last 4 or so pages
And remember: you never have to "install"/"uninstall" PND packages; just download them and it's done. So we don't have to have the headaches of implementing diff packages or stuff like that; we just download the new version of a package, delete the old one and that's all we have to do.

And yes, the web application is just a web interface for the repository; of course we'll be able to do stuff outside of the app. After all, that's what the API is there for.

 

[polyroy]

As long as .pnd packages were associated to open with a "package installer" app, then it would be like windows...
You download the ".pnd" installer, double click and it would install the application...
Now you could make the app save it's repository location, version info, and local installation directory to some kind of "universal install log" file that an auto-updater could check against for "modified file" type of updates.
Then just download, extract, install the new version with different methods like merge or complete replacement...
Just an idea...

There is more to it than what you describe. Typically, a PND which is somewhere on the system (SD card, NAND flash) will already act as if it is installed magically, no need to tell it to become installed at all. "clicking" it would actually run it's main function instead. A repository could also be as simple as an FTP archive, but we want something which has a nice humane interface, simmilar to the File Archive, which makes it more usable for the general public who don't want to use a Pandora to browse PND packages.

Then someone would only have to run the auto updater once to check for newer versions, then download and install them...
Then turn off the updater.
Might end up being less work than this whole server, web interface, automatically taking screenshots and organizing the archives thing...
Without having multiple repositories dedicated to pandora apps, a simple auto-updater would be much easier, but it would have to be combined with a package installer so they could work together... Then you still get props for doing something awesome

An auto updater feature is not going to be in the web-based system that is proposed currently, AFAIK. We'd need a local client for that which reads the versions of all PND's on your system. I agree that an updater function has a high priority, but that would be part of the local PND client app, to which currently no developer is assigned.

EDIT: dflemstr: you beat me to it. I should really be working...

 

[dflemstr]

So I'm currently working on creating an automated system that copies the whole PND repository out to a file system (for rsync, FTP and whatnot; I've not considered creating an *importer* because a repository should be write-once read-everywhere; we can't deal with randomw rites coming from all over the place)

...and was just thinking: The (kind of) current repository.xml file contains static absolute canonical URLs to various resources (PND files, screen shots, etc), aka in the style of "http://repo.com/packages/foo-1.2.3.4.pnd" which is suboptimal if you want to create copies of the repository, since all URLs then need to be rewritten to match the new repository location.

Now, I've done some more research and studied the RPM and APT package repository meta data formats, and have seen that they tend to use semi-relative paths to locate package files. This of course has some advantages and disadvantages. The primary advantage is that you can move the entire repository tree to another location, and you won't have to change any files within to still be able to use its contents. The disadvantage is of course that all of the repository related files have to stay inside of the repository tree in order to always be accessible, since there aren't any absolute URLs that guarantee a fixed location for the files.

For now, I've thought that the advantages outweigh the disadvantages, and have thus created the following directory structure (yes, I've changed a lot):

/meta/repository.xml (1)
/meta/packages/1.xml (2)
/pnds/foo_bar-1.2.3.4.pnd (3)
/icons/1.png (4) (oh and these are 128x128 pixels in size)
/smallicons/1.png (4) (these are 64x64)

(1) The repository descriptor. Contains a list of all packages, including a "details"-URL for each package that point to the meta files described below
(2) Meta data about a package. Currently, the magic file name in this case is the database ID of a package (I was lazy), and it shouldn't really matter what it is, since you simply are able to find a package in the repository.xml file and use the URL within to locate the meta file. So the package meta files' names and locations are moot; I just put them there for convenience (and again, since we're using semi-relative URLs, they should be close at hand in case we want to move the repository). If someone creates an alternative implementation of this, they'll be able to store the files in whichever location they choose as long as they keep the repository.xml file in the same location.
(3) These are the actual PND files, conveniently named by name. Their name/location doesn't really matter either, since, as above, they are referenced from the master repository file and the package meta data file, but I gave them good names because they thus don't create naming conflicts when downloaded.
(4) The icons belonging to a package, and again referenced by DB id, also since their URL is in the meta data.

This system would then work as follows:

1. The actual repository location that an user would enter into some package installer would be the canonical path to the repository root dir. Examples:
   • Repofile loc: http://pndmanager.org/meta/repository.xml ; What the user enters as the repository path: http://pndmanager.org/
   • Repofile: ftp://192.168.2.7/packages/pnd/meta/repository.xml; path: ftp://192.168.2.7/packages/pnd/
2. The various metadata files reference other files by using the repository location as their "root directory" (à la APT/dpkg). So in a package metadata file, a pnd file URL would be "/pnds/mypackage-1.2.3.4.pnd".
3. The whole thing is portable and can be accessed over any protocol, including http, ssh, ftp, whatever

Anything that I've missed or something I should change?

Oh and in case anyone looks at the repository file and notices lots of '<icon href="..."></icon>' instead of the usual '<icon href="..."/>'; this is because expanded tags are easier to read by many XML parsers, and since this is the default Scala XML mode anyways, I stuck with it.

 

[polyroy]

Anything that I've missed or something I should change?

I believe this will work great the way it is setup, just a few (minor) remarks:

Your choice of using those magic database ID's as identifiers will make merging somewhat difficult. If, for these magic ID's, you also use the full package name as per (3), the operation of rewriting the repository.xml after a merge will become the only required operation and will avoid mass renaming. This could also prevent forcing your "import" operation to overwriting a possibly existing database.

Is there any particular reason you do not extract screenshots from the PND's? I know icons would be especially useful for appstore-like applications, but for exploring purposes, wouldn't screenshots also be quite useful? Or will you ask us to query for screenshots via the repository API only?

Also, just a few thoughts here, but have you thought about how large PND's will be handled? Are you going to store them all in the same backend, or is it possible to introduce some external referencing for distributed content provision. Large popular stuff such as Quake 3 and the corresponding map packs could be candidates for remote fileservers. (Edit: references containing protocol and host portions could then indicate external resources outside the "root" of the repository, as a means of exceptions for uncommon unreasonable content) I remember you saying it'd have to be inside memory for a brief period. Just in case someone uploads something which does not fit in memory, i.e. on a limited memory repository: What exactly is the maximum upload limit (in relation to available memory), especially when you have multiple simultaneous uploaders?

And again, thanks for your much appreciated work.

 

[dflemstr]

Your choice of using those magic database ID's as identifiers will make merging somewhat difficult. If, for these magic ID's, you also use the full package name as per (3), the operation of rewriting the repository.xml after a merge will become the only required operation and will avoid mass renaming. This could also prevent forcing your "import" operation to overwriting a possibly existing database.

Yes - it's only laziness on my part that has prevented me from just naming the files correctly. It will be done

Is there any particular reason you do not extract screenshots from the PND's? I know icons would be especially useful for appstore-like applications, but for exploring purposes, wouldn't screenshots also be quite useful? Or will you ask us to query for screenshots via the repository API only?

I haven't found a good way to extract them yet. I created numerous GIT-branches with various implementations of ISO-readers, but none of them were abstract enough that they could handle ISO file tables, Joliet and RockRidge in tandem reliably (And I've grown to hate the "double-endianness" 32-bit integers). It's quite complicated to write a FS-driver that only is allowed to operate in RAM. If there only was some reference implementation I could take a look at; even something in C, Haskell or even Erlang would suffice, just ANYTHING! ISO isn't really a file format that's meant to stay in RAM, so no one has bothered making readers for it.

What I'm basically looking for is a simple function that takes an ISO byte array + a file name and spits out an array containing the files contents. Quite simple, right? Well, it's not; i at least haven't had any luck so far.

Also, just a few thoughts here, but have you thought about how large PND's will be handled? Are you going to store them all in the same backend, or is it possible to introduce some external referencing for distributed content provision. Large popular stuff such as Quake 3 and the corresponding map packs could be candidates for remote fileservers. (Edit: references containing protocol and host portions could then indicate external resources outside the "root" of the repository, as a means of exceptions for uncommon unreasonable content) I remember you saying it'd have to be inside memory for a brief period. Just in case someone uploads something which does not fit in memory, i.e. on a limited memory repository: What exactly is the maximum upload limit (in relation to available memory), especially when you have multiple simultaneous uploaders?

There is no memory limit
First of all - a PND package, when loaded into memory, will only take the size of the actual PND file + about 64 bytes of memory.
Then - The default file server is set up to handle 8 MiB multipart POST-requests I believe (meaning that the max PND upload size is 7.9 MiB), but that is easily changed, so people can upload packages of any sizes really (as long as the server has the bandwidth). The thing is that since this is a Java WebApp (or it really isn't but I make it look like it is to the server) it can be molded to whatever you want it to be. If you run the app on a Pentium 4 server with 128 MiB of ram, then it will run single-threaded, have very aggressive garbage collection, and utilize the database pool very efficiently (I've set the thread pool scheduler to only allow 4 or so simultaneous database accesses, which isn't a bad thing; you get more even loads and maybe 3 ms extra latency as seen by the client). Or if you run it on a Cray machine with 128 cores and 64 GiB of RAM, it'll spawn its usual 500 threads and cluster requests very efficiently. Thus it's really hard to tell how well it will perform on a server before knowing exactly how much oompf the server has. On the machine I'm at at the moment (4GB RAM, Intel Q6600 quad CPU), the app will allocate 300 MB of RAM, of which it uses ~40 MB for the actual app (= about 40 threads if there are enough clients) and leaves the rest available for cache/files/whatever. It'll allocate even more if it needs it. I haven't tested uploading files larger than 10 MB (because seriously, is there a game out there that's larger? OpenArena and Quake etc might be the exceptions, but they are still max 100 MiB which SHOULDn't be a problem unless there are 20 different versions that are downloaded simultaneously)

About that keeping in memory stuff - a PND file needs to stay in memory while it's being downloaded to the user, or being uploaded from the user, and there's no avoiding this. Even if you use external storage, this is something that must be (unless you use disk streaming, but then you risk data integrity, and you need to write very efficient down-to-the-metal libraries to do this for you). Note also that if two users download the same file, no extra RAM is needed; they share the same RAM data (unless I've forgotten some setting somewhere). Again, you can cluster the app (put up multiple machines running it that all access the same DB) if you have many machines with very little RAM, and you can even shard the database if you know your stuff, but that shouldn't be necessary.

Oh, and the relative URLs are still valid URLs; you can easily swap any of them for a "ftp://" link or something; the local Pandora installer program must of course be prepared to read multiple protocols but this is handled transparently by for example Qt; what I'm saying is that there wouldn't be any problems with manually changing meta data files. Including this in the main application, however (aka using an external storage unit for the PND Manager while still using the default implementation for everything else), wouldn't really change anything from a performance POV, since the files would have to be loaded into memory to extract meta data anyways, and also loaded into memory whenever they are downloaded from the external data source (either on the data source's server, or on the application's server)

EDIT: I think that I should rebuild the PND Manager a little; currently I've made all of its components reusable and very abstract, which could have some impact on performance (for instance, the whole repository interface is based on a "generic entry digest system" that could be used elsewhere in other projects, but really doesn't to anything but complicate things for the PND Manager). If I made it more integrated, I might gain some speed. (An OOP purist would kill me if he saw this)

 

[polyroy]

What I'm basically looking for is a simple function that takes an ISO byte array + a file name and spits out an array containing the files contents. Quite simple, right? Well, it's not; i at least haven't had any luck so far.

You could check out do_isofs_readdir and others from the Linux kernel source, and convert it to something which doesn't do any disk access and kernel structures. I know, it's a pain. Seems to handle both Joliet and Rock Ridge extensions properly though.

(because seriously, is there a game out there that's larger? OpenArena and Quake etc might be the exceptions, but they are still max 100 MiB which SHOULDn't be a problem unless there are 20 different versions that are downloaded simultaneously)

Well, I could see people wrapping stuff in PND's which they'd otherwise not do, such as a 2.7GB PSX game called Final Fantasy IX (ugh!) but otherwise, I guess for all public use PND size would be somewhat limited. It is good to consider future storage requirements however, and of course hosts for your application which are not that generous with memory, especially if someone of the community has to pay for that!

About that keeping in memory stuff - a PND file needs to stay in memory while it's being downloaded to the user, or being uploaded from the user, and there's no avoiding this.

What about fetching only a section (say 1MB each time) of the file which is currently downloaded from the db, and advancing to the next section whenever needed? This would help since downloads are often slow, and you only keep data local which you really need at that time. Since you are doing strictly sequential access, the memory issue will be negated for downloads at least. To add in HTTP resume support (random access from the client in PND's) would make it easy for DOS-attacks however, but that is already a huge problem.

Perhaps an upload could be done in a simmilar way, the moment you have received a certain amount, it is appended to the db blob until it has been received completely. After upload, you could access the PND "array" through a cache which fetches the pieces from the DB as they are read. You won't need the complete ISO then, just a few select parts.

 

[dflemstr]

You could check out do_isofs_readdir and others from the Linux kernel source, and convert it to something which doesn't do any disk access and kernel structures. I know, it's a pain. Seems to handle both Joliet and Rock Ridge extensions properly though.

Looked it through already; the code depends too much on all of the "struct file" and "struct inode" APIs for my taste (the kernel is very integrated and uses lots of other APIs to do the job for it in the ISO driver). If I'm left without another choice, however, I'll have to look more closely at it. I don't really have too much time to spend on this, however.

Well, I could see people wrapping stuff in PND's which they'd otherwise not do, such as a 2.7GB PSX game called Final Fantasy IX (ugh!) but otherwise, I guess for all public use PND size would be somewhat limited. It is good to consider future storage requirements however, and of course hosts for your application which are not that generous with memory, especially if someone of the community has to pay for that!

Well, if people wrap stuff like that in PNDs, we can't accept it. It's that simple. Because no matter which storage system you chose to use (even vsftpd has to store the file in memory while transmitting; if you use other systems you're just moving the memory needs to another server). But really, is there any PC out there today that doesn't have at least 2GB of memory? I mean come on. Especially with the current RAM prices, it's often easier to just install more RAM than it is to create some separate storage system for the PND Manager (but I guess that I should start working on it nonetheless)

What about fetching only a section (say 1MB each time) of the file which is currently downloaded from the db, and advancing to the next section whenever needed? This would help since downloads are often slow, and you only keep data local which you really need at that time. Since you are doing strictly sequential access, the memory issue will be negated for downloads at least. To add in HTTP resume support (random access from the client in PND's) would make it easy for DOS-attacks however, but that is already a huge problem.

Perhaps an upload could be done in a simmilar way, the moment you have received a certain amount, it is appended to the db blob until it has been received completely. After upload, you could access the PND "array" through a cache which fetches the pieces from the DB as they are read. You won't need the complete ISO then, just a few select parts.

DB queries are atomic; either you get the whole data blob or you get nothing. That's the beauty of relational databases: all entries are atomic. I could of course create a fragment system that would require sorting algorithms and linking algorithms to function properly etc, but it's just not worth it. On a server, RAM is the most abundant resource (if it isn't, you're doing it wrong ) so if I reduce RAM usage and increase CPU usage instead, it'll make the manager run on fewer platforms, really.

Oh, I just don't know what's best; maybe I should just create the option and let the end-hoster decide what's best. Tomorrow, I'll try to abstract the whole file management system so that it can be handled by anything. It needs doing, anyways; my system has enough coupling as it is.

 

[Kramy]

I mean come on. Especially with the current RAM prices, it's often easier to just install more RAM than it is to create some separate storage system for the PND Manager (but I guess that I should start working on it nonetheless)

How about redirecting to an external web address? Or does that defeat the purpose of a repository?

I'm just thinking... Commercial games(and there may be a few) will want the benefits of a unified update checker, while still maintaining control on distributing their stuff.

If you don't host the content, does it really matter what's inside it?

 

[dflemstr]

I mean come on. Especially with the current RAM prices, it's often easier to just install more RAM than it is to create some separate storage system for the PND Manager (but I guess that I should start working on it nonetheless)

How about redirecting to an external web address? Or does that defeat the purpose of a repository?

I'm just thinking... Commercial games(and there may be a few) will want the benefits of a unified update checker, while still maintaining control on distributing their stuff.

If you don't host the content, does it really matter what's inside it?

The way I will make the external storage system available will probably be by making a trait (think: interface) that looks like this:

trait PNDFileStorage {
  def saveFile(pndName: String, pndVer: (Int, Int, Int, Int), data: Seq[Byte]): Unit  //alternative implementations will exist for this
  def getURLFor(pndName: String, pndVer: (Int, Int, Int, Int)): Option[String]
}

...so it will be possible for the data to be stored absolutely anywhere on the planet. This would make it possible for commercial apps to be distributed, but it still wouldn't be possible for them to have some kind of payment system. Oh, and I don't know yet if it would be a good idea to allow for different storage systems in parallel (so that some packages would be in the manager itself and some on an external FTP server) but it would be possible to implement at least.

 

[may88]

Voted Yes as there was not a "probaly" option. Looking good. Had a bit of a read but this Topic is a bit too verbose and detailed for the average Joe.

Dflemstr, If you're as prolific with your coding as you are with your postings then SpaceRTS'll be ready and waiting for me to download via PND Package Manager (aka something else by then). I promise to only use this to download it.

Me, I'm a leach. Happy to encourage those of you with skills beyond my own with the odd comment now and again. I'll be back to study (PND.P.M) as soon as I have something to donwload and something to download on to. Say in a couple of months time.

 

[polyroy]

Warning, this post has some opinions flying about

Well, if people wrap stuff like that in PNDs, we can't accept it. It's that simple. Because no matter which storage system you chose to use (even vsftpd has to store the file in memory while transmitting; if you use other systems you're just moving the memory needs to another server). But really, is there any PC out there today that doesn't have at least 2GB of memory? I mean come on. Especially with the current RAM prices, it's often easier to just install more RAM than it is to create some separate storage system for the PND Manager (but I guess that I should start working on it nonetheless)

To be fair, most virtual hosts you order are from places where resources such as memory can be quite expensive. The less memory they give users, the more they can share a server. Not everyone can run their own hardware on a fast pipe. You'd need some premium (colocation or some kind of sponsored pipe) for that here. All resources are essentially limited, be it disk space, memory, or the number of times an application logs to output. It is about finding the right balance. The less an application requires (and the more can be configured), the more of it you can run and the less it costs.

If possible, we might even get this hosted next to an existing app (shielded, of course) on someones host, and it could cost us nothing if we don't intrude too much on their resources. Not having to purchase an additional server alone could be a good reason

There are plenty of storage systems which can cope just fine with huge files. I'm sure any decent FTP server will allow transfering of files larger than main memory allows, it can't be that silly. I do in fact transmit files in the TB range and never encounter any issues (except in some isolated instances, but those were mostly configuration related) Keeping complete files in memory, especially where they can grow big, is not at all common.

DB queries are atomic; either you get the whole data blob or you get nothing. That's the beauty of relational databases: all entries are atomic. I could of course create a fragment system that would require sorting algorithms and linking algorithms to function properly etc, but it's just not worth it. On a server, RAM is the most abundant resource (if it isn't, you're doing it wrong ) so if I reduce RAM usage and increase CPU usage instead, it'll make the manager run on fewer platforms, really.

I agree with you that the principles of atomicity are great, and I do understand relational databases, but I always try to consider what atomicity means. In any case where a portion of the data can be desired in favour of the complete dataset, and this has an impact on some relevant factors, the data should not be considered truly atomic. If your database can't deal with nonatomic data properly, or you can't find a way to make the considered data truly atomic, it is better to store it in a way which those advantages are offered.

It makes sense to always approach things from their efficiency standpoint to make things scale properly. Of course it is easier to just treat a file as if it is atomic, but it isn't the best way if this causes other issues (IMHO).

I really don't think that there should be a reason for PND's to be restricted in filesize other than storage concerns. If there are 1TB disks we can attach to a pandora, why not allow PND's to become that size if someone thinks that that would in fact be exactly what they want?

But this is becoming somewhat of a rant, so you are free to ignore if you wish

Edit: last line

 

[skeezix]

Not sure if I parsed that right but..

- you're not actually putting .pnd files into a database are you? (That sort of thing always makes my teeth itch; maybe I'm old school, but I prefer to keep relational data in a relational DB, and big giant zip files out of it, with references from the DB to the files (ie: name, whatever.) No reason to keep the entire repository as one big file (which introduces problems of its own, when trying to move the repo via flash drive, or onto a hard drive with a file system with 2GB file size limits, etc and so on. It also means ytou have one big file which is hard to work with, instead of many small files more easily split across things.) And most importantly, 1) DBs can go corrupted, and 2) maybe someone wants to keep the repo, and then later ditch the software for another repo software, and still have all the pnd files. Its never wise to suck agnostic data into a specific apps domain.

- if you're talking about pnd downloads to a pandora/some-device, then streaming is the way to go; it would be insanity to pull the files into memory while serving them out. *madness* I say, madness! You'll be wanting to pull in reasonably sized chunks .. madness to keep multiple MB/GB in RAM while sending down to someones slow-ass dial modem for hours on end. Make a Stream object pair (In on server, Out on client), and they can just be dummies at first, who does whole file in RAM sillyness. But as time permits, down the road, you can make them have smarts and manage intelligently.

jeff