dflemstr said:
I am not responsible for package management on the Pandora; I am not an official developer. I just threw the idea out there a month ago that we'd need a central way to manage PND files - the only package format supported by the Pandora - and some people, including some devs I recall, agreed that I should at least try. This is nothing official, and therefore I ask for your opinions and not "You are doomed to get this, what do you feel about that, eh?". I am very happy that you can come with constructive input; of course the PND manager could be rebuilt to use a file system for actual PND file storage; when I built the database system, I made sure that this was possible. You would lose a lot of performance, however, unless you want to stream packages directly from disk, which exposes you to file integrity issues.
And about IPKG: it won't be possible to use, since it requires you to have persistent storage on the Pandora. Since Pandora lacks internal, always-present storage (beyond the 512 megs on the fragile NAND), it won't be possible to use a management system that expects files to stay where they are and always exist until removed. We need a system that can handle the constant swapping of SD cards.
Also, they (aka the Pandora devs) wanted to remove the "installation" step of packages, and have something that never needed to be installed. As you might or might not know, PND packages aren't even extracted before being used, they are used as-is. This wouldn't be possible with IPKG packages that have various install scripts attached to them.
I do think that core updates to the core Ångström OS (such as firmware, bootloader and kernel updates) will happen through IPKG, but that all other packages should be installed through PNDs. And remember: it's probably possible to write scripts that automatically convert IPKGs into PNDs, so this won't be an issue (just simulate a Pandora env, run the install scripts and preserve the output tree, wrap it in a PND and done).
I did not realize PND files would be operated this way. Your choices seem to make a lot more sense now. There are two kinds of packages; those "portable" PND packages, and those internal continuously available to the OS. You should not confuse the two, as they cannot really be used interchangeably. I sure hope there is a way to make some PND packages integrate with the OS, in the way that they create the appropriate binaries and shared files in the proper locations so that e.g. a music player provided through a PND is always available through the classical UNIX mechanisms (mpd, music playing daemon, for example).
Your last statement makes me irk; if you were to wrap part of /usr/bin and part of /etc/ in PND's, you'd have to search all PND's on the system for providing those files. So I don't think this is viable. It only works with simple apps which are not integrated in the system at all. For all other purposes, I believe there is no alternative to either IPKG or a complete reinstall. Or you'd have to make PND's which spew content all over your NAND when run, in the traditional "big fat update" kind of way. For simplicity, the latter might be the choice, but for developers, IPKG is really something you cannot go without. You need inter-dependencies and dependency handling as much as possible when you only have 512MB to work with. AFAIK you can't have half the /usr/bin entries from one source and the other half from another, unless you do some kind of fancy symlinking, which is something quite tedious when it comes to installation (deciding which files go where). You're really talking about thousands of IPKG packages with interdependancies which PND cannot even have knowledge about.
dflemstr said:
I don't believe in tradition. I don't even celebrate Christmas
And simple FTP will not suffice. We cannot, for example, let people upload unvalidated PND files to somewhere (I have encountered various possible exploits in the current format, and unless those are fixed, we can't let PND files spread without control).
But it would of course be possible (As mentioned above) to move the storage of PND files out of the PND manager and to a file system. People would upload files in the PND manager and then have FTP read-only access to them. As I said,it's only been me working on this project so far; I'd really appreciate if someone could assist me in improving it in whichever way is required.
I've learnt to reuse existing systems as much as possible. If you innovate too much and people have a hard time interfacing, you lose it all. Simply documenting interfaces isn't enough, you need to ensure proper availability of the data in as much ways as possible. The proper way to do that is use classical widely accepted readily available tools. I hope this explains my reasoning.
About direct package acceptance, well, that's the grand issue isn't it. PND files will have to be signed by the developer that uploads it (thereby making it immutable). You can't know if the executable is malicious or not, they might've added a backdoor somewhere anyway. So it makes sense to allow any uploads immediately (you can do only basic validation without guarantees), and use a feedback system for validation (number of downloads, trust of author, removal after reports).
dflemstr said:
Hey, what makes you think that I haven't already done this? I understand how systems like this work (albeit not the itty gritty details) and thought about them when designing the current repository format. Believe me, it's not that bad.
Here are some principles that I've adhered to with the current system (tell me if there's something that you think is missing on the list):
- Packages are immutable. In the PND Manager, you can never edit a package, and a package can only exist once in the system. It is possible to delete a package, but this is required to enable moderation.
- Packages are uniquely identified. I have had a very lengthy discussion about this a while ago here (skip the first part; it's a proposal that was scrapped). A package is identified by name + version, and there can't be two packages with the same name and version. Versions are also enumerable, so there won't be issues with in which order upgrade paths should be ordered.
- Package repositories can be used synchronously. If you have two or more PND Managers on multiple hosts, and merge their repository file lists, you will only get conflicts if there are two packages with exactly the same name and version in both repositories, an issue that also exists in dpkg/ipkg. Otherwise, there are no issues whatsoever with having multiple repositories.
- PND files have strict naming rules. These aren't enforced upon the users, but when downloaded, PND files will always have the name "<name>-w.x.y.z.pnd" to avoid name collisions.
- PND repository entries are always identified by name/version, never by database ID (screenshot links and details links are; but with the REST system this shouldn't matter since you can always rely on the URLs being in a specific location, and thus the contents of the URLs are insignificant)
Ah, I'm sorry. I am more of a minimalist KISS kind of person, dealing with the tools we already have. Your plan was to make it "Web 2.0" (please forgive me for using that term), and my plan was to make it use conventional systems. Because of that I assumed you didn't know about the KISS-approach.
I acknowledge that, in order to do more with a file repository than is offered here, we need some metadata techniques. Since there are so much candidates for metadata, the REST API you have implemented might be the better of the candidates.
And I agree with Urjaman's use case scenario. That'd be a priority requirement if it were for me. Flag for retrieval, install whenever I say it's possible, but it does require the local client to be developed. I'd prefer a local client interfacing the API itself directly to any web interface using AJAX anyway, and I'm probably not alone
hedwards said:
I don't really think we need or want to have packages spread across multiple servers, it's confusing and a pain. What would be potentially quite useful would be to get a site like this for the platform. www.freshports.org basically to help us know when to look for updates and quickly find things that might be of interest.
Spreading files, no matter how well intended, is unlikely to be worthwhile for really anybody.
Well, having no master repository would be a bad thing. I was thinking of mirroring only for faster access on different hosts, and the ability to appoint another master whenever one went down (even temporarily). If it were to grow big (with lots and lots of game data), a distributed system (at least for the backend) would be needed, and it is good to be ready for that.
dflemstr, thank you for your attention and the implementation so far, I wish you lot's of success in perfecting it.
