dflemstr
It's a ball.
I would compare them to either Mac OS .dmg files, or just simply console "roms". No installation needed, but they also have auto-discovery, something portable apps don't.lulzfish said:
Last edited by a moderator:
The Pnd Package Manager
- Thread starter dflemstr
- Start date
dflemstr said:OK, some random replies to everyone:
- The PND "Metadata" file, aka the repository file, only contains text information about packages (description, creation date, version, etc). Screenshots should NOT be downloaded unless absolutely necessary; when you download the repository metadata you'll only get a link to the screenshot. I want to reduce traffic as much as possible
- You don't have to create an index at all for installed packages, IF you just:
The thing with an index is that an index doesn't allow for PND files being moved (remember, they aren't "installed", they are just dropped in a folder somewhere, and the system loads their contents on demand.
- Create a file that contains all the PATHs to the folders where packages are dropped.
- Maintain the strict naming scheme. Currently, packages have the name <name>-w.x.y.z.pnd when they are downloaded. If we tell people that they shouldn't rename files, we can simply organize them by looking at their names.
- If the medium where the PND is isn't inserted, we simply pretend that the PND doesn't exist (because it practically doesn't). Simple
Y'all, just forget the concept of "installation"; PND packages don't get installed: they are atomic. So we don't have to keep track of installed applications, we can simply specify some folders that should be scanned, and the found PND files are searched for in the repository index. If an update is found, we remove the old PND and replace it with a new PND.
Installation here would be "download and store on an SD card", removal would be "remove the thing from SD card". I'd want to keep track of my installed applications accross all my SD cards. The majority of Pandora users will probably limit themselves to 2 SD cards most of the time, but with multiple SD cards tracking installed stuff is going to be a problem. I really think we can't forget the concept of installation. If applications were always available regardless of you having inserted a specific SD card then I would agree with you, but you don't have access unless you have the right SD card. You are just trying to rename "installation" into "download and store", which is not that different
Also, if you have a lot of PND's, I can really see an issue with not indexing those PND's. First off; you'd have to scan the first x kb of every PND to get the metadata, which means you'll have to read and partially parse quite a lot of them to get your listing. It would be heaps faster to do the following: display all apps from the local cache (before displaying, stat a file to check if it still exists) in the list (you don't want to keep all this in the limited memory anyway) and simultaneously, in the background, scan for yet unknown changes on the SD cards (from the supplied paths) and supplement the current list you are browsing with that data. Of course you should be able to choose not storing those indexes, but I guess some people will find them really useful.
About reducing traffic. What you are asking is for the local client to download screenshots upon request, multiple times every time they visit unknown apps. It is better to cache those accesses so they download the screenshots only once when they are remembered from previous visits. You can see it as a kind of browser cache
Also, the download of a single PND file is probably equivalent to the download of about a hunderd screenshots most of the time, so in terms of traffic this wouldn't matter that much. If you were to download every single screenshot from all PND's to speed up your browsing experience, this could get quite large, but it would be useful for a lot of people. Nothing says more about a game/app than a visual impression. Instead of screenshots, you might also wish to consider making the application icon available, since this would enable the app-store method.Also, about your naming scheme, I hope it is not TOO strict. If I have a package, to which I make local modifications (outside of the main development tree), what do I do? Do I update the minor version number and risk versioning conflicts, or do I rename the base name of the package entirely because I cloned it? The versioning scheme on many systems is therefore something which starts with w.x.y.z, but allows indefinite .additional.version.parameters-with.alternatives to differentiate between those same versions even.
Edit: not to mention applications which already exist and are put inside a PND file, which do not match the versioning scheme you 'decided'. Wouldn't it be better to allow any kind of version numbering scheme, as long as it satisfies the property that it would keep a certain ordering so you can decide the 'latest' version without imposing restrictions? It would be quite easy to define such an ordering function such that ww-1.2.33b4 would be 'newer' than ww-1.2.5-ogles
Last edited by a moderator:
dflemstr
It's a ball.
Well, as it looks now it's not going to be me who develops the client. The people that are going to do it can do whatever they want to; I just provide a repository for them to pull from. If they want to create an index, sure, why not; it wont interfere with the repository.polyroy said:Installation here would be "download and store on an SD card", removal would be "remove the thing from SD card". I'd want to keep track of my installed applications accross all my SD cards. The majority of Pandora users will probably limit themselves to 2 SD cards most of the time, but with multiple SD cards tracking installed stuff is going to be a problem. I really think we can't forget the concept of installation. If applications were always available regardless of you having inserted a specific SD card then I would agree with you, but you don't have access unless you have the right SD card. You are just trying to rename "installation" into "download and store", which is not that different
This sounds like a good solution, but we'd have to reduce the amount of IO as much as possible. The index file shouldn't be saved on NAND for instance, but instead on the root of every SD, so as to save on NAND writes, methinks.polyroy said:
(BTW: The PXML aka PND metadata is in the last few KB, just mentioning it for completeness sake. A PND is currently produced by "cat mydata.iso PXML.xml screenshot.png > mypnd.pnd". There are plans to maybe change this in the future (because there are numerous exploits possible with the current format).)
The application icon/screenshot are essentially the same thing. The PND author can choose to use any picture as the preview/screenshot/icon.polyroy said:About reducing traffic. What you are asking is for the local client to download screenshots upon request, multiple times every time they visit unknown apps. It is better to cache those accesses so they download the screenshots only once when they are remembered from previous visits. You can see it as a kind of browser cache
And the advantages of having completely separate screenshot files that are downloaded by installer apps on demand (app store apps or whatever) are:
- Since the repository.xml file is changed constantly, it cannot be cached by relay servers all over the world (HTTP cache). Since the screenshot files, however, are separate, immutable and identified by an UID (their URL), they'll be cached in routers or proxy servers, thus reducing traffic to a minimum for the main server.
- A local store app will never have to send requests for the same screenshot twice, since it (the screenshot) is immutable and thus can be cached on the Pandora in any way you like; I guarantee that a screenshot URL always will return exactly the same content, byte for byte (again, a REST principle).
- The repo data is currently a pure XML file. I wouldn't want to invent a custom binary format just so that you could download previews alongside the repository in one go. It would be bloated and non-KISS.
I don't control the PXML format. Yes, I (re)made it a while ago, but it was a collaborative process, and all the devs back then agreed that it always should be absolutely clear which package comes after which in the upgrade path. If you would use the Debian/whatever naming scheme, that allows for versions like "1.1-svn0", you'll have the problem that it becomes difficult to tell the order of packages. Which comes first, "1.0-svn0" or "1.0-snapshot-fork0"?polyroy said:
And no naming conflicts will happen; if you create a package for the Pandora, you control that package and can decide upon a versioning scheme. If you fork a package, it is a different package, and should have a different name. You haev to contact the original author if you want to create an "official update" for a package so that you can coordinate version numbers.
Also, since PND packages don't have dependencies, you don't have to adhere to upstream version numbers, and may release "kdebase-workspace-4.2.4" as "kdeworkspace-1.0.0.0" on the Pandora for all I care. The only important thing is that the system needs to know which package comes after which; everything else is cosmetics.
BTW, the current PXML format is currently being reworked a little, so if you want to make some official complaints, make a thread in the Dev forum to get the devs' attention and utter your opinion (as long as you can argue for it, of course) and something might change.
EDIT: Maybe a small status update: I'm currently implementing cache systems for the Manager, because as it is now, the database is accessed for every single request, something that I do not want. The PND Manager is speedy, but not speedy enough for my standards
I just love the sight of this:
Code:
INFO - Service request (GET) /pndmanager/api/repository.xml took 10 Milliseconds
INFO - Service request (GET) /pndmanager/api/repository.xml took 1 Milliseconds
INFO - Service request (GET) /pndmanager/api/repository.xml took 1 Milliseconds
INFO - Service request (GET) /pndmanager/api/repository.xml took 1 MillisecondsOh, and if you guys hate the name "PND Package Manager", then assemble and come up with a new name (So far, everyone has made a new proposal, and I'm just telling you: I'm not going to be the one who decides on a new name; if you want something to change, then tell me what to change in chorus so that there won't be fights about it)
Last edited by a moderator:
dflemstr said:
Wasn't there some kind of wear-leveling planned? Even use of JFFS2 (or UBIFS for those who like speedy experimental stuff) as the primary NAND-FS would mitigate just about all issues. But that is a discussion for another thread. I'd make the location user-selecatable, since the index file does not need to grow that big if you limit in categories what is kept recent. Besides, SD block devices also suffer from limited writes
dflemstr said:
That is indeed VERY ugly. Aargh, don't get me started on that. A much better way would be to put the three files in a simple TAR with the pnd.xml a fixed name referencing the rest...
dflemstr said:
Well, it seems to me that the repository.xml itself would be very ugly if complete updates had to be done all the time. Can we opt for some kind of universal diff between versions, or for rsyncing the repository.xml? Usually, you download a series of static diffs since the last time you updated, which are generated every six hours or so, including a changing diff which is from the last diff up to current. I looked through your code, but I could be mistaken. I also see no way of querying part of the repository based on limits, e.g. first 10 entries or a certain category. Might be worth looking into.
dflemstr said:
I agree with you in this, this would simplify versioning at the cost of losing upstream transparency, but it is worth it. I just hope people won't be making kdeworkspace{a..z} whenever they think they can do better. Do you have a list of "registered" package names to force uniqueness? An uploader would have ownership of that name, and can delegate ownership to others so that others can upload new versions if the owner chooses to, but the owner will keep final verdict on the versioning of "his" package. Also, if we work with multiple repositories (I know I'll be rolling out my own), how do we force this uniqueness? Repository-specific prefixes for packages?
OpenID at least will guarantee that user x for repository y is really user x when using repository z, as long as you assume the repositories to be honest, so the owner/author faking issue is pretty much nonexistent across repositories.
dflemstr said:
I'll look into it, to see if I agree
Last edited by a moderator:
dflemstr
It's a ball.
There is wear leveling, but why use the NAND if there are alternatives?polyroy said:Wasn't there some kind of wear-leveling planned? Even use of JFFS2 (or UBIFS for those who like speedy experimental stuff) as the primary NAND-FS would mitigate just about all issues. But that is a discussion for another thread. I'd make the location user-selecatable, since the index file does not need to grow that big if you limit in categories what is kept recent. Besides, SD block devices also suffer from limited writes
That's what I said to the devs when I was told about itpolyroy said:
(But I proposed ZIP instead, because of the ability to randomly access files in a ZIP, and because there are devs out there using Windows)
(First of all, my code is Scala, so good luck reading it unless you have some solid functional + object oriented, component based programming language knowledge; Scala is a very difficult language to fully understand, even I don't. Good luck reading it.)polyroy said:
I have not heard of a conventional package system that would diff the package index (RPM/YUM and DPKG don't do this, or at least I've never seen this happen; maybe I should read through the complete standard and look for something like this). And if you think about it: is it really worth it? Should the server really have to care about which version of the index file each and every visitor has, and generate a diff depending on that (or have several diff indices cached, or whatever)? Or should we just let the user download the 50 kb file and let the client sort out the whole thing... Remember: I currently generate the file in 10 ms, and serve it in 1 ms. This is fast enough already, wouldn't you say? And if someone wants to mirror the package DB, then they can just stat the file every hour or so and download any new packages; it's not like there will be vital updates for the Pandora that have to be distributed to everyone within 10 minutes.
BTW: The whole API currently still is a meta system and is not being copied out to a file system. There therefore isn't an actual file lying somewhere that you could RSync. This can be changed of course (it's 20 lines of code or less), the project is still in a very early stage (I've spent, like what? a total of 30 hours on it?), but still.
I do not have an index of who "owns" a package series because I don't think that such a precaution/system is necessary. We won't be dealing with 1000's of contributors here; the Pandora is a small project so I don't think that people will randomly come up with two separate projects that happen to share the same name. An user owns a package, thats it. If an user suddenly leaves the project, it should be possible for someone else to step in and continue the package series, and it should be possible for multiple contributors to all administrate a series of packages if they feel that it's necessary. If we want to limit who is allowed to upload packages and who isn't, this should happen on the administrative level, and not be system-enforced.polyroy said:
There is of course the problem with having random users upload dangerous packages with the same name as an existing package series to the repository, thus triggering an auto-update on all Pandoras, essentially creating a virus, but I'd say that we can handle that on the client-side in a similar way that openSUSE handles vendor changes for exactly the same situation(Novell actually do some things right, occasionally, believe it or not). We just have to keep in mind that this system is open; I'd really hate to force users to obey stupid security rules "just because...".
So here is what the system similar to SUSE's would be:
Since the PND Manager API exposes a package owner's name in the package index (I don't know if I've actually set the flag to do this atm, but it can be done; as I said, the API isn't set in stone and I would really like to get comments on it), the local Pandora application would be able to simply check (against an index) if an update has a different owner's name compared to the existing package, and warn the user if this is the case. This would still leave the possibility open for package series "take-overs" (the friendly and evolutionary kind), but if the end user doesn't trust the new package maintainer, no upgrades will happen.
Oh, and if there would be a global package series owner index across all repositories... that would complicate things by a fair bit, wouldn't you say? We would have to either create a central and/or distributed, concurrency-optimized transaction system that managed all of the package namespace owners (complicated), or we'd have to simply stick with 1 single repository and make an integrated system there (limiting). That thus wouldn't be a solution, methinks. (wow, why do I keep saying methinks??)
And prefixes? We'd have to ask the community if they would be OK with having such a restriction put upon them.
Last edited by a moderator:
dflemstr said:
Because SD-cards also use NAND Flash, but use nonoptimal filesystems (FAT32, EXT2/3 perhaps) which do suffer from this kind of wear, whereas the builtin NAND flash does have wear-leveling (due to having the proper FS) and therefore will last substantially longer.
dflemstr said:That's what I said to the devs when I was told about itpolyroy said:
(But I proposed ZIP instead, because of the ability to randomly access files in a ZIP, and because there are devs out there using Windows)
ZIP uses compression, so you can't read from it without having to run expensive decompression routines. With TAR, you can just read from a certain offset and have your data 1-on-1, you can wrap it in gzip or something if you want the compression too, and it will be better compressed (and more efficiently decompressed) than ZIP. Also, I believe every Windows user should have installed 7-Zip by now, since you don't need anything other than that for just about all archive formats.
dflemstr said:
Heh, luckily I study Computer Science and I do have that knowledge. Doesn't seem too difficult to grasp
But that begs the question - if you really think that no-one else can read your code, why did you choose to use it in a community project? I'm not blaming you or anything, I know how easy it can be to use a complex language you grasp.dflemstr said:
Actually, DPKG has been doing the diffs for a while, it is optional though. They have a single set of diffs to bring the user from every x-hour version up to the next, and the users download them sequentially to reconstruct the current version. The server doesn't have to think to do this, it just serves those static files it generated once whenever requested. Just press u in aptitude and you'll see the diffindexes scroll by (on Debian Lenny at least). They use pdiffs (partial diffs), but I don't know the specifics.
If you have the memory, there is no issue with keeping it there instead of on disk. Serving time depends on the client's connection speed, not your local LAN throughput
. But for the API it is useful to provide an as-flexible method of access as possible. Not every client via their GPRS connection feels comfortable with waiting for even 10KB of data It'd make sense to use gzip content-compression for files >1KB on clients that support it (virtually all, these days).dflemstr said:
Sure, for smaller repositories it makes sense to fix things as they happen as to not frustrate in the general sense. Good thought, Wikipedia seems to cope with it just fine too
dflemstr said:
Novell's method seems allright, but the "owner names" (should be unique ID's) would have to be validated by the repository as OpenID URI's, or you can't have this done properly. And even then, it makes sense to implement package signing through a public/private key system. This is real "trust" and you wouldn't rely on the honesty of the repository either, however we'd need a tool to generate PND files for this since it won't be that trivial anymore.
The nice thing about OpenID is that it facilitates that owner index. An OpenID URI is unique, and when validated, it will guarantee everything you need about that user. There cannot be any conflicts in merging owners that way, since you know the same OpenID URI across repositories to be the same person. It is just a matter of storing that URI as the "owner id" field, instead of a chosen name, which CAN be faked.
Prefixes are quite ugly, but how else can you guarantee lack of conflicts when using multiple repositories? The best thing I can say is that whenever a person launches his/her own personal use repo, they "blacklist" a prefix on the "master" server which they then use for themselves, to prevent "official" common packages suddenly interfering with a user's own packages. In general, no repository that is common would require prefixes, especially not when they mirror each other. But to prevent doom-1.2.3 messing up with a user's own copy of a completely different platform doom ROM, this makes sense.
Last edited by a moderator:
dflemstr
It's a ball.
...but SD cards are replaceable, something that the NAND is not. Yes, I know, we'll have a lifetime of 5 years even if we use the internal NAND for swap, but hey, there won't really be any performance differences, so why not use a more sustainable system if we can? We won't need information about packages on a SD card that is unmounted anyways, so why not keep all of the information in one place?polyroy said:
No shit, Sherlockpolyroy said:
Agreed, TAR is suitable at least for server storage, albeit not the optimal solution. Let's leave the discussion about this for the appropriate time and place, however; the format is insignificant (only the metadata matters) when discussing the repository part.polyroy said:so you can't read from it without having to run expensive decompression routines. With TAR, you can just read from a certain offset and have your data 1-on-1, you can wrap it in gzip or something if you want the compression too, and it will be better compressed (and more efficiently decompressed) than ZIP. Also, I believe every Windows user should have installed 7-Zip by now, since you don't need anything other than that for just about all archive formats.
Well, Scala in itself is a very simple-ish language; I just thought that maybe my way of writing Scala in combination with the libraries I use could seem a bit incomprehensible to outsiders. I haven't really cleaned up the code in some places, so some parts, especially the whole self-contained and extremely coupled "entry system" that I've made, can seem a bit messy.polyroy said:Heh, luckily I study Computer Science and I do have that knowledge. Doesn't seem too difficult to grasp
Also, I am really a more language theorist kind of guy, so I try to exploit every little feature that a language has in order to gain performance and reduce the number of LOC (you should have seen the code a few weeks ago; I actually had a class produce closures that in turn would produce database access monad types just so I wouldn't have to create what now is the Entry System), so some things in there will seem foreign. I'll clean it up a bit before I announce it in the dev forum however, but not too much; partly because the code really is clean in some places already, partly because if a dev cannot understand how a simple iterator map operation works, I don't want him tampering with "my" code.
EDIT: Clarification:
Scala is the kind of language that would fit in this situation: "Hmm. I would really need the ability to overload lazy private singleton object factory definitions." *looks in the documentation* "Oh, there's a library for that!"
That's why I call it difficult to comprehend; there are so many uses for it that you cannot understand them all; you simply learn those that you need and use and abuse them. I have been studying Scala for a long time as a hobby and would say that I know all of its syntactical and core library features, and viewed form that point of view, it's really easy to understand, kind of like Python; but it's impossible to completely understand all the wonders that library implementors have done with it.
Well, a diff system could be implemented, I'm already convinced of that, but I still don't see the point; the size reduction gained is too small. Oh, and the times I provided before aren't latency measurements from the client's POV; I have an actor that analyzes requests on the server and measures how long the server needs to process the request before sending back a response. And the measurements probably are inaccurate anyways; there is a +-1.5 ms failure because of the inaccurate timer that is being used.polyroy said:Actually, DPKG has been doing the diffs for a while, it is optional though. They have a single set of diffs to bring the user from every x-hour version up to the next, and the users download them sequentially to reconstruct the current version. The server doesn't have to think to do this, it just serves those static files it generated once whenever requested. Just press u in aptitude and you'll see the diffindexes scroll by (on Debian Lenny at least). They use pdiffs (partial diffs), but I don't know the specifics.
If you have the memory, there is no issue with keeping it there instead of on disk. Serving time depends on the client's connection speed, not your local LAN throughput
Since you have to be logged-in as the owner of the package in the PND Manager (still waiting for a new name proposal!), you will get an OpenID associated with each package for sure. Note, however, that the actual PXML format doesn't contain any owner information, and it really shouldn't either, so this will have to be handled on a repository level. And yes, I know that OpenIDs can be used as UIDs, as can all URLs; that's why I chose it for loginspolyroy said:
How to guarantee a lack of conflicts? Practically impossible; the community will not accept prefixes (you might have heard about what happened when they merged the various Fedora unofficial repositories into what now is RPM Fusion; there were feuds and they ended up just scrapping conflicting packages altogether and using the version from the most popular repo, Livna)
But if, as you say, prefixes only would be used for personal use anyways, then it would be up to the individual owning the repository to set up a naming scheme anyways, so it won't be a problem? We'll just have people check the official repository/ies and see how a specific package is named, then they can create customized package series from those packages, name them whatever they want, and be done with it. What's the problem?
Last edited by a moderator:
dflemstr said:
Well, that makes sense, I might go and give it a try too. I prefer learning all there is to a language before actually using it, so Scala would certainly be a challenge and make me more confident in my language skills and comprehension.
dflemstr said:
I was actually thinking of placing entire PD ROM libraries for a lot of platforms in a repository, and that'd produce tens of thousands of PND packages, increasing frequently. Just to make it a great and accessible platform, and a repository would need to be able to scale somewhat for this to be possible.
dflemstr said:Since you have to be logged-in as the owner of the package in the PND Manager (still waiting for a new name proposal!), you will get an OpenID associated with each package for sure. Note, however, that the actual PXML format doesn't contain any owner information, and it really shouldn't either, so this will have to be handled on a repository level. And yes, I know that OpenIDs can be used as UIDs, as can all URLs; that's why I chose it for logins
You chose right. And my proposal still stands for "PND Software Repository", so if some people would agree and we'd get it over with? Thanks
I'd still consider optional package signing for PND's, to detect tampering and such. Every decent package manager has this btw. But this is a PND file format discussion and does not belong here.
dflemstr said:
I was thinking of the situation in which someone knows about a private repository prefix and uploads legitimate looking conflicting packages to the mainstream repository to cause conflicts there. Now, the owner has to either convince the mainstream repository to remove the package on the grounds that it conflicts with his own naming scheme (not likely to be approved), or rename his own packages causing massive issues with the client(s) using that repository. If anyone could reserve these prefixes to ensure they are not used, this would be entirely prevented. I simply like to consider all theoretical abuse first, I am very focussed on correct behaviour of systems
Last edited by a moderator:
dflemstr
It's a ball.
The PND Format has some really serious issues, one being that any kind of signing, checksumming and encryption with metadata attributes just is plain impossible to do, because the metadata, aka the PXML file, is part of the data being signed/checksummed (and there's no deterministic way of telling where the ISO data ends and the PXML data begins; one of the possible exploits with the format), and thus cannot contain a checksum/GPG signature obviously, so you would have to add headers to the format that would contain signature data, or just use a separate signature file (à la "private-signed" files or emails), or just rework the format completely. Again, we'd have to contact skeezix to get some changes in.polyroy said:
Oh and BTW, the current "official" library for reading PNDs and PXML files can be found here: http://git.openpandora.org/cgi-bin/gitweb.cgi?p=pandora-libraries.git;a=summary
I have written the PXML reader and the PXML specification in there, so the PXML format is really well-defined (IMO; of course my opinion is subjective
) and shouldn't be problematic. I wouldn't say the same about the way PNDs are handled however (No offense to you skeezix if you happen to be reading this; you are as aware of these problems as I am and there's no way that you could have foreseen them when designing the PND format)
Last edited by a moderator:
dflemstr said:
I agree fully. Adding a PGP/MIME signature would be good. Since PGP signatures are so well-defined and noiseless (without MIME), the binary signature could simply be appended (concatenated) to the end of the file, much the same way as is done now. I don't think the same security issues are applicable as this format is so well-defined, but we'd need some verification first.
dflemstr said:Oh and BTW, the current "official" library for reading PNDs and PXML files can be found here: http://git.openpandora.org/cgi-bin/gitweb.cgi?p=pandora-libraries.git;a=summary
I have written the PXML reader and the PXML specification in there, so the PXML format is really well-defined (IMO; of course my opinion is subjective
Great, just some remarks about pxml, and they might've been dealt with before (don't want to start or resurrect a seperate thread for this):
- "Example file" "<clockspeed frequency="600"/><!--Frequency in Hz-->" contradicts with earlier "frequency in MHz"
- We do seem to have support for an application icon in addition to "preview pics" screenshots
- Author information IS kept inside the package, but is completely optional. No association is made with the package submitter yet. This can only be done if an OpenID url is used as Owner Identifier. In my opinion, we should not match package owners in the pxml to package submitters. Owner information is informal, the package's signature (when we do this) will tell us all about the real party responsible for the package.
- Build information does not fit in the w.x.y.z.pnd versioning scheme, minor issue, but needed perhaps for local development.
- Interpretation of "clockspeed" value is lacking. It does not make the specified clockspeed required, optional or recommended. For running multiple apps at once, this will most certainly cause conflicts. Some apps absolutely require a certain clock speed, be it fixed (ugly apps), or a minimal (heavy apps running into timing problems). Some apps just run better on a certain clock speeds, but do not require it for providing that functionality. The way it is described now, I can't imagine it being used properly.
Last edited by a moderator:
dflemstr
It's a ball.
(REM: edited the list in the quote to use numbers)polyroy said:Great, just some remarks about pxml, and they might've been dealt with before (don't want to start or resurrect a seperate thread for this):
- "Example file" "<clockspeed frequency="600"/><!--Frequency in Hz-->" contradicts with earlier "frequency in MHz"
- We do seem to have support for an application icon in addition to "preview pics" screenshots
- Author information IS kept inside the package, but is completely optional. No association is made with the package submitter yet. This can only be done if an OpenID url is used as Owner Identifier. In my opinion, we should not match package owners in the pxml to package submitters. Owner information is informal, the package's signature (when we do this) will tell us all about the real party responsible for the package.
- Build information does not fit in the w.x.y.z.pnd versioning scheme, minor issue, but needed perhaps for local development.
- Interpretation of "clockspeed" value is lacking. It does not make the specified clockspeed required, optional or recommended. For running multiple apps at once, this will most certainly cause conflicts. Some apps absolutely require a certain clock speed, be it fixed (ugly apps), or a minimal (heavy apps running into timing problems). Some apps just run better on a certain clock speeds, but do not require it for providing that functionality. The way it is described now, I can't imagine it being used properly.
- Typo in the example file It's always the spec that should be the reference
- Yes, but these are made for menu launchers and shouldn't be used by systems that don't mount the ISO files
- The author information is also just for menu launchers. I kept all this in btw, because the library had to stay compatible with the other, previous, non-XML-compatible PXML format. I would have scrapped it otherwise, if I had had the power to. So most of the PXML information is for menu launchers, including the menu category fields, btw.
- The build field is also a legacy name; it's just a name for the smallest version number element.
- Clockspeed is also a legacy thing; it is optional in practice (so if it isn't specified, the app should be able to run at any speed) but again, I had to make the format compatible with the old one because the discovery apps require certain things of libpnd. I would much prefer to have something similar to the way you specify version numbers in Maven, eg. "[1.0.3,)" for stable numbers over 1.0.3, or similar.
Last edited by a moderator:
Vorporeal
Yes, no, I, this is.
One piece of the discussion that I noticed (I've read most of this, but not all - I get a bit lost at times haha) is how the syncing app on the Pandora would keep track of "installed" PNDs to update them. Wouldn't it just make sense to have a plaintext file in the root of each SD that lists the PNDs present on that SD card and their locations? This way, the package manager (local app) would know what files it should check for updates and where they are located, and you wouldn't have to worry about writing a list of installed apps to the NAND or something like that. I suppose one way to get this file would be to have libpnd create it when it searches the SD cards for PND files - this would solve the problem of a user adding it manually and the package manager not knowing that it exists. Also, if the file wasn't found, it could just use libpnd to search the SD for PND files and recreate the file that way.
This might be totally last page, depends on how much you guys have discussed in the last few MASSIVE posts. Just thought I'd throw that out, as it popped into my head while I was trying to read some of what you guys wrote recently.
This might be totally last page, depends on how much you guys have discussed in the last few MASSIVE posts. Just thought I'd throw that out, as it popped into my head while I was trying to read some of what you guys wrote recently.
dflemstr
It's a ball.
Great idea; this should definitely be handled by libpnd, and applications using it should be able to expect that an index file exists. I would prefer if the index file was XML but otherwise this should be possible. After all, the PND discoverer will run as a daemon; why not let it index stuff too?Vorporeal said:
(I wish we had had something like the KDE system on the Pandora, then we'd just have been able to use Nepomuk to do the indexing for us. KDE is very far ahead when it comes to modern frameworks in many cases.)
But we really shouldn't be discussing this too deeply here; if we make huge plans that end up being impossible to carry out, or if the PND devs just say "no", we'll have wasted a lot of work.
Last edited by a moderator:
skeezix
Internal Development
dflemstr said:
Sorry I'm a few days behind .. I lose weekends, and work has been killing me for the last few months so I missed this whole thread
Anyway --
ipkg is supported of course and will primarily be used for updates to the firmware, or for people to install packages right into the device.. which is not really wise. ie: I need to put up a FAQ to avoid re-answering this 5 times a week
The filesystem and such is in flash, but of course of limited size (and number of 'writes' other flash implications.). I envision people keeping some data in there, and the launcher caching icons and configs and such there, and data you want to maintain across SD ejects ,but really flash should be used for firmware and not much else.Further since most Unix systems are used to working on a hard drive and whatnot (ie: one big filesystem, without parts of it randomly popping out with SD ejects all the time), they're not used to splitting data across flash and SD cards and so on. It'll work fine, but its not advisable. (If we have time, we'll use unionfs so that ipkg updates will actually get unioned off to SD anyway, say, or at least a user partition in flash... then a 'system reset' could just wipe the user partition and you'd be back to the actual flashed firmware. Awesome. But if we don't have time to set that up, ipkg's will literally be modifying your firmware.) Imagine the pain of installing firefox where half the distribution is on SD and half in firmware, and then you eject the SD.. just a nightmare. So traditional package managers don't really work too well.
So anyway, because we want to leverage the flash filesystem for the firmware and perhaps core packages (firefox etc), we created the whole pnd and pxml system, so applications could be stored on SD cards. Much easier for the user.. no installation needed. Fetch and run and thats it. If we can really make this work as well as it does in our heads, I'd be surprised if othe rUnix distros don't pick up on the concept.
Anyway, ipkg will be supported, but should _not_ be used for normal applications (homebrew, emus, commercial apps, etc.) Ipkg should be used almost exclusively for firmware updates.
(Consider -- you've got X (small) amount of flash. You might also have a pair of 16GB SD cards jacked in. Do you really want to use ipkg and be restricted to installing a few small packages into your flash, while you have 32GB of space sitting around unused? Or sure, you could try to configure things just-so so that ipkg installs apps into SD .. but then what does your PATH return? A list of apps that are there? Then you eject an SD.. do you update ytour path and find out.. oh shit, some of my shared libs and confs and apps are missing, and god knows what happens. Extra fun when you're using a ipkg and instlaling some to one SD, some to another. I've been there, we dont' want to do that
But instead, we install core firmware apps to flash, and you just use pnds, nice and clean. (And we cna extend the pnd system as well, and let you copy pnds into flash and such..)But people can use whatever they like.
jeff
Sadly time has been killer for all of us devs, so we're keeping ourselves constrained. Which is why it sgreat that fine peopel like dflemstr are trying to help out on the user and client side -- while we created pnd and pxml stancdards (and I really should post some docs and APIs and such in a more central place eh?), we're not going to be writing fancy package managers and such. The community hopefulyl will step up there .. we're throwing our spare time into this, but we only have so much. But as a whole we are strong .. it is hoped everyone will pick up these beginnings and grow everything
Last edited by a moderator:
dflemstr
It's a ball.
Hey skeezix, good to see that you're back, finally
The situation with IPKG vs PND was cleared up on the ensuing page, so you don't have to have any worries about that there are angry unenlightened users that have been ignored on these boards
When I presented my system here, I expected it to be pretty rough and lots of people complaining and mentioning issues that I ought to have thought about etc. and of course this was the case, but so far there haven't been any unreasonable complaints and we've been able to have a constructive discussion.
I really think that we should have a session sometime somewhere where we discuss the PND and PXML issues in depth; it seems like a lot of people have found issues with the formats and/or would need to have features that the formats don't provide, so it would be really helpful if we could discuss and solve them all once and for all.
Issues so far (that I can recall without looking through 20+ pages of posts in various threads):
Don't see this as an extra weight lain on your shoulders; the community is happy to help out with any of these issues if you let it.
The situation with IPKG vs PND was cleared up on the ensuing page, so you don't have to have any worries about that there are angry unenlightened users that have been ignored on these boards
When I presented my system here, I expected it to be pretty rough and lots of people complaining and mentioning issues that I ought to have thought about etc. and of course this was the case, but so far there haven't been any unreasonable complaints and we've been able to have a constructive discussion.
I really think that we should have a session sometime somewhere where we discuss the PND and PXML issues in depth; it seems like a lot of people have found issues with the formats and/or would need to have features that the formats don't provide, so it would be really helpful if we could discuss and solve them all once and for all.
Issues so far (that I can recall without looking through 20+ pages of posts in various threads):
- The current PND format makes it impossible to deterministically tell where the binary (ISO) data ends and the PXML data begins. The same is true for the PNG screenshot
- There is currently no official support for PGP signing of PND packages
- There are potential exploits that can be ...well, exploited with the current PND format, by creating a file that starts and ends with the "<PXML></PXML>" pattern but contains dangerous data, and putting this file in the middle of the PND ISO image
- The ISO format is suboptimal for packages (there are multiple KB that are wasted and the ISO byte alignment makes files difficult to read); so far, the uncompressed TAR format has been suggested as a good alternative
- There are feature requests for libpnd and the PND discovery daemon; for instance, we'd want there to be an index file available in a known location containing all the mounted and discovered PND files along with metadata
- There is no official repository spec for the PND system at the moment; I had to make one up in my PND Manager.
Don't see this as an extra weight lain on your shoulders; the community is happy to help out with any of these issues if you let it.
skeezix
Internal Development
dflemstr said:Hey skeezix, good to see that you're back, finally
The situation with IPKG vs PND was cleared up on the ensuing page, so you don't have to have any worries about that there are angry unenlightened users that have been ignored on these boards
When I presented my system here, I expected it to be pretty rough and lots of people complaining and mentioning issues that I ought to have thought about etc. and of course this was the case, but so far there haven't been any unreasonable complaints and we've been able to have a constructive discussion.
I really think that we should have a session sometime somewhere where we discuss the PND and PXML issues in depth; it seems like a lot of people have found issues with the formats and/or would need to have features that the formats don't provide, so it would be really helpful if we could discuss and solve them all once and for all.
Issues so far (that I can recall without looking through 20+ pages of posts in various threads):
- 1 The current PND format makes it impossible to deterministically tell where the binary (ISO) data ends and the PXML data begins. The same is true for the PNG screenshot
- 2 There is currently no official support for PGP signing of PND packages
- 3 There are potential exploits that can be ...well, exploited with the current PND format, by creating a file that starts and ends with the "<PXML></PXML>" pattern but contains dangerous data, and putting this file in the middle of the PND ISO image
- 4 The ISO format is suboptimal for packages (there are multiple KB that are wasted and the ISO byte alignment makes files difficult to read); so far, the uncompressed TAR format has been suggested as a good alternative
- 5 There are feature requests for libpnd and the PND discovery daemon; for instance, we'd want there to be an index file available in a known location containing all the mounted and discovered PND files along with metadata
- 6 There is no official repository spec for the PND system at the moment; I had to make one up in my PND Manager.
Don't see this as an extra weight lain on your shoulders; the community is happy to help out with any of these issues if you let it.
- 1 -- we debated (endlessly!) various options, as we've talked about before; using zip files, cram files, etc and so on. And just using plain simple .iso, and prepending or appending a header/footer 'table of contents' to let skipping around be easier. (In fact we even implementeed that for awhile.) However, in the end, 'simplicity' won out over going for complxity. The simpler it is, the more we can optimize and extend later, without being 'over commited' to complicated things. Further, with some testing (hey, this is spare time, we only had so much..) we foiund iso's nice and fast, and reliable. (Remembering we have to fight with drivers and speeds and all sorts of nightmare scenarios.) file + append was easy, and also works with zip (since zip comments are just appended to the end), and a few others. so we went with it. Its not perfect, but its a start, and it works. (ie: Like RSS and other standards.. if its heavy, no one will adopt; you start small, gain acceptance and usage, and _then_ you grow and extend.) There _are_ ways to calculate the end (if its an iso you can often determine the image size and such and skip around, but it is not pretty of course.) A reverse rewind is relativelyt easy and generally works.
Yes, there are edge cases, or attack cases, but thats true all the time. The package sites can vet out the bad files perhaps
If the PXML is not appended, and since iso is plaintext format (uncompressed/etc), then a reverse find will _Still_ find the PXML. If theres no PXML at all, the reverse find is I think capped to stop near the end (I think I added that, I forget).. worst case it seeks and fails. (Well, worst case it seeks and finds binary data for an .exe with a PXML parser, but that should fail PXML content validation and be rejected.)
The _icon_ should work okay, but again, it can be tortured. (Screenshots are not handled thia way. For awhile when we had the 'toc' (table of contents) header, we could skip right to screenshots, but we decided (blame me I think
that it was easier to just pnd_mount the .pnd or PXML-dir, and then pick up the screenshots, and then unmount. (ie: Thus putting off the caching of screenshots etc to the consuming applications, which seems wise in this case.)2 -- an interesting question. Of course, 99.9999999% of the world has lived without PGP signing till this day, and it was added later to existing protocols/etc too. So we don't need to solve every possible problem day zero, either. (And also recall, a few people in their spare time. If IBM and MS can't pull off something, its amazing people sometimes think OSS hackers can in a few hours
Anyway, you've proven yourself, you can help think up something. (Really, you've probably spent as much time on your tool as I have on pnd/pxml myself )3 -- exploit in what way? ie: If theres a buffer overrun etc, let me know; as to file format, that sort of thing shouldn't really be any more dangerous than putting the 'executable is rm -rf /' into the PXML. (And if that didn't occur to you the very first day ED mentioned PXML, then you're slipping
4 -- iso is not really wasteful. No one is oging to quible over a very _K_ on a multiGB card. Someone mentioned the minimum iso size was XXXXMB, but they werew qway off. You can have a 20k iso if you want. .iso is very easy to work with by every OS, is easily mountable and so on and so forth. I don't know if a tar (tar is always uncompressed, a container compresses it. .Z or .gz or .bz etc) is mountable, but its an option. The PXML can specify alternative package types, and using 'magic' or any number of techniques .. it should be pretty easy to add to libpnd to handle it. (ie: We've _assumed_ there would be more types, like cramfs and zip and so on, since day one.)
Again, it doesn't have to be 100% super right (there is no true 'correctness').. it has to work, and be a compromise of everyones goals. (Some want size, some want speed, some want ease of condtruction or repacking..)
5 -- If someone wants to make requests.. they shoudl be sending them to me (duh), or offering to make the change (the source has been available publicly since day one.) Since I've not heward of it, there is no request, probably
Certainly, once we 'go live' (or even before, be my guest) I'm going hands off. I had really hoped to pour a few hundred more hours in that I had time for, but this whole economy, layoffs and general life has really killed most of us (myself, vimacs, djwillis, notaz, the core devs, we're all suffering
.. So no way do I want ot be a 'gate keeper' .. people are free to submit patches, or take it right over. We can add permissions ot check in any time, if someone seems to have a clue
Anyway, there 'so far' will be few mounts at once, and _all_ of them will be in a well defined location. (The config files and sh scripts pnd_run etc make it all ovious.. but all mounts will be in consistent locations. The raw ount and the iunioned app-data mount.) The libpnd API includes functions to ask for what these places are. ie: No index is needed, as the filesystem is that index.
In the future, we allowed for the idea (but we didn't code it in yet) to have somehting like a dir that has 'pre-mount' pnds.. say you want to have shared libs bundled together (or a codec pack, say) .. put them in a dir on flash or SD and at boot time, have all those mounted permanently, say. In that case there would be many pnds open at once, but we've not coded that in yet ('worry later' is better than 'take on too much at once.'), and further, it should be as easy as the above.
Right now, however, mounds should be all in one place, and likely only one or two mounts open at once anyway (probably one, the current app), so not a big deal.
But as to requests anyway. fire 'em over.
Most of my interest right now, in what little time I've got, is fixing up some robustness. I mean, theres some lame-ass stuff in there -- the important bit is the API is half useful and half-well-designed. The implementation details might be great, or might be crap in any given spot. Thats okay, we the community have many hands that can improve over time. (It will be included as shared lib so in theory non binary breaking changes should be easy to magicly apply to all apps.)
(Further, since source is all available, I would fully expect peopel to improve as they see fit. Again, it will not be "100% done" on launch, and it certainly will not be "skeezix's baby, hands off". No, we all (the dev team and us, and you guys, doubly so yourself) habve had a hand it in, and we put it out for everyone to hack on
6 -- yep, you are right. I know I've not the cycles for it, and I am very thankfully you're taking the charge. If you've noticed, I've been referring peopel to bug you about it (teehee
--
I hope that doesn't sound like I'm washing my hands of it.
I'm actually pretty good at managing some of these things, and what I'm aiming for her is .. "get it good enough, and reasonably well designed for the future obvious needs". If we over design, you risk the complexity hangfing you, and we really dont' want fragmentation. we want adoption.
All told, I don't evne know if a single person will use any of this though, but I hope it works and people use it
If anyone wants to help with the coding and testing, feel free to hook up. We've really not had the resources to test the hell ouf of this.
I realyt shoudl be working on a public documentation wiki, to copy in these long posts, and to start a 'how to' and 'API manual' and such.
But for now, suffice to say -- the API and suh is by no means perfrect. Feel free to take a look and let me know if you hate my function names, module design, etc. None of it is set in stone, at least unti llaunch day
jeff
Last edited by a moderator:
dflemstr
It's a ball.
Whoa, you sure can provide meaty posts when we least expect it
OK, first of all, the 'suggestions' I mentioned originate from the discussion in this thread (people came to me with requests and I try to recall what they were now) so you probably haven't discovered them yet
Basically, people have come up with scenarios that could present security issues. For example, should there be an "owner" of package series? AKA, should there only be one user that may upload packages with the name "gedit"? If not, how do we prevent people from uploading viri to a repository with the name of a popular package and a very high version number, thus triggering updates on all Pandoras? Then we started to talk about owner identification, signing, the requirement to have an index file that would contain metadata for all "installed" PNDs so that you could tell who had owned the package when it was downloaded, etc.
We could of course just keep it all simple and let people deal with it themselves (notification dialogs for new updates, but no automatic downloads etc) but it would still be a security issue for novices.
Oh, and ISO: what bugged first and foremost me and polroy about it was that ISO is a disk image format, bringing the consequences that you have to mount it to read data (as you say) and can't just randomly read a specific file in it (say, "/PXML.xml") without having to mount the whole thing. But, as you say, it might just be the easiest route. I as a web developer who has to deal with keeping PNDs in RAM for as short times as possible, don't enjoy the fact, however It now is impossible for me to extract anything from PNDs beyond the PXML and icon.
PGP signing: just a small issue; isn't really important. Forget that we mentioned it
Exploits: Imagine that you put the following byte sequence in the middle of the PND:
(It's the PNG header enclosed in <PXML></PXML>, ISO-8859-1 encoding)
This sequence, if put somewhere near the end of a PND file, preferably, will make my scan algorithm detect a (invalid) PNG file + (invalid) PXML data, while yours will find the "PXML file", parse it file, NOT determine that it's invalid (because TinyXML skips certain garbage if the file is prepared correctly), and then completely miss the PNG file. We have a single conditional read here, that could be used to provide other metadata depending on the reader. Yes, I know, not a huge deal, but it could be avoided; people probably don't want to download a PND and find that it has a completely different icon after having been downloaded, or that it has completely different metadata (exec info, etc etc). Also, if the PNG image happens to contain the pattern "</PXML>" (it could be but it's unlikely; but someone could do it on purpose), chaos will present itself (or not on my side; I actually made a blocker for this in the webapp, but I don't know about libpnd).
Oh and I am using screenshot and icon here to mean the same thing; it wasn't really clear exactly what the appended PNG was supposed to represent until now So it is an icon, something supposed to be square-shaped? Good to know, I'll adjust a few things now having that knowledge. I currently would like to have quite large icons, the standard 128x128 size perhaps; would this be something that I could expect from PND files, or did you want the icon to be something much smaller?
You probably have better things to do than reading my novels, so I'll stop here. I think that I've said everything that's important ATM.
OK, first of all, the 'suggestions' I mentioned originate from the discussion in this thread (people came to me with requests and I try to recall what they were now) so you probably haven't discovered them yet
Basically, people have come up with scenarios that could present security issues. For example, should there be an "owner" of package series? AKA, should there only be one user that may upload packages with the name "gedit"? If not, how do we prevent people from uploading viri to a repository with the name of a popular package and a very high version number, thus triggering updates on all Pandoras? Then we started to talk about owner identification, signing, the requirement to have an index file that would contain metadata for all "installed" PNDs so that you could tell who had owned the package when it was downloaded, etc.
We could of course just keep it all simple and let people deal with it themselves (notification dialogs for new updates, but no automatic downloads etc) but it would still be a security issue for novices.
Oh, and ISO: what bugged first and foremost me and polroy about it was that ISO is a disk image format, bringing the consequences that you have to mount it to read data (as you say) and can't just randomly read a specific file in it (say, "/PXML.xml") without having to mount the whole thing. But, as you say, it might just be the easiest route. I as a web developer who has to deal with keeping PNDs in RAM for as short times as possible, don't enjoy the fact, however
It now is impossible for me to extract anything from PNDs beyond the PXML and icon.PGP signing: just a small issue; isn't really important. Forget that we mentioned it
Exploits: Imagine that you put the following byte sequence in the middle of the PND:
Code:
3C 50 58 4D 4C 3E 89 50 4E 47 0D 0A 1A 0A 3C 2F 50 58 4D 4C 3EThis sequence, if put somewhere near the end of a PND file, preferably, will make my scan algorithm detect a (invalid) PNG file + (invalid) PXML data, while yours will find the "PXML file", parse it file, NOT determine that it's invalid (because TinyXML skips certain garbage if the file is prepared correctly), and then completely miss the PNG file. We have a single conditional read here, that could be used to provide other metadata depending on the reader. Yes, I know, not a huge deal, but it could be avoided; people probably don't want to download a PND and find that it has a completely different icon after having been downloaded, or that it has completely different metadata (exec info, etc etc). Also, if the PNG image happens to contain the pattern "</PXML>" (it could be but it's unlikely; but someone could do it on purpose), chaos will present itself (or not on my side; I actually made a blocker for this in the webapp, but I don't know about libpnd).
Oh and I am using screenshot and icon here to mean the same thing; it wasn't really clear exactly what the appended PNG was supposed to represent until now
So it is an icon, something supposed to be square-shaped? Good to know, I'll adjust a few things now having that knowledge. I currently would like to have quite large icons, the standard 128x128 size perhaps; would this be something that I could expect from PND files, or did you want the icon to be something much smaller?You probably have better things to do than reading my novels, so I'll stop here. I think that I've said everything that's important ATM.
skeezix
Internal Development
The appended PXML and icon is for convenience of launchers (but not so convenient as to put screenshots, audio clips, and the entire pnd contents .. ie: a launcher could easily scan for a candidate list of pnds (using libpnd or whatever) and then use the libpnd functions to get the icon and PXML (say), and show something. Then in the background it could pnd_mount, fetch screenshots (if the PXMl says there are some), and then pnd_unmount.
But also, pndnotifyd will dump the icon and the .desktop file in the configed place (in flash, to persist across SD ejects) .. this way pretty much ebvery darned launcher in unix/linux will show the executable with the correct icon by definition. (As to definition of 'icon', thats an exercise for the reader .. since every window manager/launcher expects differnet things, so I suggest to sticking with small and making it pandora specific, and desktop systems can upscale it
As to embedding nasty sequences.. I figure if the .pnd fails to mount, thats good enough. The repositories could automagi caly detect a bad pnd and not show it (or drop it from the index.) But a lot of user policing will handle it -- same as crap websites or shovelware now is never really found. Sites with bad software won't rank high in google so generally wont' et found, and download services will downrank things wiht few downloads.
the only real risk is 'exophase_psp_emulator_1.5' gets uploaded, and 5000 people download it within 20 mins before a mod kills it, since it didn't get uploaded by Exo. But the existing repo has a moderator to vet uploads before they show, so that is handled. I bet you'll do the same
Or do it like a blog -- registered ('cluefull') users could see the inbox, and after an app gets 5+ votes (say) byu those peeps, it goes public, but those peeps should know enough to run it first.
Likewise with issues with autodownloading versions -- thats not directly a pxml or pnd issue, thats you auto-downloading bastards creating issues for yourself
jeff
.. ie: a launcher could easily scan for a candidate list of pnds (using libpnd or whatever) and then use the libpnd functions to get the icon and PXML (say), and show something. Then in the background it could pnd_mount, fetch screenshots (if the PXMl says there are some), and then pnd_unmount.But also, pndnotifyd will dump the icon and the .desktop file in the configed place (in flash, to persist across SD ejects) .. this way pretty much ebvery darned launcher in unix/linux will show the executable with the correct icon by definition. (As to definition of 'icon', thats an exercise for the reader .. since every window manager/launcher expects differnet things, so I suggest to sticking with small and making it pandora specific, and desktop systems can upscale it
As to embedding nasty sequences.. I figure if the .pnd fails to mount, thats good enough. The repositories could automagi caly detect a bad pnd and not show it (or drop it from the index.) But a lot of user policing will handle it -- same as crap websites or shovelware now is never really found. Sites with bad software won't rank high in google so generally wont' et found, and download services will downrank things wiht few downloads.
the only real risk is 'exophase_psp_emulator_1.5' gets uploaded, and 5000 people download it within 20 mins before a mod kills it, since it didn't get uploaded by Exo. But the existing repo has a moderator to vet uploads before they show, so that is handled. I bet you'll do the same
Or do it like a blog -- registered ('cluefull') users could see the inbox, and after an app gets 5+ votes (say) byu those peeps, it goes public, but those peeps should know enough to run it first.
Likewise with issues with autodownloading versions -- thats not directly a pxml or pnd issue, thats you auto-downloading bastards creating issues for yourself
jeff
dflemstr
It's a ball.
I already have a dummy validation algorithm built in; I'm just not using it atm (similar to "def valid(p: Package) = true") so I'll have to build something in, and this seems to be as good a solution as any, thanks!skeezix said:
I'll have to hear with the others who've been frequenting this thread if they have any complaints first, however
Last edited by a moderator:
About the ISO format for PND's: There are some libraries (one being libMirage) which can query (read files from) ISO files without the expense of OS-dependent mounting, so it should be possible to put the pxml INSIDE the ISO file to circumvent at least the issues of format ambiguity With respect to validity of archives, I guess we could exploit the ISO format spec and append an optional signature right after the ISO, since ISO does not care about additional appended data. So the choice for ISO should not cause signing issues when we do that later on. Also, a PGP signature includes a checksum so we can easily verify corruption too. Big bonus here Since most metadata will get stored in a cache, the overhead of accessing through a library is negligible as it will only be done once (every time the file gets a new location or so.)
A disadvantage of TAR (which I proposed previously) would be that every file has its own header which is not neccesarily stored at the start of the TAR file (so for enumerating files, you'd have to skip quite a few blocks every time). ISO does seems nice to me now.
A PND file would then look like [[ISO [pxml][app data]][optional pgp signature of complete prior ISO (including checksum and other typical stuff such as the signer's fingerprint)]], which would seem pretty simple, robust and agile I'll make this my final recommendation and let the issue rest now.
Seems nice, one of the things I didn't comprehend the first time too But that won't be useful without having fixed locations of executables since with package-dependent mount names we'd get nowhere. Say, could we add "provides" to pxml in the way that when such a PND (say vlc) is mounted, it will create a symlink from (among others) /usr/bin/vlc to the appropriate files inside the mount? Then a PND would really have the capability to "extend" the system's firmware too.
I'd vote against this. For starters, people should not HAVE to register to download files or vote for them. I'd say authentication is only needed for uploads. My suggestion: if a developer uploads a package with a specific name for the first time, he'd need approval by an elected moderator. If he later uploads the package of the same name, e.g. updating it, it would go right through and depend upon community disapproval to be removed (the developer IS trusted by his prior submission at this point, seems fair to grant him that trust afterwards). If another dev uploads a package with a name which already exists, but hasn't been uploaded by that same developer, they'd also need to be approved - either by a moderator or the original submitter.
This should kill most forms of malice.
With respect to validity of archives, I guess we could exploit the ISO format spec and append an optional signature right after the ISO, since ISO does not care about additional appended data. So the choice for ISO should not cause signing issues when we do that later on. Also, a PGP signature includes a checksum so we can easily verify corruption too. Big bonus here Since most metadata will get stored in a cache, the overhead of accessing through a library is negligible as it will only be done once (every time the file gets a new location or so.)A disadvantage of TAR (which I proposed previously) would be that every file has its own header which is not neccesarily stored at the start of the TAR file (so for enumerating files, you'd have to skip quite a few blocks every time). ISO does seems nice to me now.
A PND file would then look like [[ISO [pxml][app data]][optional pgp signature of complete prior ISO (including checksum and other typical stuff such as the signer's fingerprint)]], which would seem pretty simple, robust and agile
I'll make this my final recommendation and let the issue rest now.skeezix said:
Seems nice, one of the things I didn't comprehend the first time too
But that won't be useful without having fixed locations of executables since with package-dependent mount names we'd get nowhere. Say, could we add "provides" to pxml in the way that when such a PND (say vlc) is mounted, it will create a symlink from (among others) /usr/bin/vlc to the appropriate files inside the mount? Then a PND would really have the capability to "extend" the system's firmware too.skeezix said:the only real risk is 'exophase_psp_emulator_1.5' gets uploaded, and 5000 people download it within 20 mins before a mod kills it, since it didn't get uploaded by Exo. But the existing repo has a moderator to vet uploads before they show, so that is handled. I bet you'll do the same
Or do it like a blog -- registered ('cluefull') users could see the inbox, and after an app gets 5+ votes (say) byu those peeps, it goes public, but those peeps should know enough to run it first.
dflemstr said:I already have a dummy validation algorithm built in; I'm just not using it atm (similar to "def valid(p: Package) = true") so I'll have to build something in, and this seems to be as good a solution as any, thanks!
I'll have to hear with the others who've been frequenting this thread if they have any complaints first, however
I'd vote against this. For starters, people should not HAVE to register to download files or vote for them. I'd say authentication is only needed for uploads. My suggestion: if a developer uploads a package with a specific name for the first time, he'd need approval by an elected moderator. If he later uploads the package of the same name, e.g. updating it, it would go right through and depend upon community disapproval to be removed (the developer IS trusted by his prior submission at this point, seems fair to grant him that trust afterwards). If another dev uploads a package with a name which already exists, but hasn't been uploaded by that same developer, they'd also need to be approved - either by a moderator or the original submitter.
This should kill most forms of malice.
Last edited by a moderator:
