Vimacs
Don't be evil!
not sure what you mean by that, but libpnd allows for accessing other pnd'sAethix said:
Last edited by a moderator:
Pandora Libpnd - The Pandora Library
- Thread starter skeezix
- Start date
skeezix
Internal Development
Libpnd includes routines for exec of pnd, but also mount/unmount should a tool wish to browse a pnd, pull up sample art/audio for a launcher, etc. Cpasjuste launcher depends on libpndfor these sorts of things
ie: people should use (or extend!) libpnd for all pnd and pxml-Dir abilities
note you can treatpxml-Dir and pnd more or less the same in libpnd.. Vey handy.
Also note pxml canextend to multiple executables, but .desktop cannot iirc.
Posting from my phone way too much
we'll just make dflem do the heavy lifting around pnds, right?
jeff
we will launch, then people can go nuts. We designed with the hopeof providing direction, example, stubs, etc... But who knows how it will evolve?
ie: people should use (or extend!) libpnd for all pnd and pxml-Dir abilities
note you can treatpxml-Dir and pnd more or less the same in libpnd.. Vey handy.
Also note pxml canextend to multiple executables, but .desktop cannot iirc.
Posting from my phone way too much
we'll just make dflem do the heavy lifting around pnds, right? jeff
we will launch, then people can go nuts. We designed with the hopeof providing direction, example, stubs, etc... But who knows how it will evolve?
skeezix
Internal Development
I suppose it would be easy to emit multiple .desktop files during discovery, once the pxml and behavior are well-defined. An uninstalle is easily added by I have deliberately avoided it to encourage not needing it.
A data manager for delete andcopy is on psp and xbox and ps2 etc so we could all create one later
A data manager for delete andcopy is on psp and xbox and ps2 etc so we could all create one later
Vimacs
Don't be evil!
to do that you are meant to use the appdata folder, contents in that will override the files in the pnd.Aethix said:
Last edited by a moderator:
skeezix
Internal Development
You would not need to modify the pnd itself; you would drop the new .cfg into the /pandora/appdata/wesnoth-1234/ dir, and the Wesnoth pnd app would see the cfg as if it was inside the pnd. ie: The pnd could be copied to someone lse and be 'just like new', but on your system it'd see all the new or modified files. *magic*
jeff
jeff
skeezix
Internal Development
Damn, I've written way too much documentation lately:
http://pandorawiki.org/Libpnd_hub
Maybe I wont' go into too much detail of the actual API .. peopel can read .h files right? MAybe I will go and mention workflow and order of operations.
Daaaamn
jeff
http://pandorawiki.org/Libpnd_hub
Maybe I wont' go into too much detail of the actual API .. peopel can read .h files right? MAybe I will go and mention workflow and order of operations.
Daaaamn
jeff
dflemstr
It's a ball.
So, now comes the question: "Where next?", right? Since developers are encouraged to make changes to the code base now, then why don't we?
First there might be a need for some cleanups. Fix everything from indentation to broken scripts etc (look at "testdata/scripts/pnd_make.sh@25,1" to see what I mean; I don't think that everyone will have PND scripts in their "~/pndtest/" folder...). I can volunteer to do such cleanups, just give me a shout.
Then there's the thing about new features...
For example, there was some talk about adding support for ZIP-files for PNDs. That would be something that I'm currently missing. No need to use "weird" scripts to create PNDs, just drop in your PXML, previewpic and go. This shouldn't be too difficult, since ZIPs can be randomly accessed if you know the file name of what you're looking for, and that's the case with PXML.xml and preview.png... Also, with libzip, such things can be decompressed in memory, and by using for example fuse-zip, ZIP files can be mounted (that's of course using FUSE, but I guess that some code can be stolen GPL-style, or should it be accepted that root uses FUSE?).
I really don't know what we may and may not do, seeing as the discovery code is really unmodular (no offense) in that it's optimized for seeking ISOs, but then I look at the pnd_run.sh and see that there's support for fuse-zip mounting.
So, should libpnd depend on fuse-zip? And what about libzip; I thought that dynamic libs should be avoided at all costs, at least for libpnd itself? What about copy-pasting the relevant code from the libzip source like how it was done with TinyXML?
First there might be a need for some cleanups. Fix everything from indentation to broken scripts etc (look at "testdata/scripts/pnd_make.sh@25,1" to see what I mean; I don't think that everyone will have PND scripts in their "~/pndtest/" folder...). I can volunteer to do such cleanups, just give me a shout.
Then there's the thing about new features...
For example, there was some talk about adding support for ZIP-files for PNDs. That would be something that I'm currently missing. No need to use "weird" scripts to create PNDs, just drop in your PXML, previewpic and go. This shouldn't be too difficult, since ZIPs can be randomly accessed if you know the file name of what you're looking for, and that's the case with PXML.xml and preview.png... Also, with libzip, such things can be decompressed in memory, and by using for example fuse-zip, ZIP files can be mounted (that's of course using FUSE, but I guess that some code can be stolen GPL-style, or should it be accepted that root uses FUSE?).
I really don't know what we may and may not do, seeing as the discovery code is really unmodular (no offense) in that it's optimized for seeking ISOs, but then I look at the pnd_run.sh and see that there's support for fuse-zip mounting.
So, should libpnd depend on fuse-zip? And what about libzip; I thought that dynamic libs should be avoided at all costs, at least for libpnd itself? What about copy-pasting the relevant code from the libzip source like how it was done with TinyXML?
skeezix
Internal Development
No offence of course. I'm not particularly proud of the code, but it works very well. Lik I've said, spare time and not a lot of it
feature freeze ntil release, but you can totally got nuts then
ie: I be simple rather than go nuts.. Let us grow the usage before adding zip. Adding zip won't help push the format, but your distribution tools might. Windows pnd builders too.
Shrug
jeff on phone
feature freeze ntil release, but you can totally got nuts then
ie: I be simple rather than go nuts.. Let us grow the usage before adding zip. Adding zip won't help push the format, but your distribution tools might. Windows pnd builders too.
Shrug
jeff on phone
Vimacs
Don't be evil!
Well, that isnt the newest version you are looking at, also those scripts will not ship in this state.dflemstr said:
Most of those scripts still need quite some work before im prepared to ship them.
Especially genpxml and pnd_make need to allow for user interaction and maybe get a nice gui.
Regarding zip:dflemstr said:
At the first planing stage we wanted to use zip cause, as you say, its simple to use.
However with that a few problems appeared, for example when accessing a file inside a zip the whole file has to be decompressed into ram first, even if you only want to read a small block. This is slow, takes ram, cpu, and creates useless io load.
For that and other reasons we removed all zip support and switched to iso, the zip stuff in pnd_run.sh is just a leftover which will be removed before shipping.
Oh, and if anyone wants to create a nice pnd creation utility, go ahead! The format is frozen. (well, maybe a small pxml change, but nothing that should break anything for now)
Last edited by a moderator:
Rockthesmurf
Advanced Member
Great job Jeff (and everyone else who has been working on this)!
Steve
Steve
skeezix
Internal Development
At a real keyboard this time.. I write way too much crap over a phone
(and I prolly write way too much crap in general; less chat and I'd have more time to code and design right? hmm.. my poor fried mind
It may not be obvious, but one thing I've been trying to avoid is 'over doing it'; ie: adding zip support, cramfs, and so on is all well and good .. but best avoided unless needed. (and being open source, someone might want to do it just for fun, but it is often better to _not_ do something and keep it simple.) There has in fact been lots of junk tossed in there, and then removed entirely -- less to get underfoot is better sometimes. ie: Less supported formats is easier for people to use, and means less unused code lieing around to bug-up. (Theres defiantely some minor memleaks and a few bugs (mostly bugs of omission) in there, but I'm just saying.. if we supported the half dozen file formats we went through, only one shoudl end up being used for 90%, and the rest would just be code 'in the way'.)
(You can see evolution in the design of course, with how the separation of disco-t and pxml-t drives me nuts; further, the really ugly structs in place for containing the parsed pxml data, and really the pxml xml spec altogether needs huge changes, but tat'd just delay stuff and we don't need any of that!)
So I am trying to resist doing anything 'too much'. For something to catch on, it should be simple _first_, and then when it catches on, you grow it. If you make a spec far too big, no one will read it. "tl;dr" syndrome. This is how RSS and such got off the ground.. barely a spec at all, so easy to implement.. bam. But you throw a big fat spec out, and _flop_.
So this is why I didn't define uninstall and other things; I don't even mention it. It is both best left out for now, a limit of the manpower we've had, and something that can be discussed and managed down the road. If I go and bring it up, I encourage more discussion, when really I want pnd's to ideally not even need an uninstall in the normal case.. It is not me being short-sighted, it is me deliberately trying to manage things so that they can be crafted better in the future. (You may disagree, and thats fine. Someone else can take it over once it ships, I don't care So worry later, rather than worry now. The same as the great coding practice (from Kjnuth in fact) of .. optimize later, never early. Optimization and over-speccing means necessarily complicating things up front when you need it not complicated. LAter when you see how it is used in practice, you will know what needs optimization. (And certainly iso is fast and easy to work with, we had tonnes of performance issues when fiddling with other formas and layer. We've also had enough issues squeezing things into the kernels and such .. so many parts devved at the same time, gotta keep it simple.)
Anyway, this is why I'm a bit bullish about some things; it is better to keep small, and worry about growing it after we get something that works and shipped. I'm _very_ good at shipping things, which is rare. (That said, yes, I'd love to have more testing in this, and a dozen more hands, and free time from my little one to actually mull things over quietly once in awhile, but you make the best with what you have
So once we ship, I fully expect half of this stuff to get rewritten, and thats the way it should be. But not right now, it works pretty darned well so lets go with it
And definately keep up the ideas and log them; enhancements are good, and theres good folks in these forums, and dflem -- you definately seem like a good man on it when you can muster the time, so I expect to see you checking dstuff in down the road
jeff
(and I prolly write way too much crap in general; less chat and I'd have more time to code and design right? hmm.. my poor fried mind
It may not be obvious, but one thing I've been trying to avoid is 'over doing it'; ie: adding zip support, cramfs, and so on is all well and good .. but best avoided unless needed. (and being open source, someone might want to do it just for fun, but it is often better to _not_ do something and keep it simple.) There has in fact been lots of junk tossed in there, and then removed entirely -- less to get underfoot is better sometimes. ie: Less supported formats is easier for people to use, and means less unused code lieing around to bug-up. (Theres defiantely some minor memleaks and a few bugs (mostly bugs of omission) in there, but I'm just saying.. if we supported the half dozen file formats we went through, only one shoudl end up being used for 90%, and the rest would just be code 'in the way'.)
(You can see evolution in the design of course, with how the separation of disco-t and pxml-t drives me nuts; further, the really ugly structs in place for containing the parsed pxml data, and really the pxml xml spec altogether needs huge changes, but tat'd just delay stuff and we don't need any of that!)
So I am trying to resist doing anything 'too much'. For something to catch on, it should be simple _first_, and then when it catches on, you grow it. If you make a spec far too big, no one will read it. "tl;dr" syndrome. This is how RSS and such got off the ground.. barely a spec at all, so easy to implement.. bam. But you throw a big fat spec out, and _flop_.
So this is why I didn't define uninstall and other things; I don't even mention it. It is both best left out for now, a limit of the manpower we've had, and something that can be discussed and managed down the road. If I go and bring it up, I encourage more discussion, when really I want pnd's to ideally not even need an uninstall in the normal case.. It is not me being short-sighted, it is me deliberately trying to manage things so that they can be crafted better in the future. (You may disagree, and thats fine. Someone else can take it over once it ships, I don't care
So worry later, rather than worry now. The same as the great coding practice (from Kjnuth in fact) of .. optimize later, never early. Optimization and over-speccing means necessarily complicating things up front when you need it not complicated. LAter when you see how it is used in practice, you will know what needs optimization. (And certainly iso is fast and easy to work with, we had tonnes of performance issues when fiddling with other formas and layer. We've also had enough issues squeezing things into the kernels and such .. so many parts devved at the same time, gotta keep it simple.)Anyway, this is why I'm a bit bullish about some things; it is better to keep small, and worry about growing it after we get something that works and shipped. I'm _very_ good at shipping things, which is rare. (That said, yes, I'd love to have more testing in this, and a dozen more hands, and free time from my little one to actually mull things over quietly once in awhile, but you make the best with what you have
So once we ship, I fully expect half of this stuff to get rewritten, and thats the way it should be. But not right now, it works pretty darned well so lets go with it
And definately keep up the ideas and log them; enhancements are good, and theres good folks in these forums, and dflem -- you definately seem like a good man on it when you can muster the time, so I expect to see you checking dstuff in down the road
jeff
I did follow the development of it early on then stopped.
If there's something like this in place then sorry, but maybe a thread with the 1st post kept up to date with feature requests, what's already done etc, it would stop the same questions being asked all the time.
At least that way people can see that it's been asked, noted or even ruled out. By no means does a request mean it's in development or ever will be (unless stated otherwise of course)
Maybe you've already done something like that on the wiki.
If you have ignore my blabbering
If there's something like this in place then sorry, but maybe a thread with the 1st post kept up to date with feature requests, what's already done etc, it would stop the same questions being asked all the time.
At least that way people can see that it's been asked, noted or even ruled out. By no means does a request mean it's in development or ever will be (unless stated otherwise of course)
Maybe you've already done something like that on the wiki.
If you have ignore my blabbering
dflemstr
It's a ball.
Paraphrased:
Now don't get me wrong; in the state that the PND format is now, it's the wisest decision to just let it be for a while; it's just that I don't wish that this was the case.
I'm just hoping that, when the format is revised, it will not become even worse than it is now (with everyone adding extensions to the PXML etc...)
What bugs me the most at the moment and what probably will be annoying to a lot of other people is that while ISO is an OK format, there's really no library for writing an ISO file from scratch, or modifying it etc. Yeah, sure, there's mkisofs and the iso9660 driver itself but those are specialized tools and not something that are accessible in a {secure,chrooted,SELinux-ed,Windows,web server} environment. There's not something like "libiso9660" that you can use to create actual ISO images; you'll always have to do 'system("mkisofs ...");" in your code. That, I think, and also that ISO isn't a popular general purpose storage format, will depopularize the PND format, and that's why I say that supporting ZIP or some other common format should be top priority.
No wonder that DEB and RPM are so messy today, with that kind of perspectiveskeezix said:
Now don't get me wrong; in the state that the PND format is now, it's the wisest decision to just let it be for a while; it's just that I don't wish that this was the case.
I'm just hoping that, when the format is revised, it will not become even worse than it is now (with everyone adding extensions to the PXML etc...)
What bugs me the most at the moment and what probably will be annoying to a lot of other people is that while ISO is an OK format, there's really no library for writing an ISO file from scratch, or modifying it etc. Yeah, sure, there's mkisofs and the iso9660 driver itself but those are specialized tools and not something that are accessible in a {secure,chrooted,SELinux-ed,Windows,web server} environment. There's not something like "libiso9660" that you can use to create actual ISO images; you'll always have to do 'system("mkisofs ...");" in your code. That, I think, and also that ISO isn't a popular general purpose storage format, will depopularize the PND format, and that's why I say that supporting ZIP or some other common format should be top priority.
Last edited by a moderator:
skeezix
Internal Development
RPM just seems bad imho; deb is a bit better off, but their DB rules are a serious disaster and kludgefest.. now this of course is pragmatic and perhaps necessary and evolutionairy, buts its some nasty We are doomed to repeat it We could do a lot of fixes to PXML's xml spec and all that, but we just odn't havce calendar time anymore, and in the past we didn't have the energy I think.. and this is always the case. You can't know where you going until you look back and see what you've been right?
What stops you from getting the code to mkisofs and wrapping a lib around it? If it has a main, you can turn it into a lib, no? Perhaps its a lot of work, I've not looked into it. (Certainly you are correct that is is not so common as the standard zip libs floating around.) it also proved (at the time with that kernel and set of libs and so on) to perform poorly as well as be problematic. We've not tried it lately of course, but by definition it should be possible to support many kidns of pnds without annoying the user. Still, I think it wiser to let it sit by for awhile -- muddying things up sounds more harfmul than other things we could be doing right after launch. (At the same time, it is not really hard to bring into libpnd, since it has been there before, so feel free to work on it.. you can do whatever you like
For a web tool, actually invoking mkisofs .. thats not really a crime; its not like we don't have 10 years of internet history with webservers invoking CGIs (external tools) which in turn invoke other external tools (ie: How much of thw web was perl scripts for the last 15 years?) How many photo CGIs werre using 'convert' to resize, rotate etc, either live or in a queue? How much video transcoding is handed off to the command line? (for that one, I can safely say nearly all of it
Its not such a kludge.
jeff
We are doomed to repeat it We could do a lot of fixes to PXML's xml spec and all that, but we just odn't havce calendar time anymore, and in the past we didn't have the energy I think.. and this is always the case. You can't know where you going until you look back and see what you've been right?What stops you from getting the code to mkisofs and wrapping a lib around it? If it has a main, you can turn it into a lib, no? Perhaps its a lot of work, I've not looked into it. (Certainly you are correct that is is not so common as the standard zip libs floating around.) it also proved (at the time with that kernel and set of libs and so on) to perform poorly as well as be problematic. We've not tried it lately of course, but by definition it should be possible to support many kidns of pnds without annoying the user. Still, I think it wiser to let it sit by for awhile -- muddying things up sounds more harfmul than other things we could be doing right after launch. (At the same time, it is not really hard to bring into libpnd, since it has been there before, so feel free to work on it.. you can do whatever you like
For a web tool, actually invoking mkisofs .. thats not really a crime; its not like we don't have 10 years of internet history with webservers invoking CGIs (external tools) which in turn invoke other external tools (ie: How much of thw web was perl scripts for the last 15 years?) How many photo CGIs werre using 'convert' to resize, rotate etc, either live or in a queue? How much video transcoding is handed off to the command line? (for that one, I can safely say nearly all of it
Its not such a kludge.
jeff
dflemstr
It's a ball.
As always, I have the role of the pessimist and perfectionist.
You're right, it's not that bad, I've seen worse (The original MS Office Word .doc format for instance... brr... (I'm lucky that there are libs for reading that (On Linux/without having Word/in server environments (because NO, most hosts and companies DON'T allow CGI in any shape or form (since that binds the webapp to one specific platform (lambda (arg) (concatenate 'string arg too-many-parens)))) now))
You're right, it's not that bad, I've seen worse (The original MS Office Word .doc format for instance... brr... (I'm lucky that there are libs for reading that (On Linux/without having Word/in server environments (because NO, most hosts and companies DON'T allow CGI in any shape or form (since that binds the webapp to one specific platform (lambda (arg) (concatenate 'string arg too-many-parens)))) now))
