Rethinking PNDs and standard packaging approach

Discussion in 'Pyra OS (Debian GNU/Linux)' started by Wally, Jul 16, 2014.

  1. bzar

    bzar A Commando

    Joined:
    Sep 22, 2008
    Messages:
    4,444
    Location:
    Finland
    I don't mean purposefully relying on them, but only ever testing with versions with those effects. This is one case where neither party knowingly does any wrong, but the end result is a dysfunctional piece of software.
    I actually do agree with you. I'm just pointing out it's not as clear cut as I interpreted you made it out to be.
    I'll just head back under the bridge now :p
     
    levi likes this.
  2. thatgui

    thatgui Advanced Member

    Joined:
    Apr 2, 2009
    Messages:
    2,901
    You still need to show a reason why this should not happen, in reference to keeping a container format where portability and independency from the basic system are essential parameters.
     
  3. _wb_

    _wb_ Microbe Staff Member

    Joined:
    Apr 5, 2012
    Messages:
    5,387
    Location:
    Brussels, Belgium
    Here are some reasons:

    1. Smaller packages -- so less to download, less space needed on your SD cards.

    2. Automatic updates of all dependencies -- this would almost certainly not be the case if the updates give only minor improvements

    3. Better forward compatibility possible -- e.g. if your package depends on libsdl1.2 but instead of just having the dependency in metadata, it includes or is statically linked with a particular version of it, then chances are slim that it will work (well) on different/future hardware.

    4. More freedom in the required base system -- there does not need to be a fixed, non-mutable base system that we have to design now and never ever change afterwards. Also users can install and remove whatever they want without breaking anything.

    5. Probably easier to get things to work in non-standard distributions -- even if those other distros are not apt-based, at least the explicit dependencies will make it easier to figure out what is missing.
     
  4. thatgui

    thatgui Advanced Member

    Joined:
    Apr 2, 2009
    Messages:
    2,901
    While saving space is always nice, I am for giving it a low priority in that particular discussion: space only gets bigger and or cheaper, the download & "install" experience may be improved, but most of the libraries we are talking about here are in the kB range, assets are probably eating most of the bandwith
    This is indeed usefull, but again I'm asking to focus on the expected cases: Games. If speed is of importance in that matter, the author / maintainer is inclined to update it anyway. So whats the gain in having a new version other then that its newer ?
    I guess we can expect a basic set of libs just to be there, preferable only those that have to be specifically adapted anyway.
    There seems to be a vital flaw in my understanding of the situation: Wouldn't including more libraries into the pnd itself be actually more beneficial in order to achieve that ?
    I currently inclined to make a "broader" statement about the topic at hand:

    I'm just fearing that making things too complicated and altering "external" components (like the package management system of debian) will result in system that is less reliable and harder to maintain as there is no guarantee that someone will step up to keep things going. The current implementation of the pnd system is quite robust (but not ideal of course), because its a rather simple implementation (not the design, just the implementation)
     
    klapse likes this.
  5. _wb_

    _wb_ Microbe Staff Member

    Joined:
    Apr 5, 2012
    Messages:
    5,387
    Location:
    Brussels, Belgium
    Space does not necessarily only get bigger and cheaper. The Pyra will have mobile data (3G), which tends to have less speed and be more expensive than wifi.
    Games are not the only expected cases. I expect the Pyra to be useful for much more than just games.
    But even for games: if it already runs full-speed, would a maintainer still care about a speed increase in a library? Probably not. But that speed increase will probably still be useful since it translates to a cpu that is more idle and thus consumes less power.

    And how do you determine which libs are considered "basic" and have to be "specifically adapted"? These are not at all clear-cut notions.
    No, not necessarily. You would put a version of the dependency in the PND that was compiled for Debian, and that may or may not work out of the box on a different distro.
    We don't need to alter external components: we can use the debian package manager as is. The change would be in the PND install/run system, and it would not be particularly hard to implement. Just a few calls to dpkg, essentially.
    We can of course just keep doing what we did before: have just an "OS version" field in the PXML, which is ignored by everyone and everything, and try to minimize base system changes to avoid things breaking. And if something breaks anyway, then let it break ungracefully -- the /tmp/pndrun.out might contains some hints on how to downgrade your system to make it work again. We can just do that again. Have a spec that basically says "it works if it works", where everything is based on implicit assumptions and the only way to figure out how much will break if the pre-installed system is modified, is to try it out.

    As far as I'm concerned, the goal here is to come up with a new PND system that is robust and flexible in the long term. A field in the meta-data for dependencies (with a default corresponding to the Pandora base system) seems to me to be an essential part of any such system. Package authors can choose for themselves whether they want to use this feature or not -- they can just keep the field empty and make a self-contained package.
     
    Last edited by a moderator: Jul 24, 2014
  6. thatgui

    thatgui Advanced Member

    Joined:
    Apr 2, 2009
    Messages:
    2,901
    Are there many libraries that are big enough to add to the size significantly in comparison to other parts of the package, Qt comes to my mind, but anything else ?
    Of course the Pyra will be usefull for much more than games (and if things are working out as they look like, gameing will be thing I will do the least on the Pyra). But I have my doubts that there will be much besides games that will see a release as pnd.
    How often do speed increases of significance are introduced by changing the code of the library only ?
    Everything that needs adjustments to make better use of pyras specific hardware and are deemed to have a more broader use, the only thing that comes to my mind is SDL. Good starting point to evaluate that further would be the current repo.
    You obviously have more knowledge and experience in that field, so I can't really argue against that. I can only ask (more out of curiousity) are there cases where libraries need adjustments while the binary that depends on them does not need them ?
    Ah okay, I thought the changes would affect "standard" components of the system. Thanks for clearing that up
     
    Last edited by a moderator: Jul 25, 2014
  7. _wb_

    _wb_ Microbe Staff Member

    Joined:
    Apr 5, 2012
    Messages:
    5,387
    Location:
    Brussels, Belgium
    For example: GTK, ImageMagick, Java, Webkit, GHC.


    But I'm mostly worried about dependencies that are not libraries, e.g. LaTeX, Lilypond, Gimp, LibreOffice, most interpreted programming languages, etc.

    It is true that it is mostly games that will not already be present in Debian and/or that are big enough (because of data files) to not install it on the internal storage.


    But I can imagine the development of Pyra-specific applications that would be released as a PND. For example a GPS application that comes bundled with lots of maps, and which gives you an option to export views to PDF (maybe depending on LaTeX to do that) and PNG (maybe depending on ImageMagick to do that). Or for example a music application that comes with lots of instrument samples, which uses LilyPond to nicely engrave sheet music.

    Lots of applications spend most of their time in executing library calls. A typical game will spend most of its time in calls to SDL or some other graphics library, music and sound libraries, and maybe some physics engine or something like that. During loading they will typically spend time in decoding libraries like libpng.

    Define "needs adjustments" and "have a more broader use". Nearly everything could be adjusted to make better use of Pyra's specific hardware and nearly nothing needs to be adjusted (since it would also work as is, just slower). Almost by definition, libraries "have a broader use", because that's exactly the point of a library: to be used in more than one application. Of course some of them are more popular than others, and some of them are closer to the root of the dependency tree than others, but these things change over time and there is no clear-cut notion of that.

    Also the current repo is not a good starting point to evaluate the importance of libraries on the Pandora, because there is no easy way to tell which libraries are actually used by some PND. We don't have a "depends on:" field yet, so the only way to check is to run ldd on the binaries, inspecting which libraries are included in the PND (although many PNDs include more stuff than what they actually use), and hoping that the binaries don't do any dynamic loading of libraries (or hoping you can make sense of the dlopen() calls). Also, some PNDs depend on other PNDs using some shell script magic, but there's no general way to figure that out.

    Sure. In a lot of cases the library offers some kind of abstraction to the application that uses it, so the application does not have to worry about hardware-specific or distro-specific stuff. When compiling the library, you would have to let it detect (or specify manually) where everything is located, what the conventions are, maybe which backend to use if there are alternatives, which optional functionality to enable (perhaps needing additional dependencies), etc. A lot of that configuration may be distro-specific. The applications themselves typically don't have that much distro-specific configuration: they're mostly happy when they can find their dependencies.
     
  8. Silent-Hunter

    Silent-Hunter Hardcore Member

    Joined:
    May 29, 2010
    Messages:
    3,005
    Some things will be more convenient to install in the onboard flash, like I want COBOL support, because I'm learning COBOL. The best way to have things that use command line like that is to install on NAND/eMMC/whatever.
     
  9. thatgui

    thatgui Advanced Member

    Joined:
    Apr 2, 2009
    Messages:
    2,901
    Fair point , but many of them are needed to be preinstalled already (like GTK to use XFCE). And most of these examples apply to applications. But I specifically asked and you specifically answered...
    Why is a deb package no option in that case? Its a far more tried and tested mechanism, that also provides complete integration into the system, without any additonal effort from the pryas system dev team side
    In my personal book, thats more a bad design decision - problem (to bundle the binary with asset data), than a regular case we need to handle (and will counter your own stance of wanting to reduce size of pnds where possible).
    Of course everything can be adjusted to fit some scheme to counter an argument. But that is not the point:The pyra will deliver with certain amount of software preinstalled, this already determins the basis (like GTK). Going on from that, there will be some libraries that would "make sense" to include, like "SDL", which would be a case of "needs adjustmets to make full use of its potential. Of course this is a "cherry pick" process, as there is no universal sorting algorithm and this process will happen anyway independant of what mechanisms are included in the new pnd format/system.

    So its usually more of "set path" problem then anything else ?
     
  10. _wb_

    _wb_ Microbe Staff Member

    Joined:
    Apr 5, 2012
    Messages:
    5,387
    Location:
    Brussels, Belgium
    Maybe GTK will be preinstalled, but what if someone decides he likes KDE more and removes GTK? I want such things to be possible, without causing the PNDs to just crash. I want them to say: "This PND requires GTK, do you want to install it?".

    Yes, that is a perfectly fine option if you can assume that everyone who wants the application always wants to install it on the internal storage.
    Good point. Still, it makes sense to store the binaries at the same physical place as the assets, because one is useless without the other. So sure, you could distribute the GPS application as a smallish .deb package to be installed on the internal storage plus a large data pack to be installed anywhere, and that would be a reasonable workaround. But it would be nicer to be able to put the data either in a PND together with the binary, or in the appdata folder of the PND.
    Yes, we already established that doing the same thing that was done for the Pandora can be done again. I'm arguing that this "cherry picking" process is not ideal since it unnecessarily forces everyone to have the same "cherries" or at least to manually try to keep track of which "cherries" he can remove (and then when he downloads a new PND try to figure out why it isn't working).
    Compiled-in paths are one aspect of the problem, but also compile-time options like which back-end libraries it uses and which functionality is included (sometimes some not-so-frequently-used features are disabled by default).If it was just a question of paths then you could probably make everything work with some environment variables and/or symlinks, but there are other potential problems.

    Anyway, for FOSS stuff I think it makes most sense to submit sources to a build server like what Cloudef is proposing, and then automatically get whatever we want out of that: light PNDs (with dependencies not included in the package itself, but mentioned in the metadata), heavy PNDs (stand-alone, all except the basic dependencies included), fat PNDs (work out-of-the-box on both Pandora and Pyra), Pandora PNDs, debian packages, x86 debian packages, etc.
     
  11. thatgui

    thatgui Advanced Member

    Joined:
    Apr 2, 2009
    Messages:
    2,901
    hmm, that makes a whole lot of sense, especially in context with the automated build server clouddef is proposing.
    While there is always the possibility that someone does the unexpected, how likely is it ?
    Argh, no, if its data that is likely to change, never, ever put it in same place. Picture Viewers/Video Players/Emulators/etc. are all useless without some form of "data", still usually you don't stuff this into the binary path (or one of its subdirectories). This even makes sense for some games, especially if they allow custom asset (like levels) creation.I define the appdata directory as a mobile/portable extension of the users home dir, so I am also against putting big stuff there, as this makes syncing/backing up userdata a lot more complicated/resource intensive without any real reason I could think of.
     
  12. pmprog

    pmprog Advanced Member

    Joined:
    Apr 25, 2011
    Messages:
    3,767
    Hmmm, on my desktop I always prefer keeping them together. ie,

    G:\Amiga

    G:\Amiga\Roms

    G:\Amiga\ADFs

    G:\C64

    G:\C64\Games

    However, on the Pandora, I really hate having to navigate the whole /media/Applications/pandora/appdata/<emulator/ path

    If I could create a folder in the root of my SD card called "C64" and have the VICE.PND (or PND-alternative) along with all the D64 files, that'd be great.
     
  13. thatgui

    thatgui Advanced Member

    Joined:
    Apr 2, 2009
    Messages:
    2,901
    Don't know if this is comparable as (depending on your age) you might be trained by MS that there is no difference between binaries and data. The concept of having a directory for user specific data was introduced rather late in the Windows world. Your problem could easily solved with an emulator that remembers either the last used dir or a "standard" path to look for games (which do most emus on the Pandora, afaik).
     
  14. pmprog

    pmprog Advanced Member

    Joined:
    Apr 25, 2011
    Messages:
    3,767
    I don't know why, but this made me chuckle. Is Windows NT 3.1 late in the Windows world? Anyway...
    Given the "portable" nature of the PND(-replacement); it makes sense that you can grab the directory that has the program and all it's related data from one place and drop it into another (SD card for example).
     
  15. thatgui

    thatgui Advanced Member

    Joined:
    Apr 2, 2009
    Messages:
    2,901
    It is not about system mechanisms to prevent you from puttings things were you think they belong, you can do this freely in any Linux system (you have elevated rights on) too, its about how users are "trained" to where stuff belongs: Like parents telling their kid in which drawer socks, underwear, trousers belong. Sure the kid can put them where he wants them to, but he can't expect that after the next loundry his parents will have adapted to the arrangement he thinks is best, but how they think its best.Since the early days of DOS/Windows seperating user data and system data is a complete mess that MS iterates over again with every new version released, which gets really annoying in the latest versions (like 7/8 ) with a whole bunch of virtual directories (with additional localized versions) that try to consolidate the mess they created over the years.

    So the ideal situation would be that you (letting legal questions aside for a moment) can download emulator x and all the games for that specific system are included, it would be ultra portable right ? What if emulator x provides overall the best ui/emu performance but has a flaw in its emulation routines that makes some games not enjoyable, while emulator y provides a lesser overall experience but lets you play these particular game without problems ? Should every game be packaged into that pnd too ? Or in your case, would you copy your library in both appdata dirs, just for convenience ?
    Where is the big effort in just copying a second directory ? which would be also more convenient, if you just want to copy the games itself as you don't have to navigate through the whole appdata tree to find the games.

    Or if you just want to sync your save states and emulator configs with another device (like I do), without wanting the additional "bloat" of having to sync your whole game library too ?
     
    Last edited by a moderator: Jul 30, 2014
  16. pmprog

    pmprog Advanced Member

    Joined:
    Apr 25, 2011
    Messages:
    3,767
    Sure.
    That depends. The closest I've got to this scenario is Street Fighter III - where I had to move the games from MAME to FBA; because it ran slooow in MAME. But in such case, I wouldn't leave SFIII in MAME, what's the point if it doesn't play it properly.
    Alternatively, you could have both emulators in the same directory (Oh, I can already hear the cries of indignation). I would love to get rid of the /pandora/* tree on the SD card, and it just find PNDs anywhere on the card - and use the PND's current directory as the "AppData" path. But there we go...
     
  17. slaeshjag

    slaeshjag ¯\_(ツ)_/¯

    Joined:
    Apr 8, 2010
    Messages:
    2,687
    Location:
    ~Stockholm, Sweden
    There's a number of reasons not to do that. A major one being that it'd need to watch every directory on every SD-card, which will add significant overhead if you got a deep directory structure. And then the "PND"-system would get notified of every file change/creation/deletion on that SD-card, and be expected to react to that. 
     
  18. pmprog

    pmprog Advanced Member

    Joined:
    Apr 25, 2011
    Messages:
    3,767
    Yeah, I understand why it's not like that, but doesn't mean I still wouldn't prefer it like that :)
     
    Last edited by a moderator: Jul 30, 2014
  19. thatgui

    thatgui Advanced Member

    Joined:
    Apr 2, 2009
    Messages:
    2,901
    So you would rather clutter your games over several directories than have them conveniently placed in one directory ? What if a new emulator comes up promising that it can do better than both before him, time to start digging through the file system ? Or there is a new version you hesitate to install because of possible drawbacks (some user may reported) ? Or you want to move to a new (e.g. bigger) SD-Card, but don't want to drag all the old crap that accumulated there (like several appdata directories of pnds you don't use anymore), time to start digging again ?This list goes on and on and on. Why have all these drawbacks? Just because of an old habit ?

    which can get tricky if there are file name conflicts (like using different version of an emulator).
    This was orginially planned for the pnd system too, but scraped because of overhead it produced while scanning the whole directory tree. You would also need another mechanism to decide where to place the "link" to the binary in the UI, which is also problematic as not every UI that will be possible provides the same mechanism for the user to customize the placement - one idea that quickly came to my mind was to just place an entry in the "start" menu and then let the user make a copy of that on the desktop if he wants it to appear there too. But thats not easy to do with XFCE and you would have to face the problem that there will be dead links as the pnd deamon would not know anything about copies.Edit: should update the thread before posting, so somewhat ninjad
     
    Last edited by a moderator: Jul 30, 2014
  20. pmprog

    pmprog Advanced Member

    Joined:
    Apr 25, 2011
    Messages:
    3,767
    Not just habit, but preference. I can see we're not going to see eye-to-eye on this, and that's fine.
    Yep, but I've yet to encounter it...
    Yep, as mentioned above, I understand why it's not done.
     

Share This Page

Loading...