Confliting requirements

[_wb_]

I don't propose to merge .deb and .pnd or whatever it is you think I propose.

I merely propose to replace the "OS version" field in the PXML specification (a field that is now simply not used and usually ignored) with a more useful "Depends on:" field, which lists the base system libraries and tools the PND (or .pyra, because for other reasons we may want to define a new file format too) uses. The best way to represent those dependencies is to do it in the same way as .deb packages do it (assuming our base system is Debian or Debian-based).

 

[vinipsmaker]

I don't propose to merge .deb and .pnd or whatever it is you think I propose.

I merely propose to replace the "OS version" field in the PXML specification (a field that is now simply not used and usually ignored) with a more useful "Depends on:" field, which lists the base system libraries and tools the PND (or .pyra, because for other reasons we may want to define a new file format too) uses. The best way to represent those dependencies is to do it in the same way as .deb packages do it (assuming our base system is Debian or Debian-based).

Put software on SD card (this is what happens in PND) and put dependencies also (this is what happens in .deb). Merging.

What do you think to replicate apt-get to install software on SD-card? We need to focus on solutions or this talk will go nowhere.

 

[EvilDragon]

Well, that's why suggested to have as many standard libraries as possible into the OS.

Have a meta package for the base installation for the Pyra.

And whenever there's a library that's needed for a new PND that could also be useful for other devs, update the meta-package and the next OS update will include it.

Sure, the dependency lists in the PNDs are a good idea as well and would have a similar result.

 

[vinipsmaker]

Well, that's why suggested to have as many standard libraries as possible into the OS.

Have a meta package for the base installation for the Pyra.

And whenever there's a library that's needed for a new PND that could also be useful for other devs, update the meta-package and the next OS update will include it.

I like this solution.

Sure, the dependency lists in the PNDs are a good idea as well and would have a similar result.

It's too fragile. Just use .deb packages.

 

[_wb_]

I would keep the forced common base system relatively small, instead of letting it grow and grow (with never a possibility to shrink or to drop old stuff in favor of a non-backwards compatible newer version). If someone wants to have a Pyra without Python or Java for some reason, why not let him. Of course, as soon as he installs a PND that depends on Python, Python will get installed, and that makes sense.

Having explicit dependencies in PNDs is easy to implement. If the field is missing, it defaults to a big list of everything that is currently in SZ. That should help to keep Pandora PNDs working on the Pyra. If the field is not missing, the Pandora PND installer/launcher just gives up and says "This a Pyra PND, it does not work on Pandora", while the Pyra PND installer/launcher essentially does a "sudo apt-get install [deplist]".

 

[vinipsmaker]

Having explicit dependencies in PNDs is easy to implement. [...]

It's easy to implement and it's easy to break the system.

 

[ShleeDragon]

Couldn't we have library snapshots - Compile your application with the pyra2014-01 toolchain, and it'll work with that... six months later, we can recompile the newer libraries as pyra2014-07 and people can recompile their .PYRA for that snapshot..

 

[vinipsmaker]

Couldn't we have library snapshots - Compile your application with the pyra2014-01 toolchain, and it'll work with that... six months later, we can recompile the newer libraries as pyra2014-07 and people can recompile their .PYRA for that snapshot..

The discussion in the other topic was way heater than here. Anyway, looks like the people wanting dependencies found a solution to make it work in the PND world.

When the code for makepnd becomes open, I'll contribute to make it support flat pnd and light pnds. We could have both on the repo and let the user choose. And the idea to support a minimal set of installed packages for the foresable future will together make everyone happy.

Only requirement is a nice person with commit access to core tools that will accept the patches.

 

[_wb_]

When the code for makepnd becomes open, I'll contribute to make it support flat pnd and light pnds. We could have both on the repo and let the user choose.

This is a very good idea, and I completely support it.

 

[vinipsmaker]

When the code for makepnd becomes open, I'll contribute to make it support flat pnd and light pnds. We could have both on the repo and let the user choose.

 
This is a very good idea, and I completely support it.

Click on the "Like This" button to show your support in this facebook era (I'm just kidding, hate facebook).


It would be good to read an official "I approve" comment from EvilDragon anyway.

 

[Rockthesmurf]

It would be interesting to know a break down from the current Pandora repo, what % of PNDs run into problems of being 'excessively large' due to packing up lots of large dependencies. I can't help but get the feeling these are actually the minority case (by a huge margin). I appreciate some software is difficult to package up due to massive dependencies, but from reading the boards, I get the impression this is only an issue for a very small % of people/applications.

I think I prefer the idea of having a 'Compiled for your convenience' page, where libraries are prebuilt into static libraries and made available for everyone to use. If you are writing some software that requires libpng, zlib, imagemagick and libcurl then just grab the latest packages from the prebuilt page, statically link, and you are good to go. You get a PND file out the other end, it is self contained, doesn't require dependencies managers, doesn't require any strange mounting, you just insert your SD, the OS sees the PND and lets you run it.

I think I prefer the idea of having dependency PNDs or something, like a QT5 PND, you have that on one of your SDs, then applications that require QT5 can run. If you don't have an SD with QT5 mounted when you launch an application that requires this it just specifies that this is missing and needs installing. Seems like quite a straight forward error that any user would be able to resolve. Right now how many cases of these are there? Unless the dependencies are using up some reasonable chunk of space does anyone really care?

I come mainly from a commercial handheld/console background, and am used to having minimal dynamically linked stuff, and mainly trying to stick to a executable with as much statically linked as possible. From this background I have seen many cases where 50-100MB executables can be stripped down to 5-10MB with the right compile/link options. I wonder how much of this side of 'optimization' has been looked into? If we are creating PND's with 100's of MB's of dependency libraries I have to question what is going on. How on earth can there be that much compiled code? What is the break up of that code (text, data, debug information, etc.)?

 

[vinipsmaker]

It would be interesting to know a break down from the current Pandora repo, what % of PNDs run into problems of being 'excessively large' due to packing up lots of large dependencies. I can't help but get the feeling these are actually the minority case (by a huge margin). I appreciate some software is difficult to package up due to massive dependencies, but from reading the boards, I get the impression this is only an issue for a very small % of people/applications.

Yeah, most of the people here "wanting" deps don't even know what they are wishing for. The proof: Just read their arguments to offline installation and you'll see that most of them come with a non-working solution. Before I was suggesting them to just use .debs, because their solution don't work as offline installation and they would need to be connected anyway, but after so much heat someone pointed a solution that actually works.

Anyway, your argument is statistical. This is already a bad thing to begin with, but moving on... we don't have the source scripts to build the PNDs in the Pandora repo, then we cannot say shit about how useful or unuseful something is.

And maybe my argument might confuse you guys about my opinion, then I'll reaffirm it: I like the light PNDs + flat PNDs suggestion.

I think I prefer the idea of having a 'Compiled for your convenience' page, where libraries are prebuilt into static libraries and made available for everyone to use. If you are writing some software that requires libpng, zlib, imagemagick and libcurl then just grab the latest packages from the prebuilt page, statically link, and you are good to go. You get a PND file out the other end, it is self contained, doesn't require dependencies managers, doesn't require any strange mounting, you just insert your SD, the OS sees the PND and lets you run it.

makepnd can solve this and most of the pain-related issues. In fact, your proposal reminds me the nightmare weekends when I was still using Windows. New version is out, then you need to download manually.

With ArchLinux's makepkg (original code for Cloudef's makepnd), you won't bother with the trouble to go to the site, download the new version... it's all automatized.

I think I prefer the idea of having dependency PNDs or something, like a QT5 PND, you have that on one of your SDs, then applications that require QT5 can run. If you don't have an SD with QT5 mounted when you launch an application that requires this it just specifies that this is missing and needs installing. Seems like quite a straight forward error that any user would be able to resolve. Right now how many cases of these are there? Unless the dependencies are using up some reasonable chunk of space does anyone really care?

Replicate apt-get to work with removable devices or give the nightmare of dependency management to the user?

Difficult choice, but I still would prefer replicate apt-get, because of the superior quality. But I don't like this solution, because it's too fragile (transfer the PND to your friend's SD card and system/app doesn't work).

I come mainly from a commercial handheld/console background, and am used to having minimal dynamically linked stuff, and mainly trying to stick to a executable with as much statically linked as possible. From this background I have seen many cases where 50-100MB executables can be stripped down to 5-10MB with the right compile/link options. I wonder how much of this side of 'optimization' has been looked into? If we are creating PND's with 100's of MB's of dependency libraries I have to question what is going on. How on earth can there be that much compiled code? What is the break up of that code (text, data, debug information, etc.)?

I don't see a game of such size as a problem. If a software needs that much space, then might be because the software just need such space.

 

[levi]

I can see the argument for just building stuff against the standard libs, but I still think that's a slightly luddite suggestion, if you'll excuse the term. Sure, copying the pnd to your friends system won't work out of the box, but with proper dependencies it'll tell you why and offer to fix if for you.


You may be right that not many PNDs are artificially inflated due to including updated system libs, but I know at least some are, and I suspect more people would use the functionality in newer libs to create better software if they knew it could be installed for them automatically. Also note, we could have PNDManager in the firmware on Pandora (and not have to wait for .next) if QT could be updated in situ where needed.


But I'm not completely convinced that dependencies are needed for PNDs. It kind of worked out for Pandora, and people will and have included libraries where needed; who cares if the PND get a little bigger? And different people having different libs depending on what they've installed does make it harder to debug problems people have, and more work for those building and testing a firmware image for Pyra. That's probably less of an issue than it was with Pandora due to things like the RasbPi testing the water for us, but it's a valid concern still IMO.

 

[Cloudef]

When the code for makepnd becomes open, I'll contribute to make it support flat pnd and light pnds. We could have both on the repo and let the user choose.

 
This is a very good idea, and I completely support it.

https://github.com/Cloudef/makepnd


I already have test dev env also as well that I've been sharing on IRC. So far B-ZaR has been using it and been very pleased.


When I release PUR (AUR Clone), I'll release the env here as well and maybe do opkg feed for updates.


Theoretically you could do cross-compile env from the the dev env, but most likely many recipes will fail that way, so I suggest using distcc and fast hdd or SD (or tmpfs) instead, this way you can acquire near cross-compilation speed (of course distcc server will have cross-compiler, linking and configure is ran on pandora).


I'll make guides, best practices and standards for packaging when this is done, so no worry there though.


As for PUR, its code will show up eventually on my github as well, but here is sneak peek:

(python3 + bottle.py)

As for light pnds, you could acquire that by making another pnd without running the 'copy libs found by ldd and not in default OS' logic.


However some recipes may have libs that the logic can't detect in pnd_post_package() function (same so name, but newer minor version. Or lib compiled with hacks/optimizations whatever)


To fix this, I could create pnd_copy_libs() function which won't be run for light pnds, and recipe creators should put the lib copy there if needed..


Edit:

For light pnds I could just do cleanup post-trigger that cleans all so files in LD_PRELOAD path from PND actually..

No changes needed.

I come mainly from a commercial handheld/console background, and am used to having minimal dynamically linked stuff, and mainly trying to stick to a executable with as much statically linked as possible. From this background I have seen many cases where 50-100MB executables can be stripped down to 5-10MB with the right compile/link options. I wonder how much of this side of 'optimization' has been looked into? If we are creating PND's with 100's of MB's of dependency libraries I have to question what is going on. How on earth can there be that much compiled code? What is the break up of that code (text, data, debug information, etc.)?

GLhck is really static compile friendly, so some glhck games already out there for pandora (spacerocks, qb (eh?)) are mostly static compiled.


With makepnd you can also do ipk packages, and I should probably setup perhaps some git repo where we can share the recipes. Those ipks would contain both dynamic and static libs.


Currently I think makepnd strips .a files from package.. (Though it's user configurable switch, maybe this should be disabled when building ipks)

http://cloudef.eu/pandora/dchrt/ipk/ <- some of these are built with makepnd, those that are not are manually packaged and thus poor quality (not correct conflicts, depencies or provides in metadata etc..)

 

[comradekingu]

So i have 10GB / on my machine, used just over half. I dont think ive ever gone past 15 on any machine, as i typically go with 15 and then a 10 for a second os if i want to tinker, but it was a little less this time on a 120GB SSD.

The rest is on my /home, which is a trivial change to mount and transfer over to an SD card. No problem using up the space there, and so the equivalent is transferring that same data from one SD to another bigger one, possibly making a mdadm array if you have only two smaller SD cards and are running out.

What I dont understand is why a solution must be made for those that require more than i do on this desktop machine.

Is the plan to have less than 16GB not extendable nand space or something?

Or is it a slightly taller fence garden where one doesnt contribute back to debian? I think the attention given by sharing is huge and very much a selling point for pyra. Debian is gigantic compared to pyra, and to have it as a sort of approved platform means alot. Other such devices have been nslu2, eee-pc, n900 and n950 and many more to varying degrees. Not many generic user desktop products ship debian. Sure steam is shipping on debian, but they contribute zero games back, i think this is a slot to fill for pyra.

Edit: what is "pnd packaging standards"?  I certainly hope it will include lisence as a required field with source if provided in a standard place with dependencies stated. I think i got shot down by craig when i said that was a good idea to have when the pnd format was made.

 

[vinipsmaker]

Or is it a slightly taller fence garden where one doesnt contribute back to debian? I think the attention given by sharing is huge and very much a selling point for pyra. Debian is gigantic compared to pyra, and to have it as a sort of approved platform means alot. Other such devices have been nslu2, eee-pc, n900 and n950 and many more to varying degrees. Not many generic user desktop products ship debian. Sure steam is shipping on debian, but they contribute zero games back, i think this is a slot to fill for pyra.

See http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412/URL]

 

[comradekingu]

Having the card you use now, work in debian, is an ugly fix, it will work, but its no solution.

Getting software to work the way it should, without the issues pnds lead to, is a better solution long term.

Certainly there is a lot of overhead in packaging new libraries every time, so overall debs is much easier on overall space. 60GB and no export of /home and /var is a cornercase.

I think regular users, like løra, will be much more at ease with a package manager that works and thats as far as you have to think about it. Pnd manager is a wonderful front-end, were it isnt so good is where aptitude and apt-get, even dpkg alone works wonders.

Pandora specific config should also be sent upstream, much like eeepc_laptop is for all the acpi speciality stuff that goes on there.

Edit: how about http://pandorawiki.org/Extend_Utils  or a chroot, or any solution besides complexity and trouble for everyone. I guess this debate is partly to do with http://boards.openpandora.org/topic/15651-poll-emmc-vs-usd/

Its possible to get ext support on windows, and moving sd cards manually to begin with, is not what the one regular user who posted in here prefers.

EDIT: https://romanrm.net/mhddfs  seems like the perfect fit.

It's even more than that; if a certain drive runs out of free space in the middle of a write (suppose, you tried to create a very large file on it), the write process will not fail; mhddfs will simply transfer the already written data to another drive (which has more space available) and continue the write there. All this completely transparently for to the application which writes the file (it will not even know that anything happened).

Its FUSE, but still, seems like it does exactly what it needs to do.

 

[comradekingu]

What do you think about mdadm mhddfs or aufs.  The way i see it, running the pnds in chroot is the only feasible pyra solution that works out of the box. The right way to go about it is imo porting to deb, the sooner the better. If space runs out i dont think that is the fault of the package system, but rather a storage question.  Besides, debian can read fat32, ntfs, and exfat just fine, where you can put your deb files and then dpkg -i *.deb once inserted into the pandora.

 

[vinipsmaker]

EDIT: https://romanrm.net/mhddfs  seems like the perfect fit.

It's even more than that; if a certain drive runs out of free space in the middle of a write (suppose, you tried to create a very large file on it), the write process will not fail; mhddfs will simply transfer the already written data to another drive (which has more space available) and continue the write there. All this completely transparently for to the application which writes the file (it will not even know that anything happened).

Its FUSE, but still, seems like it does exactly what it needs to do.

mhddfs seems an excellent piece of software, complementing the type of solutions we had before (eg. lvm, mdadm, aufs...). I like its "interoperability" and anti-too-clever approach.

But software is more complicated. It'll have dependent files not known by mhddfs. Software A needs click.ogg and software B needs logo.png. What prevents mhddfs from put "Software A" and "logo.png" in one sd card and "Software B" and "click.ogg" in the other one, making the two softwares unusable when only one SD card is plugged? I think mhddfs will work better for the home partition, where you usually put personal media files that will continue working in separate.

What do you think about mdadm mhddfs or aufs.

Let's wait for current PND maintainers's answers.

I know aufs only in theory. Never used beyond small experiences in pre-built solutions.

The way i see it, running the pnds in chroot is the only feasible pyra solution that works out of the box. The right way to go about it is imo porting to deb, the sooner the better. If space runs out i dont think that is the fault of the package system, but rather a storage question.

Debian uses a good package management with dynamic linking, dependencies and orphan packages. It will waste little space and most of the software can be used via .deb packages.

Besides, debian can read fat32, ntfs, and exfat just fine, where you can put your deb files and then dpkg -i *.deb once inserted into the pandora.

Care to elaborate? I don't see clearly what you're implying.

https://github.com/Cloudef/makepnd

Congratz Cloudef.

And the PUR screenshot looks exciting.

 

[comradekingu]

I was implying the offline manual sd in-out installs are still possible for those who prefer it. That was one point of the ed post.

If we assume the internal storage is the fastest, so the OS is there, with mhddfs the first mount is prioritized, so you would need to run out of space at less than a debian install <2GB in any event for it to be an issue.

So the system runs fine, not ejecting the drive i was using, while i was using it, is a bit of a strange thing to do for someone technically apt enough to run out of nand space and use mhddfs, or read the tutorial if its included in the base os.

Besides, the hardcore storage option is running e-sata on the go

Can someone lay out a holistic approach to why pnds/supersaxxon/next/pyrafiles make sense? I see the goto quickfix, i dont see relying on it as anything sustainable.