Drop support for SuperZaxxon in new PND format

[EvilDragon]

Well, if you find something that offers the same features and is as easy to handle for the end user, sure, fire away.

We've thought a lot about that and that's the best we could find.

I have no idea how you would run an executable within a package without mounting it.

The missing source code doesn't even have anything to do with the PNDs - people can also release Debian packages without the sourcecode.

(Btw. The sourcecode isn't even needed - the person who offers the compiled package has to give you the source on request, but it doesn't even need to be released on the website).

What does AUFS have to do with FAT32 partitions...?

Rescanning of every PND can be fixed within libpnd, it just has to watch for file changes in a folder...

I have almost every existing PND on my card, but adding / removing a PND only takes a couple of seconds for me.

As mentioned, you're welcome to propose something better, but I have no clue how you would achieve that.

 

[comradekingu]

In this thread http://boards.openpandora.org/topic/15574-confliting-requirements/ Ive proposed a unified way to carry backwards compatibility, forwards, maintainability, deal with obsolesence, offline installs etc.

Its the same discussion, and should be merged with this one.

How about doing a union mount at which point you actually begin to desire the need for a solution, namely as you run out of space on the root /

Which with modular eMMC or uSD isnt a hard limit anyhow, so it could even be solved in hardware.

Modularity is good for its own set of additional reasons http://boards.openpandora.org/topic/15651-poll-updated-emmc-vs-usd-modular-emmc/

The real issue is making debs out of all those pnd files, even theoretical best case that would be manual.

Packages without source in a -src wouldnt be debian packages, that would be debian non-free packages.

Having sourcecode here and there is a system that cant be maintained the right way, no matter how much you stick to the plan.

The software as cartridges, dont touch the system model is broken, and it _always was_

Android has fragmentation galore, what they do atop the regular way of linux, is junk, they are google and still have problems doing it. Even cartridges shipped with extra processors inside them for each separate game is how it was done in the old days.

The world moves faster now, just adopt. Debian doesnt break its own package management, debian stable is rock solid, and its featureset explains away the need for trying to outdo it at what it does best.

https://wiki.debian.org/DebianStability

There is no need for repackaging regular packages, nor is there anything special about the pyra that needs customization, if you want to do that, do it in one file in the home, do one input layer config, someone has to do it the non sdl way _and_ make it non configurable, and even that can be solved without repackaging.

Its a standard display resolution, it will have capable hardware. Sure you could adorn unix-philosophy enough to carry optimizations in a package specific way, but thats mostly aimed at emulators.

The regular d-i can support the pyra, thats how it works, send patches upstream and you have device specific support. Thats what TI is doing for the hardware. Having things mainline is best for everyone.

 

[_jr_]

The missing source code doesn't even have anything to do with the PNDs - people can also release Debian packages without the sourcecode.


(Btw. The sourcecode isn't even needed - the person who offers the compiled package has to give you the source on request, but it doesn't even need to be released on the website).

This wasn't true the last time it was mentioned and is still simply wrong. Source redistribution is implicit in any copyleft type license. The GPL only makes that more explicit than others and offers some shortcuts for e.g. distribution on physical media (to overcome their size constraints; mostly irrelevant today). Look at the GPL or any FAQ if you don't believe me: redistribution by download requires equivalent access to the source. The only way to guarantee that in practice is to include source in the PND or offer a source repository in parallel to the PND repository (just like you are already doing with the firmware packages).

I agree with your other points, though The PND format is quite well thought out and works good enough in practice. Minor improvements are possible, but it is unlikely that reimplementing it from scratch will result in significantly a better system. In the end it depends on who will do the work for the Pyra. If nobody steps up to port the PND system it won't be there. And reinventing the wheel is typically a lot more fun.

 

[comradekingu]

Nobody is reimplementing anything from scratch, what i described was the opposite. I know debian is a whole lot of fun, because it works.

"Those who don't understand Unix are condemned to reinvent it, poorly." – Henry Spencer

 

[_jr_]

The problem with your proposal is, that a lot of us want the software as a cartridge model. And PND demonstrates that this could work quite well, independent of how broken you think it is (and I agree it is broken to some extent).


But that is not a real problem. The Pyra has enough root fs space that you can easily pursue your dream of only using the removable storage as some kind of root fs extension or a data transfer mechanism. Fortunately that doesn't preclude us using removeable storage the same way we do now: having self contained packages that can be popped in and out on demand and run most of the time (even after system upgrades) without the root fs knowing about that at all.

 

[sebt3]

@sswam, I agree with you to some extent. but...

All the features of pnd_run.sh are either implementation of a request or an other. There is only one single feature I think that is not used anymore (one specific for daveC). All the other features of this script is used dayly by the community. Beside, I find it simple and easy to maintain (you should have a look at the size of some script I maintain at work...). If you dont, you just dont have enough bash-fu

I have no idea how you would run an executable within a package without mounting it.

We could assocciate squashfs files and iso using binfmt_misc. The down side is that the kernel will also try to do this for normal ISO file...

What does AUFS have to do with FAT32 partitions...?

There is actually a bug where the system wont create the missing directories on vfat, so the PND script actually have to. Nothing too hard to handle realy...

 

[Linopoly]

I have no idea how you would run an executable within a package without mounting it.

It's possible, but now the code that loads the executable has to understand the format that you're no longer mounting, which means that you have to write that part as well. I get the impression that outside of compatibility with formats you don't control (as with wine), it's probably not worth the effort. It's also possible there's a performance benefit to doing it that way, but I don't know that, it could also be the slower way to do it.

 

[vinipsmaker]

We could assocciate squashfs files and iso using binfmt_misc. The down side is that the kernel will also try to do this for normal ISO file...

Couldn't binfmt be configured to look in the file extension instead of file magic number?

 

[sswam]

Well, if you find something that offers the same features and is as easy to handle for the end user, sure, fire away.

No, I mean to continue using PND but handle it a bit more nicely.

I have no idea how you would run an executable within a package without mounting it.

via a wrapper script in PATH which mounts the PND, then execs your executable

The missing source code doesn't even have anything to do with the PNDs

It's a real pain not having the source code to our "open source" PNDs available in a consistent repo.

What does AUFS have to do with FAT32 partitions...?

AUFS doesn't work as it's supposed to when the appdata is on a FAT32 filesystem.

Rescanning of every PND can be fixed within libpnd, it just has to watch for file changes in a folder...

Yeah, that should be fixed.

All the features of pnd_run.sh are either implementation of a request or an other.

I appreciate the PND system and pnd_run.sh, my "request" is that we simplify it. Here's some suggestions:

- We don't need to support ISO 9660 filesystem. It's hardly used by any PNDs, and not necessary.

- The appdata dir / mount point should be the same as the stem of the PND file name, after stripping extension and any version number.

- Version numbers in package filenames should be done consistently as in Debian, or omitted.

- We should not change the clock speed. If an app really needs that, it can run some script/s in its wrapper.

- Should have only one application in a PND, at least as far as the PXML is concerned; and a single desktop icon.

- pnd_run.sh should exec a program in the PND, not run in the background as a "supervisor" process

- pnd_run.sh should not unmount a PND. A background process can unmount unused PNDs as needed.

- We should factor out the mount and unmount scripts into separate scripts.

As for my bash fu, I have plenty of bash fu.

Couldn't binfmt be configured to look in the file extension instead of file magic number?

Not a bad idea, but even if it can't we can still check that.

Firstly, a PND must be marked executable before it will try to exec it. Although on FAT everything is executable

Secondly, the pnd_run.sh script can check that the PND is indeed a ".pnd" file.

If not it should exit fail, with an error message.

 

[sebt3]

- We don't need to support ISO 9660 filesystem. It's hardly used by any PNDs, and not necessary.

Since you pretty much have a copy of the repo at hands, would you tell me a few stats about this ? how many pnd do you have, how many are iso ?

- The appdata dir / mount point should be the same as the stem of the PND file name, after stripping extension and any version number.

The appdata name is a parameter in the XML file and distinct from the actual PND name. If some put a version number here, they should fix it, not the pnd system... that would actually add some clutter to the solution not remove...

- Version numbers in package filenames should be done consistently as in Debian, or omitted.

See reply above. Bad usage of the system, not a problem of the system per see.

- We should not change the clock speed. If an app really needs that, it can run some script/s in its wrapper.

That a requested feature. Might be remove based on a forum pool I guess... (Oh I think it's ED that requested that from the very start ;P)

- Should have only one application in a PND, at least as far as the PXML is concerned; and a single desktop icon.

On pyra the use case of the PND system will change slightly. That should be attainable. But I thing having the ability to more than one apps is still nice. I'm using it for some of my PNDs for good reason. One have to think at putting the "main" app as first app though as it's the only app pnd_run will start.

- pnd_run.sh should exec a program in the PND, not run in the background as a "supervisor" process

At first, the number of concurrent opened PNDs was limited to 9. Being able to automagically unmount them was a requirement.It's now less of a game changer. Yet having a simple log system is a nice feature used every day on these board. I doubt it would be a good idea to actually change that...

Beside, are you whining for a freaking bash process ? A 2M of used memory (even less if you count that a a good part of it is actually shared) and a number in the process list in the kernel ?

- pnd_run.sh should not unmount a PND. A background process can unmount unused PNDs as needed.

So you're arguing against a simple bash sleeping process but want a deamon that check on a regular basis if any process are still using using a set of directories ?That's not going to be batteries and CPU friendly...

- We should factor out the mount and unmount scripts into separate scripts.

Why for ? it's actually 2 distinct functions of the same script. Each of these could be called independantly using a simple flag to the command line (-m or -u).

As for my bash fu, I have plenty of bash fu.

That was meant as a joke not as an insult. sorry for that.

Couldn't binfmt be configured to look in the file extension instead of file magic number?

Good idea, I posted an howto in the other thread, here you go :

Could be done right now. Reading this, I think we could just add a :

/sbin/modprobe binfmt_misc
mount -t binfmt_misc none /proc/sys/fs/binfmt_misc
echo ':pnd:E::pnd::/usr/bin/pnd_run:' > /proc/sys/fs/binfmt_misc/register

on a current pandora at boot and have it right now (some path might be wrong, i dont have my pandy near hands).

 

[sswam]

Since you pretty much have a copy of the repo at hands, would you tell me a few stats about this ? how many pnd do you have, how many are iso ?

approx 107 / 1320 PNDs use ISO 9660. I guess none of them need to, nor any benefit from it, and we could automatically convert them to use squashfs instead.

The appdata name is a parameter in the XML file and distinct from the actual PND name.

Yes, but that was not a good idea. It would be better if the stem of the filename is used for the appdata name and mount point. I.e. the stem of the filename after stripping extension and any version should be the same as the package ID. Then we can mount and unmount, automatically move PNDs with their appdata, etc without looking in the PXML.

If PND wants to change clock speed, this can be done with separate scripts. pnd_run.sh doesn't need to be involved with it.

I wouldn't mind multiple apps per PND so much, if we would use a simpler file format for the package metadata.

I think the pnd_run.sh should not stay running, it should exec the program inside the PND and not print anything on the terminal. This allows to transparently run a program inside a PND, and avoids cluttering up the process tree (which also might break things). This transparency is important when attempting to simulating a normal Linux environment, where everything is installed on the base system. As for logging, yes it is good to redirect stdout and stderr to a log file and print some debug info from pnd_run.sh, when a PND is run from the desktop, but not when it is run from a terminal.

So you're arguing against a simple bash sleeping process but want a deamon that check on a regular basis if any process are still using using a set of directories ?

That's not going to be batteries and CPU friendly...

I think there's a way to do it efficiently.

 

[sebt3]

I think the pnd_run.sh should not stay running, it should exec the program inside the PND and not print anything on the terminal. This allows to transparently run a program inside a PND, and avoids cluttering up the process tree (which also might break things). This transparency is important when attempting to simulating a normal Linux environment, where everything is installed on the base system. As for logging, yes it is good to redirect stdout and stderr to a log file and print some debug info from pnd_run.sh, when a PND is run from the desktop, but not when it is run from a terminal.

Ho so pnd_run.sh is bugging while debugging ?But then nothing stop you from mounting the pnd first (use the -m flag to pnd_run.sh) and then having fun in your /var/utmp/<appdirname> without any background process.

I for one have a few bash function in my .profile on pandora to do this automaticly out of a .desktop file (my workflow put PNDs I'm working on in the desktop, so these desktop files are easy to find).

I even use the aufs features to upload only binaries to the appdata dir and have a pseudo updated PND at hand fast (works even without remounting the PND).

When done just umount the pnd using the -m flag.

I even know a few PNDs automating this to use other PND features

I think there's a way to do it efficiently.

Show me your code (warning to be included in the pandora git this will need to be GPL or BSD licenced.)

 

[ptitSeb]

Honeslty, I find the PND system very fine.

It's a bit complicated the first time, but once you get all the possibilties, many many things are doable! I even know now how to autodiscover and automount "Runtime PND".

Really, PND is very good, and so easy to end user. Why changing (or worse dropping) it?

 

[EvilDragon]

I have no idea how you would run an executable within a package without mounting it.

via a wrapper script in PATH which mounts the PND, then execs your executable

But then you ARE mounting it... just with a wrapper addition.

The missing source code doesn't even have anything to do with the PNDs

It's a real pain not having the source code to our "open source" PNDs available in a consistent repo.

I agree, but why should devs include it when they build .debs or similar?

Usually, there's a documentation included which should also link to the sourcecode.

What does AUFS have to do with FAT32 partitions...?

AUFS doesn't work as it's supposed to when the appdata is on a FAT32 filesystem.

Well, all PNDs found a workaround for this.

However, that's also something you can't really change - or is there an alternative for AUFS that DOES work with FAT32?

- We don't need to support ISO 9660 filesystem. It's hardly used by any PNDs, and not necessary.

While that's something I basically agree, why remove it if it's already there?

It would break compatibility to those few PNDs.

- The appdata dir / mount point should be the same as the stem of the PND file name, after stripping extension and any version number.

The reason to NOT do that was because some people love to rename their PNDs into a different name.

Then the full upgrading system via PNDManager would stop working, and the appdata would need to be renamed as well.

- Version numbers in package filenames should be done consistently as in Debian, or omitted.

Again, not a fault of the system, but of the dev.

And the version numbers are already being used for upgrading via PNDManager, so most devs use them as well.

- We should not change the clock speed. If an app really needs that, it can run some script/s in its wrapper.

That's a leftover from Zaxxon, before we had the power saving features included.

Basically, back then power consumption changed according to the CPU speed, so you wanted to run the desktop with as little MHz as possible and emulators with the overclock ratio you needed.

Since SuperZaxxon, that's not needed anymore.

So this can definitely be removed, yes.

- Should have only one application in a PND, at least as far as the PXML is concerned; and a single desktop icon.

And how would stuff like Mednafen, MESS or LibreOffice work? You have more programs for different tasks here.

- pnd_run.sh should exec a program in the PND, not run in the background as a "supervisor" process

Then it can't unmount.

- pnd_run.sh should not unmount a PND. A background process can unmount unused PNDs as needed.

A background process would need to permanently check if that PND is still in use, which needs CPU time.

A sleeping pnd_run.sh doesn't need CPU time, it simply waits for the PND to quit.

- We should factor out the mount and unmount scripts into separate scripts.

Umm... why?

There many command line functions with multiple options for one function.

Would you like an apt-install, apt-remove, apt-clean, apt-upgrade, apt-update, etc. instead of one apt-get command with parameters?

 

[fantomid]

It's a bit complicated the first time, but once you get all the possibilties, many many things are doable! I even know now how to autodiscover and automount "Runtime PND".

Correct, for the first time.

Will it not be possible to have HardFloat PND and SoftFloat PND on the same SDCard ?

Only the SoftFloat will be discovered on the SoftFloat OS and HardFloat on the HardFloat OS.

On the repo, you will have the two versions for the same application.

 

[sswam]

I have no idea how you would run an executable within a package without mounting it.

via a wrapper script in PATH which mounts the PND, then execs your executable

But then you ARE mounting it... just with a wrapper addition.

Yes, that's what I was getting at. The point is that I can run programs such as "mplayer" or "make" or "gcc" that are inside a PND, directly and transparently from the shell, without having to think about mounting or unmounting a PND, it happens automatically.

why should devs include [source code] when they build .debs or similar?

Debian maintains a complete source code repo including upsteam sources and patches.

e.g. `apt-get source gcc` will fetch the gcc source from your Debian mirror, and apply the Debian patches.

We should do similar.

Well, all PNDs found a workaround for [AUFS / FAT bug].

However, that's also something you can't really change - or is there an alternative for AUFS that DOES work with FAT32?

It makes things difficult for PND authors. I suppose we could fix AUFS.

While that's something I basically agree, why remove [iSO 9660 PND support] if it's already there?

It would break compatibility to those few PNDs.

It complicates the system for no benefit, makes it a little harder to write tools that work on PNDs, etc.

It's trivial to fix the PNDs.

some people love to rename their PNDs into a different name.

Tell me who they are, so I can smack them around the head with a brick!!

Well maybe that's a little harsh. If they rename the PND, they should rename the appdata too.

PND manager could still look in the PXML for the "standard" name.

Again, not a fault of the system, but of the dev.

And the version numbers are already being used for upgrading via PNDManager, so most devs use them as well.

The system should enforce or at least encourage sensible conventions for our benefit.

Some PNDs are using "-", "_" or nothing to separate package name from version; they should all be using "_" for that, and not in the main package name.

Glad you agree about killing the CPU speed function. If an app really needs that, it could call a script to do it.

And how would stuff like Mednafen, MESS or LibreOffice work? You have more programs for different tasks here.

Yeah, maybe it's a good feature to allow have multiple "desktop icons" / apps per PND. I think I was disliking it because I hate the extra complexity of the XML. XML is bad (for this).

A background process would need to permanently check if that PND is still in use, which needs CPU time.

A sleeping pnd_run.sh doesn't need CPU time, it simply waits for the PND to quit.

pnd_run.sh does sleep and repeatedly poll using "pidof" in some cases.

If you didn't want a background process to do it, we can unmount unused PNDs lazily from pnd_run.sh, or if an SD card is removed.

- We should factor out the mount and unmount scripts into separate scripts.

I think it's a better design, can make the script much simpler (no need for parsing options).

Anyway I'd better shut up and do it, then talk about it later.

 

[sebt3]

pnd_run.sh does sleep and repeatedly poll using "pidof" in some cases.

You seems to have not understand.While the start command of the PND is running, the bash process is in a sleep state kernel wise. The kernel wont bring back that process to life up to and until the start command finished.

At this point, there have been a few case where the PND mount point was still not immediatly free. So I added a simple wait loop at the end to try to umount the PND cleanly. It will wait up to 2 minuts until it gave up and leave the PND mounted.

It's a security mesure that is not triggered 99% of the time, but still save ass once in a while.

It's the current safety. You want o build a deamon doing this 100% of the time. I definatly think the current way is better.

 

[levi]

Will it not be possible to have HardFloat PND and SoftFloat PND on the same SDCard ?


Only the SoftFloat will be discovered on the SoftFloat OS and HardFloat on the HardFloat OS.


On the repo, you will have the two versions for the same application.

We could have softfp binary-containing PNDs in the pandora/menu (etc.) folder and hardfp ones in pyra/menu (etc.).

 

[vinipsmaker]

- We should not change the clock speed. If an app really needs that, it can run some script/s in its wrapper.

I think it needs to be handled differently for different Pandora versions (and probably disabled in Pyra).

sswam, on 20 Feb 2014 - 06:21 AM, said: I think there's a way to do it efficiently. Show me your code (warning to be included in the pandora git this will need to be GPL or BSD licenced.)

Every container (and even some non-container related) tool is jumping in the cgroups wing.

Debian maintains a complete source code repo including upsteam sources and patches. e.g. `apt-get source gcc` will fetch the gcc source from your Debian mirror, and apply the Debian patches. We should do similar.

See Cloudef's makepnd and PUR.

 

[klapse]

I have no idea how you would run an executable within a package without mounting it.

via a wrapper script in PATH which mounts the PND, then execs your executable

But then you ARE mounting it... just with a wrapper addition.

Yes, that's what I was getting at. The point is that I can run programs such as "mplayer" or "make" or "gcc" that are inside a PND, directly and transparently from the shell, without having to think about mounting or unmounting a PND, it happens automatically.

My understanding is that standard programs like that which we run from the command line will be provided by .debs and will be in the users PATH.