The software side

[Linux-SWAT]

Post count increase is a full-time job.

 

[ShleeDragon]

From a non-technical side. I like the idea of the binary and the appdata being separate.... I like the idea you can upgrade easily. I like that the binary is read only and doesn't "corrupt".. I like that it's idiot proof.

 

[Cloudef]

Here is discussion related recipes and the suggested new PND format.

11:05:17     Cloudef    B-ZaR: morning
11:05:39     B-ZaR    so you're building PNDs from recipes
11:06:36     Cloudef    Yes
11:07:08     Cloudef    I thought I should take first step as the thread is not going anywhere
11:07:36     sebt3__    Cloudef: thanks for doing so
11:07:58     sebt3__    but is any games buildable this way that shouldnt be patched to support the weird control schema ?
11:08:12     Cloudef    sebt3__: this is why you can include patches with recipe
11:08:15     Cloudef    sebt3__: similar to aur
11:08:22     Cloudef    sebt3__: the website will have button to send improvements to recipe
11:08:23     B-ZaR    which sounds awesome
11:08:25     sebt3__    I dont see any apps usefull as PND in the target system
11:09:05     Cloudef    history of the recipes will stay though, and maintainer of recipe needs to accept the improvement for it to show up as default for PND
11:09:23     Cloudef    however if maintaner is jerk, moderator can move his maintanership for the better recipe submitter
11:09:56     Cloudef    sebt3__: yeah, the pnds are only useful for big stuff and games
11:10:05     Cloudef    but recipes should increase quality a lot
11:10:25     Cloudef    I also do prototype of the new PND format
11:10:38     Cloudef    but first I demo something that works with our current system
11:11:10     B-ZaR    you're a hero
11:11:12     Cloudef    depending how things go, you could also generate deb from the recipe
11:11:46     Cloudef    I still think though the pnds should be self contained (static linked, or contains libs that cannot be static linked)
11:11:54     Cloudef    but I won't go into that discussion again 
11:12:40     B-ZaR    kinda like canonical's app containers?
11:12:41     B-ZaR    or what were they called
11:13:14     Cloudef    B-ZaR: IIRC app containers contain runtime
11:13:25     Cloudef    similar to how steam does it
11:13:51     Cloudef    they just provide runtime with libs you should link against
11:14:11     Cloudef    if you link dynamically something else, you are fucked so that should be either included or statically linked
11:16:17     B-ZaR    this, I think http://news.slashdot.org/story/13/05/08/2038243/ubuntu-developing-its-own-package-format-installer https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037074.html
11:17:51     Cloudef    there are even containers where each has own userspace 
11:17:57     Cloudef    I don't remember the name
11:18:02     Cloudef    But those are mostly for servers
11:18:59     _wb_    Cloudef: static linking is perhaps good when making PNDs that should also work on Pandora
11:19:49     dreamer    Cloudef: lxc? openvz? 
11:19:51     Cloudef    _wb_: yeah, you could generate softfp PND from recipe
11:20:08     Cloudef    Too bad we have that ABI difference
11:20:39     _wb_    but for Pyra-only PNDs, real dependencies are better imo
11:21:04     Cloudef    That's something that needs to be discussed better when there is working prototype I think
11:21:15     Cloudef    Maybe try different things
11:24:25     dongs2    Hooorah
11:24:25     _wb_    e.g. consider a PND that uses LaTeX and qt4 and a bunch of tools like ImageMagick
11:24:56     _wb_    you don't want to include all of that in the PND, you just want to specify what the dependencies are
11:25:22     _wb_    maybe the recipe can build a huge PND with all dependencies included too, for Pandora
11:26:32     _wb_    but on Pyra it makes more sense to install pdflatex and convert etc on the root fs
11:26:33     Cloudef    _wb_: Yeah or some other distro for those who want to use one
11:29:36     _wb_    basically you would have .deb for general-purpose stuff, and .pyra for big apps you don't want installed in the rootfs, like games with lots of media
11:29:46     _wb_    the .pyra files have metadata for dependencies on debian packages
11:30:24     Cloudef    Yeah and the unlike PND we would have real header structure for the file
11:30:32     Cloudef    I just make recipe generate UBJSON or something for metadata
11:30:36     Cloudef    and make sure the file has icon embedded
11:30:45     Cloudef    also screenshots will be embedded, so they are faster to read
11:30:52     _wb_    and .pnd files can be interpreted as .pyra files with a default set of dependencies that corresponds to SZ
11:30:53     comradekingu    Just put the media in .homedir like everything else
11:30:57     Cloudef    only package files are under RO filesystem
11:31:05     Cloudef    _wb_: we can also use squashfs instead of squashfs
11:33:21     _wb_    whatever can be mounted is fine
11:34:19     Cloudef    other thing we can do is add $PATH to pyra binary files
11:34:33     Cloudef    so you can run the program inside from terminal with just typing program name
11:35:01     Cloudef    though, that's not important as the .deb files are better for such things
11:35:12     comradekingu    http://www.filesystems.org/project-unionfs.html
11:35:28     Cloudef    but at least recipe will generate right thing for correct argument passing and file associations
11:35:52     Cloudef    comradekingu: you do realize that package manager doesn't like if half of the stuff disappears when SD breaks or is switched?
11:35:58     _wb_    yes, if it's something you want to run from terminal or from another program, it should usually be a zdeb
11:36:16     Cloudef    indeed
11:36:35     comradekingu    Thats why i dont opt to put system libraries on removable media
11:36:42     Cloudef    Keeping it simple will be good
11:37:58     comradekingu    keeping it non-proprietary is good
11:41:47     _wb_    just a tarball of metadata, icon, screenshots and squashfs
11:42:11     Cloudef    _wb_: no tarball for them, I would just include them in the binary file
11:42:16     Cloudef    For faster reading
11:42:33     Cloudef    Image files are already compressed, and if metadata is UBJSON it's small
11:42:53     _wb_    no gzipped tarball, just tar
11:43:09     Cloudef    that's not needed
11:43:14     Cloudef    it's just waste of space
11:43:35     _wb_    it makes it easier to modify if it's already existing glue
11:44:08     Cloudef    you modify the recipe, not the pnd itself
11:44:20     _wb_    true, but still
11:44:47     Cloudef    Plus, it will be simple for programs such as PNDManager or launchers to read
11:44:53     Cloudef    simpler*
11:45:38     _wb_    users may want to modify the .pyra, say they want to change some metadata or the icon for whatever reason
11:45:57     _wb_    it's simple enough to read a .tar
11:46:12     Cloudef    they should be encouraged to contribute to recipe in that case
11:46:38     _wb_    not if it's a user-specific customization
11:46:49     Cloudef    Then the user is just being jerk honestly
11:47:05     Cloudef    You could fork the recipe for your own purpose
11:47:50     dongs2    I don't think we should support pandora PNDs... just recompile :|
11:47:56     _wb_    say I want DraStic to appear as "Nintendo DS" in my menu, with the actual DS logo as an icon
11:48:23     Cloudef    _wb_: If that becomes a big issue, I can create tool to edit that stuff
11:48:25     _wb_    not gonna fork a recipe for that, just tweak the .pyra a bit
11:48:43     _wb_    anyway, it does not matter much
11:48:55     _wb_    so how do you propose to glue the stuff together?
11:49:09     Cloudef    _wb_: what stuff exactly?
11:49:29     _wb_    fs image, metadata, icon
11:49:34     Cloudef    ah
11:49:38     Cloudef    I explained quite well on the thread
11:49:53     Cloudef    it would be format with magic header, offsets and sizes for relevant data
11:49:53     _wb_    the stuff that makes a .pyra package
11:50:17     Cloudef    first data is metadata, then comes icon, and screenshots, and lastly squashfs
11:50:17     _wb_    isn't that exactly what .tar is?
11:50:29     _wb_    ok, the filenames are overhead
11:51:12     Cloudef    tar does contain file permissions and such too no?
11:52:00     _wb_    it's not going to make a huge difference in size
11:52:23     comradekingu    apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted).
11:52:37     Cloudef    _wb_: you would still need tool to open it though (or at least know the format enough to extract the tar part)
11:52:40     comradekingu    How about a little script that does that, on the off chance you should feel a need for it
11:53:20     Cloudef    _wb_: but you have point, tar could do as well
11:53:44     Cloudef    _wb_: and maybe more future proof actually
11:53:56     Cloudef    if we decide to embed more files..
11:54:12     _wb_    if we ever decide to add more stuff, exactly
11:55:09     Cloudef    To cover the simplicity part, I could just do small lib to read the file
11:55:25     Cloudef    not that tar is rocket science 
11:59:11     _wb_    you need something to read the file anyway, but if it's just a .tar then lots of things can be done with just shell scripts, even on servers that only have default tools installed
11:59:38     Cloudef    yeah, with dd you can extract the tar part and so

 

[fantomid]

Waouh. Lots of reply.

I didn't read all the posts.

But in my mind, I think for the Pandora, effort has to be made on making .next stable and with clear instructions to build it from a host PC.

OE is quite powerfull but not really easy to understand.

With automated firmware builds and automated application and game builds, it could be possible to have up to date version of the most attractive application for the devices.

If we change how we make/build softwares for the Pandora, it will be profitable for the Pyra even if it's Debian based.

On Pandora, with SD boot, testers may try firmware upgrades and applications, do they ?

Giving just some feelings/ideas.

 

[God Ginrai]

What is wrong with PND? It's an easy plug'n play system, usable by everyone, and working well.

What is the limit of the PND system, really?

With the PND, I can advise everyone to buy a Pandora. It's easy to use as a Console. Can be easy as a SmartPhone (with the Minimenu has main GUI, and PNDManager to add/remove PND), or as advanced as a Linux Full PC (with distro like Slackware on a SDCard).

Without PND (or somthing equivalent), that plug'n play easy to use side maybe lost. Don't create a hardware for just a small linux elite.

 

I am all for PND as well, the Pyra should be comfortable like the Pandora was in terms of Software.

 Whoa, whoa, whoa there you two. We're talking about improving PND, not getting rid of it. We like what PND does, we just want it to do it in a more manageable way from the system side.

The existing PND system accomplishes this but it is messy and has several known issues. This is the perfect opportunity to redesign it from the ground up with an eye to solving those issues.

 How is it messy exactly ? Yes it has issues but most of them are transparent for the user.

This is exactly the mindset that leads to all those bugs you mentioned that exist in ubuntu. A user-centric feature becomes harder and harder to maintain if you don't try to make sure that you are doing things right where the user can't see.

-God Ginrai

 

[Galaxis]

I don't quite get what the package manager fuzz is about: That the Pandora has a way to "install" applications independently from the base system is one of the things that makes it so easy to use. It's a non-optional feature.

I don't currently have an overview of the possible alternatives, but just look at Mac OS X for a different solution to the same problem: OSX application bundles are not managed by the same infrastructure as the base system either. In fact, before the Mac app store showed up, they were not "managed" by the system at all.

 

[bzar]

One big plus for a real package management system is dependency management. Another one is library upgrades without rebuilding every single container/bundle containing it. Especially important for stuff like security updates and major bug fixes. Yet another is ready repositories filled with oodles of already packaged software like Debian's. And to top it off, the package management clients are a lot more feature-complete compared to, say PNDManager/Milkyhelper.

With all that said, there's still a role to play for PND-like application containers, which I've outlined above in a previous post.

 

[EvilDragon]

Well, we'll have a lot more space available - we can include a lot more libraries than before!

And if a game needs a library that could be useful for other games, why not include that in a firmware update right away, instead of including that library in the PND?

With a meta-package (like meta-pyra-base) it would be easy to add new libraries as standard to an existing system.

 

[Klumpen]

So, modding Synaptic and using SD-Slot1 as /home/usr is not a good idea?

Everyone has some SD in Slot 1 all the time anyway and I personally use the second slot for ISOs, Music/Audiobooks, Videos, etc...

 

[Granitehead]

Well, we'll have a lot more space available - we can include a lot more libraries than before!

And if a game needs a library that could be useful for other games, why not include that in a firmware update right away, instead of including that library in the PND?

With a meta-package (like meta-pyra-base) it would be easy to add new libraries as standard to an existing system.

.deb-dependencies .pyras like discussed in that irc log would be better.

 

[levi]

So, modding Synaptic and using SD-Slot1 as /home/usr is not a good idea?

Everyone has some SD in Slot 1 all the time anyway and I personally use the second slot for ISOs, Music/Audiobooks, Videos, etc...

Modding synaptic, or more likely using some of its libs to figure out the dependencies with a new front end might be a way to go still IMO.  I'm not sure what you mean by /home/usr - that's not a path I have on any of my Linux boxes.

I'd imagine the system will still allow a fake home directory per program to be put onto your SD card somehow - that seems to be a pretty solid requirement to me, although how it will be achieved is still up for debate.

 

[EvilDragon]

So, modding Synaptic and using SD-Slot1 as /home/usr is not a good idea?


Everyone has some SD in Slot 1 all the time anyway and I personally use the second slot for ISOs, Music/Audiobooks, Videos, etc...

Nope. Lose your SD Card somewhere and the OS thinks files are installed even though they are gone.


Aditionally, how will Windows users put that card into their card reader and copy over some roms?

.deb-dependencies .pyras like discussed in that irc log would be better.

I couldn't read it yet as I'm on-the-go and tapatalk shows it as big block of text

 

[Klumpen]

Nope. Lose your SD Card somewhere and the OS thinks files are installed even though they are gone.
Aditionally, how will Windows users put that card into their card reader and copy over some roms?

I totally forget about stuff like that sometime now since you guys brought me to Linux.

 

[skeezix]

I've not had time to read the whole thread.. but so far its..

- we've not really kicked off the whole software discussion yet at all, so its all up in the air so far

- Deprioritizing pnd seems like a good idea; we've sort of been assuming that everything will just go with normal Linux conventions.. if we do not have to invent something, it is best not to   Now, it woudl be nice to back-support pnds, and actually support a pnd or pnd-like system shoudl not be too hard, but in general it should not be the main distribution; there will be more sizable internal storage this time. PNDs primary motivator back so many years ago was because there was not much NAND and therefore apps _had_ to be installed on SD, with ejectable state, swappable while on or off, etc etc. Those days are gone in the new device....

- Keeping pnd or libpnd around is okay; we've not discussed it much, but should be doable

- Supporting a new pnd-like system (ie: self contained apps, actual requirements TBD!) is also cool!

- Supporting existing pnds from Pandora on new one .. well, I think it should be started up front 'no promises' but some of them may just work depending on level of co,plexity, and level of pandora-ness in the port etc (how many assumptions were made.)  Conceivable that some will for sure work (just sh-scripts in a pnd file may port easy), though I woudl expect most apps not to work (ABI issues, or at least device name changes, and assumptions about device peripherals and so on.)

Anyway, I'm just injecting and running, but thought I'd throw that out there

jeff

 

[aTc]

Wow, a lot of stuff going on here

Yes, so the first thing to be done is setting up a forum section for pyra firmware dev.

Everyone shouting random stuff in one thread obviously doesn't work.

And this thread was meant to set up the large scale stuff first, not for fighting about implementation details.

There is an awful lot of infrastructure required to make what's basically your own distro.

It may be based on debian, but debian isn't actually designed to produce complete images. The debian way is to give you an installer, a lot of packages, and you have to install and configure everything yourself from there on.

A build server will have to be set up that can generate all the pyra specific .deb packages from source, a git repo for those sources,  and some way of turning a collection of packages from various sources into a user friendly, pre configured, easily installable firmware image.

As mentioned above, the Pandora had issues with an OS that became outdated quickly, mainly because OE seems to be a bit picky on whether it wants to build stuff or not.

I've been following aTc great work on .next, and it still looks like that an image that built fine two days ago doesn't build properly two days later because of random recipe changes by someone... this is something we really don't need anymore.

Therefore, I think Debian should be a fine OS for us. We can use the official Debian repositories, so it stays up-to-date.

And then provide our own repository with higher priority that has stuff like nub, LED, etc. configuration tools or optimized SDL ports like the one from notaz for the Pandora.

This way, we don't have to maintain too much for the base system - as Debian is updated automatically.

Let me know if you think there are any issues - I'm certainly no expert in that stuff

It was actually a major advantage to not have to worry about any updates in the pandora firmware oe tree.

Having library versions change all of a sudden and having stuff break was never an issue

Most breakage in .next comes from exactly that, angstrom moved from 2012.12 to 2013.06 , stuff got updated, pandora stuff that depended on it broke too.

Angstrom en OE are really meant for devices a lot more limited than the pandora, where there isn't a full multi user desktop setup, and certainly not something that has to survive updates.

I think the recommended OE method is still to just make a copy of the most recent tree that works, and then base all your work on that.

Of course you can't compare angstrom to debian when it comes to updates, or everything else really.

Debian is a lot more careful to make the upgrade process as smooth as possible, and is actually designed to be upgradable.

Next, onto anything we add ourselves... One thing I learned from notaz during the last years is that we should stick to mainline kernel as close as possible. Otherwise, you will have hacks that will make it hard updating to new kernel versions in the future. And if people can simply build the kernel from mainline, it's easier to run any ARM Linux distribution on the Pyra.

This not only applies to the kernel, but also to the distro itself.

You want to avoid having to do custom versions of stuff already in the debian repos.

So whatever we plan here - no evil hacks.

As far as I can see, at the moment the only "hack" that would be required to keep everything (kernel and distro) as mainline as possible is the input handling, as detailed in my previous post.

I also don't need 100% backwards compatibility - many things will probably be re-ported or exist in Debian anyways.

That's all you had to say

Now, onto the PND System... This is what I appreciate about the PND System.

<long list>

Yes, and something LIKE pnd would be great, and is certainly a thing to go for.

But seriously, don't expect the current .pnd system to cleanly port over to a new device. 

The current .pnd is nearly unusable anywhere else, it's not like there's one script that handles everything, its a combination of daemons running in the background, libraries that are hardcoded for the pandora hardware, and a lot of scripts that assume a LOT of things about the os it's running on. And let's not forget the horrors that go on inside the .pnds themselves.

A new system, properly designed, taking into account all the issues .pnd has and avoiding them, that's the way to go.

Extra bonus points for a design that can easily be used on other distros and devices as well.

If there has to be any compatibility, it should be that the pandora should also be able to use the new format.

And if the pnd successor is cleanly implemented, that should be no problem whatsoever.

 

[Askarus]

Yes, and something LIKE pnd would be great, and is certainly a thing to go for.

The .pra system (for Pyra)?

I would use a new extension and build a new system.

 

[CrazyIvan]

if we talk about 'past gen' handheld console usage type when a user could just get a game (download some file in our case), insert it and run then we'll eventually have to deal with new software being installed on a system that hasn't been updated for a long time. In that case having dependencies in pnd itself is a good idea since you don't depend on outdated system libraries. But it becomes a bad idea in reverse scenario - when .pnd is old and the system is new. Depending on system libraries sometimes could solve the issue of badly behaving libraries in .pnd but it's not an option here since they're in .pnd itself.

.pnd could have only dependencies but in old system that is not connected to Internet it's impossible to download them, so in dependency-based .pnd format there should be some 'fat pnd' variant with full dependency tree included in form of OS packages which could be installed on first .pnd run (or detection). There are two problems with this approach: final 'fat .pnd' size and the process of building it. There could be some general 'system upgrade .pnd' which will, well,  upgrade all system to the most current state with packages contained within but as the time passes it'll eventually get bigger and bigger and, well, if we're aiming towards 'download, insert and play' usage simplicity, I think it's too much a burden to force a user to check if he have a recent system every time he downloads a .pnd (in case he doesn't do this very often). Besides .pnd should depend only on software within this 'system update pnd' that is a little restricting. 

I hope that 'fat .pnd' wouldn't be that fat but it's needed to be checked. But to create such 'fat .pnd' one need to generate a full dependency tree that could be changed with every repository update (I mean full debian repository). That means that ED would have to host entire debian arm repository copy and update dependency trees for every .pnd stored every time repository is synced. The need to actually generate and store 'fat .pnd' could be mitigated with a (probably) simple script that could make 'fat .pnd' on the fly from dependency tree and local debian repository (since packages themselves are compressed there is no need for additional compression so it could be fast).

But do we need such a system or can we just treat pyra as a self-sufficient computer (like steam does actually when it automatically updates itself)?

 

[Exophase]

Just a couple random thoughts..

1) I agree that a system development forum would be a good first step and the sooner the better. Back on the old boards there was a forum for this before Pandora was released, but it was hidden. Something public seems much better this time around, but it should be heavily moderated because it's one place where you really don't want noise to creep in. Maybe even make a forum specifically for the spillover that will definitely accumulate, and encourage people to start the more user-oriented or opinionated discussion there. This needs to be a technical forum with a clear goal of determining specifications and managing work.

2) Is it possible to do a /dev/input driver with something in userspace (like FUSE), communicating with the normal kernel interface? Can we not do this now on Pandora as a proof of concept? This would already make our lives easier, especially for those who support/would like to support USB joysticks.

 

[God Ginrai]

Yes, and something LIKE pnd would be great, and is certainly a thing to go for.

 The .pra system (for Pyra)?

I would use a new extension and build a new system.

.pyr would be a better extension to use.

-God Ginrai

 

[aTc]

2) Is it possible to do a /dev/input driver with something in userspace (like FUSE), communicating with the normal kernel interface? Can we not do this now on Pandora as a proof of concept? This would already make our lives easier, especially for those who support/would like to support USB joysticks.

Yes, and as I mentioned in one of my previous posts, that actually existed, but due to lack of interest from the rest of community it never went anywhere.

I might still have it in my archives, it must have been nearly 5 year ago,  i think it was JayFoxRox who did it, there might still be something on the forums about it, but i think most of it happened on irc.

And as I also mentioned, the existing way the input works on pandora caused some major issues for it.