A possible PND-system replacement

[skeezix]

The concatenation sucks, fully agreed; right before going with that idea, we'd tried using a catenated index and then catenating the chunks.. which is more or less exactly what zip does for comments and junk. But it was shoddy, and we thrashed around trying options,and decided to go for 'this is easy and dumb, we can replace it later'; but in software we all know how that works,.. 'shit lives forever'

But squash (or whatever, the payload container type doesn't matter much, since you can detect and mount it easy enough) + zip appended was really always the goal; someone run some tests.. if the zip handlers are smarter now, that could be the way to go (unless some completely drastic other idea comes up.) Zip for main container is same boat.. mounting the entire thing to RAM was madness.

If anyone mentioned it (I didn't have time to read up, sorry , the executable pnd idea is something else to consider; I fiddled with it, had it work (its not hard to do, the binfmt thing .. hell, you can enable it in current firmware easy enough), but of course it doesn't support much .. when it occurred to me that having documentation handlers and multiple executables in there, it pretty much killed off the executable pnd idea. (By which I mean ./my.pnd to run it directly.)

The documentation handling is not bad as it is, but its not great either; somethign else to mull over

Oh, and using .desktop is the way to go imho, or JSON .. we all hate XML At least the XML makes self validating easy; .desktop is goofy as heck too, but at least its common and semi-standardized; we can add a pile of extra fields for the meta we want. At the time we thoguht of it a bit late, but I seem to recall there were some things we didn't think we could shoe-horn into .desktop, but I couldn't think of what they were a couple years back when I was ondering wtf we'd done there

jeff

 

[slaeshjag]

ZIP for metadata is handled cleanly with current versions of libarchive, so appending that to the squashfs isn't really much of a problem. The relevant files are extracted by dbpd and automatically cleaned up when the package disappears. The .desktop files are hot-patched to point to the real icon path, but are otherwise left unchanged. Uncompressed zip is probably best for meta-data, but compressed zip works too, al though a bit slower (but I don't know how big of a problem that is with small archive as it's just meta-data?)

The executables are exported as per a list in a separate section of default.desktop. The application launcher is also the default action of a package, which is runned then you do ./package.dbp.

The idea isn't to use dbp's for exporting libraries, although it *could* be done by requesting that dependency to be mounted and LD_LIBRARY_PATH to that mount, as they are predictable from the package ID and daemon configuration file.

 

[pmprog]

Could you not set up the binary handler to accept


./my.pnd %app% [%arg1% %arg2%, ...]
But then, I like that you can mount PNDs without requiring to run anything (myself and petitSeb use that with Mono).
In fact, I think the ability for the new PND system to support non-application PNDs would be great (ie, you could have GAME.PND and GAMEDATA.PND).

You could also kind of use this a dependancy system too, so you could have an SDL.PND, ALLEGRO.PND, IRRLICHT.PND, MONO.PND etc., and your apps load your requirements?

 

[slaeshjag]

You can run any executable within the dbp if you go via dbp-run. All exported executables also go into $PATH.

and yes, when the dependency system is in place, you could have a package depend on other packages, and from the launch script request those packages to be mounted at a predictable location. Mounts are ref-counted inside dbpd.

 

[slaeshjag]

Did another pass on dbp.

Now I've implemented dbp-cmd, which is a dbus wrapper for shellscripts etc. to do low-level dbpd operations like mount package, get its path etc..

I've also done all needed preparations in dbpd for the future desktop daemon.

Next up is descriptive error messages (with full gettext support)

After that, the desktop daemon. It runs as a regular user, creates application icons on your desktop if they are placed in a path where the last directory name is "desktop".

And after that... Dependency check, I guess?

 

[slaeshjag]

So, everything but useful error reporting + translation of that is in place now, from the my list of things anyway. H4xxel is working on the error reporting stuff, so that should hopefully be done within the next days.

To recap, the dbp-system in its current form. All the features below are implemented and to some degree tested.

* Squashfs for application data (not needed if no aplication data is needed,) appended zip for meta-data. This allows both containers to be accessed with standard tools.

* Meta-data for the dbp-system is contained in .desktop-files. Package meta-data is located inside meta/default.desktop. default.desktop and all other .desktop files inside the zip may contain program launchers. default.desktop is the default action when a dbp-file is invoked directly.

* Programs inside the dbp have executables exported into path. These can be launched from other programs, the terminal or .desktop launchers. Or anywhere else for that matter that don't do binary linking to it.

* Run-time dependency checking. Checks if all package dependencies are met. Both for dbp-packages and installed deb's.

* File type association in desktop enviroments that pay attention to installed .desktop-launchers

* Packages can like in the PND-system specify an appdata-directory that is different from the package ID. There is also support for an additional read-only appdata directory.

* stdout/stderr are written to a log-file when you run an executable, if the exported executable has enviroment override set in the package metadata. If no enviroment override is set, it behaves just like any other command installed to system. In both cases, output is always sent to stdout/stderr as well.

* Multiple programs in the same package can have different icons (yay?)

* Does automatic crawling and file change detection ofc.

The dbp-system have two main parts

* dbpd, handles all system-wide stuff like package mounting, executable export etc.

* dbp-run, the user-part that requests dbpd to mount the package. then dbp-run sets up the enviroment and executes the real binary. dbp-run is invoked from inside an exported executable.

There's also some optional parts

* dbp-run-path, allows you to execute the default action of a dbp-file just by passing it a path.

* dbp-desktopd, needed for program launchers to appear on your desktop. Runs as your regular user.

* dbp-cfg, extracts configuration values from dbp_config.ini. Nothing special, saves some work in scripts I guess.

* dbp-cmd, allows you to do low-level commands to dbpd. This is over dbus and uses the same API as dbp-run. Commands are: mount dbp, umount dbp, register dbp, unregister dbp, get path from pkgid, get pkgid from path, get package media mountpoint, list all packages

For package managers etc, there's also two dbus signals of interest that announce newly detected and newly removed packages.

Some details about the meta-data is yet to be standardized. This include stuff like version numbers and other details that the dbp-system itself doesn't worry about. For this, I want feedback, as I personally don't have much experience with the issues that may pop up with this.

aTc has done some work on packaging it all, so I dunno, maybe that it'll appear in the pyra-specific repo soonish?

That's all I could think of right now.

 

[jA_cOp]

So, everything but useful error reporting + translation of that is in place now, from the my list of things anyway. H4xxel is working on the error reporting stuff, so that should hopefully be done within the next days.

This refers to graphical error reporting, i.e. issue #13, right?

* Meta-data for the dbp-system is contained in .desktop-files. Package meta-data is located inside meta/default.desktop. default.desktop and all other .desktop files inside the zip may contain program launchers. default.desktop is the default action when a dbp-file is invoked directly.

Are the .dbp files executable? Do they have relevant icons when viewed in a file manager? Although of fairly little importance, it would be intuitive behaviour.

* Programs inside the dbp have executables exported into path. These can be launched from other programs, the terminal or .desktop launchers. Or anywhere else for that matter that don't do binary linking to it.

Cool, so no more need for special terminal environments like the one the Pandora CodeBlocks package ships with?

* Run-time dependency checking. Checks if all package dependencies are met. Both for dbp-packages and installed deb's.

What happens if the dependencies are not met? I think the smoothness of the path to installing the dependencies is crucial for dependencies to catch on from developers.

The dbp-system have two main parts


* dbpd, handles all system-wide stuff like package mounting, executable export etc.


* dbp-run, the user-part that requests dbpd to mount the package. then dbp-run sets up the enviroment and executes the real binary. dbp-run is invoked from inside an exported executable.


There's also some optional parts


* dbp-run-path, allows you to execute the default action of a dbp-file just by passing it a path.


* dbp-desktopd, needed for program launchers to appear on your desktop. Runs as your regular user.


* dbp-cfg, extracts configuration values from dbp_config.ini. Nothing special, saves some work in scripts I guess.


* dbp-cmd, allows you to do low-level commands to dbpd. This is over dbus and uses the same API as dbp-run. Commands are: mount dbp, umount dbp, register dbp, unregister dbp, get path from pkgid, get pkgid from path, get package media mountpoint, list all packages

So dbp-run is an internal command, while dbp-run-path is designed for a user-facing environment like a terminal or script?

Some details about the meta-data is yet to be standardized. This include stuff like version numbers and other details that the dbp-system itself doesn't worry about. For this, I want feedback, as I personally don't have much experience with the issues that may pop up with this.

For versioning numbers, I strongly recommend using SemVer; its handling of backwards-compatibility is ideal for dependency systems. With this system, packages should specify version spans like ">=1.3.0" (higher than or equal to major version 1, minor version 3), "==1.3.0" (exactly major version 1, minor version 3) or ">=1.3.0 <=1.3.4" (major version 1, minor version 3, patch between 0 and 4). This is robust enough that an application can even link to dynamic libraries in its dependencies, which pretty much means it's robust enough for everything else too.

 

[slaeshjag]

So, everything but useful error reporting + translation of that is in place now, from the my list of things anyway. H4xxel is working on the error reporting stuff, so that should hopefully be done within the next days.

This refers to graphical error reporting, i.e. issue #13, right?

That and issue #20

* Meta-data for the dbp-system is contained in .desktop-files. Package meta-data is located inside meta/default.desktop. default.desktop and all other .desktop files inside the zip may contain program launchers. default.desktop is the default action when a dbp-file is invoked directly.

Are the .dbp files executable? Do they have relevant icons when viewed in a file manager? Although of fairly little importance, it would be intuitive behaviour.

Yes. I am still working on them having icons in the file managers (issue 24.) The idea though is that their main use-case is that the user have them installed in a special folder like in the PND system. Loose files scattered around your system isn't going to show up in menues, in your path etc..

* Programs inside the dbp have executables exported into path. These can be launched from other programs, the terminal or .desktop launchers. Or anywhere else for that matter that don't do binary linking to it.

Cool, so no more need for special terminal environments like the one the Pandora CodeBlocks package ships with?

Since you'll be able to just install that stuff normally via apt-get now, not really. I guess it would be somewhat easier to implement with the dbp-system too.

* Run-time dependency checking. Checks if all package dependencies are met. Both for dbp-packages and installed deb's.

 What happens if the dependencies are not met? I think the smoothness of the path to installing the dependencies is crucial for dependencies to catch on from developers.

The idea is to pop up a message box giving the user the bad news, with the option to continue anyway or exit. When a package manager exists for dbp-files, it could have a button to launch that to install the missing deps.

The dbp-system have two main parts

* dbpd, handles all system-wide stuff like package mounting, executable export etc.

* dbp-run, the user-part that requests dbpd to mount the package. then dbp-run sets up the enviroment and executes the real binary. dbp-run is invoked from inside an exported executable.

There's also some optional parts

* dbp-run-path, allows you to execute the default action of a dbp-file just by passing it a path.

* dbp-desktopd, needed for program launchers to appear on your desktop. Runs as your regular user.

* dbp-cfg, extracts configuration values from dbp_config.ini. Nothing special, saves some work in scripts I guess.

* dbp-cmd, allows you to do low-level commands to dbpd. This is over dbus and uses the same API as dbp-run. Commands are: mount dbp, umount dbp, register dbp, unregister dbp, get path from pkgid, get pkgid from path, get package media mountpoint, list all packages

So dbp-run is an internal command, while dbp-run-path is designed for a user-facing environment like a terminal or script?

dbp-run is the main command. dbp-run-path is a hack that provides the possibility for people who really really wants to run packages by path. This is against how the system works internally, but it makes it possible to execute .dbp's as executables.

Some details about the meta-data is yet to be standardized. This include stuff like version numbers and other details that the dbp-system itself doesn't worry about. For this, I want feedback, as I personally don't have much experience with the issues that may pop up with this.

For versioning numbers, I strongly recommend using SemVer; its handling of backwards-compatibility is ideal for dependency systems. With this system, packages should specify version spans like ">=1.3.0" (higher than or equal to major version 1, minor version 3), "==1.3.0" (exactly major version 1, minor version 3) or ">=1.3.0 <=1.3.4" (major version 1, minor version 3, patch between 0 and 4). This is robust enough that an application can even link to dynamic libraries in its dependencies, which pretty much means it's robust enough for everything else too.

At this moment in time I don't know who exactly will implement the package manager and the repo, but I think they are the people to comment on that. Sounds reasonable though.

 

[slaeshjag]

Well, finally making some progress on error reporting... Probably needs to prettifying though.

 

[Silent-Hunter]

Does DBP stand for DragonBox Package?

 

[slaeshjag]

That's what it stands for in my head, mainly because it isn't clear to me if this system will outlive the pyra or not

 

[LEOXD]

If in doubt do the RPM and name it DBP Boxing Package.

(I wonder if there's any chance to get DBP into the LSB, as package format to be used on removable drives.)

 

[ryan27968]

Looks like a good idea to easily install things offline but will it have apt-get or pacman when online?

 

[TrashyMG]

There will be all the standard Debian repo things, including our own repository for Pyra related things... However having a PND like system is still handy for dealing with removable media and limited NAND space.

EDIT: So far the dbp system is a lot more approachable for new developers than the PND system.

 

[slaeshjag]

So, I've gotten some requests on explaining how to actually package stuff into dbp-files. I've hold off from this since all the relevant features aren't implemented or decided yet. However, see this as a preliminary guide that will change in the future.
 
If you read this post after the Pyra has been released, or after a package manager/repo has been released, it is most likely outdated. There will probably be better ways to create packages, eg. something like makepnd.
 
First off: All packages need a package ID, this is something unique like "someprogram-maintainer" that is unlikely to ever conflict with other packages. Packages of different versions should have the same ID if they are compatible (ie. upgrading from version x to version y won't break program z.) See below about default.desktop for details about the package id.
 
The first step is to gather all the files that your program will need inside a folder. If you need to compile the program with a installation prefix, use  /mnt/dbp/images/package-id
This directory is not writable. However, in the event that multi-user support is implemented, the writable directory will not be predictable, it'll be known at run-time as cwd unless the running executable is exported without environment override, see below.
This directory should contain all read-only data needed by the program in the structure it expects.
Generate a squashfs image from this directory. using:
 


mksquashfs directory_with_files image.sqfs

The next step is assembling the meta-data. This is where things will change over time. Since to repo (or package manager exists, and no real discussion has been held about it, this is very much in draft state.
 
Create another directory. In this directory, you'll need at least two more directories, meta and icons.
Inside the meta/ directory, you should put all the .desktop files for your application. One file per program launcher. For .desktop files in general, see the example below.
 


[Desktop Entry]
Version=1.0
Type=Application
Categories=Game;ArcadeGame
Name=OpenTyrian
Comment=OpenTyrian
Exec=opentyrian
Icon=tyrian-128.png
Path=
Terminal=false
StartupNotify=false
Exec name is the file name of the executable to launch without any leading directories. Icon name is a .png file inside the icons/ directory. All icons inside icons/ are exported, so only put the ones used there, and keep their file sizes reasonable. Icon names will automatically be mangled at run time to avoid collisions. The rest of the file is a pretty standard FreeDesktop file.
 
Additionally, all valid .dbp files have a meta/default.desktop. This contains package meta data under a special section, "[Package Entry]". Note that this file can also be a file launcher, and a "[Desktop Entry]" will describe its default action when the package file is executed directly. An example "[Package Entry]" is shown below.


[Package Entry]
Id=SomeProgram_slaeshjag
​Name=Some Package!
Name[sv_SE]=Ett paket!
Arch=armhf
Exec=bin/someprogram;someotherprogram
NoEnvExec=some-term-program
SysDependency=vim>=7.4
PkgDependency=some-dbp;other-dbp
Version=1.3.2
Appdata=notthepkgid
Icon=package_icon.png

Id may only contain the characters A-Z a-z 0-9 -_. (regex: /[-_\.a-zA-Z0-9]+/ ) No spaces!
 
Name is a human readable string with the name of the package. Ignored by the dbp-system, but will probably be used by a future package manager and/repo.
 
Arch is currently not used, but should be armhf for anything compiled for the Pyra. If the package only contains platform independent scripts, you may instead use:


Arch=any
Exec is a ;-separated list of executables inside the squashfs to export into path. So if I export bin/some_exec, some_exec will be in path.
The Exec= key is used to export executables to run in an environment simlar to PNDs. CWD changes, environment variables are overridden to direct writes to appdata etc. For games, this is how you want to export your executable. Since the exported executables end up in path, they should have reasonable names, like "someprogram", not "run.sh".
 
Binaries exported with NoEnvExec= will not have its environment altered. This is preferred for any command line tools.
 
SysDependiencies is a ;-separated list of dpkg packages that must be installed. Currently only one version requirement is allowed (>= or == or <=,) however this will change in the future.
 
PkgDependency is a ;-separated list of dbp's that needs to be installed (listed by package id.) Currently, no version numbers are supported, but these will be handled in the same way as with dpkg packages.
 
Version number is currently ignored, however this will be used in the future for PkgDependency checking. Version numbers must adhere to SemVer.
 

Appdata= specified a different name for the appdata directory. It follows the same naming rules as the package ID.

Icon works the same way as for the .desktop launchers, however inside the [Package Entry] section, it defines the icon for the package file itself.
 
 
Inside [Package Entry], only the entries "Id" and "Version" are mandatory. If Arch= is omitted, Arch=any is assumed.
 
 
Finally, zip these up. The resulting zip file should have "meta/" and "icons/" directly in the root. Append the zip file to the squashfs file and rename the resulting file 


<package name>.dbp
This file should be recognized by the dbp system. In the future, a utility will be provided that validates a dbp file and lists information about it.
File suffix is preliminary and may change after a couple dozen forum polls.
This probably ended up slightly messy, someone with better writing skills than me will have to re-write it at some point.

 

[_wb_]

File suffix is preliminary and may change after a couple dozen forum polls.

A couple dozen forum polls! Woot Woot!

How about the filename suffix ".de", for "Dragon(box) Egg"? Eggs are the packages of nature, and dragon eggs are obviously way cooler than, say, chicken eggs.

It's not a completely original name (there are also Python Eggs, for example), but it is nicer than the generic word "package". Also, it could help to avoid confusion to use the word "package" only for Debian (or whatever distro) packages, so (dragon) eggs are clearly something else. Finally, in my mind, packages are things you still need to install, while eggs can just sit there, in their nest directory.

 

[NoeleyC]

There's always something about PND as a term that made it feel unique to this community ,because of that in my mind I envisaged PYND as a possible successor term for the Pyra. The suffix ".de" instantly just suggests German websites to me!

 

[TrashyMG]

It will always be dbp to me...

 

[Cralex]

dbp works fine for me, but I like the DragonBox Pyra Egg thing too. We could do .pegg or .peg if we want to play with the egg theme, without getting confused with German websites .de

 

[Wally]

I know .PND has a nice ring to it but .DBP makes a lot more sense and we can differentiate between old and new.