A possible PND-system replacement

[slaeshjag]

So, for the past week I've been trying to hide from some real life stuff and forget about this 30°C summer heat. What better way to do it, if not writing software nobody will care about?
 
So far, it is still very early in development, but I've tried really hard to not use more hacks than I need to. There's plenty of time this time. For reference in this post, I will refer to it as the DBP system. This is not in any way final, and for any PND-system successor, it'd be mandatory to have at least 10 polls about its name. I just want something to call it when comparing it to the PND-system, k?
 
The currently implemented features are not final, and they are not the final list of features that will be implemented.
All dependencies so far exist in Debian Jessie, and I intend to keep it that way.

Design goals
* As few hacks as possible. No Zenity. Doing a system()-call instead of just properly implementing it in code is forbidden.
* Correct the meta-data situation with PND's (raw append, wtf?)
* Keep it extendable
* Allow for terminal integration with DBP-installed programs
* Potential for multi-user support
Probably more stuff I can't remember

 
DBP-files
DBP-files consists of an optional SquashFS file, and an appended ZIP-file with meta-data. The core meta-data consists of at least one .desktop-file. All .desktop-files in a DBP are exported if they contain a "Desktop Entry" section. DBP-files themselfs have some core package data inside the "Package Entry" section inside "meta/default.desktop". These .desktop-files are exported mostly unchanged, with the only exception being that the Icon path is changed to an absolute path into the icon cache.

Apart from desktop-files, executables are also exported, via a shellscript template, that invokes run-dbp with the arguments preserved. This template will also set up enviroment variables, if needed. The idea is that executables can be exported without having their enviroment mangled to redirect writes to appdata. As a result of executables being exported separatly, it means that if the .desktop-files themself only refer to executables already present on the system, and no other data needs to be included, it's possible to create a working package without a SquashFS image. This is unlikely to become a norm though, as the idea itself is a bit silly. It's merely a side-effect.

DBP-system
The DBP-system consists of two main parts.
* A daemon running as root, that does package crawling and mounting.
* A user program (dbp-run) that runs a program using package ID and executable path within the DBP as argument

The daemon is configured via one configuration file, that sets stuff like icon/exec/image/union/appadata directories. It also depends on a shellscript template being available, that is used for emitting the executables.

The daemon and user program communicates over dbus's system bus, so for this reason, dbus needs an additional config-file.

Currently implemented features
* Executable exports
* .desktop and icon installation
* Automatic crawling on media mount/umount
* Very, very rudimentary executable launcing (requests to mount the aufs, does a system()-call, requests a umount.)
** Supports multiple instances of a program running without trying to umount when one quits

* Directory watch for removed/added/modified DBPs

* Proper application launching

* Proper error logging (like pndrun_*.out) when launching DBPs.
** Needs to split stderr/stdout so that terminal stuff still works properly.

Features I want to implement Soon™
* Modifying the exec shellscript to set up the enviroment, if needed

* Automatic mimetype association for packages
* Other stuff I've forgotten

Features I'd like to see
* run-dbp spitting out error messages with full gettext translation support
* Some form of dependency tracking
* Allow for some packages to show on the desktop
* Multi-user support (with stuff like per-user appdata)

* Maybe perhaps possibly sometime PND backwards compatibility
* Other stuff I've forgotten


Contributing
Preferably, this thread should focus on this specific implementation, rather than the general direction a PND-replacement should go in. For general talk about a PND-replacement, there's already a thread about it.

Code is currently hosted on github: http://github.com/slaeshjag/dbp
The issue tracker there is probably the best place to file serious feature requests at. Anyone wanting to help is welcome to do so, and may ask on IRC what current issues are the most relevant.

 

[rygD]

Not really related to the topic:

this 30°C summer heat

 
Some days that is the temp inside my house with the AC on, and in most places I have lived that temp outside would make for a cool summer.

 

[_wb_]

Looks good!


The metadata .desktop files is a good idea. How do you plan to do the global metadata? Package name and description for the repo, maybe some other things...


I think there should also be some way to specify dependencies on particular .deb packages (and .dbp packages?), and the launcher should check that the dependencies are met.

 

[slaeshjag]

Currently, the idea is to keep that data under a separate section in default.desktop. Like this: http://p.rdw.se/default.desktop.txt

A longer description should probably be placed in a separate file though, with bigger freedom for line breaks, etc. 

 

[milkshake]

Why not keep the xml data but inside the zip?

Or even better json file

 

[slaeshjag]

Because the desktop files needs to be created anyway. Why not just let the desktop files be and put the required additional information inside them, rather than using some other Cool and Hip format just to generate new files at run-time?

 

[milkshake]

Because json or xml has more support in more languages than the dot desktop file so it's win win for developers.

 

[slaeshjag]

.desktop files are hardly difficult to parse anyway. The time it takes to write a parser is about the same as it takes to google up a library supporting them. They're just ini-files with locale-support, pretty much. With that in mind, I couldn't justify using some convoluted multi-level data description format for something that will only result in flat key-value pair files...

 

[milkshake]

It allows for nested data like multi language title, description collections as well as things like collections of screenshots locations.


It's much better to have the flexibility of a format that is easy to work with and not hard to create, than it is write a new parser with every different language u use.

 

[sebt3]

So that's the reason you were bugging about d-bus on IRC ?

You should have told

Anyway great work

 

[notaz]

XML may have lots of libraries but is rather painful to write and maintain, I wish we hadn't used it..

I think Exophase once said that one of harder parts of making a pandora release was writing the PXML.

 

[milkshake]

There can be tools made to generate pxml, foxblock wrote one and an online one could be done too.

 

[Neelix]

There can be tools made to generate pxml, foxblock wrote one and an online one could be done too.

You could do that...   or you could use a format that is easily readable, and easy to construct in any text editor. 


I think slaeshjag made the right choice.

- Neelix

 

[levi]

There are always corner cases that if you're writing your own parser you'll decide on differently to the next person, making compatibility a mess.  So writing your own parser when there's already one out there is very rarely the right choice IMO.

I'm as anti-XML as the next man, but at least it covers concepts like escaping the syntax characters when you actually want to use them in your text.  Ideally you want something that has a defined way to do stuff like that, but doesn't go to the lengths that XML goes to to acheive that.  Whether that's .desktop files, JSON or something else.  I used YAML in a recent project, but that seems to be missing documentation on how to escape commas in lists, or referencing lists, and while I worked out how pyYAML did it I have no confidence that that's the proper way.

 

[milkshake]

You could do that... or you could use a format that is easily readable, and easy to construct in any text editor.


I think slaeshjag made the right choice.


- Neelix

Yeah it's nice being able to write it easy but when it comes to parsing that data that's a different story.
Anyway I'm not that bothered which way he does it I was just thinking ease of use for developers to take advantage of gathering the meta data, not just users or the packager.

 

[_wb_]

The easiest way to find PNDs on the Pandora is not by extracting the PXML from PND files, but by grepping the .desktop files in /usr/share/applications.

For flat key-value pairs, nothing fancy is needed. Parsing can be done easily enough from shell scripts using simple tools like awk and grep if needed.

It should be easy enough to convert between plain key=value lines, XML and JSON when needed. The important thing is that the discovery daemon and the launcher have minimal overhead, and that things are easy to use from shell scripts -- preferably without needing hacks but by using some standardized tools as proposed: at least dbp-run (probably not that necessary if executable names can also be used directly), but maybe also things like dbp-mount and dbp-umount to just mount the squashfs (dsp-mount would output the path), dbp-info which returns metadata like version numbers and file location, dbp-get to download and install a new dbp package (like apt-get) including its dependencies, and so on.

 

[milkshake]

For example which ever way you do it, it should be pretty easy to update PndAid to get access to the information it needs... assuming your little project was taken on as the defacto format for the Pyra.

The easiest way to find PNDs on the Pandora is not by extracting the PXML from PND files, but by grepping the .desktop files in /usr/share/applications.

Afaik this is not how it works anyway, the PXML is only for the meta data not pnd detection.

 

[_wb_]

The easiest way to find PNDs on the Pandora is not by extracting the PXML from PND files, but by grepping the .desktop files in /usr/share/applications.

Afaik this is not how it works anyway, the PXML is only for the meta data not pnd detection.

I mean if you want to do things with other PNDs from your PND, like ptitSeb's LibreOffice launcher PNDs, or like my Game Soundtracks: it finds installed games and audio players not by looking at PXML but simply by looking for .desktop files. That is a kind of hacky way to do it, so it would be better to have standardized tools for such tasks (dbp-info, dbp-run, dbp-mount etc). But I think this shows how irrelevant it is what file format is used for the metadata -- the only thing that needs to look at that format are those standardized tools/libraries (and maybe the repo server, although it could also use such tools/libraries instead of doing its own parsing).

 

[Wally]

Admit it Milkshake, you don't want to change because you have the repo as you want it

 

[sebt3]

I could wrote a .desktop file parser in every language i know in half an hour. Xml on the other hand (hint bash is one of them )