A possible PND-system replacement


_wb_

Microbe
Staff member
Joined
Apr 5, 2012
Messages
5,387
Age
39
Location
Brussels, Belgium
Let's just use Prolog syntax for the metadata, so we have a Turing complete language to express things and everything is in a logical notation that can be natively parsed provided you are programming in Prolog ;)

(I'm only half kidding here, it would actually not be a bad idea. But I still like simple .desktop files more.)
 

Neelix

Insecticidal Maniac
Joined
Jan 8, 2011
Messages
3,240
Location
Melbourne, Australia
Xml brings benefits I.e. You can validate xml against a schema xsd file, you can transform xml into various formats (probably even into a .desktop) using xslt.


Both xsd and xslt are also written in xml which is neat.


I can only think of these a positives over plain .desktop files.


Plus xml is very well suited to structure data in a logical format.
It may be neat if you happen to like reading and writing raw XML,  less so otherwise.

A few months ago, I exported a bunch of data to an XML file with a view to importing it into another program. (XML was the default export option and I figured it was a fairly well defined format so importing the data shouldn't be a problem)

Then I discovered it wasn't that easy.  I couldn't find any ready made tools to automatically parse it and in order to transform the data with xslt (which was the destination program's recommended way of importing it) I would have had to first write another XML document describing how to translate the first one, which would have required a considerable amount of work.   I gave it up as more trouble than it was worth, and  re-exported the data as CSV instead.

In any case I'm struggling to see how that's an advantage in this context.   The format the data needs to be translated to for use in this context is .desktop.

Case for XML:  there are cumbersome to setup translation tools

Case for .desktop: it doesn't need to be translated.

That's a clear win for .desktop if you ask me.

While the inbuilt validation scheme may be a point in XMLs favour it's not enough to convince me that XML is better for this use case, especially given the simplicity and readability of the .desktop format.

- Neelix
 

milkshake

Advanced Member
Joined
May 18, 2009
Messages
3,735
Age
36
Location
Rotherham, UK
I take it the repo uses a PXML schema to validate submissions, but I don't actually know where to find it, and wouldn't have anything installed that I know of to test a PXML file against a schema currently.
Yes there is an official schema which the repo does use but it wasn't written for the repo.


There is a while bunch of information on the wiki for the pxml specification.

http://pandorawiki.org/index.php?title=PXML_specification


Validation and schema:

http://pandorawiki.org/index.php?title=PXML_specification#Validation


I'm not trying to say xml is the greatest thing ever but it's mature and already used.


Using a schema to validate xml is not hard just Google "validate xml with xsd <your language here>".


Anyway this is just an idea and experiment not official (that I am aware of), but if it was and xml or json or similars wasn't chosen I wouldn't complain I was just get on with it and deal with any fallout (if any) later.
 
Last edited by a moderator:

slaeshjag

¯\_(ツ)_/¯
Joined
Apr 8, 2010
Messages
2,687
Location
~Stockholm, Sweden
The dbp-system keeps track of which executables are exported by which packages, so it'll clean those up as packages comes and goes.

As it is right now, that would be kind of annoying to implement copied binaries. As part of the start-up routine for dbpd, I nuke all stale wrappers. For that, I depend on a magic comment on line 2 to determine if it's created by dbpd or not. I guess a workaround would be to have it on a tmps and assume dbpd will never crash or be restarted, but that's kinda ugly.
I was expecting more like a flat-file db on tmpfs of the exported wrappers. An alternative would be to have a tmpfs folder that contains all of them and only all of them and just nuke it on daemon start.

--

But I'm okay with you not breaking that meta / binary (=zip/sqfs) barrier :p

Alternatively, are you looking to eventially 'cache' the mounts for a little while (atleast when using these wrappers)?
These wrappers will also be used by the desktop files. However, I could add an umount-delay. It'll be a bit messy and probably involve another thread though... (Already got 4 of them <.<) But I'll consider to do it later. A reason against would be that you can't umount your SD-card for X seconds after you've closed all your programs running from it.
 

slaeshjag

¯\_(ツ)_/¯
Joined
Apr 8, 2010
Messages
2,687
Location
~Stockholm, Sweden
So, I've worked some more on this. The core functionality is now more or less in place. I still need to do some more in-depth testing of them, but so far it seems to be working just fine.

run-dbp is still a bit hackish, but a friend said he wanted to rewrite it his way, so I'm going to leave it be for now and let him do that. It's not a massive part of the system anyway. The needed functionality in it is there, though.

So, a current feature-list:

* Packages are discovered both when media is inserted (or if it was already there,) and are discovered when they are downloaded to a watched directory.

* .desktop-files, icons and executables are exported and cleaned up as packages are added or removed

-> Apparently, .desktop-files in /usr/share/applications have their mime-types automatically added to the DE, so mime-type registration is theoretically already in place. Need to test it myself though, next time I'm in a graphical DE.

* The union is mounted, and writes to cwd is automatically redirected to a userfolder "appdata/<appdata>", the subdirectory is either the package ID, or a different name defined in default.desktop.

* Packages can be executed from both exported executables, and launching the package-file by path. When the latter is done, the desktop entry default.desktop is used to determine what it'll run.

* Argument-passing to executables works just like you'd expect.

* Executables with enviroment-override set will automatically get their output logged to a configurable directory. stdout and stderr will still go out into the terminal unmodified if it's logged.

Dependency-checking is probably the next big thing on the list, but I want to iron out any remaining bugs in the base first. I will probably also work on other minor features in the meantime.
 

sebt3

homebrew player (P. & C.)
Joined
Sep 9, 2008
Messages
4,805
Age
40
Location
France
Website
sebt3.openpandora.org
Dependency-checking is probably the next big thing on the list, but I want to iron out any remaining bugs in the base first. I will probably also work on other minor features in the meantime.
Understoud, especially since there are 2 kinds of dependencies to support :- OS packages deps

- other dbp (like the java.pnd, mono.pnd and alike. If it had been built in, it would have been way more used /me hides as he feels guilty here)
 

slaeshjag

¯\_(ツ)_/¯
Joined
Apr 8, 2010
Messages
2,687
Location
~Stockholm, Sweden
I suppose it makes the most sense to do the dependency checking at runtime? Since deps can come and go after the package has been registered.
 

Cralex

Member
Joined
Mar 29, 2014
Messages
34
Even though I don't know enough to understand all the technical details, it's exciting to see this project develop! Keep up the good work!
 

skeezix

Internal Development
Joined
Mar 11, 2003
Messages
8,062
Website
www.codejedi.com
Ooh, fun thread but I didn't notice it till now .. gotta read up.  Great to see everyone carrying the flag!

Keep in mind compatibility or migration strategy/requirement (ie: pnd's coexisting, meaning libpnd/etc running same time.. or forcing everyone to migrate.)

BTW.. zip was not used as metadata on purpose (feel free to ask why we did things the way we did.. it was a huge crazy rush of course, but we did try a dozen variations first.. we had some reasons ;) .. at the time (possibly today even), the zip management had a huge memory overhead (and also at the time, zip didn't work at all for mounting, but did later in firmware life; but recall we had only a couple weeks to put together a system, so thats why :) ; Various mount strategies tended to put the entire thing in RAM at once, which meant including a couple png's in there ate up a lot of memory in a worst case. At the same time, it was considered to use an unpacked zip since the content should be small and the large content wouldn't compress well anyway, and that could permit manual scanning through anyway; but in the end, we've all written out own unzip handlers before and just jumping to the piece in question and unzipping it is probably okay too. Just be ware, that using a mount against a zip chunk is usually a bad idea for performance :p

Before nailing down anything, try it out on your desktop and check the performance in terms of memory usage, kernel support, feature parity, raw usage performance .. though with todays hardware its probabyl a very different situation than last round!

jeff
 

skeezix

Internal Development
Joined
Mar 11, 2003
Messages
8,062
Website
www.codejedi.com
Feature list above is so far more or less what pnd is .. good to reevaluate the foundation, but the hard stuff is..

- package dependancies? Is that an inclusion, or an exclusion to worry about? (probably better to include it this round!)

- package exports .. like a package that can export shared libraries, dictionaries, etc; packages that dump stuff in unusual places (outside of their own tree.. like JVM.)

   -> still need to run installer pnd and let it muck with fileystsems/paths, or can mount at boot and go .. and dependancies on RAM if lots of these, or order of operation loadups, etc

jeff
 

Neelix

Insecticidal Maniac
Joined
Jan 8, 2011
Messages
3,240
Location
Melbourne, Australia
BTW.. zip was not used as metadata on purpose (feel free to ask why we did things the way we did.. it was a huge crazy rush of course, but we did try a dozen variations first.. we had some reasons ;) .. at the time (possibly today even), the zip management had a huge memory overhead (and also at the time, zip didn't work at all for mounting, but did later in firmware life; but recall we had only a couple weeks to put together a system, so thats why :) ; Various mount strategies tended to put the entire thing in RAM at once, which meant including a couple png's in there ate up a lot of memory in a worst case. At the same time, it was considered to use an unpacked zip since the content should be small and the large content wouldn't compress well anyway, and that could permit manual scanning through anyway; but in the end, we've all written out own unzip handlers before and just jumping to the piece in question and unzipping it is probably okay too. Just be ware, that using a mount against a zip chunk is usually a bad idea for performance :p
Some time ago in another thread _wb_ suggested using tar as the metadata container format.  If there are still performance or memory issues with zip, could tar be a viable replacement?

- Neelix
 
Last edited by a moderator:

_wb_

Microbe
Staff member
Joined
Apr 5, 2012
Messages
5,387
Age
39
Location
Brussels, Belgium
Well, tar is not mountable out of the box, afaik.

Maybe that would work: http://avf.sourceforge.net/
We can still use squashfs for the main content, I was just talking about the glue to add the metadata to the content. What we're doing now is just concatenating first the squashfs file, then the PXML file and then the PNG icon. Concatenating stuff is not a very clean way to do it, and it has its limitations. It would be better to glue things together with e.g. tar, so if at some point we need more metadata files (e.g. extra icons for multiple applications in one PND), there's an easy way to do it.
 
Top