Docker vs DBP

[WizardStan]

I've been working with Docker at my new job, and I realized, they're basically PND/DBP packages[*]. Download a squashFS file, union mount, and overlay a data directory, run/restart a command. Everything is automatically sandboxed.
Docker containers can be stored anywhere, even on removable storage, but once mounted they function as if they are part of the filesystem: no need for modifying paths or home or whatever, if something expects to "install" into /etc it can, but that gets sandboxed within the container. No container can access files outside of itself unless explicitly stated, so malicious containers are kept to a minimum. You can, however, make one container dependent on the filesystem of another, so it would be trivial to, for example, create a "midi" container which has installed all the necessary files for doing midi, and then any further containers that want to play midi music can simply depend on this midi container; nothing needs to run in this midi container, it doesn't take up any additional resources, it's simply a read-only filesystem overlay, exactly what a lot of people complained was desirable but not possible with PNDs. A java or mono docker container would be no problem; you could even have multiple versions, a "java6" container, and a "java8" container, in the event that some apps don't work with the new java machine. There's even some progress getting Android apps to run inside of a docker container on Linux.
It even has a really easy and powerful command line installation and execution: "docker run [assortment of options] repo.openpandora.org/TheGame"
If "TheGame" has been downloaded before, it'll start. If it hasn't, it'll download it automatically. In the future you just run "docker start TheGame" and it goes.

It has a lot of pre-existing packages: starting an apache webserver is no problem, add on a mysql container, you can have yourself a complete LAMP setup in minutes.
Not to disparage all the hard work of those working on DBP, but I would strongly suggest looking into using this as a packaging system, even if that meant scrapping DBP. It's well supported, well tested, and will continue being supported for a long time to come.

[*]Seriously, everything about Docker makes me think whoever came up with the idea is also a Pandora owner, and a highly contributing member at that... so which one of you is it?

 

[levi]

PNDs do employ a read-only filesystem overlay; the union mount is setup so that the mounted pnd is (obviously) read only, but the appdata folder union mounted with it is where stuff gets written to. So I'm not sure where you got the idea people were saying it couldn't be done.

But I was under the impression that docker containers were used a little like chrooted environments, containing a whole OS+libs+application, but it seems I was mistaken. That's worth knowing, and I'll have to think a little more about this.

Presumably we'd want some way to automatically generate .desktop files from some embedded pxml file. I guess you could include the .desktop file in the environment, but if you also need a more easily parsable pxml file for other purporses, you'd be introducing redundancy by doing that.

Also, where do programs get logged to? I guess if you're using the old syslog /etc/log files you could probably bodge something in there easily enough, but it must also work with systemd/journald somehow?

Also, out of interest, what format are the mounts?

 

[WizardStan]

containing a whole OS+libs+application,

That was my first impression as well, but no, a container has only what you put into it; you can start with an empty container, copy a single executable into it, and that's all it gets. Or if you need a specific version of a library and can't guarantee that that library will be on the system, you can "apt-get install" that sucker in your container and it'll get installed, just for that container.

Presumably we'd want some way to automatically generate .desktop files from some embedded pxml file.

We can consider the details later, after deciding if this is even viable. I'm imagining we can piggyback off the install process, come up with a standard, like the "run" command which installs and starts the application for the first time actually generates the .desktop file, copies that file (and maybe an icon) to somewhere accessible, and then exits. Then we don't need to keep scanning PND files for PXML and regenerating it every time, the mount process just needs to look at this directory for .desktop files. All it needs to contain is "docker start <Name>" and the docker daemon takes care of the rest: no need to know paths or anything like that.

Also, where do programs get logged to?

They're logged by the container in plaintext, and you can attach a shell to the container to read them out of /var/log, or simply "docker logs TheGame"

Also, out of interest, what format are the mounts?

It's customizable when the docker daemon starts. You can choose aufs, overlayfs, btrfs, vfs, or "devicemapper" (though I'm confused as to how that works, exactly). From what I can gather, it downloads a base container into a directory, creates a new FS file in the selected format, and then union mounts them (plus any additional you may select) at time of execution.

By default docker installs containers into /var/lib/docker. The only problem I can see is how do we get it to look in /mount/*/docker (or wherever) instead, and if more than one exists, select between them for install. We could probably spin up a custom docker daemon that does that with a command line switch pretty easily.
[doublepost=1454251156,1454250812][/doublepost]

So I'm not sure where you got the idea people were saying it couldn't be done.

Oh, sorry, I meant people complaining about the difficulty of having one PND depend on another, like some kind of library PND. You could have such a library PND, mount it, but the libraries it contains go into /mnt/utmp/whatever, not into /lib or /usr/lib as most programs would expect, so there would be some futzing, at least, to get it to work.
With docker you create a new container, install the libraries you want, they go into the container at /lib or /usr/lib, or wherever, exactly as standard, but that only affects the container, the host system remains untouched. Then you can commit that, push it up as its own library container, and any other container can simply union mount against it and they will all share the installed libraries in that container, at /usr/lib, exactly as they'd expect.
[doublepost=1454251578][/doublepost]As a bonus, since all "applications" would be containers, it's dead simple to simply call "docker pause <container>" and SIGSTOP is sent to whatever the running process is. Loop through all running containers and you can effectively create a nice little "system pause" feature, say for suspending, or an overlay menu. One of the problems I had when I did my Pandora Overlay POC was getting a list of PNDs that were currently running. Containers solves that problem perfectly.
[doublepost=1454255632][/doublepost]Actually, having just tried creating an empty docker container I have discovered that it is actually a bad idea. In order for the container to access the hosts bin and lib folders it needs to mount them, and since containers run with root privileges by design (within their own containers it is perfectly safe) that means that anything running in such a container is able to mess with those host directories. So this doesn't actually happen the way I thought it did.

This means any application really does need all of its dependencies included. If a game uses SDL, it needs to include the SDL libraries, it cannot use the ones on the host system. All games using SDL need their own copies of the SDL libraries, for security reasons.
It's a great idea if you have no, or very few dependencies (static linking for everything) but as soon as you start sharing objects you start running into a lot of duplication. But that may not actually be an unfair trade-off; a couple dozen MB of duplicated libraries doesn't seem so bad vs what we'd be getting.
There is a way around that: all common libraries, graphics and audio and any other SO we might think is needed, can be stuffed into a single container, and all other containers can mount this one container.
The downside to this is that someone would need to maintain this library container: whenever a new version is released it would need to be tested, and added to the container; this wouldn't actually be that difficult, just apt-get update the package inside the container. It would also mean that we would still have two copies of many libraries on the system: one for dpkg installed apps and the other for docker installed apps.
The upshot of this is that we could have a fixed library container with known versions of libraries for developers to code towards without worrying about it changing and becoming incompatible. We could even have custom versions of a library stored here, like if the libsdl in the Debian repo is meh because it's armhf generic, but Notaz comes along and does his magic and produces a Pyra specific libsdl, it's pretty easy to put that custom libsdl into our Pyra library container while leaving the default Debian one in the base filesystem.

I dunno, there's pros and cons. I can look more into things like disk usage from duplicate libraries, or how to get it to use containers in multiple storage locations, if people are in any way interested. But if this seems like a dead idea and y'all want to stick with DBP then I won't bother.

 

[levi]

Or if you need a specific version of a library and can't guarantee that that library will be on the system, you can "apt-get install" that sucker in your container and it'll get installed, just for that container.

Can you really? I can imagine dpkg getting awfully confused if it starts concerning itself with stuff on removable media.

Still, even if they haven't got a workaround for that set of problems, it's not hard to unpack a deb file into a container I assume.

Edit: Everything is a container might be worth investigating, and I don't immediately see why you couldn't write a systemd module to mount it before login. Shame you can't interface with the host OS libs and so on, for us at least (though I can see why they've done it for their primary use cases). Don't spend too much time on a workaround though; I get the feeling that if it turns out to be hard it's because we're subverting their use cases, and could have similar trouble down the line.

 

[WizardStan]

Can you really? I can imagine dpkg getting awfully confused if it starts concerning itself with stuff on removable media.

Well that's the beauty of it: it doesn't know it's removable. The container is self contained, it gets mounted from external storage but the system just sees /lib, /bin, /etc...
It is actually a little more complicated than I originally thought (see my last edit, if you haven't) because in order to (safely and securely) install a package the container would need to also contain a copy of apt, the repository cache, and all required libraries up to that point, but once what you need is in the container you can them strip out what you don't, like apt, the repository cache, and any no-longer-required libraries.
But within the confines of the container, docker doesn't actually care where the file is, it sticks it where it is supposed to go.

it's not hard to unpack a deb file into a container I assume

It isn't, you can create a docker image from a tar file, and deb files are basically just tar.gz files with extra stuff. Even easy to add to an existing container.

 

[levi]

Well, one of the arguments against docker files last time round was that if you have different copies of the dkpg database in different containers, the number of permutations those containers can get installed in means the database will often be wrong.

But if you either have the dpkg database in the root fs, or in the 'everything' container, and say that other containers mustn't contain a database, then it should work.

 

[EvilDragon]

But if you either have the dpkg database in the root fs, or in the 'everything' container, and say that other containers mustn't contain a database, then it should work.

But then that database would be lost in case you reflash the OS...

 

[WizardStan]

Why would there be a conflict? There aren't "different copies". The host system has its copy, the library container can have its copy, and never the two shall meet. Reflash the OS and it just downloads the library container again from wherever it came from.

 

[EvilDragon]

Why would there be a conflict? There aren't "different copies". The host system has its copy, the library container can have its copy, and never the two shall meet. Reflash the OS and it just downloads the library container again from wherever it came from.

So, you have e.g. 100 docker files on your SD Card and dpkg keeps them in the database (though I have no idea why it should even do that, if the files are self-contained) and the database is deleted due to a reflash, it recreates the database based on the files on the SD Card?

And it removes the entries from the database when you remove the SD Card?
That sounds like a lot of work to do (knowing how long dpkg takes to update 100 packages on a normal Debian system).

How fast is scanning the files?
Can hundreds of files be scanned and included into the system within seconds (as fast as PND files)?

 

[WizardStan]

So, you have e.g. 100 docker files on your SD Card and dpkg keeps them in the database (though I have no idea why it should even do that, if the files are self-contained) and the database is deleted due to a reflash, it recreates the database based on the files on the SD Card?

No, there's no dpkg database of docker containers. That's only for installed packages. A docker container is a fully self contained application. The idea is that you can enter a docker container as a chroot, use apt-get to install whatever you need just for that container, and the host system has absolutely no awareness of what you've done in that container. You can then exit, save that container, push it to a repository, and then someone else can download it, complete with everything that you had installed and setup to run.
If you wipe the host OS but still have the containers on the SD card then they're still on the SD card, waiting for docker to be reinstalled. Nothing to remove, nothing to scan.

 

[EvilDragon]

That sounds a bit overkill for us, to be honest.
It's awesome for VMs or other virtualized things (servers, etc.), but I'm not sure it suits us.

How easy is it to create a simple docker package?
Any special tools needed? PNDs and DBPs can be created without any special stuff (well, except that you need to install squashfstools if you want to create squashfs'ed files).

Visiting the website, it seems like they want me to register for something... why?

 

[vinipsmaker]

I like the idea of docker. In fact, it's the idea I liked the most from all the discussions that I've read.

We just need to make sure the guest environment has proper access to any hardware accelerating device (gpu...). I run Steam inside a LXC container on my local machine and everything runs fine (even 3D games). There is the need to properly configure pulseaudio so users can use the host mixer to control the volume of guest applications. In my machine, I just the set the PULSE_SERVER envvar, but we might be interested in KDBUS for better performance in a distant future like one or two years from now.

 

[TrashyMG]

I already made an Opentyrian DBP package, there is no turning back now.

 

[Wally]

I've got 200 working DBP Packages.. No turning back here either

 

[Plume]

If making a package is so hard, I doubt that the DBP system is such a good idea.

Anyway, if there is a working system which is used and supported by others, there is no point to reinvent the wheel.

 

[Wally]

It's not hard.. It's just reconfiguring all these packages is quite a task.

If we do decide to take a different approach then fine, i'll do it. DBP is quite functional and built for purpose and uses the .desktop system and slaeshjag has worked quite a bit on it.

 

[WizardStan]

That sounds a bit overkill for us, to be honest.

I don't see why, it is almost exactly PND with a few extra features.

How easy is it to create a simple docker package?

You need docker to be installed wherever you are building it, but it's all the same tool, you could do it on the Pyra. It's part of the the debian and RHEL package repository so as easy to install as squashfs tools.
Put everything you need into a directory, create a docker command file (similar to a PXML), "docker build -t TheGame .". Done, you have a docker container named "TheGame" which you can push to the repo for others to download and run. At the most basic it is no more difficult than building a PND.

Visiting the website, it seems like they want me to register for something... why?

So that you can create a docker repository for them to host for you, it's how they monetize the work. That's not required, you can host your own docker repository wherever you want.
[doublepost=1454273543,1454273294][/doublepost]

DBP is quite functional and built for purpose and uses the .desktop system and slaeshjag has worked quite a bit on it.

That's why I was hesitant to bring this up, because a lot of work has been done already and I'd hate to see it scrapped, but I also don't think we should continue using it just because a lot of work has already been done when it may be worthwhile using an existing and well tested solution.
But docker may also be overkill, or have too many drawbacks of its own.

 

[slaeshjag]

If nobody ends up using DBPs, I won't hesitate to kill it off. It has been hard enough to find motivation to finish up what is left anyway.

 

[WizardStan]

If someone has a working dev board they wouldn't mind giving me access to I could try a proof of concept, maybe "dockerize" firefox or something just to see how big such a container would be vs our existing PND.

 

[slaeshjag]

I doubt you need a devboard for that. I've developed the entire DBP system without touching an ARM system.