Proposal For A Package Management System

[cosam]

As I mentioned elsewhere, I've been thinking about the possibility of implementing a "proper" package manager which can handle removable storage. This would be fully compatible with the PXML/.zip system and works by adding a few (entirely optional) tags to the XML file itself. The location (i.e. path and storage device) of packages modified in this way can be tracked. The system will also happily install unmodified PXML archives according to the user's preferences.

I'm posting what I've come up with so far. Bare in mind it's a very early, very high-level design. I would very much appreciate any comments.

Apologies for the excessively long post - I'm not entitled to upload attachments. Maybe a moderator can sort that out if deemed necessary.

CODE

===============
Package Manager
===============


1. Functional Requirements
==========================

1.1 General
-----------
The package manager's function is to facilitate the installation, removal and
updating of software.


1.2 Packaging
-------------
Software is to be distributed in package form, consisting of one file in which
all necessary data for the installation and removal of that package can be
recorded. The package format is to be compressed and use open standards.
Packages should be compatible with the PXML standard.


1.3 Dependencies
----------------
The system must be able to automatically download and install any dependencies
for a given package. Removal of packages on which others depend must be
detected and deterred by means of a clear message to the user.

Packages are required to list all of their dependencies (including version
information) so that these may be catered for during installation.


1.4 Removable Storage
---------------------
The system must be able to handle the removal of any part of the filesystem
that resides on removable media. A system must be implemented by means of which
the media on which a package resides can be determined. Removable media must
be identifiable by a user defined, natural language identifier.


1.5 Database
------------
The package manager's database will keep track of the status of all installed
packages and optionally an easily updated list of available packages, including
version information. The database must be available independent of which
removable storage devices are mounted.


1.6 Program Launching
---------------------
A means of checking for dependencies when a program is launched must be
implemented. This mechanism will prompt the user regarding which removable
media must be mounted in order to run the requested program. If the program's
dependencies span more devices that is catered for (e.g. the data is stored on
three storage devices, and harware limits mounting to two) the user should be
prompted and suggestions should be made as to how the situation can be
resolved (e.g. by duplicating data from one device to another).


2. Packaging
============

2.1 Applicable requirements
---------------------------
"Software is to be distributed in package form, consisting of one file in which
all necessary data for the installation and removal of that package can be
recorded. The package format will be compressed and use open standards."

"Packages are required to list all of their dependencies (including version
information) so that these may be catered for during installation."

"Packages should be compatible with the PXML standard."


2.2 Package Format
------------------
The package format is based on that used by the PXML system. Packages consist
of a single directory compressed using PKZIP-compatible compression. The
directory (including any subdirectory thereof) must contain at least one PXML
file. Such ZIP archives which support the package management system may include
extra tags in the PXML file(s), enclosed between <package> and </package> tags
directly under the root element. All such tags are optional.


2.3 Dependencies
----------------

Dependencies are indicated by adding the following XML tag:

<depends minversion="2.0" maxversion="2.4">glibc</depends>

Would be used to register a dependency on the "glibc" package which of version
2.0 thru 2.4 inclusive. Multiple "depends" tags are permitted.


2.4 Scripts
-----------
Two extra XML tags can be used to run scripts before (preinstall) or after
(postinstall) package installation. These can be used to shutdown/restart
services, prepare or clean up files and directories, etc.

<preinstall>scripts/runfirst.sh</preinstall>
<postinstall>scripts/cleanup.sh</postinstall>

Similar tags can be used to run scripts before (preremove) of after
(postremove) a package is removed:

<preremove>scripts/shutdown.sh</preremove>
<postremove>scripts/realclean.sh</postremove>

All paths are relative to the PXML file in question. No more than one of each
of these tags may be present in one <package></package> element.


3. Dependencies
===============

3.1 Applicable requirements
---------------------------
"The system must be able to automatically download and install any dependencies
for a given package. Removal of packages on which others depend must be
detected and deterred by means of a clear message to the user."


3.2 Repository
--------------
The packaging system is dependent on a yet to be determined type of repository
where software packages are made available for anonymous download. Multiple
repositories can be configured for used by one system.


3.2 Installation
----------------
Installation of a package implies that this package be downloaded form a
previously configured repository. Ideally the repositories will provide an
index of available packages so that dependencies can be resolved before
downloading begins. This is however not required as any package can be
downloaded, examined for dependencies, and these dependencies can subsequently
also be downloaded. This process can continue recursively until all depend-
encies are met.

Once dependencies are resolved, the package manager will download the package
(and if dependencies have already been resolved, the packages on which this
depends), extract them, and run the appropriate scripts. The package manager
database is updated to reflect the new status of each installed package.


3.3 Removal
-----------
Removal of packages entails the deletion of the extracted packages from their
storage location and updating the database to reflect this new status. Packages
on which others are dependent cannot be removed without confirmation from the
user. Packages which are no longer required, due to being installed solely to
fulfil the dependencies of another (now removed) package, can optionally be
removed after user confirmation.


3.4 Updating
------------
Updating a package is tantamount to removing the old package and installing a
later version.


4. Removable Storage
====================

4.1 Applicable requirements
---------------------------
"The system must be able to handle the removal of any part of the filesystem
that resides on removable media. A system must be implemented by means of which
the media on which a package resides can be determined. Removable media must
be identifiable by a user defined, natural language identifier."


4.2 Package/storage Tracking
----------------------------
Each removable device attached to the system must be able to be uniquely
identified. Depending on the media type, this may be a software-accessible
serial number, a volume label, or something else. Where possible, an identifier
which does not change over time should be used. The package manager uses this
unique identifier to track the storage device on which each package is
installed.

During installation of a package, the user should be prompted to select a
storage device onto which the package will be installed. It should also be
possible to configure a default device to avoid this prompt.

The same package may be installed on multiple storage devices to allow programs
to access their dependencies from different sources when necessary. All
locations are recorded in the database. It is necessary that all copies of a
particular package be updated to the same version. This can be performed when
a package is updated, i.e. the user is prompted to insert all media on which
the package is known to reside. Should the user decline, the status of the
package on the out-of-date media should be marked. Upon mounting an out-of-date
device, the user should be prompted to update the package.

Packages may be removed from devices independently (i.e. a package can be
removed from one card yet remain installed on another) or globally (i.e. all
copies are removed from all devices known to contain the package) at the user's
discretion.


4.3 Natural Language Labels
---------------------------
The package manager database will record a natural language label for each
storage device. This label will be mapped to the device's unique identifier
as defined above. The natural language label may be altered at any time without
further affecting the working of the package manager.


5. Database
===========

TBA


6. Program Launcher
===================

6.1 Applicable requirements
---------------------------
"A means of checking for dependencies when a program is launched must be
implemented. This mechanism will prompt the user regarding which removable
media must be mounted in order to run the requested program. If the program's
dependencies span more devices that is catered for (e.g. the data is stored on
three storage devices, and hardware limits mounting to two) the user should be
prompted and suggestions should be made as to how the situation can be
resolved (e.g. by duplicating data from one device to another)."

6.2 Launch Wrapper
------------------
A wrapper can be used to assess the availability of dependencies when the
user requests to launch a program. The launcher must also be able to start
regular (i.e. non-packaged) executables.

The launcher will use the package manager database to analyse the dependencies
of the program to be launched and ensure that the required storage devices are
available. Should the required devices not be mounted, the user will be
prompted to insert/attach the appropriate device. Should this not be possible
due to hardware constrains (e.g. the dependencies are spread over three devices
and only two can be inserted simultaneously) the launcher will suggest to the
user how the problem can be resolved. This may involve copying or moving a
package to a different storage device. This copying or moving should be
automated as much as possible.

Calling the launcher may be achieved from within a program directly via a
function provided by the package manager library. The prototype of this
function will follow that of the the exec() system call. A stand-alone tool
will also be created that uses this aforementioned library and can be run from
the command line with arguments, e.g.:

$ launch myprogram -option1 -option2

Where "launch" is the wrapper program, "myprogram" the executable to launch.
The remaining arguments are passed as arguments to the program being launched.

 

[Zoxc]

I would like a package manager for each SD card. This way it would be easy to manage them via a computer and it doesn't rely on flash storage.

 

[atomicthumbs]

I would like a package manager for each SD card. This way it would be easy to manage them via a computer and it doesn't rely on flash storage.

Uh. What? :blink:

 

[Squidge]

I think he means he wants a package manager, but he also wants to manage the packages from both PC and Pandora.

 

[javaJake]

I've read the whole thing through, and I like your ideas. Here are my thoughts:

• First, if the packages required are already available in a specified directory, then the package manager would not attempt to download from the internet. (This is good for situations where your Pandora doesn't have 'net, but you still want to install something.)
• Let's say you have Pidgin installed, and you have GNOME, DBus, and libpurple+Pidgin installed on three different SD cards. (Yea, I know, terrible example, but work with me here.) As you said, the package manager should detect this situation when you try and launch Pidgin. At this point, though, instead of merely mentioning what the user could do to resolve the issue, the package manager should instead provide an option to automatically move one of those packages. It should either merely copy installed data from the original disk to one of the other three available disks (NAND, plus two other SD cards), or if that won't work, it should provide an option to uninstall the package and then reinstall it on another disk.This part is definitely optional, but probably very good nonetheless. The package manager should have a function where it determines the most optimized situation for the two disks. Using this, you could preform cleanup on the cards (to bunch together the dependencies as close together as possible) or resolve the above problem.

[*]I can see this split-dependencies-over-multiple-cards feature becoming an issue. Thus, the package manager should discourage, but still support, that kind of operation, especially when more than two disks are involved.

 

[bzar]

I don't have time for a very long post, but I think each storage device should have it's own dependency tree (each device satisfies dependencies of software installed on it). This would cause package duplication between storage devices, but would:
1. ensure that all software on the device is runnable without other dependencies than those satisfied by packages on the device (or in "static storage" eg. internal flash)
2. get rid of the requirement to unique-id all storage devices
3. still save disk space by having each required library only once per storage device (if needed)

EDIT:
A command-line interface could be something like:

CODE
$ pinstall foobar
Package name: foobar
Version: 1.2

1. SD-1:
libraries needed: libfoo, libbar
space needed: 250KiB
2. SD-2:
libraries needed: libbar
space needed: 210KiB
Choose media: 2

Installing libbar
retrieving...
installing...

Installing foobar
retrieving...
installing...

done.
$



Or something

 

[cosam]

First, if the packages required are already available in a specified directory, then the package manager would not attempt to download from the internet. (This is good for situations where your Pandora doesn't have 'net, but you still want to install something.)

Good one. I've been thinking of defining different types of repository, and one type could be a simple local directory.

Let's say you have Pidgin installed, and you have GNOME, DBus, and libpurple+Pidgin installed on three different SD cards. (Yea, I know, terrible example, but work with me here.) As you said, the package manager should detect this situation when you try and launch Pidgin. At this point, though, instead of merely mentioning what the user could do to resolve the issue, the package manager should instead provide an option to automatically move one of those packages. It should either merely copy installed data from the original disk to one of the other three available disks (NAND, plus two other SD cards), or if that won't work, it should provide an option to uninstall the package and then reinstall it on another disk.

I think this is covered by 6.2. I want to automate it as much as possible, but I think it's important to prompt the user before copying/moving packages. Something along the lines of "You can't run this because X is on card Y. I can fix this by copying X to card Z. Shall I go ahead and do this?"

This part is definitely optional, but probably very good nonetheless. The package manager should have a function where it determines the most optimized situation for the two disks. Using this, you could preform cleanup on the cards (to bunch together the dependencies as close together as possible) or resolve the above problem.

Yes, I'm thinking about some kind of optimisation tool which would do this. Probably with different modes of operation like "save as much space as possible", "minimised media swapping", etc. Something for later though, I think.

I can see this split-dependencies-over-multiple-cards feature becoming an issue. Thus, the package manager should discourage, but still support, that kind of operation, especially when more than two disks are involved.

Absolutely. It's not going to scale well to more than a few cards. Default behaviour would be to install a package along with its dependencies on the same card. Maybe making use of some of the optimisations above.

I don't have time for a very long post, but I think each storage device should have it's own dependency tree (each device satisfies dependencies of software installed on it). This would cause package duplication between storage devices, but would:
1. ensure that all software on the device is runnable without other dependencies than those satisfied by packages on the device (or in "static storage" eg. internal flash)
2. get rid of the requirement to unique-id all storage devices
3. still save disk space by having each required library only once per storage device (if needed)

That's one way of doing it, yes. I would imagine this uses a subset of the "full" functionality proposed and could therefore be offered as an option. Then again, it's also a possible starting point on which to build the multiple device version later.

I would like a package manager for each SD card. This way it would be easy to manage them via a computer and it doesn't rely on flash storage.

Not what I originally had in mind, but definitely do-able. Nothing stopping you from having the a database for each card, stored on the card itself. You could then manipulate the packages from any computer.

 

[bigtruck]

Hmm... I like the idea, but it does seem as though it'll require cooperation from the developers of individual apps. For example, if a several apps are statically linked to the same library, it's still wasted space. Ideally developers would release a plain zip containing all the libraries for those who just want to drag and drop, as well as the app by itself with a list of dependencies. It will probably also require volunteers; for example there will need to be separate packages for the required libraries which will probably not be created by the developers of the app which requires said libraries.

Though I do see the benefits of the approach you outlined, I am more in favor of Zoxc's suggestion (assuming I am interpreting it correctly): Completely independent databases of installed packages on the SD cards. Well, independent in the sense that the package manager won't keep track of things on non-present cards. I explained this in more detail in the Apt Repositories thread in General Talk. Honestly, it seems that if you're going to have more than one SD card with apps on it, you're asking for a headache. If you want more than two, you're insane. I'm probably going to have one with apps and a second with music, movies, ROMs, etc.

 

[sindbad]

Though I do see the benefits of the approach you outlined, I am more in favor of Zoxc's suggestion (assuming I am interpreting it correctly): Completely independent databases of installed packages on the SD cards. Well, independent in the sense that the package manager won't keep track of things on non-present cards. I explained this in more detail in the Apt Repositories thread in General Talk. Honestly, it seems that if you're going to have more than one SD card with apps on it, you're asking for a headache. If you want more than two, you're insane. I'm probably going to have one with apps and a second with music, movies, ROMs, etc.
Apt-walkabout is at least proof of concept.

 

[cosam]

Hmm... I like the idea, but it does seem as though it'll require cooperation from the developers of individual apps. ... It will probably also require volunteers;...

Although co-operation would indeed be required in order to make use of the full potential of the system, one of the major goals is to ensure that compliance is not mandatory. After all, "regular" packages are functionally the same as "package management aware" packages with no dependencies. In the end it will come down to whether people find such a system useful enough that it becomes adopted by a significant number of users. Hopefully the fact that the package manager itself is essentially just another app will help.

Though I do see the benefits of the approach you outlined, I am more in favor of Zoxc's suggestion (assuming I am interpreting it correctly): Completely independent databases of installed packages on the SD cards.

Yes - the more I think about it, the more it seems this is the logical starting point. Tracking multiple devices would be nice, but it's probably not going to be required by the majority of users.

 

[dentrado]

UnionFS should be the solution if it works as described here, (I know you already know SteveM but I think I should post it here if somebody missed it)

 

[cosam]

Yep, I really hope the UnionFS method pans out. It's a much more elegant solution and has the advantage of using established tools. If there are enough people interested making the .deb packages, it could really take off.

 

[javaJake]

Ideally developers would release a plain zip containing all the libraries for those who just want to drag and drop, as well as the app by itself with a list of dependencies.
I imagine that the package management software wouldn't split off and create an entirely new database, but use the existing ZIP and PXML files developers would have created already. Edit: ...and supplement where needed (such as dependencies).