Apt Repositories For Pandora

[TaG]

What exactly goes into one of these zips (I know roughly)? If someone wants to try an Ubuntu/Debian Armel app, how do you you get the app, libs, dependancies etc, package them and use it? Are there anticipated to be so many apps packaged this way that "an average user" won't go looking for more?

I'm certain I've missed something obvious (which is why I'm asking), but this seems harder than apt-get install.

I do remember comments some while back that all apps would need porting to Pandora. Is this still true with the latest incarnation of the OS/firmware, and therefore when porting one might as well create the zip?

That's not really true.
Lots of apps have ARM ports and most desktop apps might need some porting if they make some nasty assumptions, but porting them would be relatively easy.
All the apps will need to be packaged for the Pandora, though, as the devs don't want to use a well-established packaging system like DEB or RPM.

With Debian Armel, you'd just have to type apt-get install *something* and as long as it's in the repository, it will be installed without further problems. The devs argue that not everyone would have an internet connection and some special system would have to be created to manage packages, as we're dealing with removable media.
This is what I don't get - apt-get is simple. Squidge is saying the zips are really simple, If I want to try a valid Ubuntu/Debian Armel app what do I have to do?

 

[Squidge]

95% won't require any libraries? What kind of magic applications are these? Written totally from scratch, not using a single graphic library (be that qt, gtk, opengl, whatever)? Talk about bloatware. Presumably you are

I mean libraries which will not by default by in the firmware. Things like libc, qt, gtk, opengl, will be in the firmware by default, and thus any applications that use these will not have any dependancy issues.
QUOTE

This is not at all about saving space on an SD card. I think you're missing the point completely. For example, if you just unpack a zip file, how are you supposed to remove that piece of software?

You tell the installer to delete it, or you just delete the directory yourself. The installer knows what software is installed by the PXML files, so it doesn't matter if you installed it via the installer or just copied the files from someone elses sd card.
QUOTE

The point about libraries is that I of course want every application which uses lib X to use _the same copy of lib X_.
and you can do that if you wish, but that will not be the default configuration.
QUOTE

As you can see I made the same argument again, and I'm totally baffled as to why that would not be a good idea. The only time it wouldn't matter is if you want to run only a single program at a time, at all times. If that's so then I can't see why you would want something as powerful as the Pandora to begin with.
Theres nothing stopping anyone from including all the libraries with there app instead of static-linking, and those people who want the libs seperate can do so. However, including all libraries may create a significantly larger archive to install, and you are no longer including just the routines you are using. (and no, having the libraries as a seperate archive would be no good, they need to part of the application archive)

 

[Pleng]

Sorry for my late entry into this thread. Hope I'm not repeating anything said before...

QUOTE

The only way zip files will be a mess on Pandora is if you try and save a couple of meg of space on your sd card because you want all the libraries seperate, rather than leaving them with the application, which can easily be fixed by static linking.



This is not at all about saving space on an SD card. I think you're missing the point completely. For example, if you just unpack a zip file, how are you supposed to remove that piece of software? Without a package manager? (And no, I don't think 'remove manually, file-by-file' is a good argument. That's messy on a desktop, that'll be messy on Pandora.)

It's hardly messy. It was the way things were done in the early DOS days and for self-contained apps (which ALL Pandora apps should be), it's the easiest way. Things are going full circle, with the adoption of portable apps on the Windows. And WHY are such apps popular? Because they can be put on an SD card or portable hard disk and run without the need to install. They are designed with PORTABILITY in mind (see any similarities with Pandy?)

I download MYGAME.ZIP. I unzip it to a folder called MYGAME. When I no longer want it, I remove the MYGAME folder (sorry, directory). How difficult, or messy is that? Compared to any kind of install routine which, if not implemented properly, could end up leaving files all over my computer?

QUOTE

QUOTE

QUOTE

If every package is self-contained, this would mean that an application uses its own copy of every library it uses. (With the possible exception of 'core' libraries that are presumed to always be available somewhere on the NAND). That'll completely bloat the memory footprint of the applications.


That argument is becoming boring now, everyone states the same argument, completely ignoring the reasons why we don't think it's a good idea.


As you can see I made the same argument again, and I'm totally baffled as to why that would not be a good idea. The only time it wouldn't matter is if you want to run only a single program at a time, at all times. If that's so then I can't see why you would want something as powerful as the Pandora to begin with.


It's really not that difficult. I, for instance, have several SD cards. A couple of 2Gbs a couple of 1Gbs, a 4Gb and soon a 16Gb. I will have my apps stored on a variety of different cards. Therefore if SquidgeSNES, on my 2Gb cards, needs the same library as CoolGame on my 1Gb Card, I'll need the library on BOTH.

Package management systems really only make sense on a system with fixed storage. The Pandora is not a laptop. If you want to install a Linux distro on a big SD card and use it in the same way you'd use a laptop, complete with package management, then go ahead. But you are not using the device in the manner it was designed for, so you can't expect your ideals of how a system should be to be supported as the official method.

The thing was designed to be a portable gaming system. It has the potential to do a lot more. But apps will need to be packaged in a convenient form which makes logical sense to the user, and is the only logical way to do so when taking into account the design goals of the system.

 

[Squidge]

This is what I don't get - apt-get is simple. Squidge is saying the zips are really simple, If I want to try a valid Ubuntu/Debian Armel app what do I have to do?

On a desktop PC with Internet access, "apt-get prog" is indeed simple. However, how simple is it on a desktop PC /without/ internet access, compared to a self-contained zip file you can unarchive wherever you like?

 

[bigtruck]

I've never used apt-walkabout (it's been a long time since I've even used apt), but after reading the page I don't think it would address the issues the Pandora would have. It seems to be intended to upgrade regular systems which have no internet connection through sneakernet rather than be used on a system which has frequently changing storage media. Correct me if I'm wrong, but apt-walkabout has the package cache on removable media, not the installed packages' files. The Pandora would likely require a package manager written from scratch, or at least a heavily modified version of an existing one.

QUOTE
On a desktop PC with Internet access, "apt-get prog" is indeed simple. However, how simple is it on a desktop PC /without/ internet access, compared to a self-contained zip file you can unarchive wherever you like?

Which is why as I previously mentioned there should be a version of the package manager for regular desktop OS's that upgrades and installs Pandora packages directly to the SD card. It wouldn't really have to be all that different from the version the Pandora would use. Also, I don't quite understand what the big deal about being able to put apps where you want is. Personally I like that I don't have to care where my programs are, they're all somewhere in /usr, /opt, etc. (no, not /etc )

 

[TaG]

This is what I don't get - apt-get is simple. Squidge is saying the zips are really simple, If I want to try a valid Ubuntu/Debian Armel app what do I have to do?

On a desktop PC with Internet access, "apt-get prog" is indeed simple. However, how simple is it on a desktop PC /without/ internet access, compared to a self-contained zip file you can unarchive wherever you like?
Thanks Squidge. I do get what you are saying, and I am not arguing with it - it does sound simple enough. I just don't understand what I would have to do if I wanted to use an app that no one else had packaged.

 

[Squidge]

Thanks Squidge. I do get what you are saying, and I am not arguing with it - it does sound simple enough. I just don't understand what I would have to do if I wanted to use an app that no one else had packaged.

You could compile it from source? Or, if you could override the settings in the package file(s), you could force installation and database to an SD card and use apt-get or similar. Sure, it would download lots of useless guff due to the empty database, but it should work. Maybe there can be a default database on the firmware to stop the useless guff, I don't know. Thats up to the file system creators. (apt-get will most likely be on the firmware simply due to the reason that it runs Ubuntu).

 

[AireTamStorm]

apt-get will most likely be on the firmware simply due to the reason that it runs Ubuntu
That is plenty. That means that we can have a repository if we want, and you can keep your zips. Everyone is happy. You did however, contradict this in your original post.

If APT is included in the firmware, I will be making packages with my own repo for all the software on the OpenPandora archive, as well as all of their libraries. I'll also write how-tos for doing union mounts.

 

[EvilDragon]

Short question: will there be an easy option to have it check for updates in the archive? So we don't have to browse the archive and check for ourselfs?

Yes, it is planned that also auto-updating will work.

If every package is self-contained, this would mean that an application uses its own copy of every library it uses. (With the possible exception of 'core' libraries that are presumed to always be available somewhere on the NAND). That'll completely bloat the memory footprint of the applications.

Wrong, since ALL standard libraries you'll need on the Pandora WILL be included in the OS.
When new libraries appear, they will be included within the next update.

The few programs that use other libs need to include them - but that bloating will be about 3 meg for 100 programs or so...

I don't know about you, but the last time I had a package management database corruption / out of sync issue was two years ago on a known-to-be-buggy SuSE installation. I've never had it happen since with RPM and never seen it period with APT.

I actually had quite a few with Debian on a server. DPKG refused to do anything until that got fixed. I managed to take care of that - but what should a user do, who runs into such problems and has no idea how to fix it? Reformat the card? And reinstall everything? Those users will hate the Pandora.

What happens, if a normal user deletes programs from an SD Card using a card reader? The database is out of sync. Sure, you SHOULDN'T do that. But it's better if nothing like that can even happen.

QUOTE

The GP2X way of doing things is a headache.



It's not really the GP2X way. Thanks to PXML, the apps will be automatically added to the menus. Thanks to an installer, they can be downloaded and installed automatically - just like you can do with a package manager. But without the problem, that anything goes wrong when a user moves or deletes some programs around on the SD Card.

Okay, let's do it the other way round:

If anybody shows me a way to include a package manager which makes the following without any problems possible, we can think about including it.

It should be possible to:

- Install the programs WITH and WITHOUT the Pandora. A lot of guys like to prepare the SD Card without having the handheld around. So it should be possible installing the programs onto the card by simply using a card reader, regardless of the OS.

- Organize the SD Cards the way you like. Move programs around, delete them, etc. Without corrupting any database or getting out-of-sync.

- If the card corrupts and a few programs need to be deleted because they have faults or even worse the database has faults, there should be no problem rebuilding it somehow. The user SHOULD NOT have to reinstall EVERYTHING.

Also, if anybody could tell me the ADVANTAGES of a package management system on a REMOVABLE media, I would be glad. As said, the Firmware will use package management, as you can't change the NAND and it makes it easier to upgrade the firmware.
But on the cards, there are A LOT of DISADVANTAGES in my opionion (like I told above).

SD Cards DO corrupt. If there's a power loss or the user removes it before the cache is written to it, the database can be corrupted. How do you recover it?

 

[scarath]

apt-get will most likely be on the firmware simply due to the reason that it runs Ubuntu

That is plenty. That means that we can have a repository if we want, and you can keep your zips. Everyone is happy. You did however, contradict this in your original post.

If APT is included in the firmware, I will be making packages with my own repo for all the software on the OpenPandora archive, as well as all of their libraries. I'll also write how-tos for doing union mounts.

Agreed, this is the perfect solution, if APT (or as Chero suggested 'ipkg') is included in the firmware (and includes only the firmware repo to start with) then there will be no problems. Us fans of APT will use it and people who think its a bad idea will not have to.

AireTamStorm, i hope more devs follow this idea and put their software on a repo as well as in zip format on the Panda library site.

 

[bigtruck]

Hmm, I'm actually starting to lean towards the other side (no package manager). I've read some of the information about PXML, but I don't know everything about it, so I do have some questions/suggestions.

First, the installer. I didn't know PXML was going to have one. Could you elaborate more on this? Is it going to function similar to a package manager, in the sense that you the installer will be on the Pandora and you can download a list of packages and install them without ever touching a web browser?

There should also be a PXML file (or another file) that keeps track of all the files installed by the package so that it can be removed automatically, which would in turn make automatic upgrades possible. Is this what you meant by auto-updating, or did you just mean updating the database of available packages?

Essentially, doing it this way the differences with this and a package manager would be the lack of dependency resolution (I'll suppose I'll have to trust that nearly all the libraries used will in fact fit in the NAND. If so, I don't have much of a problem with this approach) and the lack of an actual database of installed files. However, such a database could be automatically generated by scanning the SD card for all PXML files, which might be useful for certain operations.

However, to play devil's advocate, I don't really see why people need to organize the actual contents of the packages. People can manipulate the packages themselves, they don't have to deal with their contents. The SD card could have a .packages folder or something at the card's root that has the packages' files.

PXML seems like it could reach a good middle-ground between the GP2X's simple zips and full-blown package management with dependency resolution and such.

 

[hessiess]

linux - package management and dynamic linking = bloated and useless OS like windows.

 

[EvilDragon]

First, the installer. I didn't know PXML was going to have one. Could you elaborate more on this? Is it going to function similar to a package manager, in the sense that you the installer will be on the Pandora and you can download a list of packages and install them without ever touching a web browser?

The PXML is just a file format that has to be included with every program within the ZIP archive. It's standard XML and contains all the informations needed for every launcher.
So basically, to install a program, all you need to do is extract it anywhere on the SD Card. You can do that yourself. Or the Installer can extract it (organizing it simply by using the category structure within the PXML file). If the user moves a directory around, the GUI won't be affected by this, as it simply uses the PXML file within the directories.
So you can extract the ZIP file yourself, move it onto the SD Card and tell the installer to install (extract) it or use an online-based interface to download and install it.

QUOTE

There should also be a PXML file (or another file) that keeps track of all the files installed by the package so that it can be removed automatically, which would in turn make automatic upgrades possible.



That's not needed - as all you need to do is crawl through the SD Card, find all the PXML files and either check with the website if there are more recent versions and download and install it, if you like.
This can all be done by the installer. Also removing is easy: Simply remove the directory of the program. Can be done with the installer or by simply deleting it from the SD Card with your computer.

QUOTE

Is this what you meant by auto-updating, or did you just mean updating the database of available packages?



As said, no database needed. Of course, GUIs can use databases if they want to (or cache the PXML files somehow), but it cannot corrupt, as it can be rebuilt anytime by simply crawling throug the card and reading the PXML files again

QUOTE

However, to play devil's advocate, I don't really see why people need to organize the actual contents of the packages. People can manipulate the packages themselves, they don't have to deal with their contents. The SD card could have a .packages folder or something at the card's root that has the packages' files.



Well, they shouldn't. But you can be pretty sure a lot of them will.

It's also easier for devving or manipulating games (i.e. graphics, levels, etc.) as you can simply copy the stuff you want to try out on your SD Card, delete it if you're finished with it and don't have to worry about databases or any package manager.

linux - package management and dynamic linking = bloated and useless OS like windows.

You didn't even read or understand anything I said, did you?

There WILL be a package manager for the firmware, simply not for removable medias like SD Cards as it leads to a lot of problems.

And there WILL be 99% dynamic linking - except for any non-common libraries.

 

[hessiess]

QUOTE
You didn't even read or understand anything I said, did you?

There WILL be a package manager for the firmware, simply not for removable medias like SD Cards as it leads to a lot of problems.

And there WILL be 99% dynamic linking - except for any non-common libraries.


just didn't read the whole thread

 

[bigtruck]

Alright, I'm satisfied. From what I understand, automatic package updates and downloading new ones is going to be fairly simple. That was my main concern; I just didn't want to have to do everything by hand.

As for memory and storage usage, is it really that likely that many apps will be using the same library not included on the NAND? We might only have 256MB of space, but I've been using my laptop for over a year and that only has 250MB worth of libraries installed, and that's with 64bit and 32bit copies of certain libraries installed. If multiple apps start using the same non-included library, we could submit a request that that library be added to the Pandora's package management.

 

[cosam]

If anybody shows me a way to include a package manager which makes the following without any problems possible, we can think about including it.

The system I envision would use a database, local to each card, that is in essence a cached copy of existing PXML files, a list of repositories, and a list of packages available from those repositories. With that in mind...

It should be possible to:

- Install the programs WITH and WITHOUT the Pandora. A lot of guys like to prepare the SD Card without having the handheld around. So it should be possible installing the programs onto the card by simply using a card reader, regardless of the OS.

With the database on the removable media, there's nothing stopping one from running the package manager form any system capable of accessing this media, provided the software has been compiled for the platform in question.

- Organize the SD Cards the way you like. Move programs around, delete them, etc. Without corrupting any database or getting out-of-sync.

To me, the point of a package manager is that I don't have to worry about where things get placed. The system should put them in a logical place, but I don't necessarily care where that is as long as I can run the software. The system would however support random placement of packages, or automatic placement according to PXML categories - as the database is based purely on the contents of the PXML files, moving or deleting packages is a matter of rescanning the PXML files (which you would need to do anyway to keep your launcher menu up-to-date).

- If the card corrupts and a few programs need to be deleted because they have faults or even worse the database has faults, there should be no problem rebuilding it somehow. The user SHOULD NOT have to reinstall EVERYTHING.

The same scanning process would rebuild the database based on the remaining contents of the card. It can also prompt you as to what's missing and therefore tell you what needs to be replaced (or even replace it for you automagically).

Also, if anybody could tell me the ADVANTAGES of a package management system on a REMOVABLE media, I would be glad. As said, the Firmware will use package management, as you can't change the NAND and it makes it easier to upgrade the firmware.
But on the cards, there are A LOT of DISADVANTAGES in my opionion (like I told above).

I don't know what the current situation of the NAND is, but space is finite and supposedly at a premium. Duplication of data is generally considered a Bad Thing™ and is to be avoided, especially if you take large libraries, interpreted languages and other such shared resources into consideration. I may be overestimating the requirements, but these things don't tend to get any smaller as time goes on. I'd also rather have as much as possible on an SD card where it can be easily replaced, than have something screw up on NAND (although admittedly, if your NAND goes bad, a few missing libraries will probably be the least of your worries).

 

[EvilDragon]

If anybody shows me a way to include a package manager which makes the following without any problems possible, we can think about including it.

The system I envision would use a database, local to each card, that is in essence a cached copy of existing PXML files, a list of repositories, and a list of packages available from those repositories. With that in mind...

Well - that's exactly what I designed PXML for and talked to some GUI devs about weeks ago. But that's not a package management system - it's basically just a cache for the PXML files.

QUOTE

It should be possible to:
- Install the programs WITH and WITHOUT the Pandora. A lot of guys like to prepare the SD Card without having the handheld around. So it should be possible installing the programs onto the card by simply using a card reader, regardless of the OS.

With the database on the removable media, there's nothing stopping one from running the package manager form any system capable of accessing this media, provided the software has been compiled for the platform in question.

There's already one disadvantage: You need a software for it, you can't use a standard explorer.
I know quite a few gp2x users use their free time at work to download new apps and games and put them onto the Pandora. Or at an internet cafe with card readers. Download, put on the card, and that's it.
Very often you can't run programs on such systems.

QUOTE

- Organize the SD Cards the way you like. Move programs around, delete them, etc. Without corrupting any database or getting out-of-sync.

To me, the point of a package manager is that I don't have to worry about where things get placed. The system should put them in a logical place, but I don't necessarily care where that is as long as I can run the software. The system would however support random placement of packages, or automatic placement according to PXML categories - as the database is based purely on the contents of the PXML files, moving or deleting packages is a matter of rescanning the PXML files (which you would need to do anyway to keep your launcher menu up-to-date).

Eeeermm... you're NOT talking about a package manager system here. You ARE repeating MY arguments I posted at the thread about the PXML definition about WHY I want PXML and NOT a package manager.
So at the moment, you are AGAINST a standard packag manager and FOR the simple PXML solution.

QUOTE

Also, if anybody could tell me the ADVANTAGES of a package management system on a REMOVABLE media, I would be glad. As said, the Firmware will use package management, as you can't change the NAND and it makes it easier to upgrade the firmware.
But on the cards, there are A LOT of DISADVANTAGES in my opionion (like I told above).

I don't know what the current situation of the NAND is, but space is finite and supposedly at a premium. Duplication of data is generally considered a Bad Thing™ and is to be avoided, especially if you take large libraries, interpreted languages and other such shared resources into consideration. I may be overestimating the requirements, but these things don't tend to get any smaller as time goes on. I'd also rather have as much as possible on an SD card where it can be easily replaced, than have something screw up on NAND (although admittedly, if your NAND goes bad, a few missing libraries will probably be the least of your worries).

Well, I doubt the complete system will need more than 100 MB for libraries.
Open2x on the gp2x has been developed for dynamic linking, it has all basic libraries needed, a complete OS and some apps and tools and fits on the 32MB gp2x NAND.
Even the bloated Fenix Runtime and python with pygame only use a few Megabytes, so those should easily fit on the NAND.
Personally, I don't think the NAND will corrupt within a Pandoras lifetime, however, even if it does, you can install the complete system onto SD Card, boot from that and use the second one for apps and tools.

 

[bigtruck]

[Exactly how I would design a package manager for the Pandora]

All of which can be extended from the PXML system, and we can all live happily ever after.

In all seriousness though, are there any remaining arguments as to why this is a bad idea and a full-blown package manager should be used instead?

To recap some of the major features of PXML (in relation to package management):

• The PXML installer will have automatic package updates.
• The PXML installer will be capable of fetching and installing packages by itself.
• If you want to, you can organize the SD card as you wish, as the PXML files use relative paths.
• The most commonly used libraries (at least 95%) will be on the NAND and dynamically linked.
• Those libraries will be updated by a separate package manager.
• Other libraries required by a package will either come with the package or be statically linked.
• Interest has been expressed in extending PXML to a system that does have dependency-resolution, which would be necessary only if multiple packages share the same library not included in the NAND.

It seems most of the discussion has turned to semantics, specifically what exactly a package management system is. The PXML installer, if I understand correctly, will be a sort of package manager, though it will differ under the hood from what is used on a typical desktop package manager. If you're really that concerned about a tiny bit of memory/space being wasted (which would only happen if two apps were using the same uncommon library), there is a discussion about a Pandora package management system using PXML taking place here.

 

[cosam]

There's already one disadvantage: You need a software for it, you can't use a standard explorer.

Ok, probably should have included that under the first point instead of the second, but a standard explorer would work fine. The system is completely compatible with non-modified PXML-based archives as they would be picked up the moment the card hits the Pandora.

Eeeermm... you're NOT talking about a package manager system here. You ARE repeating MY arguments I posted at the thread about the PXML definition about WHY I want PXML and NOT a package manager.
So at the moment, you are AGAINST a standard packag manager and FOR the simple PXML solution.

I guess that depends on your definition of package management ;-) If I'm repeating your arguments, it's because I agree with them! If it makes it easier, you could also see this as an extension to the PXML system which allows (but in no way forces developers nor users into) the separation of common components of packages and allows these to be registered as dependencies of other packages. If such features where present in an encompassing system for the installation and removal and updating of software, I'd consider that system to be a package management system (although probably not in the traditional sense). A nice extra for those who want it, but no skin off anyone's nose for those who don't.

Well, I doubt the complete system will need more than 100 MB for libraries.

I'll have to take your word on that ;-) Still, along with the rest of the OS, I can't imagine you're exactly swimming in space.

Personally, I don't think the NAND will corrupt within a Pandoras lifetime, however, even if it does, you can install the complete system onto SD Card, boot from that and use the second one for apps and tools.

I certainly hope the NAND will last too. It may be a bit old-school, but to me firmware isn't something you generally go about updating every time a new version of a library comes along.

@bigtruck: Looks like I was typing my response just as yours was posted. I won't bother quoting it all, but you really hit the nail on the head ;-)

 

[cosurgi]

There WILL be a package manager for the firmware, simply not for removable medias like SD Cards as it leads to a lot of problems.

And there WILL be 99% dynamic linking - except for any non-common libraries.

Question to EvilDragon (becasue I don't want to misinterpret your explanations):

You say that firmware will use package management. Is that apt?

Will it be possible to "spread" (??) the use of that package management onto the SDHC cards? I mean:

• I don't want to use PXML, because I have 9 years experience with .deb, and I feel like fish in a water with .deb
• I don't ever plan to remove SDHC cards - I will have internet acces on Pandora. So apt-get will be efficient.
• I'm not afraid of SDHC corruptions, because I have already set up a home-server backup box which backups twice per day all PCs that I have at home (using rsnapshot).

Will I be able to use the (your words here "firmware package management" in my case?

Or did I misunderstood you?