A possible PND-system replacement

[slaeshjag]

^ It would be quite difficult to achieve that using squashfs compressed filesystem images, so I'm afraid it will not be any more feasable than it was for the PND system.

...I guess you could potentially download additional squashfs images that contain the updated files and have them override the earlier ones, but that would make a mess of your SD card...

 

[thatgui]

^ It would be quite difficult to achieve that using squashfs compressed filesystem images, so I'm afraid it will not be any more feasable than it was for the PND system.

...I guess you could potentially download additional squashfs images that contain the updated files and have them override the earlier ones, but that would make a mess of your SD card...

no real idea what I am talking about, but couldn't a simple binarydiff do the trick ?

 

[slaeshjag]

I haven't looked into it myself, but I'd expect a small change to result in a quite different dbp file. Maybe possibly.

I haven't done any research on it myself, maybe only it would change the blocks the changed file is located in, and not all of them..

 

[WizardStan]

It would be quite difficult to achieve that using squashfs compressed filesystem images, so I'm afraid it will not be any more feasable than it was for the PND system.

It is actually extremely feasible for PNDs, even SquashFS ones. I don't have the data anymore but I remember using Wesnoth for my tests and finding that differences between versions could be reduced to 10MB binary diff files as opposed to the full 300-400MB PND. The research is all buried in a thread somewhere on the boards here, I can't remember anything about it anymore so it'll be really hard to find, may just be best to repeat the science.The only thing holding it back was a standard method of communication: milkshake's repo would need to be able to calculate the diffs and somehow send them to pndmanager which would then apply them, something I don't think anyone had the time for.

But if it's designed as part of the goals of the DBX or whatever then it can be handled early rather than as an afterthought.

edit: here it is actually.

My memory was way off, it seems. I bindiffed Wesnot 1.8 against 1.11 and got a 200MB diff file, but there's a lot of changes between the two: I'd need to rerun the same tests to compare between two consecutive versions (1.11.16 vs 1.11.17, for example)

But my notes on testing an intentional example of changing and moving just a few files suggests it should work just fine.

 

[slaeshjag]

As long as the package file is umounted at the time of applying the patch, I guess it could work then.

 

[Silent-Hunter]

I think it's a good idea! I mean, no matter what it's going to reduce the download size, even if only by a little it's better than nothing right?

 

[WizardStan]

You don't get the reduced download size for free. It takes a lot of memory and CPU to get a good binary diff. It could easily end up costing more to generate the diff than it does to just send the whole thing.

 

[Cloudef]

Binary diffs are also much harder to manage. Think of the repository side, you need to store bunch of diff files, download each of them and apply them in order. It complicates both the server and client side. If you only ever store diffs, it's also harder to retieve the full blob when you need it. That said binary diffs are wonderful, but I don't think they are ideal for packages (even though some package managers support them like pacman, their official repos don't actually use them).

 

[Silent-Hunter]

Well must we use squashfs then? Can we not use something that new files can be inserted into?

 

[WizardStan]

Files in squashFS are independently compressed. Inserting a new file just shifts things around and inserts the new file. We wouldn't gain anything as far as binary diffs are concerned by using something like ISO, we'd just lose the compression.

 

[Silent-Hunter]

Then why can't the new files just be inserted by the package manager? In fact, PNDs could be stored in an uncompressed form, so that individual files are downloaded and inserted when they are updated. That solves the problem of not being able to get the whole package if you need to as well.

 

[WizardStan]

Then why can't the new files just be inserted by the package manager?

In theory they can, that's what these binary diff programs accomplish. I mean, you can't JUST insert data in the middle of either a squashFS or ISO, you need to also manipulate the entry table at the start, but it's not an impossible task, binary diff tools take care of that as well.But again, all this requires work. Milkshake and B-ZaR (and anyone else who wanted in on that committee) would need to discuss a communication standard for sending updates, how to store and apply the diffs, Milkshake would need to update the server's code to create these package diffs and B-ZaR would need to update PNDManager to apply them, and EvilDragon would need to agree to the increased CPU usage and extra storage required.

 

[_wb_]

Then why can't the new files just be inserted by the package manager?

In theory they can, that's what these binary diff programs accomplish. I mean, you can't JUST insert data in the middle of either a squashFS or ISO, you need to also manipulate the entry table at the start, but it's not an impossible task, binary diff tools take care of that as well.
But again, all this requires work. Milkshake and B-ZaR (and anyone else who wanted in on that committee) would need to discuss a communication standard for sending updates, how to store and apply the diffs, Milkshake would need to update the server's code to create these package diffs and B-ZaR would need to update PNDManager to apply them, and EvilDragon would need to agree to the increased CPU usage and extra storage required.

An alternative would be to let package authors upload PND revisions as binary diffs (if they want to), which means they would need to do spend the CPU usage, but 1) they save on upload time/bandwidth and 2) ED's server doesn't have to do the CPU-intensive work to compute the binary diff, it just has to apply the diff to compute the new full PND file (but that's a much simpler operation).

 

[ptitSeb]

Then why can't the new files just be inserted by the package manager?

In theory they can, that's what these binary diff programs accomplish. I mean, you can't JUST insert data in the middle of either a squashFS or ISO, you need to also manipulate the entry table at the start, but it's not an impossible task, binary diff tools take care of that as well.
But again, all this requires work. Milkshake and B-ZaR (and anyone else who wanted in on that committee) would need to discuss a communication standard for sending updates, how to store and apply the diffs, Milkshake would need to update the server's code to create these package diffs and B-ZaR would need to update PNDManager to apply them, and EvilDragon would need to agree to the increased CPU usage and extra storage required.

An alternative would be to let package authors upload PND revisions as binary diffs (if they want to), which means they would need to do spend the CPU usage, but 1) they save on upload time/bandwidth and 2) ED's server doesn't have to do the CPU-intensive work to compute the binary diff, it just has to apply the diff to compute the new full PND file (but that's a much simpler operation).

I have done that already a couple of time, but it's true that without proper infrastructure support (I mean PNDManager & Repo support), it's not very andy. But for small update of large PND, it's very usefull!

 

[slaeshjag]

An idea: Built in datafile check in dbp-run.

Supply a list of files in a set (and optionally a list of acceptable checksums) and map sets of datafiles to executables or launchers, if they are missing, display a nagscreen with a list of missing (or "invalid") datafiles and possibly a description of where to get them. This could also incorporate in a future repo as a "requires non-free datafiles" flag.

Thoughts?

 

[levi]

Ah, you mean for e.g. the sort of game data you often need to get for game engine ports?  I'm not sure it's that hard for devs to provide their own nagscreen, but I can see the benefit in the future repo being able to tell downloaders that they'll need to get some other files to get the dbp to run.

 

[slaeshjag]

Yes, be it gamedata, BIOS-roms (does any emulator even need those anymore?) and whatever other files that might be needed for it to work properly.

 

[pmprog]

BIOS-roms (does any emulator even need those anymore?)

Yes, a few. Gameboy Advance and C64 (though most C64 emus ship with the roms)

It'd be great to tell there where to put it too! Depending how invested you want to get, you could even give the user the option of "I have these files, let me tell you where there are", and open a "Select Directory" dialogue, and then copy them to the right place, and for the more lazy/disorganised a "Search my filesystem..." (though you'd need some sort of confirmation of picking the correct files).

 

[levi]

That's a corner case though - GBA and DS at least mostly ship these days with an open source bios, which makes most games and things run okay.  But you can still use the genuine bios to make the system boot look exactly like on real hardware, or to get that last weird foreign game to run.

I guess that wouldn't trigger the case where you'd need a nag screen though, and it not being flagged in the repo as needing extra installs kind of works then too.  So I guess that's covered, thinking it through.

 

[Christoph.Krn]

If you have any thoughts on the ideas that I've been suggesting on the previous page a while ago ( https://boards.openpandora.org/topic/16726-a-possible-pnd-system-replacement/?do=findComment&comment=381675 ), then I'd be grateful if you shared them with me, especially if you think that there may be something wrong with them.