A possible PND-system replacement


slaeshjag

¯\_(ツ)_/¯
Joined
Apr 8, 2010
Messages
2,687
Location
~Stockholm, Sweden
^ It would be quite difficult to achieve that using squashfs compressed filesystem images, so I'm afraid it will not be any more feasable than it was for the PND system.

...I guess you could potentially download additional squashfs images that contain the updated files and have them override the earlier ones, but that would make a mess of your SD card...
 

thatgui

Advanced Member
Joined
Apr 2, 2009
Messages
3,048
^ It would be quite difficult to achieve that using squashfs compressed filesystem images, so I'm afraid it will not be any more feasable than it was for the PND system.

...I guess you could potentially download additional squashfs images that contain the updated files and have them override the earlier ones, but that would make a mess of your SD card...
no real idea what I am talking about, but couldn't a simple binarydiff do the trick ?
 

slaeshjag

¯\_(ツ)_/¯
Joined
Apr 8, 2010
Messages
2,687
Location
~Stockholm, Sweden
I haven't looked into it myself, but I'd expect a small change to result in a quite different dbp file. Maybe possibly.

I haven't done any research on it myself, maybe only it would change the blocks the changed file is located in, and not all of them..
 

WizardStan

Mega GP Mania
Joined
May 24, 2008
Messages
16,733
It would be quite difficult to achieve that using squashfs compressed filesystem images, so I'm afraid it will not be any more feasable than it was for the PND system.
It is actually extremely feasible for PNDs, even SquashFS ones. I don't have the data anymore but I remember using Wesnoth for my tests and finding that differences between versions could be reduced to 10MB binary diff files as opposed to the full 300-400MB PND. The research is all buried in a thread somewhere on the boards here, I can't remember anything about it anymore so it'll be really hard to find, may just be best to repeat the science.The only thing holding it back was a standard method of communication: milkshake's repo would need to be able to calculate the diffs and somehow send them to pndmanager which would then apply them, something I don't think anyone had the time for.

But if it's designed as part of the goals of the DBX or whatever then it can be handled early rather than as an afterthought.

edit: here it is actually.

My memory was way off, it seems. I bindiffed Wesnot 1.8 against 1.11 and got a 200MB diff file, but there's a lot of changes between the two: I'd need to rerun the same tests to compare between two consecutive versions (1.11.16 vs 1.11.17, for example)

But my notes on testing an intentional example of changing and moving just a few files suggests it should work just fine.
 
Last edited by a moderator:

Silent-Hunter

Hardcore Member
Joined
May 29, 2010
Messages
3,182
I think it's a good idea! I mean, no matter what it's going to reduce the download size, even if only by a little it's better than nothing right?
 

WizardStan

Mega GP Mania
Joined
May 24, 2008
Messages
16,733
You don't get the reduced download size for free. It takes a lot of memory and CPU to get a good binary diff. It could easily end up costing more to generate the diff than it does to just send the whole thing.
 

Cloudef

Member
Joined
Dec 30, 2009
Messages
460
Location
Gensokyo
Binary diffs are also much harder to manage. Think of the repository side, you need to store bunch of diff files, download each of them and apply them in order. It complicates both the server and client side. If you only ever store diffs, it's also harder to retieve the full blob when you need it. That said binary diffs are wonderful, but I don't think they are ideal for packages (even though some package managers support them like pacman, their official repos don't actually use them).
 

Silent-Hunter

Hardcore Member
Joined
May 29, 2010
Messages
3,182
Well must we use squashfs then? Can we not use something that new files can be inserted into?
 

WizardStan

Mega GP Mania
Joined
May 24, 2008
Messages
16,733
Files in squashFS are independently compressed. Inserting a new file just shifts things around and inserts the new file. We wouldn't gain anything as far as binary diffs are concerned by using something like ISO, we'd just lose the compression.
 

Silent-Hunter

Hardcore Member
Joined
May 29, 2010
Messages
3,182
Then why can't the new files just be inserted by the package manager? In fact, PNDs could be stored in an uncompressed form, so that individual files are downloaded and inserted when they are updated. That solves the problem of not being able to get the whole package if you need to as well.
 

WizardStan

Mega GP Mania
Joined
May 24, 2008
Messages
16,733
Then why can't the new files just be inserted by the package manager?
In theory they can, that's what these binary diff programs accomplish. I mean, you can't JUST insert data in the middle of either a squashFS or ISO, you need to also manipulate the entry table at the start, but it's not an impossible task, binary diff tools take care of that as well.But again, all this requires work. Milkshake and B-ZaR (and anyone else who wanted in on that committee) would need to discuss a communication standard for sending updates, how to store and apply the diffs, Milkshake would need to update the server's code to create these package diffs and B-ZaR would need to update PNDManager to apply them, and EvilDragon would need to agree to the increased CPU usage and extra storage required.
 

_wb_

Microbe
Staff member
Joined
Apr 5, 2012
Messages
5,387
Age
39
Location
Brussels, Belgium
Then why can't the new files just be inserted by the package manager?
In theory they can, that's what these binary diff programs accomplish. I mean, you can't JUST insert data in the middle of either a squashFS or ISO, you need to also manipulate the entry table at the start, but it's not an impossible task, binary diff tools take care of that as well.
But again, all this requires work. Milkshake and B-ZaR (and anyone else who wanted in on that committee) would need to discuss a communication standard for sending updates, how to store and apply the diffs, Milkshake would need to update the server's code to create these package diffs and B-ZaR would need to update PNDManager to apply them, and EvilDragon would need to agree to the increased CPU usage and extra storage required.
An alternative would be to let package authors upload PND revisions as binary diffs (if they want to), which means they would need to do spend the CPU usage, but 1) they save on upload time/bandwidth and 2) ED's server doesn't have to do the CPU-intensive work to compute the binary diff, it just has to apply the diff to compute the new full PND file (but that's a much simpler operation).
 

ptitSeb

Serial Porter
Joined
Aug 15, 2012
Messages
8,685
Age
47
Location
France, near Lyon
Then why can't the new files just be inserted by the package manager?
In theory they can, that's what these binary diff programs accomplish. I mean, you can't JUST insert data in the middle of either a squashFS or ISO, you need to also manipulate the entry table at the start, but it's not an impossible task, binary diff tools take care of that as well.
But again, all this requires work. Milkshake and B-ZaR (and anyone else who wanted in on that committee) would need to discuss a communication standard for sending updates, how to store and apply the diffs, Milkshake would need to update the server's code to create these package diffs and B-ZaR would need to update PNDManager to apply them, and EvilDragon would need to agree to the increased CPU usage and extra storage required.
An alternative would be to let package authors upload PND revisions as binary diffs (if they want to), which means they would need to do spend the CPU usage, but 1) they save on upload time/bandwidth and 2) ED's server doesn't have to do the CPU-intensive work to compute the binary diff, it just has to apply the diff to compute the new full PND file (but that's a much simpler operation).
I have done that already a couple of time, but it's true that without proper infrastructure support (I mean PNDManager & Repo support), it's not very andy. But for small update of large PND, it's very usefull!
 

slaeshjag

¯\_(ツ)_/¯
Joined
Apr 8, 2010
Messages
2,687
Location
~Stockholm, Sweden
An idea: Built in datafile check in dbp-run.

Supply a list of files in a set (and optionally a list of acceptable checksums) and map sets of datafiles to executables or launchers, if they are missing, display a nagscreen with a list of missing (or "invalid") datafiles and possibly a description of where to get them. This could also incorporate in a future repo as a "requires non-free datafiles" flag.

Thoughts?
 
Last edited by a moderator:

levi

Still fresh, damnit!
Joined
Oct 6, 2008
Messages
12,649
Location
Somewhere off the coast of the EU
Ah, you mean for e.g. the sort of game data you often need to get for game engine ports?  I'm not sure it's that hard for devs to provide their own nagscreen, but I can see the benefit in the future repo being able to tell downloaders that they'll need to get some other files to get the dbp to run.
 

slaeshjag

¯\_(ツ)_/¯
Joined
Apr 8, 2010
Messages
2,687
Location
~Stockholm, Sweden
Yes, be it gamedata, BIOS-roms (does any emulator even need those anymore?) and whatever other files that might be needed for it to work properly.
 

pmprog

DNF (Did Not Finish)
Joined
Apr 25, 2011
Messages
4,023
BIOS-roms (does any emulator even need those anymore?)
Yes, a few. Gameboy Advance and C64 (though most C64 emus ship with the roms)

It'd be great to tell there where to put it too! Depending how invested you want to get, you could even give the user the option of "I have these files, let me tell you where there are", and open a "Select Directory" dialogue, and then copy them to the right place, and for the more lazy/disorganised a "Search my filesystem..." (though you'd need some sort of confirmation of picking the correct files).
 

levi

Still fresh, damnit!
Joined
Oct 6, 2008
Messages
12,649
Location
Somewhere off the coast of the EU
That's a corner case though - GBA and DS at least mostly ship these days with an open source bios, which makes most games and things run okay.  But you can still use the genuine bios to make the system boot look exactly like on real hardware, or to get that last weird foreign game to run.

I guess that wouldn't trigger the case where you'd need a nag screen though, and it not being flagged in the repo as needing extra installs kind of works then too.  So I guess that's covered, thinking it through.
 
Top