General Save directory?

[double7]

At that point we drifted from my question how many apps are really affected by that feature.

It's not relevant how many applications need or are affected by the feature.

Why did you make a point about "n times" editing the exclude file then? Or unlikely the problem lies within the readonly data?

The more apps are affected the more possible users may be affected.

Affected in both directions, likes and dislikes.

I do like slaeshjag for the disabling option and i'm fine with that result.

I assume that if the user did a mess "somehow" he should better do a reset to a defined state. That is the PND content only.

you have two reset points: with custom readonly data (delete only appdata) and bare PND (delete both appdata and appdata-ro). In use cases I can think of it's very unlikely the problem lies within the readonly data. Can you give some examples?

A buggy plugin, a crc error on the filesystem, the buggy 7z support of drastic - which can't decompress 7z with some highlevel compression options. In those cases the fix lies in the RO data. I do agree, that may not be very likely, but for drastic it doesn't help if you are affected.

Notice you need to modify the excludes each time you add new stuff to appdata that needs to be excluded. With appdata-ro that information is implicit.

Your argument is valid for single file definitions. The majority (bytes) of RO data would be *.mapextension or *.romextension and needs to be excluded only one time. - Ok, one time per app, error, per RO data extension.

How should i know that a GB emu is listed at letter "c" because it starts with com.ex...

Check the appdata for the dir name. It's not that difficult. You have already run the PND once to create the appdata dir anyway,

Yes, to all PNDs i ever ran on that SD card. If i can't find the appdata i move the app to an empty SD card. As you said, some have many PNDs installed, i'm on 91 on my main SDcard but have 117 appdata dirs there. The actual maximum appdata dir counter could be above 1500, if you had the repo on your SD.

I'm trying to reduce complexity by adding implicit information here. Generally the solution to fixing issues arising from complex environments is not complex operations, but better organization. By keeping the operations simple you reduce the possibility of errors. Even automation is simpler when the data is well organized.

I do agree. Maybe it pulled me at the wrong time into this discussion. I see you try to reduce complexity but adding another feature does not reduce complexity, it increases complexity by one feature. We read "re-inventing" in the beginning of this thread and i observed a similar process in the midnight comander (mc) development. While mc-4.6.2 saves user config in ~/.mc the mc-4.8.3 changed to ~/.config/mc and added ~/.cache/mc and ~/.local/share/mc. Instead of one directory it now uses three. Other apps use ~/.appname or came with their own idea of a good structure, e.g. ~/.mozilla. They would or would not follow into the new structure and until all other apps are updated any other tool have to handle both concepts. I don't see the reason why the OS should support and workaround individual app flaws.

The RO data feature was implemented because of lack of a _configurable_ RO data directory - or lack of minimal comfort by saving last active selection directory.

Reducing complexity within the OS could be achieved by better apps organization e.g. giving the option to configure their data organization.

I think this is a matter where we just have to accept personal preferences.

Sigh, yes, i wish devs do. (Push bold font button here.) This is no offense to anyone personal! I could add a smily, but i'm to serious with that wish.

In this case i wish Wallys app could get an option for configurable RO data dir.

In other cases i wish i could disable the preview icons in the selector list of drastic. It is as it is.

Configurable features could be accepted by anyone.

 

[bzar]

(the quote end tag detection is broken and I can't seem to fix the quotes)

Ok, closing notes. I'm retreating from this conversation because coming up with the answers takes far too much of my limited free time . I'll answer the previous questions and then fade away. Someone else is free to continue on my behalf if willing.

At that point we drifted from my question how many apps are really affected by that feature.

It's not relevant how many applications need or are affected by the feature.

Why did you make a point about "n times" editing the exclude file then?

Or unlikely the problem lies within the readonly data?

The more apps are affected the more possible users may be affected.

Affected in both directions, likes and dislikes.

I do like slaeshjag for the disabling option and i'm fine with that result.

Because N is the number of times things need to be done no matter the value of N. I meant it's not relevant how many applications exactly are affected. Not that the cardinality has no effect. It doesn't matter if it's exactly 5, 10 or 50. The amount of work grows linearly compared to the amount of PNDs. For this conversation any given specific value for N is irrelevant. In this case I'm assigning an asymptotic work load (for a parallel see asymptotic complexity) to compare the options without guessing how much work specific things are.

I assume that if the user did a mess "somehow" he should better do a reset to a defined state. That is the PND content only.

you have two reset points: with custom readonly data (delete only appdata) and bare PND (delete both appdata and appdata-ro). In use cases I can think of it's very unlikely the problem lies within the readonly data. Can you give some examples?

A buggy plugin, a crc error on the filesystem, the buggy 7z support of drastic - which can't decompress 7z with some highlevel compression options. In those cases the fix lies in the RO data. I do agree, that may not be very likely, but for drastic it doesn't help if you are affected.

Both the buggy plugin and the 7z option suggest there was something wrong with the data to begin with. In this case of course you need to delete appdata-ro. The case I was presenting was one where you knew the data was good. Let me quote myself with some emphasizing: "Once you add the readonly data and see the application working as intended you can be sure that whatever mess you make after that is related to things in appdata.". This precludes cases where the data was bad to begin with. The point was about the user messing the data up himself.

Filesystem errors we can't avoid without using a filesystem that actively works against them like btrfs. Granted, this is something that can happen, but the system still lets you go to the original data by deleting the application's appdata-ro. I think it's still a good thing you can easily first try if it was just stuff you've done that made the application stop working.

Notice you need to modify the excludes each time you add new stuff to appdata that needs to be excluded. With appdata-ro that information is implicit.

Your argument is valid for single file definitions. The majority (bytes) of RO data would be *.mapextension or *.romextension and needs to be excluded only one time. - Ok, one time per app, error, per RO data extension.

Once you take into account that many game data directories contain files with don't have an extension or have a common extension such as txt you're back at at least one exclude rule per app on average. Rules using extensions reduce the amount needed by an amount, but don't change the linear relation between amount of PNDs and amount of exclude rules.

Having too broad rules may even work against you: say you create a custom map to a game on your device and it's not backed up because it has .mapextension. With appdata-ro it'd be in appdata and get backed up with the rest. With a *.mapextension exclude rule you'd miss that file and be sad when you noticed it's gone after an unfortunate incident.

How should i know that a GB emu is listed at letter "c" because it starts with com.ex...

Check the appdata for the dir name. It's not that difficult. You have already run the PND once to create the appdata dir anyway,

Yes, to all PNDs i ever ran on that SD card. If i can't find the appdata i move the app to an empty SD card. As you said, some have many PNDs installed, i'm on 91 on my main SDcard but have 117 appdata dirs there. The actual maximum appdata dir counter could be above 1500, if you had the repo on your SD.

Yes, there is an upfront cost (in work) to the appdata-ro system. Just like there is one doing the initial exclude rules. These about balance each other out as I pointed out in the part you didn't quote.

I'm trying to reduce complexity by adding implicit information here. Generally the solution to fixing issues arising from complex environments is not complex operations, but better organization. By keeping the operations simple you reduce the possibility of errors. Even automation is simpler when the data is well organized.

I do agree. Maybe it pulled me at the wrong time into this discussion. I see you try to reduce complexity but adding another feature does not reduce complexity, it increases complexity by one feature. We read "re-inventing" in the beginning of this thread and i observed a similar process in the midnight comander (mc) development. While mc-4.6.2 saves user config in ~/.mc the mc-4.8.3 changed to ~/.config/mc and added ~/.cache/mc and ~/.local/share/mc. Instead of one directory it now uses three. Other apps use ~/.appname or came with their own idea of a good structure, e.g. ~/.mozilla. They would or would not follow into the new structure and until all other apps are updated any other tool have to handle both concepts. I don't see the reason why the OS should support and workaround individual app flaws.

The RO data feature was implemented because of lack of a _configurable_ RO data directory - or lack of minimal comfort by saving last active selection directory.

Reducing complexity within the OS could be achieved by better apps organization e.g. giving the option to configure their data organization.

Ah, but you see I'm not proposing a new scheme, but extending a previous one. I'm even extending it in a way that's fully backwards compatible with the previous one. And optional to use to boot. It's analogous to if ~/.mc was symlinked to ~/.config/mc and ~/.cache/mc was symlinked to ~/.config/mc/cache and ~/.local/share/mc was symlinked to ~/.config/mc/share. The previous was guessing the same information was earlier in ~/.mc, ~/.mc/cache and ~/.mc/share. Since the directories are all mounted on top of each other during PND mounting the PND system after mounting sees no difference between how things are now and how they would be. Users wanting their data in ~/.mc could just remove the symlinks and have their data where it used to be. The application can still reference all things via the ~/.mc tree. The only difference is one more option for the user to manage her data.

It doesn't matter if applications don't follow into this system. They do not need to. The use of appdata-ro is entirely up to the user. The applications can't even affect it since they won't see the difference between if it's used or not. Things just magically are where they have always been no matter if they physically reside in a more categorized structure.

Applications having their own options for data directories does not reduce complexity, it increases it. Having a centralized solution in the platform you can rely on is way better than each PND maybe making their own that maybe works like some of the other PNDs. Having it centralized means it always works the same way and PND builders don't need to worry about it at all.

I think this is a matter where we just have to accept personal preferences.

Sigh, yes, i wish devs do. (Push bold font button here.) This is no offense to anyone personal! I could add a smily, but i'm to serious with that wish.

In this case i wish Wallys app could get an option for configurable RO data dir.

In other cases i wish i could disable the preview icons in the selector list of drastic. It is as it is.

Configurable features could be accepted by anyone.

 

[thatgui]

Ok, closing notes. I'm retreating from this conversation because coming up with the answers takes far too much of my limited free time . I'll answer the previous questions and then fade away. Someone else is free to continue on my behalf if willing.

I guess double7s point is instead of trying to work around the issue, the issue should be extirpated (sounds a lot harsher, then in my native tongue): devs not integrating a mechanism to let the user choose where application data is situated in the filesystem (for all kinds of reasons: other understanding of file organization, lazyness, not enough knowledge, etc.). While you seem to have a more "laissez-fair" approach to it: "if you can't beat it, embrace it".

But in the end its all cool, as your solution is "practically" optional.

 

[double7]

Why did you make a point about "n times" editing the exclude file then?

Because N is the number of times things need to be done no matter the value of N. I meant it's not relevant how many applications exactly are affected. Not that the cardinality has no effect. It doesn't matter if it's exactly 5, 10 or 50. The amount of work grows linearly compared to the amount of PNDs.

That applies to GUI moving files to the RO dir, but not for editing the exclude file. An --exclude=PATTERN will apply to all future apps too.

Once you take into account that many game data directories contain files with don't have an extension or have a common extension such as txt you're back at at least one exclude rule per app on average.

Now it's my turn to ask for an example. Which game data is without extension or with *.txt extension of a size that makes it relevant and worth the effort to keep it out of backup?

No good dev would use a common accepted extension name like *.txt for textures.

Rules using extensions reduce the amount needed by an amount, but don't change the linear relation between amount of PNDs and amount of exclude rules.

My understanding of linear is "1 PND 1 rule - 10 PND 10 rules" which is not valid for *.extension rules.

Having too broad rules may even work against you: say you create a custom map to a game on your device and it's not backed up because it has .mapextension.

I have to agree. No matter i can't imagine anyone doing map design on a palm sized screen, your argument applies.

Applications having their own options for data directories does not reduce complexity, it increases it.

The complexity is increased due to the users wish for the seperated savestate and RO dir function. That's no fault of the OS nor the app.

Having a centralized solution in the platform you can rely on is way better than each PND maybe making their own

Do you remember my 5 gba/gbc emu example? With the actual situation we have 5 RO dirs and have to copy the same RO files 5 times to the SDcard. Or learn how to use symlinks just to find out that they depend on the filesystem.

I don't think anyone would like to have the same RO files multiple times on his SDcard.

How could you let them use one single RO dir where it is useful?

Mount/bind one RO dir into the specific appdata-ro/appname dirs?

Controlled by the OS with another config menu or file?

I meant your backup habits. We have to accept personal preference regarding backup habits.

The point of view was limited to backup purposes from the start of the discussion but there is more. We discussed and fixed the symtoms, but the source of lack of configuration possiblilities still exist. If that would be fixed there are no problems to adjust personal backup preferences to that.

 

[bzar]

OK. One more.

Why did you make a point about "n times" editing the exclude file then?

Because N is the number of times things need to be done no matter the value of N. I meant it's not relevant how many applications exactly are affected. Not that the cardinality has no effect. It doesn't matter if it's exactly 5, 10 or 50. The amount of work grows linearly compared to the amount of PNDs.

That applies to GUI moving files to the RO dir, but not for editing the exclude file. An --exclude=PATTERN will apply to all future apps too.

Yes, but it's still linear. See explanation of linear growth below.

Once you take into account that many game data directories contain files with don't have an extension or have a common extension such as txt you're back at at least one exclude rule per app on average.

Now it's my turn to ask for an example. Which game data is without extension or with *.txt extension of a size that makes it relevant and worth the effort to keep it out of backup?
No good dev would use a common accepted extension name like *.txt for textures.

I'm not just excluding big files. As I mentioned I'd rather back up ONLY what I can't get elsewhere. There's absolutely no point backing up only a part of a game's unchanging data files. It's all or nothing. My system allows backing up only what is absolutely necessary and does so automatically after the initial setup. No need to figure out what files are big and what are not.

Rules using extensions reduce the amount needed by an amount, but don't change the linear relation between amount of PNDs and amount of exclude rules.

My understanding of linear is "1 PND 1 rule - 10 PND 10 rules" which is not valid for *.extension rules.

No. Linear growth means it can be expressed as f(x) = ax + b, where a > 0 and b is some constant. a can be 1 or 100 or 0.0001 or whatever real number greater than zero. I also mentioned average number of rules elsewhere in the post, because it's not an exact number but an expected value. So it's not an exact amount of rules per PND. Some will have none as they're covered by previous ones, some will have several. On average their number still grows in a linear relation to the number of PNDs. Say there's a probability P(none) that a PND needs no rules, a P(one) that it needs one rule and P(two) that it needs two rules (totalling into P(zero) + P(one) + P(two) = 1). For some reason no PND in this example needs more than two rules. In this case the probable number of rules for x PNDs is f(x) = 0*P(zero)x + 1*P(one)x + 2*P(two)x, or f(x) = (P(one) + 2*P(two))x, which means a = P(one) + 2*P(two) and b = 0 in the linear growth function. (Apologies if the math has mistakes. I'm just about ready to get to sleep. The idea still stands)

Applications having their own options for data directories does not reduce complexity, it increases it.

The complexity is increased due to the users wish for the seperated savestate and RO dir function. That's no fault of the OS nor the app.

No, but it's still a concern that can be addressed. If it can be addressed without harm to others, why not?

Having a centralized solution in the platform you can rely on is way better than each PND maybe making their own

Do you remember my 5 gba/gbc emu example? With the actual situation we have 5 RO dirs and have to copy the same RO files 5 times to the SDcard. Or learn how to use symlinks just to find out that they depend on the filesystem.

I don't think anyone would like to have the same RO files multiple times on his SDcard.

How could you let them use one single RO dir where it is useful?

Mount/bind one RO dir into the specific appdata-ro/appname dirs?

Controlled by the OS with another config menu or file?

You have the exact same issue with just appdata. Appdata-ro doesn't change this at all. Neither does anything else discussed here.

 

[levi]

I don't have any ROMs in my appdata folders. I don't use any emulator that doesn't let me point its browser at a folder of my choice.


It seems to me that B-Zar's idea is simple to understand, simple to implement (if aufs plays ball) and simple to use. One use I could see for it is for storing stuff an app generates on first run - a lot of recent apps unpack stuff and store it in appdata for later use. After the first run, and doing nothing else, just move that stuff to appdata-ro. You can then revert it to fresh by deleting appdata, but avoid the initial long load time.


In fact you can snapshot any appdata at any point like that. Just copy it to appdata-ro when you want to make the snapshot, and any new files or updated files will be stored in appdata, and can be deleted if you want to reset it.


I like that it doesn't change anything about the users appdata unless they want to use this new functionality. I worry slightly that for anyone not following this thread, they may never know about it, but hopefully if it's documented on the wiki and mentioned in other threads as and when, that won't happen.


Admittedly, I don't really understand double7's proposal at present. What would read this exclude file? Does that limit the backup tool we can use? I too worry about rules which are written in one context accidentally applying to the wrong files in other context as the solution is currently stated. B-Zar's solution looks simpler and to have fewer downsides at present to me.

 

[_wb_]

The number of "exclusion rules" you would need, would scale linearly with the total number of files, which can be much larger than the number of PNDs. Essentially you need one rule per file. Sure, you can sometimes be more concise, but probably not systematically and asymptotically so.

To give a simple example: say you first have a PND which has all its savefiles in *.sav (to be backed up) and all its game assets in *.dat (to be ignored by backups). Great, you can express that with one simple exclude rule: "don't backup any .dat file". Now you install a second PND which stores its savefiles in one big file called savegame.dat. Oops, you need to add an exception to the previous exclude rule. No biggie, let's make it "don't backup any .dat file, except if it's called 'savegame.dat'". But then you install a third PND which stores its savefiles in separate files called <something>.dat, where <something> is a complicated string which contains the time and date and a name chosen by the user. Now you basically have to get rid of the general exclude rule, and instead just list the names of the game assets in the first PND. And so on.

An --exclude=PATTERN will apply to all future apps too.

Yes, and this can sometimes be a good thing, but it can also be a bad thing, as my little example above illustrates.

In the end, the easiest and most future-proof way to specify which files to backup and which to ignore, is by explicitly listing them in two categories. But how to make those file lists in a user friendly way, without introducing the option to have invalid or duplicate entries, or overlap between the blacklist and the whitelist? Aha, we already have a solution for that: it's called a file system!

 

[Maschinenherz]

hm. My thoughts on this....


The system atm is ok. I havent tried the suggested backup tools, and dont know if they have this option. Are they pnd or repository? But what about... i think it is called incremental backup? I tried a programm from portableapps.com a few years ago. Witch should be able to be ported to pnd if necessary.


first backup: everything. Then you can save this job and say only chanced.


in my opinion it would be the best option. In the first backup you save everything, which for example for starcraft means, you dont have to look for the cds, install on windows, apply patch,copy files... more than once. If you have many of this games, you freak out if you want to get your stuff back. And later it saves only changed files, i.e savegames. Backup speeds up.


what do you think?

 

[Wally]

I've worked out a better solution for myself.. That is repackage all the game data into the DBP's itself and have the saves redirect to the appdata folder.

 

[slaeshjag]

The dbp system supports an rodata directory for gamedata if you want to keep the appdata dir clean from that.

 

[Neelix]

I've worked out a better solution for myself.. That is repackage all the game data into the DBP's itself and have the saves redirect to the appdata folder.

I'm curious as to what makes you think that would be a better solution... The game data can't always be included in the DBP (or PND) due to licensing issues which is why we have engine only ports on the Pandora in the first place. That would mean that the end user would have to be the one to do the repackaging which seems overly complicated when compared to simply dumping the game data into DBP specific read only data folder. End users should not need to know how to repackage a DBP.

- Neelix

 

[bzar]

How about if the DBP is actually a DBP maker with a small GUI that requests the game data and pops out a game DBP?

EDIT: Of course, updating these would be quite challenging

 

[Wally]

I've worked out a better solution for myself.. That is repackage all the game data into the DBP's itself and have the saves redirect to the appdata folder.

I'm curious as to what makes you think that would be a better solution... The game data can't always be included in the DBP (or PND) due to licensing issues which is why we have engine only ports on the Pandora in the first place. That would mean that the end user would have to be the one to do the repackaging which seems overly complicated when compared to simply dumping the game data into DBP specific read only data folder. End users should not need to know how to repackage a DBP.


- Neelix

That's not my problem, other people can do their own thing hence the solution being for "myself"

 

[Neelix]

How about if the DBP is actually a DBP maker with a small GUI that requests the game data and pops out a game DBP?

EDIT: Of course, updating these would be quite challenging

*chuckles* That could be interesting, but I suspect it would be more trouble than its worth.

That's not my problem, other people can do their own thing hence the solution being for "myself"

Well I suppose if you want to make work for yourself that's your call. Have fun!

- Neelix

 

[Wally]

That's not my problem, other people can do their own thing hence the solution being for "myself"

Well I suppose if you want to make work for yourself that's your call. Have fun!


- Neelix

Of course It's not hard to unpack / repack a DBP though