Moving Pnd Files On Sd Cards

[mindlord]

Looking at the git: It appears that libpnd uses a simple tree walking algorithm called ntfw that's included in angstrom that crawls the directories specified in the config every so often. It looks for files ending .PND and processes them. It's not using any sort of external library for discovery. I can't seem to find out how often this happens. It looks like it's always happening.

 

[skeezix]

Turns out inotify sucks imho.

Theres a number of competing systems for 'filesystem change monitoring', and it seems to vary wildly. When I first started hacking on libpnd, INOTIFY did not have the signalling version, so you had to poll inotify (very low load, a select() and block really), whereas now you can get a kick when something changes. But even then, INOTIFY seems not to work for a lot of things you should think .. be it desktop or on pandora. ie: You have it watch a directory for file creation, and then mount something with UDEV creating the dir-mount-point in the watched dir.... inotify won't catch that. Inotify is a pita for making recursive watches in (even when you do depth-first watch-setup). Now, some of that may have been to the particular version of inotify in the kernle we were using a year ago (much has changed!), but I had a lot of hairpulling where inotify just didn't do useful shit (or 'seemed to', but didn't.)

In the end, what it does is two-fold..

1) inotify to watch for normal file activity (you move a pnd from here to there, deal with it.) It uses a very light polling, as above. ie: Its not sit and spin, but its checking every 5s with a select() type thing IIRC. You can't "block forever" and wait, withotu getting really smart in the code. (ie: I ran into a thing where you'd inotify watch on a bunch of directories, and then somethign would change, and you'g get hundreds or thousands of notifications in a single instant, and sometimes oyu'd get htem spread over like 10s .. so you'd be reacitng to a notification while mor still coming, creating never ending loops, etc.) So I put in some cheesy little easy things that work really nicely, but inotify is basicly sort of annoying. At least the inotify version I was using at the time, and the code has stood and works well still.

2) I use dbus to watch for mount/unmounts. This was a recent hack when inotify was pissing me off*, so I just switched to a crappy dbus listener. I'll be altering that code quite a bit to be more lean soon, but for now works perfectly if a little goofy.

* inotify .. you have to remember that for development, we're booting off SD; Most of us are using SD1 as boot partition and firmware partition, with SD2 as like you'd use .. random pnd files and data, etc. So xfce and such would recognize that they're booting from SD1, but also show SD1 as icons on the desktop, along with SD2. ie: In effect you would be seeing in the filesystem your root and /usr and such (from SD1), but also /media/mmcblk0p[12] mounted as well. ie: /media/mmcblk1p1 == /, mounted in two places (!!). inotify really got confused here, as I have notifications across /media/*/pandora/desktop (for example) since thats what will be going live ... but that picks up what is essentially / (and as the pnd system recursively watches subdirs, so you can stick stuff into /pandora/desktop/foo/bar/baz/bing.pnd) you'd be inotify watching the entire firmware. So stick SD2 in, and inotify would catch it usually --- through UDEV mounting into /media, and thus being found on /, or SD1. Think about that.

Anyway, inotify was either buggy or confused, and missing features, when I started on all that mess *fun*

--

So the short story is yes -- we use inotify to catch file moves, deletes, creates, etc, and dbus to catch mount/unmounts.

And I'm glossing over a lot, since you don't care that much

jeff

 

[skeezix]

mindlord fail!

ntfw is a linux dir-walker; that is used during notification setup.

If you think its walking the entire dir structure every couple seconds to look for changes, then you do me a disservice

I suck and have no sleep, but I do not such that much

jeff

 

[mindlord]

mindlord fail!

ntfw is a linux dir-walker; that is used during notification setup.

If you think its walking the entire dir structure every couple seconds to look for changes, then you do me a disservice

I suck and have no sleep, but I do not such that much

jeff

Not my intent, I was just giving the git a quick glance, looking for the cheap answer. Trying to prevent you from having to write a wall of text so you could get some sleep.

 

[skeezix]

heh

yeah, I've been good at 'tl;dr' type responses lately eh? WALL OF TEXT DEFEATS ALL ONCOMERS!

jeff

Someday when someone says 'why does this look suck?', you can just point to posts like the above and say 'cause life ain't perfect, you write it, biotch'

 

[skeezix]

wireless is a bad thing when you're bored in a meeting

 

[Kramy]

Hm, should be the same with Linux. As soon as my card reader stops blinking, it means that the file transfer is finished.
That's a hardware thing, so it's not affected by the write cache.
The only thing that's affected is the copy-percent bar of your file manager.

Driver issues, then?

This is the card reader: http://www.ncix.com/...ucts/?sku=24164

Yes, you can rename the PNDs to anything you want.
The appdata path is defined within the PND.
That said, you can also override it if you want (i.e. if you want to try out a new version with your own settings, etc.)

Wonderful to hear!

Writing back into a programs dir is a Bad Idea;

Depends on the app.

Profiles works okay in Linux - but in Windows, anything unclean that doesn't write back to its own folder is probably writing to a) Your profile, b ) your Windows system directory (ICK!!), or c) the base of the drive.

That's why there were so many compatibility problems when Vista came out; some of that wasn't possible anymore.

Now, why is writing to your profile bad in Windows? Because the app isn't installed within your profile! It's installed to a global folder, and any other user can replace those files! That could cause any number of issues!

Google Chrome is one of the first apps I know of that installs totally locally, and I'm okay with that. I just have the opinion that most games would benefit from keeping everything in their own dir.

it is _convenient_ sometimes, but generally poor -- god help you on a multiuser system

It works fine on multi-user systems. But it only works for games, where you don't actually care what user is playing the game, and don't care that an admin can install, but a regular user can play. It's also annoying for businesses - since the game can run off a network drive, it can also run off a USB stick. Warcraft III is a perfect example, although it does store stuff like resolution in the registry. But if you have no registry access, then you just change the resolution every time you start playing.

[quote name='skeezix' date='26 March 2010 - 07:30 AM' timestamp='1269613843' post='832402'](you're thinking back to Win3.1 or WinXP and single user .. bvut if you store data right into the app-dir, and you switch users.. you're screwed right?) -- it also leads to security issues when an app writes over itself (or something else writes there , and means you can't generally pick up an app and copy it to somewhere else (incomplete installs). With a separate app and data dirs (every modern OS, be it app data is your homedir, an app data dir, or whatever), you have a nice clean difference. The trick with pnd files is we do it to the same SD as the pnd app is on plus all the other pnd magic (no install, etc.)
[/quote]
My whole point is that you're not screwed. Mixing any two systems is annoying, but there are times when you actually want everything stored in the app's folder. One example... running Ventrilo off a network drive, I'd like the volume levels I set for each person to be retained - but they aren't.

 

[may88]

Okay! If I put a PND on a card and I have two other identical but empty cards and placed them each under one of three cups and switched the cups quickly would the Pandora always pick the right cup?

 

[skeezix]

1 sd 2 cups? -fear-

Kramy.. You're talking different again. You're talking writeback to the app dir, where every user has a full copy of the app. You can't give your copy to someone else or dupe it, and hugely wasteful of space... Not good on a mobile

you can turn off the union if you want and unpack the pnd files and it works just like this and is fine, but it's not what our goal was .. Zero install and no risk

jeffphone

 

[Kramy]

So will write cache be off by default? These are SD cards... low access latencies. We don't need it, right?


skeezix: I don't quite understand what you're saying? I'm saying that one copy per user is fine, and one copy for all users is fine, but one copy with profile-specific settings is not fine.

Keep in mind that in Windows, you can have a program run under two different users at the same time. It creates hellish conflicts, unless you keep it clean - pick one method and stick with it.

Chrome "wastes space", as you put it. That's a good thing. Imagine how awful it'd be if I ran the Beta, but another user didn't? Now s/he starts up the browser, and his/her extensions don't work, because the core files have been replaced. :lol: Awful. Or maybe it crashes or gobbles all the bookmarks because it didn't realize it was upgraded, and needed to convert to a new storage format. Stuff like this happens all the time, and it's irritating. If you want an example program, here's one: http://www.ccleaner.com/

 

[torpor]

DISCLAIMER: I haven't looked at the code for libpnd recently, but I would like to help with some of the issues reported.

skeezix: I have a bit of experience with inotify and have used it quite successfully for all sorts of things (as a notification mechanism for high-speed data capture going to /dev/shm, for example) - are you sure you're setting up the MASK for all of the different message types? There are a lot of different events masked and they're not all well-documented .. for example if you want the mount messages, you have to MASK IN_ACCESS and then closely inspect the notify_buffer to determine why the access occurred for the given inode ..

I'm sure you're pretty well advanced into this, but in case it sheds some new light on the problem, here is the somewhat-dodgy code I've been using recently in my apps which use inotify:

void select_on_notification(int noti_fd)
{
    int ret;
    fd_set read_fds;

    FD_ZERO(&read_fds);
    FD_SET(noti_fd, &read_fds);

    ret = select(noti_fd + 1, &read_fds, NULL, NULL, NULL);
    if (ret == -1)
        perror("select");
}



// a filesystem _notification_ handler.
// this is the 'inbox' for the app.
void *fs_n_inbox(void *thread_inf)
{
    int length, i = 0;
    int wd;
    fd_set descriptors;
    struct timeval time_to_wait;
    int loop = 1;

    notify_fd = inotify_init();

    if (notify_fd < 0) {
        perror("inotify_init");
    }
//wd = inotify_add_watch(fd, POOL_BASE_DIR, IN_MODIFY | IN_CREATE | IN_CLOSE_WRITE | IN_CLOSE);
    wd = inotify_add_watch(notify_fd, POOL_BASE_DIR, IN_CLOSE_WRITE);

    FD_ZERO(&descriptors);
    FD_SET(notify_fd, &descriptors);
    time_to_wait.tv_sec = 3;
    time_to_wait.tv_usec = 0;

//in_tex = 0;
    loop = 1;
    while (loop != 0) {
//      printf("[");
        select_on_notification(notify_fd);
        length = read(notify_fd, notify_buffer, BUF_LEN);
        if (length < 0) {
            perror("read");
            break;
        } else if (length == 0) {
            continue;
        }
//      printf(".");

        while (i < length) {
            struct inotify_event *event =
                (struct inotify_event *) &notify_buffer[i];
            if (event->len) {
                if (event->mask & IN_CLOSE_WRITE) {
                    snprintf(tex_fname, PATH_MAX, "%s/%s", POOL_BASE_DIR,
                             event->name);
//sem_post(&frame_sem);
                }
                i += EVENT_SIZE + event->len;
            }
        }
        i = 0;
//      printf("]\n");
    }

    (void) inotify_rm_watch(notify_fd, wd);
    (void) close(notify_fd);
    pthread_exit(NULL);
}

 

[skeezix]

I set on all masks and it. Didn't detect mounts as I said though, it was surely (at the time) getting confused orbugged, and dbus works 100% due to being tied in nicely.

The API needn't change, so we can update the internals of libpndquite a bit over time. A lotof itneeds redoing.. As I said, at any given instant since I started it, I thought o only had weeks so it's rushed but solid and working. Feel free to take a look at pnd-notify.c and patch and I can try on the panda... But even weeks ago inotify wasignoring mounts.. In /media anyway. Maybe watching /dev would work now, didn't try it lately.

Jeffonphone