Pandora Pnd_Notify.c Listening To In_Modify

[demitar]

As can be seen here pnd_notify.c (and thus libpnd) watches for IN_MODIFY, this means that any time someone writes to a pnd it will trigger an update. This is highly inefficient (and potentially dangerous), any time you're downloading something into a pnd-watched folder the cpu will spike as it tries to re-read it over and over.

The proper way to do this would be to wait for IN_CLOSE_WRITE in it's place, this will trigger when (and only when) someone had a file open with writing privilieges and then closed it.

As an example I downloaded gemrb 0.6.3 (8.6M) into a pnd-controlled folder and outside a pnd-controlled folder. The difference? From taking 9-11 seconds it took 15-27 seconds(!).

(User and sys time will naturally not be affected as wget itself does no additional work, this was on a local wifi network without any particular traffic going on.)

Outside:

ender:/media/mmcblk0p1/testing$ time wget -q http://frolic.xvx.se/_full/10/2/gemrb-0.6.3.pnd

real 0m10.440s
user 0m0.031s
sys 0m0.172s

ender:/media/mmcblk0p1/testing$ time wget -q http://frolic.xvx.se/_full/10/2/gemrb-0.6.3.pnd

real 0m9.270s
user 0m0.023s
sys 0m0.180s

ender:/media/mmcblk0p1/testing$ time wget -q http://frolic.xvx.se/_full/10/2/gemrb-0.6.3.pnd

real 0m9.651s
user 0m0.039s
sys 0m0.211s

ender:/media/mmcblk0p1/testing$ time wget -q http://frolic.xvx.se/_full/10/2/gemrb-0.6.3.pnd

real 0m11.207s
user 0m0.063s
sys 0m0.164s


Inside:


ender:/media/mmcblk0p1/pandora/menu$ time wget -q http://frolic.xvx.se/_full/10/2/gemrb-0.6.3.pnd

real 0m26.152s
user 0m0.023s
sys 0m0.461s

ender:/media/mmcblk0p1/pandora/menu$ time wget -q http://frolic.xvx.se/_full/10/2/gemrb-0.6.3.pnd

real 0m27.352s
user 0m0.039s
sys 0m0.445s

ender:/media/mmcblk0p1/pandora/menu$ time wget -q http://frolic.xvx.se/_full/10/2/gemrb-0.6.3.pnd

real 0m15.563s
user 0m0.016s
sys 0m0.570s

ender:/media/mmcblk0p1/pandora/menu$ time wget -q http://frolic.xvx.se/_full/10/2/gemrb-0.6.3.pnd

real 0m21.373s
user 0m0.023s
sys 0m0.313s

 

[skeezix]

Will take a look for hf6 maybe, as I Think hf5 is pretty close to ready,

When that stuff was written, I was pretty much having no love at all from inotify -- our kernel was using bog standard inotify code of course at the time, but it just didn't work _at all_ at the time, so theres probabyl a number of issues in there. (I suspected at the time (I think, been a year or two that all the AUFS action and rootfs being on SD was confusing things all to hell.) There is 'spam' protection in there so it doesn't refresh ever 1second or something, but if an app is slowly downloading, it coudl trigger multiple rescans so I think you're right. (I think at the time I was testing against cp into watched dir and downlaod into watched dir small test apps.. but if you're sucking down firefox or something huge, on pandora itself over slow wifi, that probably wasn't tested much .. we only got wifi working relatively 'recently' (in libpnd terms of recent), and I didn't get wifi working at all until very recently, on my units .. eep

Feel free to inspect the pnd_notify code heavily --

The big wart in that whole subsystem is in pnd_dbusnotify where libpnd actually runs dbus-watcher external tool and 'greps' the pipe for interesting lookign events; that really needs to change so libpnd actualyl is a dbus client and not just using the external tool. (Unless it turns out being a client is a big pita/load, but runing an extenal tool is a big drag imho I've just not had time to go back and revisit that so if anyone has a couple nights to burn..

jeff

<took a quick look at the code .. hey, not too bad. But I think you're 100% correct.. thats really should be close_write and not modify! I'll fix for hf6 for sure!>

Thanks!

 

[torpor]

Good work Demitar!

The difference? From taking 9-11 seconds it took 15-27 seconds(!).

I really do have to wonder if this is a camouflaged reason for the perception that "Wifi is slow" ..

 

[demitar]

The big wart in that whole subsystem is in pnd_dbusnotify where libpnd actually runs dbus-watcher external tool and 'greps' the pipe for interesting lookign events; that really needs to change so libpnd actualyl is a dbus client and not just using the external tool. (Unless it turns out being a client is a big pita/load, but runing an extenal tool is a big drag imho I've just not had time to go back and revisit that so if anyone has a couple nights to burn..

It's a hack, sure. But by the looks of it, it's probably so low-traffic (only mount/unmount and a few battery events, possibly something else) for it to have any actual performance impact.

So unless there are any stability issues there are way bigger fish to fry performance-wise at the moment. ^^

 

[skeezix]

torpor -- no, that issue is only for while writing to pnd-app watched directories -- ie: /pandora/menu, that sort. Its not impacting anything else. (Give me _some_ credit So its a buglet that probably very few would ever notice, but when you do hit it its probably quite a drag on the system!

Demitar -- that is it exactly; given time crunches and whatnot, it works, it works well, its just goofy. A number of libpnd and daemon bits are good for replacement someday, but nothing really standing out, which is good Theres a whole bunch of stuff I'd like to redo for 'firmware 2', so we'll see.

(ie: theres got to be a better way of doing pndevmapperd than watchign all the /dev/events and discarding them. But it doesn't eat much performance, but still, with every event in the system going through it, thats uncool. It was written early on, and later the pnd_io_evdec got written, so evmapperd should refactor to use the lib instead of its own version of the stuff. But I shoudl at least put some code in to ignore the /dev/eventX's that it doesn't need .. maybe I did that already, I forget. But it should definately not be watchign dpad events if it doesn't need to, for instance.)

(I am thinking about dropping in pnd-zip support soon, as its easy to add. ie: iso+zip in addition to iso+raw+raw for pnd's .. so the appended zip can have PXML.xml and icon in it, but can also have other stuff. Very limited changes to libpnd needed (pnd_disco.c really), and could be handy. Drop screenshots or soundbites into that bit, say. Anyway.)

jeff