Excessive hard drive access

[WizardStan]

So this has been bothering me for a while and I can't figure out why. Maybe someone here has a suggestion.
Sometimes my computer's hard drive will just start spinning: very audible "vrchkvrchkvrchk", the red LED solid on, many minutes of activity during which my computer becomes almost completely unresponsive due to how much disk access is going on. This is obviously something reading or writing extremely heavily, I just can't figure out what. Because here's the puzzling part: neither top nor iotop report any issues. It's normal for iotop to tell me (between mostly 0s) 20-30K/s, occasionally spiking to a couple MB, without any problems: when this weird disk access starts happening it reports nothing; like, it shows "0, 0, 0, 0, 2KB, 0, 0, 5KB...", absolutely nothing unusual can be seen. And top is the same, nothing out of the ordinary. I'm almost reaching the point of wiping my hard drive and starting over. If it gives a clue, I have to have already been running iotop: if I try to start it while this weird access is going on it won't start until the disk quiets down several minutes later. Everything is slow, but iotop specifically so, possibly just because it's trying to access a hard drive that's really really busy.

 

[kuru]

Maybe try something like smartctl to check drive health

 

[pmprog]

I used to have something similar when I was running Mint (I think) on my laptop, and could never figure out what was going on. I ended up removing Linux from the machine.

 

[rohezal]

Maybe you have an encryption trojaner / ransomware?

 

[elw3]

What kind of disk and what FS is that?
I would assume this could happen on a Hybrid for example when the firmware decides to sync with the ssd part.

 

[WizardStan]

Maybe you have an encryption trojaner / ransomware?

This has been going on for as long as I can remember, possibly years, and my harddrive remains fully accessible and un-ransomed.
[doublepost=1522063779,1522063449][/doublepost]

What kind of disk and what FS is that?
I would assume this could happen on a Hybrid for example when the firmware decides to sync with the ssd part.

Standard platter HDD. I can't tell what hard drive it's happening too: I've got three, one for my Windows install with NTFS, one for my linux install with ext4, and one for "data" like videos and stuff with fat32. It's not frequent enough that I can unplug the two non-linux ones and "wait and see" because even if I wait weeks there's no guarantee that that was the problem. Sometimes it'll do it after a few days, other times it'll be a month or so feels like. No rhyme or reason.
I'm probably just going to wipe and start over. It has literally been over 5 years since I did the original install.

edit: yeah, I've talked myself into it. Get me a new SSD with my tax refund in May.
Unless someone really wants to help tackle this puzzle feel free to ignore me.

 

[elw3]

You can check out which harddrive does something tho by using "hdparm -C"
If you find the time to enter it while it happens.
[doublepost=1522064256,1522064105][/doublepost]And of course as kuru suggested. gsmartcontrol is the first tool to check for anything drive related.

 

[kaprikawn]

I used to have something similar when I was running Mint (I think) on my laptop, and could never figure out what was going on. I ended up removing Linux from the machine.

I say we take off and nuke the site from space, it's the only way to be sure

 

[JDTAY]

Well, this happens to me on Windows. Turns out it's just SuperFetch or whatever they call it now doing disk defragmentation and preloading popular programs, so everything will load faster when I get around to using it. Not sure if Linux has a similar feature.

Edit: Though the way you describe it, it sounds like it happens worse on your system than mine.

 

[levi]

My linux build server seems to tick over the HD - I can see it flick on every few seconds for a microsecond. I'm not sure what this is to be honest - if it's systemd logging something pointless, the system checking drive health and this showing up as an access, or something to do with the fact my home partition of encrypted. But it never blocks for io for more than say 50ms at any time, and it does it persistently, so I've never spent too much time on figuring out what it is.

Are you running a web browser on this system, and is swap configured?

 

[Autian]

I get similar issues as soon as my system needs to swap. The system gets very unresponsive, the mouse moves with 0.1 FPS and the HDD indicator glows contantly. It happens with any application that causes swapping. Cities Skylines runs miles better on Windows just because of the better swapping behavior. Arch Linux and Windows 7 share both the same HDD and there is 4 GB of RAM installed.

 

[elw3]

You all dont understand what iotop is dont you?
It shows if the system writes to disk, he clearly says it shows nothing so none of the obvious reasons make any sense.

 

[Klumpen]

Could this be journaling related?

 

[FBnil]

MLOCATE

From time to time, the database for the command "locate" is rebuild from scratch. If you have many small files, and your fragmentation is above certain levels... well then the harddisk will be having a bad time.
So when that happens, verify it is that which is using lots of IO: "ps -ef |grep -i -e updatedb -e locate" and then "sudo killall updatedb.mlocate" to kill it with fire...

Where are these small files? If you have lots of images, then "find ~/.thumbnails/ |wc -l" if you have over 9000, then clean it up: "find ~/.thumbnails -name '*.png' | xargs -I{} rm {}"

Unfortunately, there are no easy defrag tools for many Linux FS's.
There are tricks though, like moving files to another HD, then back again to your old HD.
Or put your OS on a SSD. Best small investment to improve OS boot times...

Meanwhile, check if you have CRON thingies that start it up, for example:
/etc/cron.daily/mlocate

SMART


See: https://www.techrepublic.com/blog/l...smart-status-information-on-your-hard-drives/

On Debian:
# install smart (this tool can be configured to run as a service in the background, but that's not required, just run it manually, from time to time)
apt-get install smartmontools
# List all your disks
lsblk
# check the first disk, and if it is actually capable of giving smart info:
smartctl -i /dev/sda
# get all errors:
smartctl --all /dev/sda

And don't forget the others that gave good recommendations, like "apt-get install iotop; iotop" from @Elw3
or "free -h" to check if you are running out of RAM (swap should also be full) from @Autian.
Also @Klumpen gives a good idea: maybe it is time for a fsck (corrupted journal)? Verify it with:

sudo tune2fs -l /dev/sda1 | grep "Last checked"

And for fragmentation level, the last line of this command (ignore the warning about the FS being mounted):

 sudo fsck -fn /dev/sda1

Then there are "almost impossible" things like almost being out of inodes, so check the IUse% being below 80%:

df -i

 

[TeDaDeS]

Did you defrag your drive?

If there isn't much disk activity in read/write maybe the disk itself has an issue. Did you check SMART?
Once had a drive that got hot when read/writing to much, it would stop and then trash and after a while would work fine.
That disk did die after a year or 3.

 

[WizardStan]

The short smart tests passed for all three. I'm running long tests now, will take about 2 hours.

And don't forget the others that gave good recommendations, like "apt-get install iotop; iotop" from @Elw3

Elw3 wasn't suggesting I use iotop, he was saying I already mentioned I was using iotop and it reported nothing unusual
Hard drive thrashing like crazy, system slower than molasses in February, and iotop is all "hmmm, yeah, you're writing 2kb/s sometimes, I guess"
Could be journaling maybe? Does iotop only show user processes and not kernel access?

 

[FBnil]

Journalling should be shown. I use gkrellm to graphically show my stats, and I see equidistant spikes...

iotop also shows them from time to time as:
647 be/3 root 0.00 B/s 42.80 K/s 0.00 % 3.14 % [jbd2/dm-6-8]

where jbd2 = Journaling block device


If you think you might have bad sectors, and the internal hardware of the HDD is relocating data like crazy... I recommend turning off the hdd, and cloning the data it to a new and bigger harddisk as a backup.... (and only then do the SMART test ).
You can use gddrescue (The newer version, as explained here: https://askubuntu.com/questions/211578/whats-the-difference-between-ddrescue-gddrescue-and-dd-rescue) to quickly copy all data, and then try to do the "damaged" sectors.

 

[levi]

Ah ddrescue looks interesting; thanks for that. I've a few old CDRs that are giving up the ghost by now, and while the only data I've lost is stuff I don't really care about all that much, the best solution I found so far is dding the data off with a bs=1 and then mounting the resulting image, rather than mounting them directly (if a directly mounted file has read errors, the system truncates the file there and then - dd doesn't understand filesystems so will just blank out the bits it can't read and then pick up again when it can). ddrescue should work even better for me.

 

[WizardStan]

I do occasionally see jbd2, but not any more when the problem is happening.
So even though -H says "test result: PASSED", selftest result says "Completed: read failure" with 90% remaining. Both for the long and the short tests. This bodes poorly.

 

[FBnil]

I've a few old CDRs that are giving up the ghost by now

Yeah, got lots of those too. UV light is bad for the gel inside it. Oxigen makes the layer brittle... all sorts of things that make it last about 8 years... but not much more than that...