I noticed the root drive of my home server (Debian) is at 99% capacity, which was odd to be because I don’t store anything on the root ssd. sudo df -h confirms that 99% of my 256gb drive is full. But sudo du -sh * all added up, only comes to about 30gb.
This is a pc that only runs docker containers and one virtual machine for home assistant. And yes I have restarted, Any ideas as to how to find the missing 200+ gigabytes?
EDIT: sudo ncdu allowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.
dfreports on filesystems, not drives.One more puzzle piece here is that
duwon’t report on files that have been marked for deletion but are still held on to by some process. There’s anlsofincantation to list those, but I can’t recall it off the top of my head.It used to be part of sysadmin work to detect the processes that held on to large files if
dfreports that you’re running out of space, and restart them to make them let go of the file. But I haven’t done that in ages. And if you restarted the host OS that should have taken care of that.I assume you also know how to prune container resources.
Good call on the docker prune, I didn’t think about that. That accounted for about 25GB. Still not enough but at least I’m not at 99% anymore.
Wouldn’t
du -hs *only check the space used inside the folder you are in?I’d check with
sudo du -hs /*myself if I wanted. Or vedncduto get a visual representation.sudo du -hs /*looked like it began started listing every file on the entire server.
What’s your filesystem?
If you are using something like btrfs, for example, the usage reports can be misleading, and getting the exact size can be complicated… I’d recommend using more fs-specific tools, such as btdu.
Would different filesystems really report an over 200gb difference?
In theory, it can. One possible reason with Btrfs might be that you are only mounting a subvolume even though there might be other files in the same filesystem (such as snapshots/copies of the subvolume for backup) but that are not being mounted.
Also some tools like gparted do not handle btrfs disk usage very well and will display it as if the whole partition is 100% full.
du -hs *won’t find “hidden” (.) files and folders, you’ll need a slightly sifferent regx (which I will leave as an exercise for you / I don’t have that info here)And also both
duanddfshow different results depending on the underlying filesystem, ie btrfs (and maybe ZFS?) won’t show how much deduplication is happening.Also, you might be looking at sparse files too, and from memory, you’ll need another option for
duordfto report those correctly.sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information or what’s safe to delete but I’m getting somewhere.Nice. Glad you’re on to something.
I can’t help you with docker files as I don’t use it, but, there’s usually a way to find out who / what owns a file, so I hope docker utils can tell you if it’s safe to delete
Did du give any permission errors? It can’t count the size of directories that it doesn’t have permission to access.
It did yes, a few “invalid argument” a few “permission denied” and a few “no such file or directory”
UPDATE:
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.
Check for mounts hiding the underlying drive?
Sometimes
du . -xwill help, too. (-x doesn’t cross mount points).Docker containers can eat a lot of space over time. When’s the last time you did a
docker system prune? Be sure to read up on what it does before you try it.Thanks that allowed me to clear up about 20GB! Also
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information or if it’s safe to delete but at least I’m getting somewhere.
I’d be curious to see
du -ito see what’s going on with inodes. Alternatively I did have an issue long time ago with docker containers, sparse files and dirty disk. Force-runningfsckresolved my issues in the past.du -igaveinvalid optionand fsck gavecommand not foundEDIT:
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.that’ll teach me to type in a hurry. I meant
df, notdu. Lookup man page for options
Where are you running
du -sh *? (I.e. what directory, are you definitely scanning the whole file system?) I’m sure it’s obvious, but can never hurt to check!What does
du -sh /show? (Generally, the*glob pattern in the shell will not match hidden dot-files, so is it possible they are being excluded?)EDIT
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.sudo du -sh /shows a list of threedu: cannot access '/run/user/1000/gvfs' Permission deniedanddu: cannot access No such file or directoryand a fewcannot read directory invalid argumentand at the end it shows5.4T /which I assume is my root drive combined with what’s inmnt.
If you’re using BTRFS, check if you don’t have any stray snapshots.
