Dust is a rewrite of du (in rust obviously) that visualizes your directory tree and what percentage each file takes up. But it only prints as many files fit in your terminal height, so you see only the largest files. It’s been a better experience that du, which isn’t always easy to navigate to find big files (or atleast I’m not good at it.)
Anyway, found a log file at .local/state/nvim/log that was 70gb. I deleted it. Hope it doesn’t bite me. Been pushing around 95% of disk space for a while so this was a huge win 👍
ncdu is the best utility for this type of thing. I use it all the time.
Try dua. It’s like ncdu but uses multiple threads so it’s a lot faster., especially on SSDs.
I install ncdu on any machine I set up, because installing it when it’s needed may be tricky
Try
ncduas well. No instructions needed, just runncdu /path/to/your/directory.I usually use something like
du -sh * | sort -hr | less, so you don’t need to install anything on your machine.
dust does more than what this script does, its a whole new tool. I find dust more human readable by default.
Maybe, but I need it one time per year or so. It is not a task for which I want to install a separate tool.
Perfect for your use case, not as much for others. People sharing tools, and all the different ways to solve this type of problem is great for everyone.
Almost the same here. Well,
du -shc *|sort -hrI admin around three hundred linux servers and this is one of my most common tasks - although I use -shc as I like the total too, and don’t bother with less as it’s only the biggest files and dirs that I’m interested in and they show up last, so no need to scrollback.
When managing a lot of servers, the storage requirements when installing extra software is never trivial. (Although our storage does do very clever compression and it might recognise the duplication of the file even across many vm filesystems, I’m never quite sure that works as advertised on small files)
I admin around three hundred linux servers
What do you use for management? Ansible? Puppet? Chef? Something else entirely?
Main tool is Uyuni, but we use Ansible and AWX for building new vms, and adhoc ansible for some changes.
Interesting; I hadn’t heard of Uyuni before. Thanks for the info!
Seems it just runs Salt/Saltstack?
Suse forked Redhat’s Spacewalk just before it turned into Foreman + Katello.
Then worked an absolute crapload on it to turn it into a modern orchestrator. Part of that was to adopt salt as the agent interface, gradually getting rid of the creaking EL traditional client.
To say “it just runs salt” is to rather miss all the other stuff Uyuni does. Full repo and patch management, remote control, config management, builds, ansible playbook support, salt support, and just about everything else you need to manage hundreds of machines. Oh, and it does that for Rocky, RHEL, Alma, Suse, Ubuntu, Debian and probably a bunch more too, by now. Has a very rich webui, a full API and you can do a bunch more from the cli as well. And if your estate gets too big to manage with one machine, there are proxy agents, as many as you want. I only run a couple of hundred vms through it, but there are estates running thousands.
And it’s free and foss.
Honestly, it’s pretty awesome and I’m amazed it’s not more widely known.
Oh that’s pretty nifty, thanks for the comment. Sorry wasn’t trying to minimize the tool, I was simply referring to the orchestration/config management aspect of it when I looked it up real quick.
I used to be responsible for configurations of 40,000 (yes forty thousand) VMs for a large company using puppet and then later using Ansible and that was an interesting challenge. I’ve been out of the configuration management game for a few years now though so I’m pretty out of the loop. Was familiar with spacewalk back in the day too.
I’ll have to check Uyuni out, thanks for sharing!
I’d say head -n25 instead of less since the offending files are probably near the top anyway
Or head instead of less to get the top entries
With sort -hr, the biggest ones are generally at the bottom already, which is often what most people care about.
So like filelight?
check out
dua. I usually use it in interactive most which lets you navigate through the file system with visual representations of total dir/file size.Here is a screenshot randomly found from the github issues
I also recently found this gui program called
k4dirstatburied in the repos. There are a few more modern options but this one blows them all out of the park.Screenshot from the github repo:
Too bad they used such an ugly configuration for the screenshot… It allows you to modify the visualization to look better and display information differently. Anyway just thought I’d share as the project is old and little known.
thanks for sharing a screenshot of ncdu, should help others discover it
for the visualization itself IMHO Disk Usage Analyzer gives aesthetically pleasing results, not a fan of the UX but it works well enough to identify efficiently large files or directories
Why is it similar to WinDirStats?
Sorry?
You guys aren’t using
du -sh ./{dir1,dir2} | sort -nh | head?A 70gb log file?? Am I misunderstanding something or wouldn’t that be hundreds of millions of lines
I’ve definitely had to handle 30gb plain text files before so I am inclined to believe twice as much should be just as possible
I use gdu and never had any issues like that with it