No space left on device (errno ENOSPC) is one of the most common ways a healthy-looking box falls over: services stop writing, logs go silent, and databases refuse connections. The fix is quick once you know the three things that cause it — a genuinely full filesystem, a deleted file still held open, or exhausted inodes.
1. Find which filesystem is full
Don't guess — df -h shows usage per mount. A box can have plenty of space on /home while / or /var is at 100%. Note the mount that's full; that's the only one that matters.
2. Find what's eating it
Walk down from the full mount, biggest first:
du -xh /var --max-depth=1 | sort -rh | head # -x stays on one filesystem; repeat into the biggest dir
Nine times out of ten the culprit is /var/log — an unrotated log that grew without bound. Truncate it in place (safer than rm on a file a service still has open): : > /var/log/huge.log, then fix log rotation so it can't recur.
3. The sneaky one: deleted-but-open files
If df says full but du can't find the space, a process is still holding a file you already deleted — the space isn't freed until the process lets go. Find it with lsof +L1 (lists open files with zero links), then restart or signal the offending process to release the handle.
4. Space free but still ENOSPC? Inodes.
A filesystem with millions of tiny files can run out of inodes while showing free bytes. df -i shows inode usage; if it's at 100%, delete or consolidate the small files (often a mail spool or a runaway cache directory).
Practise it on a box that's actually full
Disk exhaustion is one of the incidents the Chaos Engine throws at you. Drill the find-and-fix loop on a live box with Linux troubleshooting practice, or see the whole family in common Linux incidents.