When a service suddenly refuses new connections and its logs fill with Too many open files (errno EMFILE, 24), it has hit the limit on how many file descriptors it's allowed to hold open at once. Every socket, pipe, and open file counts — and the default ceiling is often far too low for a busy server.
Confirm which limit you're hitting
# how many FDs is the process using, and what's its ceiling? cat /proc/$(pidof nginx)/limits | grep 'open files' ls /proc/$(pidof nginx)/fd | wc -l lsof -p $(pidof nginx) | wc -l
If the open count is pinned at the soft limit, that's your culprit. A leak (descriptors opened but never closed) looks the same at first — if usage climbs steadily under normal load, raising the limit only buys time; the application needs fixing too.
The stopgap vs. the real fix
Restarting the service (systemctl restart nginx) releases the descriptors and gets you breathing room — but it will recur. The durable fix is to raise the limit where the service actually reads it. For a systemd-managed service, a shell ulimit -n won't help; you need a drop-in:
# /etc/systemd/system/nginx.service.d/limits.conf [Service] LimitNOFILE=65536 # then: systemctl daemon-reload && systemctl restart nginx
Practise it
The Chaos Engine throws descriptor exhaustion at you as a service that quietly stops accepting work. Drill the diagnose-then-raise loop in Linux troubleshooting practice, and see the related resource-limit story in the OOM killer, explained.