When HAProxy marks a backend DOWN, traffic stops flowing to it — and if every server in a backend fails its check, clients get a 503. The hard part is telling a genuinely dead backend from a health check that's testing the wrong thing.
1. Look at the stats page or logs
HAProxy's stats page (or echo "show stat" | socat stdio /run/haproxy/admin.sock) shows each server's state and the last health-check result. The log line is explicit: Server web/web1 is DOWN, reason: Layer7 wrong status, code: 503 tells you the check connected but got a bad HTTP response.
2. Is the backend actually up?
Test the real server the same way HAProxy does, from the HAProxy host: curl -v http://10.0.1.15:80/. If that fails too, it's a genuine backend outage — go fix the app (a 502 / dead upstream situation). If curl works but HAProxy says DOWN, the check is the problem.
3. Fix the health check
The usual culprits: the check hits / but the app only answers on /health; the app returns 302/401 and HAProxy only accepts 2xx; or the Host header is missing. Point the check at a real health endpoint:
backend web
option httpchk GET /health HTTP/1.1\r\nHost:\ localhost
http-check expect status 200
server web1 10.0.1.15:80 check inter 2s fall 3 rise 24. Validate and reload
Always check the config before applying: haproxy -c -f /etc/haproxy/haproxy.cfg, then systemctl reload haproxy — a reload drains connections gracefully, where a restart drops them.
Practise load-balancer incidents
Load-balancer misconfig and health-check failures are incidents the Chaos Engine throws. Work them in a full incident response shift.