When a message queue suddenly stops accepting publishes and the backlog balloons, the broker is usually in flow control. RabbitMQ protects itself by raising a memory high-watermark alarm and blocking publishing connections until memory drops — so producers stall while consumers fall behind.
Confirm the alarm
rabbitmqctl status | grep -A3 alarms rabbitmqctl list_queues name messages consumers | sort -k2 -rn | head # the broker log shows: "memory resource limit alarm set"
If consumers is zero on a deep queue, your consumers died — fix them and the backlog drains. If memory is genuinely high, the alarm is doing its job and you need headroom.
Clear it and give it headroom
Restarting the broker clears a stuck alarm and lets consumers reconnect, but if the watermark is simply set too low it will trip again under normal load. Raise it in /etc/rabbitmq/rabbitmq.conf:
# fraction of system memory before publishers are blocked vm_memory_high_watermark.relative = 0.6 # apply: systemctl restart rabbitmq-server
Longer term, a backlog that keeps growing is a capacity problem: add consumers, set queue length limits, or offload old messages — a bigger watermark only delays the wall.
Practise it
Broker backpressure is one of the simulator's messaging incidents. Work it in a full incident response shift.