Nats - Slow Consumer Detected

Running Pydio Cells in docker on linux. Out of the blue Pydio Cells became dog slow. Checking logs was presented with a recurring error of:
nats - Slow Consumer Detected: WriteDeadline of 2s Exceeded…

Restarting the container cleared up the issue.

Any suggestions on how to chase down the root cause of the issue?

Hello, can you give more info about your setup :

  • Pydio Cells version?
  • Hardware sizing (mem/cpu)
  • Possible IO limitations, ulimit value
  • Amount of users, amount of files
    -c

Hi Charles,

As of right now:

  • Pydio Cells Home Edition 2.1.2 (at the time of posting was 2.0.8)
  • 12gb RAM/Intel i7-3770 @3.40ghz
  • no ulimits for this container nor any other limitations set that would inhibit IO
  • 8 users, literally just 5 or 6 files totaling under 100mb combined

At the time of the observed error docker stats showed 500% CPU utilization

Hi @rckalex there must definitely be something wrong… CPU 500% for 5 files is not really what’s expected :wink:
DB on another container?

Hello @charles,

I completely agree. The only containers running on this hardware are the cells, MySQL, and traefik containers (MySQL only used by this cells container). The cells container was consuming 500% of the CPU until a restart at which point CPU consumption dropped into single digits and the recurring nats error went away.

Is there a more verbose logging option in cells to give a better understanding of what caused the nats error should it come back?

No, as nats is the communication layer for all inter-services requests, it’s just saying : your system is going super slow.
If that happens again, first think would be to “ps aux” / “top” / “htop” to detect which Cells process is hogging the CPU. Eventually if you are motivated, you can also send a “kill -SIGUSR1 $PROCESS_ID” which will print a trace of all goroutines in that process. Maybe that can give us some infos…