Nats - Slow Consumer Detected

Running Pydio Cells in docker on linux. Out of the blue Pydio Cells became dog slow. Checking logs was presented with a recurring error of:
nats - Slow Consumer Detected: WriteDeadline of 2s Exceeded…

Restarting the container cleared up the issue.

Any suggestions on how to chase down the root cause of the issue?

Hello, can you give more info about your setup :

  • Pydio Cells version?
  • Hardware sizing (mem/cpu)
  • Possible IO limitations, ulimit value
  • Amount of users, amount of files

Hi Charles,

As of right now:

  • Pydio Cells Home Edition 2.1.2 (at the time of posting was 2.0.8)
  • 12gb RAM/Intel i7-3770 @3.40ghz
  • no ulimits for this container nor any other limitations set that would inhibit IO
  • 8 users, literally just 5 or 6 files totaling under 100mb combined

At the time of the observed error docker stats showed 500% CPU utilization

Hi @rckalex there must definitely be something wrong… CPU 500% for 5 files is not really what’s expected :wink:
DB on another container?

Hello @charles,

I completely agree. The only containers running on this hardware are the cells, MySQL, and traefik containers (MySQL only used by this cells container). The cells container was consuming 500% of the CPU until a restart at which point CPU consumption dropped into single digits and the recurring nats error went away.

Is there a more verbose logging option in cells to give a better understanding of what caused the nats error should it come back?

No, as nats is the communication layer for all inter-services requests, it’s just saying : your system is going super slow.
If that happens again, first think would be to “ps aux” / “top” / “htop” to detect which Cells process is hogging the CPU. Eventually if you are motivated, you can also send a “kill -SIGUSR1 $PROCESS_ID” which will print a trace of all goroutines in that process. Maybe that can give us some infos…

Hi Charles,

Had a similar issue today however the mysql container was the gross CPU abuser instead of the cells container (500% vs over 100% for cells). MySQL is spun up with its default configuration with the exception of the character_set_server = utf8mb4 and collation_server = utf8mb4_unicode_ci

Is there a published guideline for recommended parameterize of SQL for use will Cells? buffer size? io threads?