Nats - Slow Consumer Detected

Pydio Cells
#1

Running Pydio Cells in docker on linux. Out of the blue Pydio Cells became dog slow. Checking logs was presented with a recurring error of:
nats - Slow Consumer Detected: WriteDeadline of 2s Exceeded…

Restarting the container cleared up the issue.

Any suggestions on how to chase down the root cause of the issue?

#2

Hello, can you give more info about your setup :

  • Pydio Cells version?
  • Hardware sizing (mem/cpu)
  • Possible IO limitations, ulimit value
  • Amount of users, amount of files
    -c
#3

Hi Charles,

As of right now:

  • Pydio Cells Home Edition 2.1.2 (at the time of posting was 2.0.8)
  • 12gb RAM/Intel i7-3770 @3.40ghz
  • no ulimits for this container nor any other limitations set that would inhibit IO
  • 8 users, literally just 5 or 6 files totaling under 100mb combined

At the time of the observed error docker stats showed 500% CPU utilization

#4

Hi @rckalex there must definitely be something wrong… CPU 500% for 5 files is not really what’s expected :wink:
DB on another container?

#5

Hello @charles,

I completely agree. The only containers running on this hardware are the cells, MySQL, and traefik containers (MySQL only used by this cells container). The cells container was consuming 500% of the CPU until a restart at which point CPU consumption dropped into single digits and the recurring nats error went away.

Is there a more verbose logging option in cells to give a better understanding of what caused the nats error should it come back?

#6

No, as nats is the communication layer for all inter-services requests, it’s just saying : your system is going super slow.
If that happens again, first think would be to “ps aux” / “top” / “htop” to detect which Cells process is hogging the CPU. Eventually if you are motivated, you can also send a “kill -SIGUSR1 $PROCESS_ID” which will print a trace of all goroutines in that process. Maybe that can give us some infos…

#7

@charles
Hi Charles,

Had a similar issue today however the mysql container was the gross CPU abuser instead of the cells container (500% vs over 100% for cells). MySQL is spun up with its default configuration with the exception of the character_set_server = utf8mb4 and collation_server = utf8mb4_unicode_ci

Is there a published guideline for recommended parameterize of SQL for use will Cells? buffer size? io threads?