Repeated 'failed to rotate keys' in logs

When Cells has been running a while this message appears in the logs, then repeats every 30s

Sep 27 18:51:20 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:51:20+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 18:51:21 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:51:21+01:00","logger":"pydio.web.auth","msg":"keys rotated, next rotation: 2019-09-27 23:51:21.284836259 +0000 UTC"}
Sep 27 18:51:51 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:51:51+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 18:51:51 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:51:51+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: keys already rotated by another server instance"}
Sep 27 18:52:21 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:52:21+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 18:52:21 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:52:21+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: keys already rotated by another server instance"}
Sep 27 18:52:51 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:52:51+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 18:52:51 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:52:51+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: keys already rotated by another server instance"}
Sep 27 18:53:21 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:53:21+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 18:53:22 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:53:22+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: keys already rotated by another server instance"}
Sep 27 18:53:52 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:53:52+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 18:53:52 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:53:52+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: keys already rotated by another server instance"}
Sep 27 18:54:22 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:54:22+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 18:54:22 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:54:22+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: keys already rotated by another server instance"}
Sep 27 18:54:52 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:54:52+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 18:54:53 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T18:54:53+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: keys already rotated by another server instance"}

and so on. This doesn’t seem to cause a problem using Cells, but it seems odd, and there are other issues where I have to restart Cells every so often. (Websocket 502 error)

Prior to the above messages is:

Sep 27 12:45:37 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:45:37+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 12:46:29 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:46:29+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: update: Error 1205: Lock wait timeout exceeded; try restarting transaction"}
Sep 27 12:46:59 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:46:59+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 12:47:52 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:47:52+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: update: Error 1205: Lock wait timeout exceeded; try restarting transaction"}
Sep 27 12:48:22 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:48:22+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 12:49:13 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:49:13+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: update: Error 1205: Lock wait timeout exceeded; try restarting transaction"}
Sep 27 12:49:43 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:49:43+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 12:50:34 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:50:34+01:00","logger":"pydio.web.auth","msg":"failed to rotate keys: update: Error 1205: Lock wait timeout exceeded; try restarting transaction"}
Sep 27 12:51:04 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:51:04+01:00","logger":"pydio.web.auth","msg":"keys expired, rotating"}
Sep 27 12:51:50 cust-upload cells[995225]: {"level":"info","ts":"2019-09-27T12:51:50+01:00","logger":"pydio.web.auth","msg":"keys rotated, next rotation: 2019-09-27 17:51:04.210933878 +0000 UTC"}

Hi,

It seems that you have a lot of problem with this instance of cells. Could you please tell me more about your system

  • OS version, name
  • DB name
  • Cells version : 1.6.1

OS: Ubuntu 16.04
DB: MariaDB 10.3
Cells: 1.6.1