# If cells fail to boot, it creates hundred of staled mysql connections

**URL:** <https://forum.pydio.com/t/if-cells-fail-to-boot-it-creates-hundred-of-staled-mysql-connections/3344>\
**Category:** Pydio Cells\
**Created:** [June 23, 2020, 3:45pm UTC](https://forum.pydio.com/t/if-cells-fail-to-boot-it-creates-hundred-of-staled-mysql-connections/3344 "2020-06-23T15:45:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![drzraf](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/drzraf/32/1207_2.png) [@drzraf](https://forum.pydio.com/u/drzraf)\
**Post date:** [June 23, 2020, 3:45pm UTC](https://forum.pydio.com/t/if-cells-fail-to-boot-it-creates-hundred-of-staled-mysql-connections/3344/1 "2020-06-23T15:45:14Z")

</div>

`cells` fails to boot (`Failed to initialize OIDC server: server: no connectors specified`), also I can’t fix the source of the issue, I noticed that after a minute or so I started getting:  
`failed to initialize storage: failed to open migrations: creating dex_migration table: Error 1040: Too many connections`.

```auto
# mysqladmin processlist|grep -c pydio
151

```

(All of them are in `Sleep` state, the `Command` column)

---

<div class="post-metadata">

**Author:** ![GwynethLlewelyn](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/gwynethllewelyn/32/2419_2.png) [@GwynethLlewelyn](https://forum.pydio.com/u/GwynethLlewelyn)\
**Post date:** [July 8, 2021, 10:46am UTC](https://forum.pydio.com/t/if-cells-fail-to-boot-it-creates-hundred-of-staled-mysql-connections/3344/2 "2021-07-08T10:46:09Z")

</div>

I know, I’m bumping a year-old thread, but I have the same issue… with a slight twist: even when Cells is running after successfully booting, it _still_ opens way too many open connections that do not get closed and remain in a sleep state ‘forever’.

The irony is that for months I have been trying to track down the `too many connections` error to all sorts of _other_ applications, with zero success. Every once in a while, the server I got will struggle to connect to MariaDB — and the WordPress installations I’ve also been running on the same server, as well as other apps, fail with this error. After all, _they_ are not to blame…

When running the many tools to optimise or fine-tune MySQL there is _always_ the mention that ‘too many connections are being aborted’ and the possible cause given for that is recurrent authentication failures. Again, for months I have been fearing that I was being the victim of a mysterious exploit that was attempting to guess MySQL passwords — although I could find nothing on the logs that could give me the slightest clue to the origin of such alleged attacks. I even tried to [go very deep into the networking level to try to figure out what’s wrong with the packets sent and received by MariaDB](https://www.percona.com/blog/2008/08/23/how-to-track-down-the-source-of-aborted_connects/). And again, this was another exercise in vain: there were _no_ mysterious ‘attacks’ or failed authentication attempts coming from anywhere.

Tracking down where exactly in the Cells code database connections are being opened but not closed will be a nightmare; to make matters worse, at the time of writing, I’m having a slight issue compiling the current Cells code on either macOS or Linux — mostly because Cells originally didn’t use Go modules, but rather a popular, pre-modules tool — so I cannot even build a custom version of Cells to help with debugging. I’ve not given up (I used to successfully compile Cells without any problems whatsoever, on earlier Go versions, so I’m sure that, eventually, I’ll manage to do it again), but, in the meantime, I defer to the developers and the community to audit the code and figure out where exactly connections are being opened but never closed, being left in a sleep state.

It would be also interesting to see if more Cells users have encountered the same issue, and, if so, how they dealt with it. Currently, my best choice is to run a `cron` job every now and then and clean up the sleeping threads; but that’s a crude hack, not addressing the core issue.

No wonder that my poor MariaDB server, although there are not that many processes running on the server, nevertheless consumes almost 5 GBytes (!!!) of _real_ memory (not virtual!) — probably because it has to keep so many open connections that never get closed…

---

<div class="post-metadata">

**Author:** ![GwynethLlewelyn](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/gwynethllewelyn/32/2419_2.png) [@GwynethLlewelyn](https://forum.pydio.com/u/GwynethLlewelyn)\
**Post date:** [July 8, 2021, 11:25am UTC](https://forum.pydio.com/t/if-cells-fail-to-boot-it-creates-hundred-of-staled-mysql-connections/3344/3 "2021-07-08T11:25:04Z")

</div>

BTW, here go a few examples of how to kill stale MySQL connections periodically, using external tools.

The following script runs _twice per second_, which seems to be overkill, but you can adjust it according to your wishes:

> **[Make MySQL Faster by Killing Sleep Connections Automatically | BaseZap](https://www.basezap.com/make-mysql-faster-by-killing-sleep-connections/)**
>
> Kill Sleep Connections Automatically to Make MySQL Server Faster, Compatible with cPanel Servers too. Fastest method, Takes very less amount of resources on the server. Newbie Friendly!

A long thread with many alternative solutions, some of them as simple as one-liners that can be run directly from `cron`:

> <https://stackoverflow.com/questions/20048017/shell-script-to-auto-kill-mysql-sleep-processes>

An alternative is to change MySQL settings to let MySQL kill it automatically (if you have access to the MySQL settings, that is; the above solutions will also work if you do _not_ have permission to change these settings directly). Apparently, by default, a thread remains in the sleeping state for 8 hours; this can be adjusted to a much lower number.

> <https://stackoverflow.com/questions/5026204/how-to-delete-sleep-process-in-mysql>

This is also the way that database administrators on StackExchange recommend:

> <https://dba.stackexchange.com/questions/1558/how-long-is-too-long-for-mysql-connections-to-sleep>

---

<div class="post-metadata">

**Author:** ![charles](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/charles/32/3_2.png) [@charles](https://forum.pydio.com/u/charles)\
**Post date:** [July 9, 2021, 4:52pm UTC](https://forum.pydio.com/t/if-cells-fail-to-boot-it-creates-hundred-of-staled-mysql-connections/3344/4 "2021-07-09T16:52:21Z")

</div>

sorry if i’m out of bound, just scanning the thread : did you check that mysql always require to raise the max\_connection value (based on the RAM consumption), as the default 151 is **not** ok for production ?  
[https://dev.mysql.com/doc/refman/5.7/en/connection-interfaces.html](https://dev.mysql.com/doc/refman/5.7/en/connection-interfaces.html)

---

<div class="post-metadata">

**Author:** ![GwynethLlewelyn](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/gwynethllewelyn/32/2419_2.png) [@GwynethLlewelyn](https://forum.pydio.com/u/GwynethLlewelyn)\
**Post date:** [July 11, 2021, 6:38pm UTC](https://forum.pydio.com/t/if-cells-fail-to-boot-it-creates-hundred-of-staled-mysql-connections/3344/5 "2021-07-11T18:38:56Z")

</div>

Speaking strictly for myself, I’ve set `max_connections` to 512 (and `max_user_connections` to zero, which means: _use the value of `max_connections` instead_).

At some point in the past, I even had raised it up to 1000, just to see if the situation improved; however, the price paid in terms of consumed RAM was simply not worth it, for the few (if any) benefits I got.

By contrast, I’m pleased to say that by changing the following settings to

```ini
interactive_timeout = 180
wait_timeout = 180

```

(3 minutes, as opposed to the built-in default, which is 8 hours)

Works wonders for me! 😊

Note that I have a very low-traffic environment; I would guess that the heavier the load, the longer connections ought to persist, to avoid the overhead of closing and creating new connections all the time. Therefore, I would guess that the defaults are reasonable — most MySQL connections are persistent these days, due to the amount of traffic between the many applications and the database server. But not everyone’s needs are the same; that’s why there is no ‘perfect’ configuration (or MySQL would come, by default, with the ‘best’ configuration built-in!) but rather each particular environment needs its own tweaks…

Since, over a year, nobody else but @drzraf and myself seems to be bothered by this particular issue, it’s a good guess that the vast majority of Cells users have completely different needs, for whom this issue is not really important…
