# Cells 4.0.0 RC is constantly writing memory-mapped files to disk, even when nothing is happening \[solved\*\]

**URL:** https://forum.pydio.com/t/cells-4-0-0-rc-is-constantly-writing-memory-mapped-files-to-disk-even-when-nothing-is-happening-solved/4735
**Category:** Pydio Cells
**Created:** [October 16, 2022, 3:56pm UTC](https://forum.pydio.com/t/cells-4-0-0-rc-is-constantly-writing-memory-mapped-files-to-disk-even-when-nothing-is-happening-solved/4735 "2022-10-16T15:56:43Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![GwynethLlewelyn](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/gwynethllewelyn/32/2419_2.png) [@GwynethLlewelyn](https://forum.pydio.com/u/GwynethLlewelyn)
#### Post date: [October 16, 2022, 3:56pm UTC](https://forum.pydio.com/t/cells-4-0-0-rc-is-constantly-writing-memory-mapped-files-to-disk-even-when-nothing-is-happening-solved/4735/1 "2022-10-16T15:56:44Z")

</div>

Hi!

I’m trying out Cells 4.0.0 RC (RC7 at the time of writing, but the same applies to previous RCs as well) and have noticed that for some mysterious reason, Cells is writing on the disk _constantly_.

While this is a _production_ installation — not a mere test — it has very few actual users and exists mostly to give easy access to 10±year-old archives to a small team of people who use them very sporadically. Although there are tens of thousands of files there — including large files such as videos and old backups — my expectations are that _most_ of the time, Cells is not being _actively_ used by _humans_. The only ‘activity’ would come mostly from the many maintenance tasks scheduled by Cells itself.

Ordinarily I wouldn’t worry much about the constant writing — the disks are, after all, enterprise-class and designed to endure the load — but it doesn’t seem ‘right’. Compared to other software running on the same server (including the database server running MariaDB), Cells contributes almost 99% of all disk writes. The constant stream is not necessarily saturating the bus bandwidth towards the disks (at least, according to what I could figure out from its specs). But it _is_ constant and non-stop, 24 hours a day.

Note that the server running this has 64 GBytes of RAM and has been tweaked to keep as much data as it can in memory (there is not even swap). Admittedly, there may be some misconfiguration — in the recent past, for instance, due to a mistake I made, the system would not consume much more than 16 GBytes of RAM, which _was_ strange, since, by default, Linux tries to use up as much memory as it safely can (there is no point in ‘wasting’ RAM, after all). In fact, one of the reasons for investigating why it wasn’t consuming more memory was the huge amount of write requests being made by Cells. I hoped that, with more RAM available, Cells could safely accommodate whatever it wanted in RAM and leave the disk in peace.

To give you an idea: the _full_ data currently being served by Cells is less than 100 GBytes. In theory, almost all of it could be in memory! In practice, only a small percentage of those 100 GBytes might be in use at each time — and only during the short periods when an actual human is browsing through the files and _actively_ uploading or downloading files, during which it is conceivable that Cells might, indeed, increase disk activity. But not _all the time_.

At first, I thought that this was just Cells writing constantly to some logs (because I might have left the settings at ‘trace’ level, for example). Cells _is_ chatty with its logs, and there are database logs (even more disk writes!) and disk-based logs.

Apparently, the amount of data written to the logs is ‘just fine’, according to my expectations. Logging is _not_ constant, but comes in bursts, every time a scheduled event fires up and/or a human accesses the system. This would be the expected behaviour and should not be significant anyway — at least when compared with the logs coming in from the web and mail servers (which _are_ constant!), or even from `systemd`’s own log journals — I mean, Cells is writing to disk about _hundred_ times more frequently than _all other systems_ added together, and, on top of that, it does so _constantly_.

To say that this is intriguing is an understatement! Note that nothing of the sort happened with version 3; it just became apparent under the first version 4 release.

Also note that the system is not especially busy, in terms of overall activity; the server is deliberately overpowered for the amount of work it is supposed to do. In fact, although there was a significant rise in CPU consumption from version 3 to version 4, most of it comes from I/O waits — in other words, Cells is managing to acquire so many file handlers to open connections to files on disk that it starves off other processes (even Cells’ own subtasks!) attempting to do the same. Unix systems have, these days, excellently design schedulers for practically everything, and this means that, in general, with a little tweaking, and having lots of RAM to spare, it’s possible to fit so much data in memory that the vast majority of the running applications do not really need to write frequently to disk. In effect, it’s _mostly_ the software RAID manager that bears the burden of figuring out when things need to be written to disk.

However, it competes with access to disk with Cells — to the point that Cells can effectively block the underlying user-space disk management subsystem to write to disk. Again, as said, this does _not_ have a dramatic effect overall. Linux is supposed to be able to handle those extreme scenarios, and it does so very effectively. Overall, I think that CPU consumption merely doubled when going from Cells 3 to 4, which could be interpreted that processes get interrupted twice as more as before to wait on I/O, and get swapped out of the scheduling queue until Cells finishes whatever it is doing, and releases a slice of disk I/O access time to other processes. Since all others are not really writing a lot to disk, the scheduler is clever enough to let them write as much as they need without disturbing their functioning. Because most of these processes are working from RAM anyway, they manage to serve whatever services they are providing effectively without delays; eventually, though, they _might_ need to write the odd file to disk (e.g. logging something to a file, saving an incoming email message on disk, etc.), and these disk writing events _might_ expect a longer delay than before (but at the scale humans perceive things, it would not be significant). As mentioned, the biggest single contender for I/O (especially _writing_ to disk) is the MariaDB database management system — there is a limit to how much it can do in memory safely without requiring to finally write _something_ on disk.

All right. This was just to give you an idea that this ‘constant disk writing’ is _not_ crucial for the overall operation of this system — I can live with it being a teensy bit slower than before — but, nevertheless, _something_ must be wrong with the absurd amount of constant disk writing done by Cells.

Ultimately, I managed to track it down to memory-mapped files.

This was by no means obvious to me, but the ‘new’ flat-file storage system that Cells introduced with 4 essentially works on slicing up the disk space in separate files (allegedly with the `.zap` extension) which are then managed from within BoltDB (how exactly this is accomplished is yet a mystery to me; aye, I know I could read the code and get an answer, but Cells’ codebase is _massive_…). While the BoltDB database file format is essentially fixed (as well as its API) and frozen, nothing prevents making subtle changes to the (open-source) code, retaining API compatibility but implementing certain details quite differently; although I suspect that it’s the `minio` middleware/framework layer that ‘decides’ how buckets are stored (to disk or other locations).

Whatever layer or level of abstraction is responsible for the actual file layout at some point there _are_ memory-mapped files. A _lot_ of them. Here is how I’ve found about them.

For convenience’s sake, I use `systemd` to manage each and every service running on my Ubuntu 22.04.1 LTS server (I’m actually agnostic about the merits and disadvantages of using a specific system manager; the point is that I _really_ want an uniform way of doing such management, instead of having different ‘startup scripts’ scattered all over the filesystem; Unix is already historically very disorganised — almost by design! — and I try to keep the chaos at the minimum manageable level…); using `systemctl status cells`, therefore, shows me the root PID of Cells.

And from there I can list all open files and sockets using `lsof -p <Cells PID>` — there are many other ways and tools, but I’m a fan of `lsof` myself, so… I see that _tons_ of files/sockets are being opened — especially because Cells is really a cluster of micro-services, all connecting to each other via Internet sockets, so that they can be deployed as a cluster on different servers, or containers, thus allowing load to be spread across a cloud. But of special interest (to me) were the memory-mapped files:

```auto
cells 3317464 <cells user> mem REG 9,2 14978 173147514 <path to cells>/services/pydio.grpc.log/syslog.bleve.0005/store/000000007782.zap
cells 3317464 <cells user> mem REG 9,2 149215 173147513 <path to cells>/services/pydio.grpc.log/syslog.bleve.0005/store/000000007781.zap
cells 3317464 <cells user> mem-W REG 9,2 65536 173147519 <path to cells>/services/pydio.grpc.log/syslog.bleve.0005/store/root.bolt
cells 3317464 <cells user> mem REG 9,2 14604 173147500 <path to cells>/services/pydio.grpc.log/syslog.bleve.0004/store/0000000078b3.zap
cells 3317464 <cells user> mem REG 9,2 11158 173147499 <path to cells>/services/pydio.grpc.log/syslog.bleve.0004/store/0000000078b2.zap
cells 3317464 <cells user> mem-W REG 9,2 65536 173147502 <path to cells>/services/pydio.grpc.log/syslog.bleve.0004/store/root.bolt
cells 3317464 <cells user> mem REG 9,2 11563 173016463 <path to cells>/services/pydio.grpc.log/syslog.bleve.0003/store/000000006585.zap
cells 3317464 <cells user> mem-W REG 9,2 65536 173016465 <path to cells>/services/pydio.grpc.log/syslog.bleve.0003/store/root.bolt
cells 3317464 <cells user> mem REG 9,2 16620 173016452 <path to cells>/services/pydio.grpc.log/syslog.bleve.0002/store/00000000d3ed.zap
cells 3317464 <cells user> mem REG 9,2 8265 173016451 <path to cells>/services/pydio.grpc.log/syslog.bleve.0002/store/00000000d3ec.zap
cells 3317464 <cells user> mem REG 9,2 11082 173016450 <path to cells>/services/pydio.grpc.log/syslog.bleve.0002/store/00000000d3eb.zap
cells 3317464 <cells user> mem REG 9,2 26789 173016449 <path to cells>/services/pydio.grpc.log/syslog.bleve.0002/store/00000000d3ea.zap
cells 3317464 <cells user> mem REG 9,2 1224448 173016448 <path to cells>/services/pydio.grpc.log/syslog.bleve.0002/store/00000000d3e4.zap
cells 3317464 <cells user> mem-W REG 9,2 65536 173016453 <path to cells>/services/pydio.grpc.log/syslog.bleve.0002/store/root.bolt
cells 3317464 <cells user> mem REG 9,2 12978 173016435 /<path to cells>/services/pydio.grpc.log/syslog.bleve.0001/store/000000008a14.zap
cells 3317464 <cells user> mem REG 9,2 8624 173016434 /<path to cells>/services/pydio.grpc.log/syslog.bleve.0001/store/000000008a13.zap
cells 3317464 <cells user> mem REG 9,2 8613 173016433 <path to cells>/services/pydio.grpc.log/syslog.bleve.0001/store/000000008a12.zap
cells 3317464 <cells user> mem REG 9,2 457424 173016430 <path to cells>/services/pydio.grpc.log/syslog.bleve.0001/store/000000008a0f.zap
[...]

```

There are about 200 of those; each, in turn, will have an open regular file descriptor — which is the one constantly writing to disk, non-stop, no matter what.

Now, each of those 200 files is not huge in size. They vary a bit in size — from as small as 11KBytes to as large as 11MBytes — and adding the sizes of all of them together is merely 3 GBytes. There is more than enough RAM for _that_. In other words: it doesn’t make a lot of sense to have the ‘constant writing to disk’ of memory-mapped files if they can all be kept in memory. After all, the whole purpose of memory-mapped files is exactly that: to keep as much of it in memory, act upon it as if it were memory, and just let the underlying operating system deal with eventual reads/writes from disk, if and when it is necessary.

As a consequence, Cells basically brings the whole system down, except, well, for Cells itself.

WIthout Cells running, the system runs at around 12-13% load, and that includes a few dozens of low-traffic websites with associated mailserver (with several layers of spam filtering), a Git repository, a video streaming server, etc. In other words: it’s not _very_ busy, but it’s not sitting prettily and idly twirling its digital fingers.

(In fact, _most_ of the actual disk I/O comes from a stupid security plugin for WordPress which _insists_ in writing constantly logs for _all_ requests on the MariaDB database; since all websites really require that specific plugin, this forces the database server to essentially work as a glorified logging system, for which it’s clearly not the best choice; although I’ve contacted the plugin authors and grumbled a lot about it, they replied that ‘having the logs on the database is essential for providing our security services’. I’d have a load of _zero_! 🤬)

That said, I believe that all the above comes essentially from the ‘new’ way that storage can be defined, i.e. as a ‘flat file’ database, which I do _not_ use (mostly because I require the traditional hierarchical filesystem layout for other services — such as the above-mentioned video streaming service, which needs to access a ‘real’ file, not something virtualised inside an impenetrable container…). Nevertheless, I understand that a few crucial services have already migrated to the ‘new’ flat file format, and that’s why I see all these memory-mapped files, which previously didn’t exist (I think).

I’m pretty sure that these cannot be ‘turned off’, but I wonder if there is any way to mitigate their impact on the disk writes? _Why_ are they constantly writing to disk, after all? And why don’t such writes use some sort of buffering? Surely, on a Cells installation which may have lots of files, but few users accessing them, most of the time, Cells will be inactive, except for the occasional scheduled task?

I also understand that the memory-mapped files are at the very bottom layer of the overall system: there is [Minio](https://minio.io) to manage buckets and export them via a S3 API; there is [Bleve](https://blevesearch.com) to handle full-text indexing; and some of those services will ultimately use a [Bolt](https://github.com/etcd-io/bbolt) database to manage everything. Decoupling those systems from each other is probably impossible. However, there _might_ be a flag or configuration option at some level where it might be possible to tweak a setting or two…

That’s what I’m hoping for!

In the meantime, I have no other option but to keep Cells asleep 🙁

---

<div class="post-metadata">

### Author: ![GwynethLlewelyn](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/gwynethllewelyn/32/2419_2.png) [@GwynethLlewelyn](https://forum.pydio.com/u/GwynethLlewelyn)
#### Post date: [October 17, 2022, 1:53am UTC](https://forum.pydio.com/t/cells-4-0-0-rc-is-constantly-writing-memory-mapped-files-to-disk-even-when-nothing-is-happening-solved/4735/2 "2022-10-17T01:53:12Z")

</div>

Huh.

This is _really_ weird, but, apparently (at least for now), as soon as [I fixed the issue I had reported elsewhere](https://forum.pydio.com/t/cells-4-0-0-gets-confused-when-a-former-directory-is-deleted-and-replaced-with-a-file-with-the-same-name-mac-user-here/4736), this unnatural behaviour disappeared…?

I’m going to monitor this issue closely, and now I’m even beginning to wonder if something completely different — which I _assumed_ to have absolutely nothing to do with Pydio! — has also magically been ‘fixed’ as well…

---

<div class="post-metadata">

### Author: ![bsinou](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/bsinou/32/476_2.png) [@bsinou](https://forum.pydio.com/u/bsinou)
#### Post date: [October 17, 2022, 1:37pm UTC](https://forum.pydio.com/t/cells-4-0-0-rc-is-constantly-writing-memory-mapped-files-to-disk-even-when-nothing-is-happening-solved/4735/3 "2022-10-17T13:37:09Z")

</div>

Hi @GwynethLlewelyn

Nice post, thanks for the precise feedback 😃

Just a few remarks to correct some misunderstandings:

> [@GwynethLlewelyn](#):
>
> the ‘new’ flat-file storage system that Cells introduced with 4 essentially works on slicing up the disk space in separate files (allegedly with the `.zap` extension) which are then managed from within BoltDB

Flat storage has been introduced in v3, not in v4 and it has nothing to do with BoltDB

> [@GwynethLlewelyn](#):
>
> although there was a significant rise in CPU consumption from version 3 to version 4, most of it comes from I/O waits — in other words, Cells is managing to acquire so many file handlers to open connections to files on disk that it starves off other processes (even Cells’ own subtasks!) attempting to do the same

This is not normal and is rather linked to another issue as you correctly described.  
For the record, what we expect / have observed during our benches, comparing v4 to v3:

- More or less same consumption of RAM / CPU resources on low usage
- Much better behavior when usage is growing bigger
- Less ports use that has also a positive effect on resources consumption

> [@GwynethLlewelyn](#):
>
> and some of those services will ultimately use a [Bolt](https://github.com/etcd-io/bbolt) database to manage everything.

This is not accurate, Bolt is **not** the database to **rule them all**. This key/value store is mainly used for indexation (search engine, logs…) and for some metadata (tags…).  
In Cells v4, Bolt can also be replaced by MongoDB for larger and HA setups. We are currently working on the documentation for the cluster that makes this much clearer.  
You can already have a look here: [Pydio | Going Stateless](https://pydio.com/en/docs/cells/v4/going-stateless)

> [@GwynethLlewelyn](#):
>
> This is _really_ weird, but, apparently (at least for now), as soon as [I fixed the issue I had reported elsewhere](https://forum.pydio.com/t/cells-4-0-0-gets-confused-when-a-former-directory-is-deleted-and-replaced-with-a-file-with-the-same-name-mac-user-here/4736), this unnatural behaviour disappeared…?
> 
> I’m going to monitor this issue closely, and now I’m even beginning to wonder if something completely different — which I _assumed_ to have absolutely nothing to do with Pydio! — has also magically been ‘fixed’ as well…

Indeed, this is weird 🙂 We will anyway also have a look at you other issue and see if you might have found a bug, but good news is that you seems to have solved your issue!

---

<div class="post-metadata">

### Author: ![GwynethLlewelyn](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/gwynethllewelyn/32/2419_2.png) [@GwynethLlewelyn](https://forum.pydio.com/u/GwynethLlewelyn)
#### Post date: [November 1, 2022, 1:35am UTC](https://forum.pydio.com/t/cells-4-0-0-rc-is-constantly-writing-memory-mapped-files-to-disk-even-when-nothing-is-happening-solved/4735/4 "2022-11-01T01:35:39Z")

</div>

Ouch. Thanks for the elaborate corrections, @bsinou — it seems that I had a _lot_ of misunderstandings, I do apologise for them. I jump too easily to conclusions based on incomplete data hehe 🙂

But I really appreciate the explanations and corrections! I’m always learning new things…

In any case, aye, I can confirm that this is definitely related to [my other issue](https://forum.pydio.com/t/cells-4-0-0-gets-confused-when-a-former-directory-is-deleted-and-replaced-with-a-file-with-the-same-name-mac-user-here-solved/4736): in other words, whenever there are folders which have been replaced with files with the exact name, Cells turns crazy, grabs all possible file/socket/resource handles, and does ‘something’ with the memory-mapped files which I could not figure out. But all these symptoms are clearly related: once I removed the structured storage and added it back again, everything started to work flawlessly again (I believe I had to restart Cells, but now I’m not even sure if that was necessary or not).

I’d guess it’s not even a bug in Cells — more likely a bug in the core Go libraries themselves, or even in the Go compiler. It’s hard to say; it’s certainly one of those nasty ‘edge conditions’ that will _never_ happen to 99.99999% of all users, and then, one day, someone ‘decides’ to upload a file with the exact name as a folder, and that triggers all these mysterious symptoms.

In any case, with the same caveats as the other issue I reported, I’ll mark this as solved — the workaround works (sorry for the pleonasm!) and that’s enough for me!

---

<div class="post-metadata">

### Author: ![system](https://canada1.discourse-cdn.com/flex032/uploads/pydio/original/2X/6/61f19fa52ec15fd4250db51aea351adba0e831fd.png) [@system](https://forum.pydio.com/u/system)
#### Post date: [November 12, 2022, 7:36am UTC](https://forum.pydio.com/t/cells-4-0-0-rc-is-constantly-writing-memory-mapped-files-to-disk-even-when-nothing-is-happening-solved/4735/5 "2022-11-12T07:36:12Z")

</div>

This topic was automatically closed 11 days after the last reply. New replies are no longer allowed.