Memory Consumption Issue?

Ubuntu 18.04 LTS
8 CPU
12GB RAM

We’ve been going through Cells 1.6.1 in our test environment and noticed that there may be an issue with memory consumption. When adding a few hundred images at a time, the scheduler would extract the EXIF data from the images and make the thumbnails as normal. However, upon hitting the memory limits (8GB) the task would fail and memory would drastically go back down to 2GB consumption. Cells runs a bit iffy afterwards unless I reboot.

I bumped up the memory to 12GB to see if this helps and it does. However, once the task finished the memory stayed at 11.2G / 11.7G used. Over the last 15 minutes memory usage dropped a little to 10.9G /11.7G.

Is there a job in place to help clear the memory cache? It seems like if memory stays high then it’s inevitable that the task will fail.

Thanks.

Hello,

There are 2 things here:

  1. The crash of the tasks micro-service that you describe first: we have found and fixed an issue on the job that generate the thumbnails. It is fixed in the master. Would you mind to check if you have the same problem using the v2.0.0-rc1?

  2. The fact that the memory usage stays high: internally, we use Bolt as a Key/Value store. One of the characteristic of this piece of software is that it takes most of the available memory to be faster and release it on demand, see https://github.com/etcd-io/bbolt#caveats--limitations :

Bolt uses a memory-mapped file so the underlying operating system handles the caching of the data. Typically, the OS will cache as much of the file as it can in memory and will release memory as needed to other processes. This means that Bolt can show very high memory usage when working with large databases. However, this is expected and the OS will release memory as needed.

Thanks for getting back to me. Would love to check v2.0.0-rc1. Are you able to shoot a download link my way?

I’ve been able to verify that this works 99% of the time using the latest git. There are a few random jpegs that will not generate thumbnails. They have a smaller file size and dimension compared to others that generate a thumbnail.

Not sure why they wouldn’t generate? Is there a way to re-trigger the thumbnail scheduler job for anything that hasn’t been processed?

Following up here on my comments above:

Looks like there was a issue with a few of the jpegs and this would cause further memory issues and the inability to load a thumbnail. After running the folder of images through Photoshop Image Processor they would even fail. I think it’s safe to assume the issue now lies with the process used to create the original jpeg.

The memory cache issue appears to be corrected as I can see memory free up as the thumbnail process takes place. Thank you!

Sorry, we’ve been in pre-release validation phase and here it is: https://github.com/pydio/cells/releases/tag/v2.0.0-rc2
Downloads are here: https://download.pydio.com/pub/cells/release/2.0.0-rc2/

So you mean that the problem is on your side or I missed something ?