I rebooted cells and I cannot access any of my personal-files on any account (docker)

[bsikar]

Describe your issue in detail

Everything was fine, I rebooted my machine and now I cannot access my files.

I can see the files however:

What version of Cells are you using?

What is the server OS? Database name/version? Browser name or mobile device description (if issue appears client-side)?

Linux pydio 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

Pydio version: 4.2.3
Mongodb version:

root@pydio:/# mongo --version
MongoDB shell version v4.4.6
Build Info: {
    "version": "4.4.6",
    "gitVersion": "72e66213c2c3eab37d9358d5e78ad7f5c1d0d0d7",
    "openSSLVersion": "OpenSSL 1.1.1  11 Sep 2018",
    "modules": [],
    "allocator": "tcmalloc",
    "environment": {
        "distmod": "ubuntu1804",
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}

Mariadb version:

mariadb from 11.0.2-MariaDB, client 15.2 for debian-linux-gnu (x86_64) using  EditLine wrapper

I am running all of these on docker containers:

version: '3'

services:
  pydio:
    image: pydio/cells:latest
    container_name: pydio_cells
    restart: unless-stopped
    volumes:
      - ./cells:/var/cells
      - /mnt/moms-data/.pydio-cells:/var/cells/data
    depends_on:
      - mariadb
    environment:
      - MONGO_INITDB_ROOT_USERNAME=XXXX
      - MONGO_INITDB_ROOT_PASSWORD=XXXX
    network_mode: host

  mariadb:
    image: mariadb:latest
    container_name: pydio_mariadb
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=XXXX
      - MYSQL_DATABASE=XXXX
      - MYSQL_USER=XXXX
      - MYSQL_PASSWORD=XXXX
    volumes:
      - ./mariadb:/var/lib/mysql
    network_mode: host

  mongodb:
    image: mongo:4.4.6 # cpu doesnt support v5+
    container_name: pydio_mongodb
    restart: unless-stopped
    environment:
      - MONGO_INITDB_ROOT_USERNAME=XXXX
      - MONGO_INITDB_ROOT_PASSWORD=XXXX
    volumes:
      - ./mongodb:/data/db
      - ./mongo-init-scripts:/docker-entrypoint-initdb.d # any .js file in this dir will get ran on init
    network_mode: host

What steps have you taken to resolve this issue already?

I have backedup the database earlier today by running

cp -r pydio .pydio_backup1_6_27_2023

I have then tried to copy the current database to a tmp dir and use the backed up version of the databases, but it did not work.

Also when this first occurred (after the reboot) I was not on the latest versions of mariadb nor pydio, I have tried running the version before the latest on both as well as the current latest version.

[bsinou]

Thanks for nice and precise description of your setup. Yet you forgot the “most” important, the logs

What do the logs say in Cells upon reboot, could you find any hint there?

[bsikar]

Hello, yes sorry about that. Here are the logs:

pydio cells: pydio cells logs · GitHub
mariadb: pydio mariadb logs · GitHub
mongodb: pydio mongodb logs · GitHub

[bsikar]

Was anyone able to detect any possible solutions?

[jaimedelano]

Is the directory /mnt/moms-data/.pydio-cells accessible by Pydio user?
In my experience, this error used to happen on two occasions:

1. There were a large number of files on a network share, or a locally mounted remote storage - slow reading caused errors in Pydio.
2. There are a large number of log files in /var/cells/services/pydio.grpc.log, which caused frequent errors.

[bsikar]

Is the directory /mnt/moms-data/.pydio-cells accessible by Pydio user?

It is:

In my experience, this error used to happen on two occasions:

1. There were a large number of files on a network share, or a locally mounted remote storage - slow reading caused errors in Pydio.

I have this mounted like this in my fstab:

//192.168.1.206/moms-data /mnt/moms-data cifs credentials=cifs,uid=1000,gid=1000,noauto,x-systemd.automount 0 0

For if there is a large number of files, I guess so (87,000 files)? I am uploading the files pydio is accessing on my nas (hosted in truenas):

2. There are a large number of log files in /var/cells/services/pydio.grpc.log, which caused frequent errors.

I have no logs:

[charles]

hi, try manually removing the .minio.sys/config folder (move it to a temporary location) and restart, it should be recreated

[bsikar]

I did that, but it doesnt look like it did anything. The files do differ, but I am still unable to see anything in cells:

[bsikar]

Any more thoughts? I would like to continue using cells.

[jaimedelano]

1. Check that the Pydio user (not root) can access your files:

su - pydio
cd /mnt/moms-data/.pydio-cells/personal

2. Count the number of files in folders personal and thumbs

find /mnt/moms-data/.pydio-cells/personal -type f | wc -l
find /mnt/moms-data/.pydio-cells/thumbs -type f | wc -l

3. Check if you can open the workspace common-files via Pydio’s UI.

[jaimedelano]

First of all, enable verbose logging (debug log level).

[bsikar]

1: Yes I can
2: 87059 and 31499, I am thinking this is because of files that are not images such as pdfs, text files, etc

3: I can

Logs (same links but second revision):

pydio cells: pydio cells logs · GitHub
mariadb: pydio mariadb logs · GitHub
mongodb: pydio mongodb logs · GitHub

[jaimedelano]

Important: Remove secrets (IPs, API keys, e-mail addresses, etc) from Pydio Cells log files before posting it.
Note: English is not my native language, so you may have difficulty understanding what I write.

I think this error is related to a serious bug Pydio has managing large datasources.
My suggestion for confirming it:
1 - Stop Pydio;
2 - Rename the “personal” and “thumbs” folders and create empty folders with same names;

mv /mnt/moms-data/.pydio-cells/personal /mnt/moms-data/.pydio-cells/_personal_
mv /mnt/moms-data/.pydio-cells/thumbs /mnt/moms-data/.pydio-cells/_thumbs_
mkdir /mnt/moms-data/.pydio-cells/personal && chown pydio: /mnt/moms-data/.pydio-cells/personal
mkdir /mnt/moms-data/.pydio-cells/thumbs && chown pydio: /mnt/moms-data/.pydio-cells/thumbs

3 - Start Pydio;
4 - Check if you can upload a file to Personal Files;
5 - Now stop Pydio and restore your files by renaming your folders again.

mv /mnt/moms-data/.pydio-cells/personal /mnt/moms-data/.pydio-cells/personal.tmp
mv /mnt/moms-data/.pydio-cells/thumbs /mnt/moms-data/.pydio-cells/thumbs.tmp
mv /mnt/moms-data/.pydio-cells/_personal_ /mnt/moms-data/.pydio-cells/personal
mv /mnt/moms-data/.pydio-cells/_thumbs_ /mnt/moms-data/.pydio-cells/thumbs

This is only a test, it will not solve your problem. But it may confirm my suspicion.

[c12simple]

Hi @bsikar

That means the datasource services has problem.
The datasource services depend on “object service” which is minio

In this screenshot, I don’t see the .minio.sys/buckets/personal and .minio.sys/buckets/pydiods1 …
I think it’s the source of the problem.

Could you please verify in /mnt/moms-data/.pydio-cells/.minio.sys, is there these buckets?

[bsikar]

Hello sorry for the late reply, I was a bit busy yesterday registering for classes for university. I did steps 1-3 and I was able to see my files again, very confused on why that worked.

I was able to do step 4 and upload a file. Should I still do step 5? It seems that I can access all of the files, but I am very confused on how and why mving those folders worked

EDIT 1:

I can view these files, but cannot download them (except for the iso that I just uploaded)

EDIT 2:

I ran the step 5 commands and now its behaving like it did before:

[bsikar]

Hello, in that screenshot I ran this command:

find . -maxdepth 2 -type d -exec sh -c 'echo "$1" && ls -lah "$1/" | head -n 5' sh {} \;  | less

Thats why not all of the .minio.sys is showing, but I do have those folders:

[jaimedelano]

Step 5 restores your files to their original location. So I think you’d better do it.
Although you are seeing your files, have you tried to open any of them?
Make sure you can open/download them. If not, perform step 5.

[bsikar]

Hello I did step 5 since it didnt let me download or open the files. I now can still not see anything like before.

[jaimedelano]

I hope Pydio developers will fix it soon. For now, I have a temporary solution (workaround).
1 - Stop Pydio;
2 - Create empty folders;

mkdir -p /mnt/moms-data/.pydio-cells/personal.empty
chown pydio: /mnt/moms-data/.pydio-cells/personal.empty
mkdir -p /mnt/moms-data/.pydio-cells/thumbs.empty
chown pydio: /mnt/moms-data/.pydio-cells/thumbs.empty

3 - Mount the datasources personal and thumbs on empty folders

mount --bind /mnt/moms-data/.pydio-cells/personal.empty /mnt/moms-data/.pydio-cells/personal
mount --bind /mnt/moms-data/.pydio-cells/thumbs.empty /mnt/moms-data/.pydio-cells/thumbs

4 - Start Pydio;
5 - Unmount the empty folders/restore your files;

umount /mnt/moms-data/.pydio-cells/personal
umount /mnt/moms-data/.pydio-cells/thumbs

Pydio Cells will be able to read your files again.

[bsikar]

I dont know if that is working:

Maybe I should wait until it is patched or I could help the developers on it, is there a ticket or an issue made already?