Disk churn after upgrade to v4

I’ve been using Cells v3 on a home server using docker-compose. I’ve just upgraded to v4.0.4, and although everything appears to be working my server sounds like it’s about to take off (HDD’s are frantically working)!

It’s now been this way for 6 hours. I thought it may be reindexing or the like but there are < 50 files stored in there.

It seems to be the pydio.grpc.data.objects.local1 process causing the issue, and it is sat at 20% CPU usage. Killing the process causes the disks to stop spinning.

Docker logs show the following:

2022-11-22T21:46:06.498Z        INFO    pydio.rest.role ready
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1  API: SYSTEM()
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1  Time: 21:46:06 UTC 11/22/2022
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1  DeploymentID: 24cbd2b8-9c89-4482-8fc2-f597173d0626
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1  Error: Unable to initialize config, some features may be missing Invalid credentials (*fmt.wrapError)
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1         3: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/server-main.go:385:cmd.initAllSubsystems()
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1         2: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/server-main.go:317:cmd.initServer()
2022-11-22T21:46:06.737Z        INFO    pydio.grpc.data.objects.local1         1: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/server-main.go:526:cmd.serverMain()
2022-11-22T21:46:06.752Z        INFO    pydio.grpc.data.sync.cellsdata  ready
2022-11-22T21:46:06.758Z        INFO    pydio.grpc.data.index.cellsdata ready
2022-11-22T21:46:06.769Z        INFO    pydio.grpc.data.index.pydiods1  ready
2022-11-22T21:46:06.769Z        INFO    pydio.grpc.data.index.personal  ready
2022-11-22T21:46:06.770Z        INFO    pydio.grpc.data.sync.pydiods1   ready
2022-11-22T21:46:06.771Z        INFO    pydio.grpc.data.sync.personal   ready
2022-11-22T21:46:06.772Z        INFO    pydio.grpc.data.index.thumbnails        ready
2022-11-22T21:46:06.772Z        INFO    pydio.grpc.data.index.versions  ready
2022-11-22T21:46:06.773Z        INFO    pydio.grpc.data.sync.thumbnails ready
2022-11-22T21:46:06.774Z        INFO    pydio.grpc.data.sync.versions   ready
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  API: SYSTEM()
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  Time: 21:46:06 UTC 11/22/2022
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  DeploymentID: 24cbd2b8-9c89-4482-8fc2-f597173d0626
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  Error: Unable to migrate IAM users and policies to new format: sio: data is not authentic (*fmt.wrapError)
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1         1: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/iam.go:617:cmd.(*IAMSys).Init()
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  API: SYSTEM()
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  Time: 21:46:06 UTC 11/22/2022
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  DeploymentID: 24cbd2b8-9c89-4482-8fc2-f597173d0626
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  Error: IAM sub-system is partially initialized, some users may not be available (*errors.errorString)
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1         1: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/iam.go:618:cmd.(*IAMSys).Init()
2022-11-22T21:46:09.326Z        INFO    pydio.grpc.tasks        ready
2022-11-22T21:46:09.362Z        INFO    pydio.grpc.mailer       Starting mailer with sender 'disabled'
2022-11-22T21:46:09.840Z        INFO    pydio.grpc.oauth        Finished auth.InitRegistry
2022-11-22T21:46:09.898Z        INFO    pydio.grpc.chat ready
2022-11-22T21:46:09.898Z        INFO    pydio.grpc.user-key     ready

Is there anything I can do to fix the IAM migration issues without a clean re-install?

Hi,

you probably have a problem of overlapping object services that points toward the same folder.

Could you check in your $CELLS_WORKING_DIR/pydio.json to see if you have more than one service with prefix pydio.grpc.data.objects.* that have same LocalFolder attribute?

(most probably pydio.grpc.data.objects.local1 and a second one, let’s call it pydio.grpc.data.objects.local2 for the sake of clarity)

If this is the case, here is a procedure to solve your problem:

  • Stop the cells service
  • backup pydio.json file, and edit it:
    • remove the extra “declaration” and “instantiation” of service local2
    • replace local2 by local1 in the ds services where it is used
  • (optional) remove the .minio.sys folder that was at "LocalFolder" path
  • restart the service

After a re-sync, the issue should be gone

Would be interesting to understand how you end up having this issue on 3.0.x. Any idea? is this a vanilla instance?

Hi bsinou!

Thank you for the reply!

To be clear this issue was occurring on 4.0.4 only after upgrading from 3.0.x. I’m using the Docker image pydio/cells:latest.

I didn’t find any other services using the same LocalFolder - in fact that variable only appeared in pydio.json once:

cheney@homesrv01:/srv/cells$ cat pydio.json | grep "pydio.grpc.data.objects."
    "pydio.grpc.data.objects": {
    "pydio.grpc.data.objects.local1": {
    "pydio.grpc.data.objects": "4.0.4",
    "pydio.grpc.data.objects.local1": "4.0.4",
    "pydio.grpc.data.objects.peer": "4.0.4",
cheney@homesrv01:/srv/cells$ cat pydio.json | grep "LocalFolder"
      "LocalFolder": "/var/cells/data",
cheney@homesrv01:/srv/cells$

Removing the .minio.sys folder does appaer to have fixed the issue though - logs suggest IAM init now completes and my HDDs are now sitting silently!

2022-11-23T17:16:40.474Z        INFO    pydio.grpc.data.objects.local1  Starting local objects service local1 on /var/cells/data
2022-11-23T17:16:40.488Z        INFO    pydio.grpc.data.objects.local1  ready
2022-11-23T17:16:40.698Z        INFO    pydio.grpc.data.objects.local1  Attempting encryption of all config, IAM users and policies on MinIO backend
2022-11-23T17:16:41.042Z        INFO    pydio.grpc.data.objects.local1  Configuration file config/config.json migrated from version '32' to '33' successfully.
2022-11-23T17:16:41.072Z        INFO    pydio.grpc.data.objects.local1  Configuration file config/config.json migrated from version '33' to new KV format successfully.
2022-11-23T17:16:41.103Z        INFO    pydio.grpc.data.objects.local1  IAM initialization complete

Thank you very much!