Disk churn after upgrade to v4

cheneysan · November 22, 2022, 10:03pm

I’ve been using Cells v3 on a home server using docker-compose. I’ve just upgraded to v4.0.4, and although everything appears to be working my server sounds like it’s about to take off (HDD’s are frantically working)!

It’s now been this way for 6 hours. I thought it may be reindexing or the like but there are < 50 files stored in there.

It seems to be the pydio.grpc.data.objects.local1 process causing the issue, and it is sat at 20% CPU usage. Killing the process causes the disks to stop spinning.

Docker logs show the following:

2022-11-22T21:46:06.498Z        INFO    pydio.rest.role ready
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1  API: SYSTEM()
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1  Time: 21:46:06 UTC 11/22/2022
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1  DeploymentID: 24cbd2b8-9c89-4482-8fc2-f597173d0626
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1  Error: Unable to initialize config, some features may be missing Invalid credentials (*fmt.wrapError)
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1         3: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/server-main.go:385:cmd.initAllSubsystems()
2022-11-22T21:46:06.736Z        INFO    pydio.grpc.data.objects.local1         2: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/server-main.go:317:cmd.initServer()
2022-11-22T21:46:06.737Z        INFO    pydio.grpc.data.objects.local1         1: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/server-main.go:526:cmd.serverMain()
2022-11-22T21:46:06.752Z        INFO    pydio.grpc.data.sync.cellsdata  ready
2022-11-22T21:46:06.758Z        INFO    pydio.grpc.data.index.cellsdata ready
2022-11-22T21:46:06.769Z        INFO    pydio.grpc.data.index.pydiods1  ready
2022-11-22T21:46:06.769Z        INFO    pydio.grpc.data.index.personal  ready
2022-11-22T21:46:06.770Z        INFO    pydio.grpc.data.sync.pydiods1   ready
2022-11-22T21:46:06.771Z        INFO    pydio.grpc.data.sync.personal   ready
2022-11-22T21:46:06.772Z        INFO    pydio.grpc.data.index.thumbnails        ready
2022-11-22T21:46:06.772Z        INFO    pydio.grpc.data.index.versions  ready
2022-11-22T21:46:06.773Z        INFO    pydio.grpc.data.sync.thumbnails ready
2022-11-22T21:46:06.774Z        INFO    pydio.grpc.data.sync.versions   ready
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  API: SYSTEM()
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  Time: 21:46:06 UTC 11/22/2022
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  DeploymentID: 24cbd2b8-9c89-4482-8fc2-f597173d0626
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  Error: Unable to migrate IAM users and policies to new format: sio: data is not authentic (*fmt.wrapError)
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1         1: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/iam.go:617:cmd.(*IAMSys).Init()
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  API: SYSTEM()
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  Time: 21:46:06 UTC 11/22/2022
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  DeploymentID: 24cbd2b8-9c89-4482-8fc2-f597173d0626
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1  Error: IAM sub-system is partially initialized, some users may not be available (*errors.errorString)
2022-11-22T21:46:06.787Z        INFO    pydio.grpc.data.objects.local1         1: github.com/minio/minio@v0.0.0-20220720015624-ce8397f7d944/cmd/iam.go:618:cmd.(*IAMSys).Init()
2022-11-22T21:46:09.326Z        INFO    pydio.grpc.tasks        ready
2022-11-22T21:46:09.362Z        INFO    pydio.grpc.mailer       Starting mailer with sender 'disabled'
2022-11-22T21:46:09.840Z        INFO    pydio.grpc.oauth        Finished auth.InitRegistry
2022-11-22T21:46:09.898Z        INFO    pydio.grpc.chat ready
2022-11-22T21:46:09.898Z        INFO    pydio.grpc.user-key     ready

Is there anything I can do to fix the IAM migration issues without a clean re-install?

bsinou · November 23, 2022, 10:49am

Hi,

you probably have a problem of overlapping object services that points toward the same folder.

Could you check in your $CELLS_WORKING_DIR/pydio.json to see if you have more than one service with prefix pydio.grpc.data.objects.* that have same LocalFolder attribute?

(most probably pydio.grpc.data.objects.local1 and a second one, let’s call it pydio.grpc.data.objects.local2 for the sake of clarity)

If this is the case, here is a procedure to solve your problem:

Stop the cells service
backup pydio.json file, and edit it:
- remove the extra “declaration” and “instantiation” of service local2
- replace local2 by local1 in the ds services where it is used
(optional) remove the .minio.sys folder that was at "LocalFolder" path
restart the service

After a re-sync, the issue should be gone

Would be interesting to understand how you end up having this issue on 3.0.x. Any idea? is this a vanilla instance?

cheneysan · November 23, 2022, 5:28pm

Hi bsinou!

Thank you for the reply!

To be clear this issue was occurring on 4.0.4 only after upgrading from 3.0.x. I’m using the Docker image pydio/cells:latest.

I didn’t find any other services using the same LocalFolder - in fact that variable only appeared in pydio.json once:

cheney@homesrv01:/srv/cells$ cat pydio.json | grep "pydio.grpc.data.objects."
    "pydio.grpc.data.objects": {
    "pydio.grpc.data.objects.local1": {
    "pydio.grpc.data.objects": "4.0.4",
    "pydio.grpc.data.objects.local1": "4.0.4",
    "pydio.grpc.data.objects.peer": "4.0.4",
cheney@homesrv01:/srv/cells$ cat pydio.json | grep "LocalFolder"
      "LocalFolder": "/var/cells/data",
cheney@homesrv01:/srv/cells$

Removing the .minio.sys folder does appaer to have fixed the issue though - logs suggest IAM init now completes and my HDDs are now sitting silently!

2022-11-23T17:16:40.474Z        INFO    pydio.grpc.data.objects.local1  Starting local objects service local1 on /var/cells/data
2022-11-23T17:16:40.488Z        INFO    pydio.grpc.data.objects.local1  ready
2022-11-23T17:16:40.698Z        INFO    pydio.grpc.data.objects.local1  Attempting encryption of all config, IAM users and policies on MinIO backend
2022-11-23T17:16:41.042Z        INFO    pydio.grpc.data.objects.local1  Configuration file config/config.json migrated from version '32' to '33' successfully.
2022-11-23T17:16:41.072Z        INFO    pydio.grpc.data.objects.local1  Configuration file config/config.json migrated from version '33' to new KV format successfully.
2022-11-23T17:16:41.103Z        INFO    pydio.grpc.data.objects.local1  IAM initialization complete

Thank you very much!

system · December 4, 2022, 11:28pm

This topic was automatically closed 11 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Returning server is starting because grpc.oauth monitor is not Up Pydio Cells	9	98	September 17, 2024
Unable to launch cells after upgrading from 3.0.9 to 4.1 Pydio Cells install	5	61	November 2, 2024
[Windows] Problems after upgrading to 4.3 - (fixed in 4.3.3) Pydio Cells win	16	611	March 9, 2024
Yet another "Returning server is starting because grpc.oauth monitor is not Up" Pydio Cells install	5	41	October 22, 2024
Upgrade, now nothing works properly Pydio Cells	7	405	April 28, 2021

Disk churn after upgrade to v4

Related topics