Massive stability problems when using s3 storage (minio)

[rudepeople]

I’m trying to build a storage server to share out videos and pictures with friends and family and I’m running into a problem with connecting to an existing storage system.

So this all comes down to pydio interacting with a minio server. I build the minio server a couple of years ago and it’s been absolutely bullet proof… the problem is, sharing files gets a little complicated in that minio will never share a file for longer than a week. When I discovered pydio, I thought about switching from one to the other, but without going into too much detail, I learned that hosting my data on minio was simply the preferred route.

Pydio is supposed to be able to handle it without issues, however I keep running into performance and stability issues. When I set up pydio, I linked it to my minio server for cellsdata pydiods, personal, etc. then I added the 3 main buckets as storage and workspaces on my server. all worked fine for a day or so, now I noticed when I try to upload files to my minio buckets, the server hangs and eventually fails to transfer. Checking the tasks logs and I’m seeing a lot of these:

{"level":"error","ts":"2021-03-15T10:23:18-06:00","logger":"pydio.grpc.tasks","msg":"Error while running action actions.cmd.resync","LogType":"tasks","SpanRootUuid":"8e69d9fe-85aa-11eb-86e7-ac1f6b9dfcca","SpanParentUuid":"8e69d9fe-85aa-11eb-86e7-ac1f6b9dfcca","SpanUuid":"8e6a40c9-85aa-11eb-8235-ac1f6b9dfcca","OperationUuid":"resync-ds-cellsdata-cd0d5bb0","SchedulerJobUuid":"resync-ds-cellsdata","SchedulerTaskUuid":"cd0d5bb0-4bb8-4cc8-847e-be4c7567b00d","SchedulerTaskActionPath":"ROOT/actions.cmd.resync$0","RemoteAddress":"[IPaddressremoved]","UserAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36","ContentType":"application/json","HttpProtocol":"HTTP/1.1","error":"{\"id\":\"\",\"code\":0,\"detail\":\"Cannot Stat Root: Object storage backend is unreachable\",\"status\":\"\"}"}

As I’ve said, minio has been effectively bullet proof so I am fairly confident the minio server isn’t the problem. But I cant seem to get pydio to play nice! am I doing it wrong? should I not have set up cells to store it’s data in minio with the workspaces?

[rudepeople]

Shameless self-reply.
So I went ahead and relocated the cells data folders to the local filesystem…
I stopped the cells service, mirrored the cells buckets to /var/cells/data/[folder], ran cells configure, and reset to store data locally (default folder).
Once complete, the pydio.log file logged a TON of broken files most of which were in cellsdata. By deleting those folders, the system appears to have become stable again, but I’m worried about long term issues. I use cells (cellsdata) for individual family members and their content (which told me who broke it) and it seems it happened when someone was moving a huge number of music files into pydio and the transfer got interrupted somehow (probably his laptop went to sleep). Is there a way to elegantly clean these files in the future? also, is there a better way to do this?

[charles]

Hi @rudepeople
Can you post a couple (not the TON) of logs that you’ve identified as showing “broken files” ?
-c

[rudepeople]

I can not. I relocated the storage to an SSD to try to improve performance and nuked the original data store. I haven’t seen any more broken files since I reported this 16 days ago.

At this point I think I’m just looking for housekeeping tips.

That said, I have run into a sync issue but I think I should create a new thread for that. Basically, we have noticed if someone creates a new folder in the common folder, no one can see that new folder (including the person who created it) until I go in and re-sync the pydiods1 datasource. New folders DO appear in /volume/cells/data/pydiods1/ folder on the server itself. Just not in pydio. It FEELS related (hence my explanation here) but it also seems like it could be a separate issue.