Error while listing on creation or resync of a DataSource


#1

Hello there,

after quite some big struggle to make pydio cells “working”, I stumbled on a new issue I cannot resolve by myself.
My env: LXC debian stretch witch mariadb 10.3 and cells Home edition 1.2.1

I had some troube for creating datasource at first because parent folder cannot contain any sub-mount / cross-device mount (folders I want to use as datasource are mount from the host).
Here is the path of the datasource: /mnt/home/mydatasource
But now that my datasource (named mydatasource) is created with “all services running”, I can see in the log the following:

2018-11-07T10:09:10.094Z INFO pydio.grpc.data.objects.local2 started
2018-11-07T10:09:10.101Z INFO pydio.grpc.data.objects.local2 Starting objects service local2
2018-11-07T10:09:10.333Z INFO pydio.grpc.data.index.mydatasource 2018-11-07T10:09:10.329Z INFO Creating root node in index
2018-11-07T10:09:10.350Z INFO pydio.grpc.data.index.mydatasource started
::1 - - [07/Nov/2018:10:09:12 +0000] “GET /config/ctl HTTP/1.1” 200 3120 “https://mydomain.com/settings/data/datasources” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0”
2018-11-07T10:09:14.063Z INFO pydio.grpc.data.sync.mydatasource Create job to start datasource resync
2018-11-07T10:09:14.098Z INFO pydio.grpc.tasks Run Job resync-ds-mydatasource on timer event
2018-11-07T10:09:14.228Z INFO pydio.grpc.data.sync.mydatasource Listing all S3 objects for path {“bucket”: “mydatasource”, “path”: “”}
2018-11-07T10:09:14.253Z ERROR pydio.grpc.data.sync.mydatasource Error while listing {“error”: “Access Denied.”}
2018-11-07T10:09:14.283Z ERROR pydio.grpc.data.objects.local2 time=“2018-11-07T10:09:14Z” level=error msg=“Unable to list objects.” cause=“Prefix access is denied: mydatasource/” source="[bucket-handlers-listobjects.go:118:github.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.objectAPIHandlers.ListObjectsV2Handler()]"
2018-11-07T10:09:14.299Z ERROR pydio.grpc.tasks cannot run action actions.cmd.resync: {“id”:"",“code”:0,“detail”:“Access Denied.”,“status”:""}

the result is that if I create a workspace pointing on this datasource only, the workspace appears empty (quite obvious as it didn’t succeed to create its index).
on the parent folder pydio is owner of the directory and has write access (it succesfully created directory .minio.sys, and not .pydio as advertised on the forum)
all folder on the share are world writable for testing purpose so pydio process have access to it.

another precisions for debuging purpose, if I create another datasource on a folder pointing inside the “mydatasource” directory, it is now working but that cannot be a solution as it forced all folder to respect S3 naming…
Please note that the mount from the host is NOT mydatasource, but it is home, so mydatasource is already inside the mount point (anyway, that was forced by design for minio as it refuses on creation due to cross-device mount).

Edit:
Another test I just made was to bind on device level my hardrive from the host to the lxc, so the mount inside my lxc is done as for a regular mount on the host. No changes at all, same issue.

thanks a lot for any help, I’m on this one since quite sometime, and it seems I’m closed to have it working now.

---- OFF TOPIC
PS: FYI, for now pydio server is far easier to install because even if cells should be easier by design, it introduces a lot of new restrictions (cross mounting, small font on directory, 2 level path, proxy not well supported (some configurations work some don’t), nfs share seems not supported…) and CPU/RAM consumption seems a lot higher which is quite surprising.
Don’t take it the wrong way, I’m struggling with it because it is a great piece of software, I’m a long time user, I’ve quite often tried to replace pydio (for performance reason or because it is a bit too much for my use case) and never could, there are a lot of other solution but they don’t do all that pydio can!
thanks again


Pydio Cells Storage on another Server
Pydio Cells Storage on another Server
#2

Hi,
exactly when you create a datasource the path should atleast be 2 levels down /a/b/"then your folder".

I’m not familiar with lxc but i’m going to try to reproduce your issue.


#3

Hello,

thanks for checking my issue :slight_smile:

as stated my datasource is /mnt/home/mydatasourc, so it is indeed 2 levels down.

I’ve tried many things, I even get past the first few errors as explained, but now I’m stuck.

I will gladly help you reproduce my issue in order to find a solution!

In parallel, I’m setting up a minio server to see if the combination works better, but it feels like a useless additional layer.

thanks again


#4

The mounted volume could be one of the issues, but will try to reproduce your exact case to atleast understand what’s happening.


#5

Hi,
i’m still experimenting with LXC, i did not have a lot of experience with it but i’ll get to it.