Pydio Cells : cannot create folders / empty files nor delete anything

[mhoebeke]

Hello,

After completing the installation of PyDio Cells behind a Proxy, I’m confronted with a new issue. Indeed, most of the functionality seems working except the creation of new directories or files, or their deletion.

File upload, and folder upload are working, and I can also create new cells.

The error messages I get when trying to create a new directory or an empty file (as administrator in my personal files, but I’ve tried in other workspaces as well) . For instance, when trying to create an empty “tf” folder in a freshly created “testcell”, I get the following messages:

{"level":"error","ts":"2018-08-09T02:02:52+02:00","logger":"pydio.rest.meta","msg":"Rest Error 404","SpanRootUuid":"11f563ab-d4eb-aa5e-47d2-18da6420f647","SpanParentUuid":"11f563ab-d4eb-aa5e-47d2-18da6420f647","SpanUuid":"8f665958-9b67-11e8-803b-005056a27e07","RemoteAddress":"192.168.ZZZ.TTT","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36","ContentType":"application/json","HttpProtocol":"HTTP/1.1","UserName":"admin","UserUuid":"809c07ec-9b62-11e8-b97b-005056a27e07","GroupPath":"/","Profile":"admin","Roles":"ROOT_GROUP,809c07ec-9b62-11e8-b97b-005056a27e07","error":"rpc error: code = NotFound desc = {\"id\":\"pydio.grpc.data.index.cells\",\"code\":404,\"detail\":\"Could not retrieve node /admin/testcell/tf%!(EXTRA int=404)\",\"status\":\"Not Found\"}"}

{"level":"error","ts":"2018-08-09T02:02:52+02:00","logger":"pydio.rest.frontend","msg":"message=Error executing \"PutObject\" on \"https://my.pydiocells.com/io/testcell/tf/.pydio\"; AWS HTTP error: Client error response [url] https://my.pydiocells.com/io/testcell/tf/.pydio [status code] 400 [reason phrase] Bad Request Unable to parse error information from response - Error parsing XML: String could not be parsed as XML","SpanRootUuid":"11f563ab-d4eb-aa5e-47d2-18da6420f647","SpanParentUuid":"11f563ab-d4eb-aa5e-47d2-18da6420f647","SpanUuid":"8f6cc334-9b67-11e8-803b-005056a27e07","RemoteAddress":"192.168.ZZZ.TTT","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36","ContentType":"application/json","HttpProtocol":"HTTP/1.1","UserName":"admin","UserUuid":"809c07ec-9b62-11e8-b97b-005056a27e07","GroupPath":"/","Profile":"admin","Roles":"ROOT_GROUP,809c07ec-9b62-11e8-b97b-005056a27e07","FrontIp":"192.168.ZZZ.TTT","UserId":"admin","WorkspaceId":"8843d83e-9b67-11e8-803b-005056a27e07","Source":"StreamWrapper.php l.738","Nodes":[]}

Did anyone encounter a similar issue and managed to solve it ?

Many thanks in advance,

Mark

[bsinou]

Hi,

we need more information to have a clue about your problem:

could you please provide:

• OS and version of Pydio Cells that you use
• The full log when launching the app
• The list of services and their status, you can get them with this command in a terminal:

./cells list

Thanks.

[mhoebeke]

Hello and thanks for looking at my problem.

• I’m running Pydio Cells version 1.0.0 on a CentOS 7.4 host. It might be useful to add that this host is (reverse)proxied by a “gateway” HTTP Apache server, configured as per the Run behind a proxy page of the Pydio Documentation Library. All communications between these host use HTTPS, as does the proxy to handle client requests. As the certificates are self-signed, “Skip SSL verification” is enabled in the Cells configuration. Likewise, the host header for S3 signature has been properly configured, with the internal IP address:port of the host running Cells.

• The full log, including, at the bottom, entries generated by an attempt to create a folder, cand be found here. In this file, IP 192.168.XXX.YYY is the address of the internal host running Cells, and 192.168.ZZZ.TTT is the internal address of the public (reverse)proxy host.

• And the output of the ./cells list command can be found here

Please feel free to ask for any further details.

Thanks again, and all the best

Mark

[bsinou]

Hello,

This shows that you have an issue with your data sources that are not started correctly:

 # datasource
 pydio.grpc.data.index                     [X]
 pydio.grpc.data.index.bibliography        [ ]
 pydio.grpc.data.index.cells               [ ]
 pydio.grpc.data.index.personal            [ ]
 pydio.grpc.data.index.pydiods1            [ ]
 pydio.grpc.data.index.wp1                 [ ]
 pydio.grpc.data.index.wp2                 [ ]
 pydio.grpc.data.index.wp3                 [ ]
 ....
 pydio.grpc.data.sync.wp5                  [ ]
 pydio.grpc.data.sync.wp6                  [ ]
 pydio.grpc.data.sync.wp7                  [X]

These services are responsible for storing the data (see https://pydio.com/en/docs/cells/v1/understanding-datasources ). It is then not surprising that you cannot manage the effective data.

So first thing, could you please try to rather use the latest version of the app (1.0.3), we have already fixed a few bugs since the first release and this might help.

Then if it does not fix your problem, could you please give again the starting log and the output of the same command?

Thanks.

[mhoebeke]

Hello,

I’m afraid the output of the ./cells list command may be misleading, as I ran it right after restarting cells (to generate the startup log file). When running it again after your reply, all data sources are checked. Moreover, I guess I would not have been able to upload files, or even folders, if the data sources were not available.

Nevertheless, I will upgrade cells to 1.0.3 and check if this solves the issue.

Thanks again for your input.

Mark

[mhoebeke]

As a followup to the previous message, I tried to perform the update (either to 1.0.1, or to 1.0.3) using the Cells Web front-end, but it failed, with the following message :

Is it possible to perform the upgrade by simply downloading the new archive and replacing the old binaries with those in the archive ?

Mark

[bsinou]

Yes, replacing the cells binary (you might have to add execution permission and run the setcap command to enable reserved port management) should do the trick.
You only have to empty the frontend php cache that is under .config/pydio/cells/static/pydio/data/cache before restarting the app.

If this is not enough, could you also give a little more info about the tree structure you use for your datasources?
It seems that you put all your root folders (wp1, wp2, wp3…) under the same subfolder, am I right?

[mhoebeke]

Hello,

Following your instructions, cells is now running in version 1.0.3. This did not however solve the REST frontend based file creation / deletion problem.

When configuring Cells, I created one data store per workspace to ease user access management. Each of the data stores is indeed a folder in the same directory on the filesystem. I can upload/download files (even directories) using the Cells Web UI without problems. And versioning also seems to work correctly. So I was thinking that the directory layout for the data stores was OK. Oh, and I didn’t mention it yet because I’m not sure it’s relevant, but I can (as a simple user) create new cells through the Web UI. But, as with workspaces, I cannot create / delete files in these cells through the Web UI.

Mark

[bsinou]

Hello,

Following your instructions, cells is now running in version 1.0.3. This did not however solve the REST frontend based file creation / deletion problem.

OK, but it has solved other pbs, among others, now the list command is reliable. Can you please run it again and double check that all services are correctly started?

Can you please also confirm that the error is still the same when you try to create / delete via the Web UI?

And last question, how many folders for datasources are they under the main folder?

E.g: if you have such a structure:

Parent_Folder 
|--- subFolderForDS1
|--- subFolderForDS2
'--- subFolderForDS3   

Under the hood, they share the same minio server that is mutualised for the above DS1, DS2 and DS3. They might be an issue we did not see when the minio server handle too many datasources.

(One trail you might explore as a workaround, until we find and solve this issue, might then be to add folder level and split the DSs in smaller “pack”)

[mhoebeke]

You will find the output of the list command here

And, I checked that the error is still the same. You will find the log trace that an attempt to create an empty file generated here

At the filesystem level, in the main Cells directory, there are 10 subdirectories. Each of these has been configured as a Cells data store.

If I interpret your suggestion right, I could try to create let’s say, 5 subdirectories in the topmost directory and recreate my data sources two by two in each subdirectory. Or better yet, it would maybe suffice to create a brand new data store located elsewhere on the filesystem, to configure a workspace to use it and to try if I can perform file/folder creation/deletion in this new workspace.

Mark

[mhoebeke]

As a followup : I just finished creating a new data store (outside the folder structure of the existing data stores) and associating it to a new workspace. When trying to create a new folder in this workspace, the same error occurs.

Mark

[charles]

hi guys, stepping in the discussion.
So basically you cannot create any folder or empty file, whatever the datasource, but uploading files work.
The 404 errors you see in the log may in fact be “normal”, as the frontend checks if the resources exist before creating them… Can you check also the php fpm error log ? If the proxy setup is messing something up, maybe it’s in the requests posted by the php to the backend…
Sorry I don’t have more ideas right now, trying to give new leads to follow

Charles

[mhoebeke]

Upon your request, Charles, I checked the php-fpm error log which is empty (apart from an error resulting from the log being re-opened after log rotation).

Mark

[Ximo]

These last 2 days I had a very similar issue as described above, all requests for file died with a timeout - this afternoon I changed the internal bind IP from 192.168…:8080 to localhost:8080.
Now all the basic functionality is there. (I am not using SSL while having both servers on the same machine).
I do not have an explanation, but it should perhaps be added in the documentation for apache prox:

• do not use the “internal IP” if it is just the same Ubuntu server, use the loopback IP instead
• the REST Endpoints must be set to exernal IP

Now I am stuck with webdav, I can login, but then it says Method Not Allowed …

[mhoebeke]

Hello Ximo,

The issue may be similar, bot not quite enough to be solved using the same recipe. More specifically, the errors are not timeouts and the proxy server is distinct from the server hosting cells. So putting localhost in the proxy configuration is not really an option.

Thank you anyway for trying to give me some clues regerding this persisting problem.

Cheers,

Mark

[bsinou]

Hello Mark,

Could you please start your instance in debug mode and report the full log until the error happens so that we can have more insights about what is going on?

Thanks

[mhoebeke]

Hello Bruno,

Here is the link to the complete log : from cells startup to the attempt to create a new (empty) file, to the shutdown.

Don’t hesitate to ask for additionnal information if needed.

Mark

[mhoebeke]

Hi,

I’m still banging my head on this one. I managed to enable debugging for the ApiClient (pydio/cells/static/pydio/core/src/pydio/Swagger) to try to follow the request<->response trail when trying to create an empty file.

What strikes me is that I didn’t find any POST request using the /a/meta/set REST API call.

When trying to create file myemptyfilename in workspace myworkspacename linked to datastore mydatastore, there are (successfull POST) calls to a/workspace and a/acl followed by a call to

a/meta/get/myworkspacename%2Fmyemptyfilename

This call returns with a 404 status code and a plain text error message in the response body:

rpc error: code = NotFound desc = {“id”:“pydio.grpc.data.index.mydatastore”,“code”:404,“detail”:“Could not retrieve node /myemptyfilename%!(EXTRA int=404)”,“status”:“Not Found”}

From what I gather, this is then bounced back to the the client with the previously discussed error:

{“Level”:“ERROR”,“Ip”:“192.168.XXX.YYY”,“UserId”:“admin”,“WorkspaceId”:“23537802740ab49ab197c530cd0df24b”,“Source”:“StreamWrapper.php l.738”,“Message”:“message=Error executing "PutObject" on "https://my.cells.instance.com/io/myworkspacename\myemptyfilename"; AWS HTTP error: Client error response [url] https://my.cells.instance.com/io/myworkspacename/myemptyfilename [status code] 400 [reason phrase] Bad Request Unable to parse error information from response - Error parsing XML: String could not be parsed as XML”,“Nodes”:}"

Obviously, I’m neither a PHP nor a Cells guru so I surely missed something. If any expert could acknowledge that this is the normal behaviour, I could start exploring alternatives.

Thanks for any feedback.

Mark

[bsinou]

I’m still banging my head on this one.

Ouch. Sorry to hear that… Unfortunately, I am not a PHP guru either, lately I’ve been rather working on the next version that is plain go / javascript. So I fear you might have to wait until next week for Charles to have a look at it.
Thanks for your understanding.

[charles]

hi @mhoebeke
i had a look at the log
I see some strange thing starting at

2018-08-20T16:29:06.894+0200	e[35mDEBUGe[0m	JavaScript Runner	{"time": "315.729µs"}
2018-08-20T16:29:06.894+0200	e[35mDEBUGe[0m	e[31mpydio.grpc.treee[0m	ReadNode	{"time": "5.893µs", "req": "Node:<Path:\"undefined/admin\" Type:COLLECTION MetaStore:<key:\"pydio:meta-data-source-name\" value:\"\\\"undefined\\\"\" > MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"admin\\\"\" > > ", "resp": ""}
2018-08-20T16:29:06.895+0200	e[35mDEBUGe[0m	e[31mpydio.grpc.treee[0m	Create Node	{"UUID": "", "Path": "undefined/admin"}
2018-08-20T16:29:06.895+0200	e[35mDEBUGe[0m	e[31mpydio.grpc.treee[0m	CreateNode	{"time": "4.083µs", "req": "Node:<Path:\"undefined/admin\" Type:COLLECTION MetaStore:<key:\"pydio:meta-data-source-name\" value:\"\\\"undefined\\\"\" > MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"admin\\\"\" > > ", "resp": ""} 

I guess in the sample you tried to create a file on the “personal-files” datasource, which is based on a template path. Can you produce a log when creating a file in a more standard workspace/datasource (simply pointing to a folder) ?
Also maybe you can PM a direct access to your test server for debugging ?

Charles