Pydio Cells : cannot create folders / empty files nor delete anything

Ximo · August 14, 2018, 3:13pm

These last 2 days I had a very similar issue as described above, all requests for file died with a timeout - this afternoon I changed the internal bind IP from 192.168…:8080 to localhost:8080.
Now all the basic functionality is there. (I am not using SSL while having both servers on the same machine).
I do not have an explanation, but it should perhaps be added in the documentation for apache prox:

do not use the “internal IP” if it is just the same Ubuntu server, use the loopback IP instead
the REST Endpoints must be set to exernal IP

Now I am stuck with webdav, I can login, but then it says Method Not Allowed …

mhoebeke · August 20, 2018, 7:54am

Hello Ximo,

The issue may be similar, bot not quite enough to be solved using the same recipe. More specifically, the errors are not timeouts and the proxy server is distinct from the server hosting cells. So putting localhost in the proxy configuration is not really an option.

Thank you anyway for trying to give me some clues regerding this persisting problem.

Cheers,

Mark

bsinou · August 20, 2018, 12:56pm

Hello Mark,

Could you please start your instance in debug mode and report the full log until the error happens so that we can have more insights about what is going on?

Thanks

mhoebeke · August 20, 2018, 2:35pm

Hello Bruno,

Here is the link to the complete log : from cells startup to the attempt to create a new (empty) file, to the shutdown.

Don’t hesitate to ask for additionnal information if needed.

Mark

mhoebeke · August 23, 2018, 2:10pm

Hi,

I’m still banging my head on this one. I managed to enable debugging for the ApiClient (pydio/cells/static/pydio/core/src/pydio/Swagger) to try to follow the request<->response trail when trying to create an empty file.

What strikes me is that I didn’t find any POST request using the /a/meta/set REST API call.

When trying to create file myemptyfilename in workspace myworkspacename linked to datastore mydatastore, there are (successfull POST) calls to a/workspace and a/acl followed by a call to

a/meta/get/myworkspacename%2Fmyemptyfilename

This call returns with a 404 status code and a plain text error message in the response body:

rpc error: code = NotFound desc = {“id”:“pydio.grpc.data.index.mydatastore”,“code”:404,“detail”:“Could not retrieve node /myemptyfilename%!(EXTRA int=404)”,“status”:“Not Found”}

From what I gather, this is then bounced back to the the client with the previously discussed error:

{“Level”:“ERROR”,“Ip”:“192.168.XXX.YYY”,“UserId”:“admin”,“WorkspaceId”:“23537802740ab49ab197c530cd0df24b”,“Source”:“StreamWrapper.php l.738”,“Message”:“message=Error executing “PutObject” on “https://my.cells.instance.com/io/myworkspacename\myemptyfilename”; AWS HTTP error: Client error response [url] https://my.cells.instance.com/io/myworkspacename/myemptyfilename [status code] 400 [reason phrase] Bad Request Unable to parse error information from response - Error parsing XML: String could not be parsed as XML”,“Nodes”:[]}"

Obviously, I’m neither a PHP nor a Cells guru so I surely missed something. If any expert could acknowledge that this is the normal behaviour, I could start exploring alternatives.

Thanks for any feedback.

Mark

bsinou · August 23, 2018, 3:44pm

I’m still banging my head on this one.

Ouch. Sorry to hear that… Unfortunately, I am not a PHP guru either, lately I’ve been rather working on the next version that is plain go / javascript. So I fear you might have to wait until next week for Charles to have a look at it.
Thanks for your understanding.

charles · August 29, 2018, 7:13am

hi @mhoebeke
i had a look at the log
I see some strange thing starting at

2018-08-20T16:29:06.894+0200	e[35mDEBUGe[0m	JavaScript Runner	{"time": "315.729µs"}
2018-08-20T16:29:06.894+0200	e[35mDEBUGe[0m	e[31mpydio.grpc.treee[0m	ReadNode	{"time": "5.893µs", "req": "Node:<Path:\"undefined/admin\" Type:COLLECTION MetaStore:<key:\"pydio:meta-data-source-name\" value:\"\\\"undefined\\\"\" > MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"admin\\\"\" > > ", "resp": ""}
2018-08-20T16:29:06.895+0200	e[35mDEBUGe[0m	e[31mpydio.grpc.treee[0m	Create Node	{"UUID": "", "Path": "undefined/admin"}
2018-08-20T16:29:06.895+0200	e[35mDEBUGe[0m	e[31mpydio.grpc.treee[0m	CreateNode	{"time": "4.083µs", "req": "Node:<Path:\"undefined/admin\" Type:COLLECTION MetaStore:<key:\"pydio:meta-data-source-name\" value:\"\\\"undefined\\\"\" > MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"admin\\\"\" > > ", "resp": ""}

I guess in the sample you tried to create a file on the “personal-files” datasource, which is based on a template path. Can you produce a log when creating a file in a more standard workspace/datasource (simply pointing to a folder) ?
Also maybe you can PM a direct access to your test server for debugging ?

Charles

mhoebeke · August 30, 2018, 12:25pm

Hello @charles

Thanks for looking into this again. Below, you’ll find the log messages generated when trying to create an empty file (EmptyTestFile) in an “ordinary” workspace (test-workspace) associated to a datastore linked to a local folder:

{"level":"error","ts":"2018-08-30T14:17:21+02:00","logger":"pydio.rest.frontend","msg":"message=Error executing \"PutObject\" on \"https://my.cells.instance.com/io/test-workspace/EmptyTestFile\"; AWS HTTP error: Client error response [url] https://my.cells.instance.com/io/test-workspace/EmptyTestFile [status code] 400 [reason phrase] Bad Request Unable to parse error information from response - Error parsing XML: String could not be parsed as XML","SpanRootUuid":"306f11af-f81b-5511-648a-e42a21d7eed3","SpanParentUuid":"306f11af-f81b-5511-648a-e42a21d7eed3","SpanUuid":"a54cb7af-ac4e-11e8-87ec-005056a27e07","RemoteAddress":"192.168.XXX.YYY","UserAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36","ContentType":"application/json","HttpProtocol":"HTTP/1.1","UserName":"admin","UserUuid":"809c07ec-9b62-11e8-b97b-005056a27e07","GroupPath":"/","Profile":"admin","Roles":"ROOT_GROUP,809c07ec-9b62-11e8-b97b-005056a27e07","FrontIp":"192.168.XXX.YYY","UserId":"admin","WorkspaceId":"23537802740ab49ab197c530cd0df24b","Source":"StreamWrapper.php l.738","Nodes":[]}
{"level":"error","ts":"2018-08-30T14:17:21+02:00","logger":"pydio.rest.frontend","msg":"message=[404] Error connecting to the API (https://my.cells.instance.com/a/meta/get/test-workspace%2FEmptyTestFile)","SpanRootUuid":"306f11af-f81b-5511-648a-e42a21d7eed3","SpanParentUuid":"306f11af-f81b-5511-648a-e42a21d7eed3","SpanUuid":"a55ec88f-ac4e-11e8-87ec-005056a27e07","RemoteAddress":"192.168.XXX.YYY","UserAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36","ContentType":"application/json","HttpProtocol":"HTTP/1.1","UserName":"admin","UserUuid":"809c07ec-9b62-11e8-b97b-005056a27e07","GroupPath":"/","Profile":"admin","Roles":"ROOT_GROUP,809c07ec-9b62-11e8-b97b-005056a27e07","FrontIp":"192.168.XXX.YYY","UserId":"admin","WorkspaceId":"23537802740ab49ab197c530cd0df24b","Source":"ApiClient.php l.295","Nodes":[]}
{"level":"error","ts":"2018-08-30T14:17:21+02:00","logger":"pydio.rest.frontend","msg":"message=Only variables should be assigned by reference","SpanRootUuid":"306f11af-f81b-5511-648a-e42a21d7eed3","SpanParentUuid":"306f11af-f81b-5511-648a-e42a21d7eed3","SpanUuid":"a5629bff-ac4e-11e8-87ec-005056a27e07","RemoteAddress":"192.168.XXX.YYY","UserAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36","ContentType":"application/json","HttpProtocol":"HTTP/1.1","UserName":"admin","UserUuid":"809c07ec-9b62-11e8-b97b-005056a27e07","GroupPath":"/","Profile":"admin","Roles":"ROOT_GROUP,809c07ec-9b62-11e8-b97b-005056a27e07","FrontIp":"192.168.XXX.YYY","UserId":"admin","WorkspaceId":"23537802740ab49ab197c530cd0df24b","Source":"SapiMiddleware.php l.118","Nodes":[]}
{"level":"error","ts":"2018-08-30T14:17:21+02:00","logger":"pydio.rest.frontend","msg":"message=ob_end_flush(): failed to delete and flush buffer. No buffer to delete or flush","SpanRootUuid":"306f11af-f81b-5511-648a-e42a21d7eed3","SpanParentUuid":"306f11af-f81b-5511-648a-e42a21d7eed3","SpanUuid":"a5676855-ac4e-11e8-87ec-005056a27e07","RemoteAddress":"192.168.XXX.YYY","UserAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36","ContentType":"application/json","HttpProtocol":"HTTP/1.1","UserName":"admin","UserUuid":"809c07ec-9b62-11e8-b97b-005056a27e07","GroupPath":"/","Profile":"admin","Roles":"ROOT_GROUP,809c07ec-9b62-11e8-b97b-005056a27e07","FrontIp":"192.168.XXX.YYY","UserId":"admin","WorkspaceId":"23537802740ab49ab197c530cd0df24b","Source":"ShutdownScheduler.php l.157","Nodes":[]}

I’ll PM you shortly regarding access to the Cells instance.

Cheers,

Mark

charles · August 30, 2018, 1:21pm

mmm, the first line seems already more interesting. And the following ones are strange. What PHP version are you using exactly?

mhoebeke · August 30, 2018, 1:43pm

PHP 7.1.8 according to the PHP configuration as displayed by the Cells settings dashboard (I’ll increase the max upload size once the instance is stabilized) :

Client : Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
Crypto Extension Detected : OpenSSL
DOM Enabled : Yes
GD Enabled : Yes
Upload Max Size : 2M
Memory Limit : 128M
Max execution time : 30
Safe Mode : 0
Safe Mode GID : 0
Xml parser enabled : 1
Server OS : Linux
Session Save Path : /var/opt/rh/rh-php71/lib/php/session
Session Save Path Writeable : Yes
PHP Version : 7.1.8
Locale : fr_FR.UTF-8
Directory Separator : /
PHP Opcode Cache extension loaded : Yes
PHP INTL extension loaded : Yes
PHP Output Buffer disabled : No
PHP File Uploads enabled : Yes
Magic quotes disabled : Yes
Upload Tmp Dir Writeable : Yes
PHP Upload Max Size : 2097152
PHP Post Max Size : 8388608
Users enabled : Yes
Guest enabled : No
Writeable Folders : [<b>cache</b>:true,<br> <b>data</b>:true]
Zlib Enabled : Yes

charles · August 30, 2018, 3:13pm

Still no luck with that. And there is nothing more in the log, just BEFORE the first line you sent?
Are you behind a proxy ? is the cells main URL the same as the one you access from frontend (https://my.cells.instance.com) ?
Charles

charles · August 30, 2018, 3:19pm

wait, i’m trying to read your thread from scratch … Currently you still can upload a file ?

mhoebeke · August 30, 2018, 3:31pm

Yes. I can even successfully upload directories.

charles · August 30, 2018, 3:33pm

and very small files as well (<1KB) ? Creating empty file is in fact creating a file with just one character in it (a space) as object store does not support really empty files. I wonder if this character is trimmed at some point

mhoebeke · August 30, 2018, 3:42pm

FYI: I just successfully uploaded à 20 byte file.

charles · August 31, 2018, 7:30am

I’m sorry right now i’m running out of ideas.
You have to give us a bit more time to try to reproduce that (or if you can maybe build a similar set up on a staging env that we could access ?)…
Just to re-summarize your case, had you previously tested WITHOUT the Reverse Proxy? I’m still in doubt whether this is linked or not.
Charles

mhoebeke · September 3, 2018, 6:07am

Hello Charles,

I did not test without the Reverse Proxy. But I agree with you that this is a prime suspect. If I can find another suitable machine, I’ll install a test intance of Cells without proxy. But I’m afraid this won’t happen until a couple of days.

FWIW : I created a one-character file containing only a space character. I could upload it without any problem into one of my workspaces.

Cheers,

Mark

mhoebeke · September 12, 2018, 7:54am

Hello @charles,

I wanted to share some (interesting?) new info about this pending bug.

In the Configuration Panel, all the endpoints are defined using the public address of the cells instance as base URL (i.e https://my-public.pydio.com/io for the S3 endpoint). As the bug seems S3 related, I changed this endpoint configuration to use the internal IP address. The S3 Endpoint then becomes something like :

https://192.168.XXX.YYY:7070/io

An lo and behold, this allows me to create files and directories (although deleting stuff still doesn’t work).
But : with this configuration, file upload stops working. Indeed, it looks as if the browser tries to access the S3 Rest endpoint directly using the OPTIONS HTTP method.

What I get from all this is that the S3 endpoint is used in two different ways;

When creating empty files or directories: wrapped in a POST call made by index.php (for which the non-proxied IP is OK, because the PHP is parsed on the server)
When uploading files or directories: through calls made directly by the browser (for which the public S3 endpoint is mandatory)

Is the remaining question then: why does the call wrapped in the index.php POST fail when using the public S3 endpoint?

As usual, any insights on this problem are highly appreciated.

Cheers,

Mark

martinw · September 3, 2019, 5:04am

Any progress with this S3 bug? I can use the Upload option to upload files and folders, but when I click on New Folder or File, I get the Access Denied error. Thanks. Martin

charles · September 9, 2019, 4:14pm

@martinw, please do not “wake up” old threads, they are hard to read…
Can you post a dedicated thread explaining your issue, your setup, a way to reproduce, etc?
Thanks a lot in advance!