Multipart upload failed - still shows file in interface

Pydio Cells
1

I’m uploading a large file which is split into multiple parts for upload, however these parts are timing out, I can see this in the Cells logs, and in the upload window as the progress bar restarting. If I then cancel the upload (or close the window) the file still appears to show in the Cells interface, even though it failed to upload, and I can see it does not exist in the file system.

There is nothing specifically reported to the user to say that something has failed.

I can see in the browser console log on cancelling that the DELETE request on the file fails with a 403 (I assume because the file doesn’t exist), but it is still listed in Cells, until there is a filesystem sync, or the file is attempted to be moved, when it just disappears.

2

I have increased the timeouts and reduced the part size to work around the problem. But the issue of if there is a timeout you get a ghost file still remains.

3

Do you mean the uncompleted file is listed in workspace? or just in the list of upload file in “upload” pop-up? Because it’s not possible for uncompleted upload file to be in the Cells. Or you should have other problem. Please post in detail your server OS name, web browser and the error message or screen captures.

4

yes, it appears in the workspace with the file size of the original file. If I look at the filesystem on the server for that workspace the file does not exist. If I try to download the file from the workspace I get an invalid download (it creates a file with that name of 0 bytes).

The original issue was reported to me by another user, but I could reproduce it. To do so:
Multipart upload settings on default:
threshold: 100mb
part size: 50mb
queue size: 3
timeout: 3

I then throttled my connection with devtools to 1mb/s, and tried to upload a 300mb file. The multipart uploads timed out after 3 mins, and were retried multiple times. I then cancelled the upload and reloaded the webpage in my browser. The file was then listed in the workspace.

When I cancelled the upload I saw in devtools a delete request that failed.

Server OS: Ubuntu 16.04
Browser: Chrome 77
example error messages (from server logs):

{"level":"error","ts":"2019-09-27T10:09:53+01:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"6a55ee81-e106-11e9-8411-00163ea366c6","RemoteAddress":"127.0.0.1","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"adurant","UserUuid":"cc3c19be-689b-4df7-a26f-484582f4ae15","GroupPath":"/company/","Profile":"admin","Roles":"ROOT_GROUP,cbaf0a8b-7869-4af6-b16f-a9375ce92ea6,ADMINS,ADMINS,cc3c19be-689b-4df7-a26f-484582f4ae15","error":"Put http://10.12.110.10:38285/personal/adurant/virtio-win-0.1.141.iso?partNumber=1&uploadId=8383aaf9-944e-49ce-8d7a-4bacd864903e: context canceled"}
{"level":"error","ts":"2019-09-27T10:09:53+01:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"6a55ee81-e106-11e9-8411-00163ea366c6","RemoteAddress":"127.0.0.1","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"adurant","UserUuid":"cc3c19be-689b-4df7-a26f-484582f4ae15","GroupPath":"/company/","Profile":"admin","Roles":"ROOT_GROUP,cbaf0a8b-7869-4af6-b16f-a9375ce92ea6,ADMINS,ADMINS,cc3c19be-689b-4df7-a26f-484582f4ae15","error":"Put http://10.12.110.10:38285/personal/adurant/virtio-win-0.1.141.iso?partNumber=3&uploadId=8383aaf9-944e-49ce-8d7a-4bacd864903e: context canceled"}
{"level":"error","ts":"2019-09-27T10:09:53+01:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"6a55ee81-e106-11e9-8411-00163ea366c6","RemoteAddress":"127.0.0.1","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"adurant","UserUuid":"cc3c19be-689b-4df7-a26f-484582f4ae15","GroupPath":"/company/","Profile":"admin","Roles":"ROOT_GROUP,cbaf0a8b-7869-4af6-b16f-a9375ce92ea6,ADMINS,ADMINS,cc3c19be-689b-4df7-a26f-484582f4ae15","error":"Put http://10.12.110.10:38285/personal/adurant/virtio-win-0.1.141.iso?partNumber=2&uploadId=8383aaf9-944e-49ce-8d7a-4bacd864903e: context canceled"}
{"level":"info","ts":"2019-09-27T10:16:15+01:00","logger":"pydio.gateway.data","msg":"Received MultipartAbort - Clean temporary node:","SpanUuid":"74ff0c5b-e107-11e9-8411-00163ea366c6","RemoteAddress":"127.0.0.1","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","ContentType":"application/octet-stream; charset=UTF-8","HttpProtocol":"HTTP/1.1","UserName":"adurant","UserUuid":"cc3c19be-689b-4df7-a26f-484582f4ae15","GroupPath":"/company/","Profile":"admin","Roles":"ROOT_GROUP,cbaf0a8b-7869-4af6-b16f-a9375ce92ea6,ADMINS,ADMINS,cc3c19be-689b-4df7-a26f-484582f4ae15","Node":{"Uuid":"257491e0-413a-4815-be10-e3e810dc97f8","Path":"personal/adurant/virtio-win-0.1.141.iso","Etag":"temporary","MTime":"2019-09-27T10:15:37+01:00","Size":316628992,"MetaStore":{"name":"\"virtio-win-0.1.141.iso\"","pydio:meta-data-source-name":"\"personal\"","pydio:meta-data-source-path":"\"adurant/virtio-win-0.1.141.iso\""}}}
{"level":"error","ts":"2019-09-27T10:09:53+01:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"5e8e51b1-e107-11e9-8411-00163ea366c6","RemoteAddress":"127.0.0.1","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"adurant","UserUuid":"cc3c19be-689b-4df7-a26f-484582f4ae15","GroupPath":"/company/","Profile":"admin","Roles":"ROOT_GROUP,cbaf0a8b-7869-4af6-b16f-a9375ce92ea6,ADMINS,ADMINS,cc3c19be-689b-4df7-a26f-484582f4ae15","error":"Put http://10.12.110.10:38285/personal/adurant/virtio-win-0.1.141.iso?partNumber=1&uploadId=4fe40551-d436-4570-8c97-463336b9d741: context canceled"}
{"level":"error","ts":"2019-09-27T10:09:53+01:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"5e8e51b1-e107-11e9-8411-00163ea366c6","RemoteAddress":"127.0.0.1","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"adurant","UserUuid":"cc3c19be-689b-4df7-a26f-484582f4ae15","GroupPath":"/company/","Profile":"admin","Roles":"ROOT_GROUP,cbaf0a8b-7869-4af6-b16f-a9375ce92ea6,ADMINS,ADMINS,cc3c19be-689b-4df7-a26f-484582f4ae15","error":"Put http://10.12.110.10:38285/personal/adurant/virtio-win-0.1.141.iso?partNumber=2&uploadId=4fe40551-d436-4570-8c97-463336b9d741: context canceled"}
{"level":"error","ts":"2019-09-27T10:09:53+01:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"5e8e51b1-e107-11e9-8411-00163ea366c6","RemoteAddress":"127.0.0.1","UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"adurant","UserUuid":"cc3c19be-689b-4df7-a26f-484582f4ae15","GroupPath":"/company/","Profile":"admin","Roles":"ROOT_GROUP,cbaf0a8b-7869-4af6-b16f-a9375ce92ea6,ADMINS,ADMINS,cc3c19be-689b-4df7-a26f-484582f4ae15","error":"Put http://10.12.110.10:38285/personal/adurant/virtio-win-0.1.141.iso?partNumber=3&uploadId=4fe40551-d436-4570-8c97-463336b9d741: context canceled"}
5

This exact issue has also happened to me too with version 2.1.5.

6

We are running into exactly the same issue on a newly installed version 2.2.8 system.

Users were reporting that they could not download a file shown in the interface. After investigation I found that the file was visible in the user facing interface, but clicking the download button would not do anything. This affected multiple files, other files worked just fine.

I then checked the filesystem on the server and saw that the affected files were not there at all. Not even 0 byte files.

I ran the “Sync DataSource pydios1” scheduler task, which then remove the false entries from the frontend.

7

Thanks for reporting, we will have a look at this while testing the next version.