NoSuchKey: Cells silently mis-uploaded files on S3 storage

Pydio Cells
#1

I’m experiencing a discrepancy of storage state and UI leading to 404 for files misuploaded (not actually uploaded) but shown are “ok” in the UI.

I’m using cells 3.0.6 (on a good hardware)

I’ve many files ranging from 30MB to 60GB in a workspace hosted onto S3 (unstructured object storage without versioning using native ETags and no encryption)

While all of them appear in the UI after upload, the download fails for many files (dozens), in particular many [but not all] of those over 2 GB (user spent a lot of time uploading them!).

This is the error returned to the client when trying to download them. It’s a 404 (no download happens) :

<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/io/common-files/IN/2021/foo bar baz.mov</Resource><RequestId>16FA7B5E7E6140F2</RequestId><HostId>3L137</HostId></Error>

It seems to come from S3. I then tried to figure out what’s actually within S3 and it seems there is a HUGE discrepancy between what has actually been uploaded and what Pydio BELIEVES it manages.

Pydio

Screenshot from 2022-06-20 21-48-21
Screenshot from 2022-06-20 21-48-21739×444 36 KB

S3

Screenshot from 2022-06-20 21-48-09
Screenshot from 2022-06-20 21-48-091267×299 82.6 KB

Database

Screenshot from 2022-06-20 22-00-20

I got into the log files regarding one of these files:

.config/pydio/cells/logs/caddy_errors.log 

17/Jun/2022:21:43:27 +0200 [ERROR 502 /io/common-files/Foo bar.mov] context canceled
17/Jun/2022:21:43:40 +0200 [ERROR 502 /io/common-files/Foo bar.mov] context canceled
17/Jun/2022:22:03:36 +0200 [ERROR 502 /io/common-files/Foo bar.mov] context canceled
17/Jun/2022:22:03:46 +0200 [ERROR 502 /io/common-files/Foo bar.mov] context canceled

.config/pydio/cells/logs/caddy_logs.log 

IP - - [17/Jun/2022:21:34:07 +0200] "POST /io/common-files/Foo%20bar.mov?uploads HTTP/1.1" 200 431
IP - - [17/Jun/2022:21:35:02 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=2&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:35:28 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=1&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:36:06 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=3&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:36:31 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=4&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:37:34 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=6&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:37:48 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=5&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:38:38 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=7&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:39:09 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=8&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:39:50 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=10&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:39:59 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=9&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:41:03 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=11&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:41:30 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=12&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:41:53 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=13&uploadId=UPLOAD_ID1 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:42:33 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=16&uploadId=UPLOAD_ID1 HTTP/1.1" 403 284
IP - - [17/Jun/2022:21:42:36 +0200] "DELETE /io/common-files/Foo%20bar.mov?uploadId=UPLOAD_ID1 HTTP/1.1" 403 284
IP - - [17/Jun/2022:21:42:37 +0200] "POST /io/common-files/Foo%20bar.mov?uploads HTTP/1.1" 200 431
IP - - [17/Jun/2022:21:43:27 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=14&uploadId=UPLOAD_ID1 HTTP/1.1" 502 16
IP - - [17/Jun/2022:21:43:40 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=15&uploadId=UPLOAD_ID1 HTTP/1.1" 502 16
IP - - [17/Jun/2022:21:44:13 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=2&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:44:15 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=3&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:44:19 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=1&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:45:51 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=6&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:46:00 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=4&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:46:09 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=5&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:47:17 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=7&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:48:01 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=9&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:48:12 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=8&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:49:30 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=11&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:49:36 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=10&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:49:50 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=12&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:51:16 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=14&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:51:32 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=13&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:51:52 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=15&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:53:11 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=16&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:53:32 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=17&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:54:06 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=18&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:55:01 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=19&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:55:29 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=20&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:56:16 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=21&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:56:58 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=22&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:57:16 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=23&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:57:54 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=24&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:58:55 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=25&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:59:27 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=26&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:21:59:43 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=27&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:22:00:38 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=28&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:22:01:19 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=29&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:22:01:54 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=30&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:22:02:19 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=31&uploadId=UPLOAD_ID2 HTTP/1.1" 200 0
IP - - [17/Jun/2022:22:02:40 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=34&uploadId=UPLOAD_ID2 HTTP/1.1" 403 284
IP - - [17/Jun/2022:22:02:41 +0200] "DELETE /io/common-files/Foo%20bar.mov?uploadId=UPLOAD_ID2 HTTP/1.1" 403 284
IP - - [17/Jun/2022:22:03:36 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=32&uploadId=UPLOAD_ID2 HTTP/1.1" 502 16
IP - - [17/Jun/2022:22:03:46 +0200] "PUT /io/common-files/Foo%20bar.mov?partNumber=33&uploadId=UPLOAD_ID2 HTTP/1.1" 502 16

Rewinding the upload

Apparently the user tried twice to upload this file. In both case, a chunk failed (403 on chunk #16 the first time and #34 the second time).
In both case this triggered immediately a DELETE which… led to another 403.
(And the remaining ongoing chunks triggered the context canceled… By the way,
this message also happen for mere assets like PydioActivityStreams.min.js)

Questions

I’ve multiple questions, deserved, I believe, by the severity of the issue:

  • How is possible that such a buggy upload behavior is still present nowadays in a v3.x?

  • Why this 403 for a chunk? Why no error-handling? Why no debug information regarding Pydio <-> S3 network/transactions? (It’s been a long requested feature to help debugging this class of issues and subsequently improve Pydio)

  • Why this chunk wasn’t retried? Why a DELETE?

  • How is it possible that Pydio missed so much the actual S3 storage and display blatantly wrong information while it was actually aware the upload failed in the first place?

  • If it’s possible can I resync with 0% risk of breaking my existing file tree (it’s my worst nightmare regarding Pydio) ? Will it drop the buggy upload or just pull the file size from AWS partial/corrupted blobs?

  • Could uploads be resumed in some way?

  • How to avoid this in the future?

  • I see random readfrom tcp 10.0.0.1:35536->10.0.0.1:38023: stream error: stream ID 289; CANCEL or readfrom tcp 10.0.0.1:47514->10.0.0.1:38023: client disconnected (although not directly during this particular upload) : Could they be related? What are they?

Note: It sounds like the worst possible class of problem because to some degree, users believed the files were uploaded : Pydio showed the size. But when trying to download them later, they realized the files weren’t reachable at all and affect the most critical component of Pydio (the upload).

Whatever is the origin (network connectivity between Pydio and S3 ? who knows) of this whole stack of numerous issues, is there any chance debugging and error-handling within upload be solved once for all?

Thank you!
(I hope I removed enough of my frustration from the wording of this post, after three passes)

1 Like
Confused about unstructured data and synchronisation
#2

Hello @drzraf,

Do you happen to be running Cells behind a reverse proxy/load balancer or something, like Cloudflare? I’m asking that because this kind of setup is known to be causing issues with multipart uploads. See here for example: Chunked-upload errors

Anyway, providing more technical details on your setup is recommended.

#3

I opened the issue you’re mentioning and consequently removed reverse-proxy from my installation. There is no reverse-proxy involved here.
(I’d say that even if it was involved, what I raised is the above post is still a set of serious reliability concerns regarding many other Pydio components)

@charles

#4

Hello @drzraf

:confused:

Thanks for removing your frustration, which must be high I assume, and you are totally right.
Unfortunately the scenario is probably pretty simple : it’s probably linked to an expiration of the accessKey used for uploads (=disconnection of the user). This leads to a failed part upload, which in turns should triggers “a clean” operation ( = the DELETE) that should remove the appearing file from the index. But that last one is failing as well, leaving the index in a wrong state.
A “structured” datasource would have led to a resync at some points and would have detected the discrepancy and fixed it in the index, but not a Flat.
We do perform super-long uploads in our tests without issues, so at that point I don’t know exactly what triggered the 403, but for sure we should put some kind of cleaning goroutines to avoid keeping un-complete multipart uploads appearing as finished.
I’ll try to experiment to 1 - have a reproducible scenario and 2 - see how to fix this. One idea was eventually to indicate an ongoing upload with a visible flag via metadata.
-c

1 Like
#5

Thank you for this preliminary answer, @charles.

Whether it’s accessKey expiration (or something else), that’s clearly where logs (and error handling) failed dramatically and in the short term I’ve to try to deal with the file-integrity disaster.
(My team is back on gdrive until the current situation is (hopefully) resolved and the software is fixed)

So, regarding my questions:

  1. Should I understand there is no “resync” functionality for flat storage?
  2. Then, could uploads be resumed in some way?
  3. Is there any existing routine to identify or remove dangling files and they S3 blob counterpart (I don’t know in which SQL tables objects references live)?
  4. If a user attempts, from the UI, to remove a file in such a state in order to retry an upload, would that create a mess or clean it?
  5. Do you have general suggestions about how I should proceed ideally?
#6
  1. Should I understand there is no “resync” functionality for flat storage?

indeed

  1. Then, could uploads be resumed in some way?

if the browser session is lost, it’s complicated

  1. Is there any existing routine to identify or remove dangling files and they S3 blob counterpart (I don’t know in which SQL tables objects references live)?

that’s what I’m looking at right now. Normally, could you verify that the “faulty” items have an “etag=temporary” in the index table (e.g. data_index_idx_tree_pydiods1)

  1. If a user attempts, from the UI, to remove a file in such a state in order to retry an upload, would that create a mess or clean it?

that would clean it : in flat, if you delete an object found in the index but not in the objects, it is ignored and cleaned.

  1. Do you have general suggestions about how I should proceed ideally?

investigating, would probably have to do with a routine as you describe to be introduced in next versions.

As for the error itself, in fact 403 is strange, an invalid token would trigger a 401 not a 403. You’ve put the caddy errors and logs but not the pydio errors/logs : nothing here about the 403 ?

#7

Here it is

{"level":"error","ts":"2022-06-17T21:08:44+02:00","logger":"pydio.gateway.data","msg":"Cannot recompute ETag :b0c9bb8d135b2ead31bd55b3abee9ff3-46","SpanUuid":"e7e87316-ee70-11ec-8f37-fa163e03ccc9","RemoteAddress":"CHART_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/605.1.15","ContentType":"application/octet-stream; charset=UTF-8","HttpProtocol":"HTTP/1.1","UserName":"USER50","UserUuid":"12427a72-7561-4f69-b826-03b319f62019","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,12427a72-7561-4f69-b826-03b319f62019","error":"The specified key does not exist."}
{"level":"error","ts":"2022-06-17T21:22:55+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"aed94ca6-ee72-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/7d035a34-67af-48ae-bc7d-8a6e0cf2e781?partNumber=11&uploadId=Ujy0OzPrPysg5ZnDM.s2qNorjecZLo6KArCTE.u0d60yKgPcfEiHsR6WcRmHl0nZrXWj98O5DyPUgDDdOKQXEA.Kv.h2muIXo3L4U_BvUoxgF9Z8V8Yv8PLN1DZSnPIf\": context canceled"}
{"level":"error","ts":"2022-06-17T21:23:03+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"a984c12d-ee72-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/7d035a34-67af-48ae-bc7d-8a6e0cf2e781?partNumber=10&uploadId=Ujy0OzPrPysg5ZnDM.s2qNorjecZLo6KArCTE.u0d60yKgPcfEiHsR6WcRmHl0nZrXWj98O5DyPUgDDdOKQXEA.Kv.h2muIXo3L4U_BvUoxgF9Z8V8Yv8PLN1DZSnPIf\": context canceled"}
{"level":"error","ts":"2022-06-17T21:23:31+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"c8f2d56a-ee72-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/3d3b72d3-0dc9-48ed-a53f-9380166575c5?partNumber=10&uploadId=uKREZ5yxRNGU3g2rP9pccIw_KGBkmzLypDRyJ0N7N6_EY_h64pGz_N9Le51ZejA6ovBV3mlGihGyUihODxsrsI.uP2Fy9boXUyvf3kfzuAaY.BiSCCVAlfVreE8Vqdub\": context canceled"}
{"level":"error","ts":"2022-06-17T21:33:27+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"2650c00b-ee74-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/204e7feb-2675-4bf0-9eef-714902647139?partNumber=28&uploadId=yqIrbCZYf5sRqbaUfPXMZCBWBEkJMRwzMQGEjrrOjy5QxVt33.rRS.IvbIIb5SdtBXYPmxlVT_Q4yLfwpZtTv_C2LtYCLnIDqQifCgJkGmsgUuwn0gqKI3BdD1r3cDkb\": context canceled"}
{"level":"error","ts":"2022-06-17T21:33:45+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"2a218bfa-ee74-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/3d3b72d3-0dc9-48ed-a53f-9380166575c5?partNumber=15&uploadId=gq9.kPgp3AiVNt0UZmQTysHcDJWOzpxW9JbCM.uIhFVxnRSywPwmjDLf7szHgRehLm2Uo3oUmoJlRvHOeuGsYLJznt67LLZYnu89Bwqm7fo.rmH_mVUCU.43MnoP_BDa\": context canceled"}
{"level":"error","ts":"2022-06-17T21:40:27+02:00","logger":"pydio.rest.frontend","msg":"Rest Error 401","SpanUuid":"56b9ff45-ee75-11ec-8f37-fa163e03ccc9","RemoteAddress":"MEGA_IP_ADDR","UserAgent":"Mozilla/5.0...Chrome/102.0.0.0","ContentType":"application/json","HttpProtocol":"HTTP/1.1","error":"No refresh token"}
{"level":"error","ts":"2022-06-17T21:43:27+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"85cad63c-ee75-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/cc34fe83-b3a7-4a00-93a7-2f133fbd175e?partNumber=14&uploadId=IS6oK_ZhqInZqjCl4NUtJMJaaPcCUkzc0.tolImzmNiI_nhQfa9YXYu8eNe7BXYJasnp_6_B9UawiYe2qh4FZ7MRCbywVE_BU_OnVlIS244m.al.CBUN5VWhgcZ3ZqzT\": context canceled"}
{"level":"error","ts":"2022-06-17T21:43:40+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"8dac201e-ee75-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/cc34fe83-b3a7-4a00-93a7-2f133fbd175e?partNumber=15&uploadId=IS6oK_ZhqInZqjCl4NUtJMJaaPcCUkzc0.tolImzmNiI_nhQfa9YXYu8eNe7BXYJasnp_6_B9UawiYe2qh4FZ7MRCbywVE_BU_OnVlIS244m.al.CBUN5VWhgcZ3ZqzT\": context canceled"}
{"level":"error","ts":"2022-06-17T21:43:56+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"9b9cd1db-ee75-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/29dbc0f6-e11e-48b2-bcaa-fba54060c92d?partNumber=17&uploadId=ZrQCxF6apopJifajiIJCWQImaVWW_YLrvJvkReyVU.I9JLs4TL7PJiwM6QXeILjpiLA1PX59FlYLiyem5GihYKgMnpTVCG36ZzuO4y70eI4HTseqGmM0brjTwEKz7Alk\": context canceled"}
{"level":"error","ts":"2022-06-17T21:52:36+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"e354b10f-ee76-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/7d035a34-67af-48ae-bc7d-8a6e0cf2e781?partNumber=49&uploadId=d5le1vv1B0Mq89dek8olXsJZuLUVhHwMpmx4ajh0ghXK3Fdqg4hXWcDQChSFw4q2uQ.7Zts9yOS6wTnsYo6Ev9n4Bcn0XV17DRT_5_OSkzNH1pz78ZcTDlaIL_n.Cnw4\": context canceled"}
{"level":"error","ts":"2022-06-17T21:53:12+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"f3d86ac0-ee76-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/7d035a34-67af-48ae-bc7d-8a6e0cf2e781?partNumber=50&uploadId=d5le1vv1B0Mq89dek8olXsJZuLUVhHwMpmx4ajh0ghXK3Fdqg4hXWcDQChSFw4q2uQ.7Zts9yOS6wTnsYo6Ev9n4Bcn0XV17DRT_5_OSkzNH1pz78ZcTDlaIL_n.Cnw4\": context canceled"}
{"level":"error","ts":"2022-06-17T21:54:26+02:00","logger":"pydio.gateway.data","msg":"Error while copying streams","SpanUuid":"71a43a8d-ee75-11ec-8f37-fa163e03ccc9","RemoteAddress":"MEGA_IP_ADDR","UserAgent":"Mozilla/5.0...Chrome/102.0.0.0","HttpProtocol":"HTTP/1.1","UserName":"othonielc","UserUuid":"24ea507b-ce23-4713-89e1-f04f528ca699","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,24ea507b-ce23-4713-89e1-f04f528ca699","error":"context canceled"}
{"level":"error","ts":"2022-06-17T22:03:36+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"58d99150-ee78-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/cc34fe83-b3a7-4a00-93a7-2f133fbd175e?partNumber=32&uploadId=E4wZBotdYabbWWRAdDsELN9kFy_L63nJRPP5X9jLoFAiKf4ntutGt.QBRn.e.MNcTJnTyQkYvt4zOjmQ7yqsrQdSpQRIJkxgC2GILwxqHsusgHhgCussIQcz997Cvk9K\": context canceled"}
{"level":"error","ts":"2022-06-17T22:03:46+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"5955bf7c-ee78-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/cc34fe83-b3a7-4a00-93a7-2f133fbd175e?partNumber=33&uploadId=E4wZBotdYabbWWRAdDsELN9kFy_L63nJRPP5X9jLoFAiKf4ntutGt.QBRn.e.MNcTJnTyQkYvt4zOjmQ7yqsrQdSpQRIJkxgC2GILwxqHsusgHhgCussIQcz997Cvk9K\": context canceled"}
{"level":"error","ts":"2022-06-17T22:03:56+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"64f91c53-ee78-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/69580bec-29a2-41e8-a2ae-64bda81c5343?partNumber=16&uploadId=6poaPHmkTJQGjqBVV4.y9Wp6jE999A5xXszV3rauF_2ltRXzNbclukTCRQvut4hbLLbOswWOzV5mZcykCXgC9imF82LW5wYso7NhHP.mq5NTVAdSJIwZC7S6wyIT1pkD\": context canceled"}
{"level":"error","ts":"2022-06-17T22:14:04+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"d2c13d41-ee79-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/69580bec-29a2-41e8-a2ae-64bda81c5343?partNumber=16&uploadId=.RgO7hEeuNyV1p.xG51bi6IY3aluxdmEu6.bqnhqnQ_9i4UwygqzD24zJrJq9WdJfFWt1lHmqGOidupuHT2btcUp40Ve5KLJvtEETDV32nJq8PvFqkfe.uuckfp8fbt_\": context canceled"}
{"level":"error","ts":"2022-06-17T22:25:22+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"52407461-ee7b-11ec-8f37-fa163e03ccc9","RemoteAddress":"CHART_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/605.1.15","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"USER50","UserUuid":"12427a72-7561-4f69-b826-03b319f62019","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,12427a72-7561-4f69-b826-03b319f62019","error":"Put \"http://IP:9001/ae-pydio/pydio/cc6f7c06-94ce-45e4-81b0-4f197226023d?partNumber=125&uploadId=KgRMXhwIt0HMwauEz20TiPzIr67HuZgSaCA3exq0JXOx98GkIrKgk3DL0kq.Xl_MPiqUApeS.2IBSDAjfB3yOKzWn__0Tst7FHahV8au1qh7uwkfMDcZWd4Zxa64Nb9I\": Connection closed by foreign host http://IP:9001/ae-pydio/pydio/cc6f7c06-94ce-45e4-81b0-4f197226023d?partNumber=125&uploadId=KgRMXhwIt0HMwauEz20TiPzIr67HuZgSaCA3exq0JXOx98GkIrKgk3DL0kq.Xl_MPiqUApeS.2IBSDAjfB3yOKzWn__0Tst7FHahV8au1qh7uwkfMDcZWd4Zxa64Nb9I. Retry again."}
{"level":"error","ts":"2022-06-17T22:25:22+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"53788de2-ee7b-11ec-8f37-fa163e03ccc9","RemoteAddress":"CHART_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/605.1.15","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"USER50","UserUuid":"12427a72-7561-4f69-b826-03b319f62019","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,12427a72-7561-4f69-b826-03b319f62019","error":"Put \"http://IP:9001/ae-pydio/pydio/cc6f7c06-94ce-45e4-81b0-4f197226023d?partNumber=126&uploadId=KgRMXhwIt0HMwauEz20TiPzIr67HuZgSaCA3exq0JXOx98GkIrKgk3DL0kq.Xl_MPiqUApeS.2IBSDAjfB3yOKzWn__0Tst7FHahV8au1qh7uwkfMDcZWd4Zxa64Nb9I\": Connection closed by foreign host http://IP:9001/ae-pydio/pydio/cc6f7c06-94ce-45e4-81b0-4f197226023d?partNumber=126&uploadId=KgRMXhwIt0HMwauEz20TiPzIr67HuZgSaCA3exq0JXOx98GkIrKgk3DL0kq.Xl_MPiqUApeS.2IBSDAjfB3yOKzWn__0Tst7FHahV8au1qh7uwkfMDcZWd4Zxa64Nb9I. Retry again."}
{"level":"error","ts":"2022-06-17T22:26:11+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"78b6eaf3-ee7b-11ec-8f37-fa163e03ccc9","RemoteAddress":"CHART_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/605.1.15","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"USER50","UserUuid":"12427a72-7561-4f69-b826-03b319f62019","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,12427a72-7561-4f69-b826-03b319f62019","error":"Put \"http://IP:9001/ae-pydio/pydio/5c7a4c26-e299-4610-8c32-e03bf651d376?partNumber=111&uploadId=jCGK2CVIklGQxgrlNNccxRj.lJ44o8n.kJKIse_Uqk6XmqAZHgnAy1P.OqOQ8vnurPrm61Ab_VQ.l0S6Uon46JV7vtPfhaYWNLNQquxFiE7AD3MyNwPtYDJkSm19QONZ\": context canceled"}
{"level":"error","ts":"2022-06-17T22:26:26+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"90f682eb-ee7b-11ec-8f37-fa163e03ccc9","RemoteAddress":"CHART_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/605.1.15","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"USER50","UserUuid":"12427a72-7561-4f69-b826-03b319f62019","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,12427a72-7561-4f69-b826-03b319f62019","error":"Put \"http://IP:9001/ae-pydio/pydio/fe21d62c-3e48-4100-953c-8115b04f6677?partNumber=158&uploadId=3oeZkE611Eh_VpP1SzKxKgSmLTkuf4OnpcNjHK5UFXzUquEqoKq9bwIO4o1g5Ij6qt_6nvX_LMkctJtnJgaM0hE1AlxxrAl5ZmCcO6mW6K6ravIHyzk7vRLxTJszys7.\": Connection closed by foreign host http://IP:9001/ae-pydio/pydio/fe21d62c-3e48-4100-953c-8115b04f6677?partNumber=158&uploadId=3oeZkE611Eh_VpP1SzKxKgSmLTkuf4OnpcNjHK5UFXzUquEqoKq9bwIO4o1g5Ij6qt_6nvX_LMkctJtnJgaM0hE1AlxxrAl5ZmCcO6mW6K6ravIHyzk7vRLxTJszys7.. Retry again."}
{"level":"error","ts":"2022-06-17T22:26:26+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"91a04545-ee7b-11ec-8f37-fa163e03ccc9","RemoteAddress":"CHART_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/605.1.15","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"USER50","UserUuid":"12427a72-7561-4f69-b826-03b319f62019","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,12427a72-7561-4f69-b826-03b319f62019","error":"Put \"http://IP:9001/ae-pydio/pydio/fe21d62c-3e48-4100-953c-8115b04f6677?partNumber=159&uploadId=3oeZkE611Eh_VpP1SzKxKgSmLTkuf4OnpcNjHK5UFXzUquEqoKq9bwIO4o1g5Ij6qt_6nvX_LMkctJtnJgaM0hE1AlxxrAl5ZmCcO6mW6K6ravIHyzk7vRLxTJszys7.\": Connection closed by foreign host http://IP:9001/ae-pydio/pydio/fe21d62c-3e48-4100-953c-8115b04f6677?partNumber=159&uploadId=3oeZkE611Eh_VpP1SzKxKgSmLTkuf4OnpcNjHK5UFXzUquEqoKq9bwIO4o1g5Ij6qt_6nvX_LMkctJtnJgaM0hE1AlxxrAl5ZmCcO6mW6K6ravIHyzk7vRLxTJszys7.. Retry again."}
{"level":"error","ts":"2022-06-17T22:28:12+02:00","logger":"pydio.rest.frontend","msg":"Rest Error 401","SpanUuid":"023eab80-ee7c-11ec-8f37-fa163e03ccc9","RemoteAddress":"CHART_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/605.1.15","ContentType":"application/json","HttpProtocol":"HTTP/1.1","error":"{\"id\":\"\",\"code\":0,\"detail\":\"invalid_request\",\"status\":\"\"}"}
{"level":"error","ts":"2022-06-17T22:33:02+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"7ea584a2-ee7c-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/399047ae-1aa0-4be0-8c3d-7a5816098e11?partNumber=31&uploadId=9BQlo4D2f.Lc3RLZSvEvwHL6ZR96_lygtTtl93QVAa1pqTf3DFCn6ORlXlLx4yTvt.Thnhk_p0imX_9eW3CnAdzKdnqUJjoDD8.VEOJ_IJiGjSP3JfolMSP.aTr9IWi8\": context canceled"}
{"level":"error","ts":"2022-06-17T22:33:05+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"8528fef3-ee7c-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/399047ae-1aa0-4be0-8c3d-7a5816098e11?partNumber=32&uploadId=9BQlo4D2f.Lc3RLZSvEvwHL6ZR96_lygtTtl93QVAa1pqTf3DFCn6ORlXlLx4yTvt.Thnhk_p0imX_9eW3CnAdzKdnqUJjoDD8.VEOJ_IJiGjSP3JfolMSP.aTr9IWi8\": context canceled"}
{"level":"error","ts":"2022-06-17T22:34:24+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"a2b1de5d-ee7c-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/91854f86-cf27-4ab7-b078-ac27a10f369a?partNumber=16&uploadId=iNXc2peBGE08psbtHTVosHRgQfcMwOrxflevrJabOBLMk0A2R43RmcwBZeDnhL_TkF9byneEXH7bj5jNuHdltEPorzOD9wC2BuFbKVc.We55j6iqW4hTqtYJxLaseMne\": context canceled"}
{"level":"error","ts":"2022-06-17T22:44:10+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"06a3d8b3-ee7e-11ec-8f37-fa163e03ccc9","RemoteAddress":"73.95.51.159","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/e7a29542-afdb-493e-abed-c7ddd7048165?partNumber=83&uploadId=ekGQbnqX0FbngqJBCeM_.eQpMyhzzPjXhz_ps2n1xR8aNW48Aw30ZU0m21Oj7eThCNxpO1O_tkSQHUMcaEPiSDdyajrTk6h.BwiVAJnajnJvV8snUucq117TtMjlo_8n\": Connection closed by foreign host http://IP:9001/ae-pydio/pydio/e7a29542-afdb-493e-abed-c7ddd7048165?partNumber=83&uploadId=ekGQbnqX0FbngqJBCeM_.eQpMyhzzPjXhz_ps2n1xR8aNW48Aw30ZU0m21Oj7eThCNxpO1O_tkSQHUMcaEPiSDdyajrTk6h.BwiVAJnajnJvV8snUucq117TtMjlo_8n. Retry again."}
{"level":"error","ts":"2022-06-17T22:44:19+02:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","SpanUuid":"05e36ed6-ee7e-11ec-8f37-fa163e03ccc9","RemoteAddress":"COMCAT_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/537.36","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"xxx","UserUuid":"e43f7a3c-ef53-4495-972e-2676f1d702cc","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,e43f7a3c-ef53-4495-972e-2676f1d702cc","error":"Put \"http://IP:9001/ae-pydio/pydio/e7a29542-afdb-493e-abed-c7ddd7048165?partNumber=82&uploadId=ekGQbnqX0FbngqJBCeM_.eQpMyhzzPjXhz_ps2n1xR8aNW48Aw30ZU0m21Oj7eThCNxpO1O_tkSQHUMcaEPiSDdyajrTk6h.BwiVAJnajnJvV8snUucq117TtMjlo_8n\": context canceled"}
{"level":"error","ts":"2022-06-17T22:47:38+02:00","logger":"pydio.gateway.data","msg":"Cannot recompute ETag :ac86e95213ceb8c0fb03e179288eef81-63","SpanUuid":"b8fa55e2-ee7e-11ec-8f37-fa163e03ccc9","RemoteAddress":"CHART_IP_ADDR","UserAgent":"Mozilla/5.0...Safari/605.1.15","ContentType":"application/octet-stream; charset=UTF-8","HttpProtocol":"HTTP/1.1","UserName":"USER50","UserUuid":"12427a72-7561-4f69-b826-03b319f62019","GroupPath":"/editors/","Profile":"standard","Roles":"ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,12427a72-7561-4f69-b826-03b319f62019","error":"The specified key does not exist."}
{"level":"warn","ts":"2022-06-17T22:49:11+02:00","logger":"pydio.grpc.tasks","msg":"Deletion : node not found in storage, removing from index","SpanRootUuid":"ef9b68a6-ee7e-11ec-8f37-fa163e03ccc9","SpanParentUuid":"ef9b68a6-ee7e-11ec-8f37-fa163e03ccc9","SpanUuid":"effbe16d-ee7e-11ec-83b1-fa163e03ccc9","OperationUuid":"delete-feb05027-283b-479e-a1a1-b9582efc9fa3-39caa0ec","SchedulerJobUuid":"delete-feb05027-283b-479e-a1a1-b9582efc9fa3","SchedulerTaskUuid":"39caa0ec-ddca-42ba-a6a7-0f8ae7ffe96e","SchedulerTaskActionPath":"ROOT/actions.tree.delete$0","UserName":"USER50","UserUuid":"12427a72-7561-4f69-b826-03b319f62019","GroupPath":"/editors/","Profile":"standard","Roles":",,,,ROOT_GROUP,f24cedbc-115a-4b2d-bda3-01a615dde621,editor,12427a72-7561-4f69-b826-03b319f62019","Node":{"Uuid":"5c7a4c26-e299-4610-8c32-e03bf651d376","Path":"aws/FILE123.MP4","Etag":"temporary","MTime":"2022-06-17T21:08:49+02:00","Size":3759602462,"MetaStore":{"mime":"\"video/mp4\"","name":"\"DJI_0303.MP4\"","pydio:meta-data-source-name":"\"aws\"","pydio:meta-data-source-path":"\"FILE123.MP4\"","pydio:meta-loaded-pydio.grpc.acl":"true","pydio:meta-loaded-pydio.grpc.activity":"true","pydio:meta-loaded-pydio.grpc.meta":"true","pydio:meta-loaded-pydio.grpc.user-meta":"true"}}}
#8

did you check for the “temporary” value of the etags in the index ?

#9

SELECT count(1) FROM data_index_aws_idx_tree where etag = 'temporary'; == 84

Contrary to what I suggested in my initial message, none of the uuid are part of the object inside the S3 bucket (not even as a partial object named after the uuid).

  • Either the bucket was automagically cleaned meanwhile but I doubt it,
  • Either none of the chunks of the affected files ever reached S3 in the first place.
    (But I don’t know if it’s “normal” since I don’t know how chunks re-assembly works behind the scene)

Since I now have the UUIDs of the faulty items I’ll batch delete them and handle the rest with the team.
Side-note: Retrieving the full-path from the UUID + mpath* is not obvious.

But afterwards the question will become whether there is a chance Pydio get fixed regarding these (advanced debugging of these 401/403, understanding the initial 403, and better error-handling) and is there anything I could help with?

#10

hi @drzraf
You should definitely have a look at the v4 branch : we’ve added end-user indication for incomplete files (temporary) plus a custom scheduler job to clean these.

We just tagged an RC1, it’s really working well, and if you have good backups (particularly for sql as there are migrations) you should be able to update without friction.
-c

1 Like
#11

Great to now and definitely great feature!

Do you also plan to resolve the original issue related to Pydio → AWS / 403 (and logging)?

Since distinct uploaders lost a total of 84 files because of it, it’s likely to bite us again in the future and I feel v4 would be worth testing if it provided either a fix or at least a powerful error-handling and logging so that this class of issues could be better identified.

#12

Excuse me the ping, but any new regarding this?

#13

We are not reproducing it currently. but we keep it in mind.