# "PutObjectPart has failed" (the S3 upload failure monster revival) **URL:** \ **Category:** Pydio Cells\ **Tags:** s3\ **Created:** [December 23, 2022, 2:48am UTC](https://forum.pydio.com/t/putobjectpart-has-failed-the-s3-upload-failure-monster-revival/4881 "2022-12-23T02:48:28Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![drzraf](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/drzraf/32/1207_2.png) [@drzraf](https://forum.pydio.com/u/drzraf)\ **Post date:** [December 23, 2022, 2:48am UTC](https://forum.pydio.com/t/putobjectpart-has-failed-the-s3-upload-failure-monster-revival/4881/1 "2022-12-23T02:48:28Z")
This is a reroll of the major issue (clearly and easily reproducible) I commented about in [NoSuchKey: Cells silently mis-uploaded files on S3 storage](https://forum.pydio.com/t/nosuchkey-cells-silently-mis-uploaded-files-on-s3-storage/4538) (now closed, but which contains all the gory details). **Multi-MB files consistently fail to be uploaded to S3 triggering a variety of errors:** ## `cells.log` ```json {"level":"info","ts":1671762577.533683,"msg":"http: proxy error: context canceled"} ``` ## `pydio.log` ```json {"level":"error","ts":"2022-12-23T03:34:59+01:00","logger":"pydio.gateway.data","msg":"PutObjectPart has failed","error":"Put \"https://xxxyyy.s3.dualstack.eu-central-1.amazonaws.com/pydio/c2960499-49d9-45ef-aeb0-86be1197abbb?partNumber=2&uploadId=lPi4HhnfP8Nj..GuzBHx0gHMArDzwyD4YZn8NG4twvqVKIGYgvPmmvbtqa4.19LpFWM5MIYwn6bRIPvgPgIBksRHoSAydcco1jKBeUifUUmJ5qIERhPTScbZktQe0v_.\": context canceled","RemoteAddress":"","UserAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0","ContentType":"application/octet-stream","HttpProtocol":"HTTP/1.1","UserName":"admin","UserUuid":"a69dd805-fd54-4c56-9b5f-xxx","GroupPath":"/","Profile":"admin","Roles":"ROOT_GROUP,ADMINS,a69dd805-fd54-4c56-9b5f-xxx"} ``` Contrary to 3.0.9, `caddy_errors.log` doesn’t output anything anymore (not even the `502` HTTP error codes it used to show). The ability to **DUMP THE TRAFFIC** and **DEBUG WHAT’S GOING ON** is more pressing than ever. I’d love to pinpoint from the traffic, where in the stack lays the error. Is it a missing header/query-string/timeout/token? A content-type? Is it between the client and Pydio? Pydio and AWS? In a request? A response? But I can’t because **Pydio makes me BLIND** \ With HTTPS omnipresence, tcpdump is of no use. So please, please, please, at the very least, add an environment variable like `DUMP_TRAFFIC=1` (modeled after Apache2 `mod_dumpio`) which would log every possible packet inside a `all-packets.log`) Because the o **nly thing more frustrating** than a 6 months old critical reproducible bug in production regarding failing file uploads is … the **inability to actually debug/pinpoint its origin**. Thank you ---
**Author:** ![ghecquet](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/ghecquet/32/514_2.png) [@ghecquet](https://forum.pydio.com/u/ghecquet)\ **Post date:** [December 27, 2022, 7:20am UTC](https://forum.pydio.com/t/putobjectpart-has-failed-the-s3-upload-failure-monster-revival/4881/2 "2022-12-27T07:20:39Z")
Hello, I’ll look into why the caddy\_errors.log doesn’t output anything. We’ve changed the version of caddy so it might just be a misconfiguration. We’re also looking at improving traffic tracking within the microservices but I doubt it is there that we’ll find answers because a context canceled usually means that an incoming request has been cancelled on the client side - or timed out (but in most cases timeouts clearly states that it’s been cancelled for that reason). I know it might have been asked already but you’re mentioning that it consistently fails, do you have any reproducible steps that we could follow ? I see in the other ticket that you have installed cells without any proxy - but do you have any firewall, network rules that could explain a client connection being cancelled ? Have you tried a bare local version of cells (we have docker for easy and small installations) to see if it is still reproducible here ? Have you tried with another s3 instance to see if it also happens there ? Thanks, Greg ---
**Author:** ![drzraf](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/drzraf/32/1207_2.png) [@drzraf](https://forum.pydio.com/u/drzraf)\ **Post date:** [December 29, 2022, 5:20pm UTC](https://forum.pydio.com/t/putobjectpart-has-failed-the-s3-upload-failure-monster-revival/4881/3 "2022-12-29T17:20:01Z")
I’ll try on a local storage, but in the meantime I found a lot of the following `HEAD /pydio/xxxx HTTP/1.1" 404 NoSuchKey` in my `AWSLogs/` (I’ll try to reproduce to relate the exact timestamps): (Line split for readability) ```auto e1bb965e7382d09f7199fed58f4e9709a9d310da395facc0ba8e559acd4e6ad4 xxxx-pydio [23/Dec/2022:02:30:03 +0000] 2001:::1324 arn:aws:iam::240676001234:user/pydio 5B0ZY1PJR73M8HVQ REST.HEAD.OBJECT pydio/ddfe5c26-3667-4fa2-a9b6-8cb4e9cd3637 "HEAD /pydio/ddfe5c26-3667-4fa2-a9b6-8cb4e9cd3637 HTTP/1.1" 404 NoSuchKey 313 - 11 - "-" "MinIO (linux; amd64) minio-go/v7.0.21" - mYoMFGWGhy52lK77SgwzFZWRUu/L9xKVQTHd8yakdfoqBIaJ8tgDrRCbQivxFmSH3siEdNuFoI0= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader xxxx-pydio.s3.dualstack.eu-central-1.amazonaws.com TLSv1.2 ``` ---
**Author:** ![charles](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/charles/32/3_2.png) [@charles](https://forum.pydio.com/u/charles)\ **Post date:** [January 2, 2023, 2:44pm UTC](https://forum.pydio.com/t/putobjectpart-has-failed-the-s3-upload-failure-monster-revival/4881/4 "2023-01-02T14:44:50Z")
Happy new year Basically, **context canceled** mostly means **request timeout** So if you are upload big files to s3 with a slow server =\> s3 connection, you may have to increase the request time defined in the “Console \> Uploader” options. ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex032/uploads/pydio/original/2X/6/61f19fa52ec15fd4250db51aea351adba0e831fd.png) [@system](https://forum.pydio.com/u/system)\ **Post date:** [March 3, 2023, 2:45pm UTC](https://forum.pydio.com/t/putobjectpart-has-failed-the-s3-upload-failure-monster-revival/4881/5 "2023-03-03T14:45:01Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.