# 403 and 502 errors when uploading large files (especially ≥ 1 GB) **URL:** \ **Category:** Pydio Cells\ **Tags:** upload, s3\ **Created:** [September 28, 2022, 10:04am UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709 "2022-09-28T10:04:35Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![furrykef](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/furrykef/32/1879_2.png) [@furrykef](https://forum.pydio.com/u/furrykef)\ **Post date:** [September 28, 2022, 10:04am UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/1 "2022-09-28T10:04:35Z")
We’re using Pydio 4.0.0 RC5, but we had the same issue with 3.0.9. We’re using DigitalOcean Spaces (an S3-compatible object store) as our storage, but also had the same issue with Wasabi. We’re using nginx 1.20.1 as a reverse proxy, but I think we had the same problem when connecting directly to the Cells web server without a reverse proxy. When I upload a large file with the web client, it will first appear to upload normally. Then eventually a chunk often fails to upload with a 403 Forbidden error visible in Chrome’s Inspector. When this happens, it seems to start the whole upload over, reuploading chunks that had been successfully uploaded: ![cells-error1](https://canada1.discourse-cdn.com/flex032/uploads/pydio/original/2X/3/3348c7a5369d596940d47c3663d30ba487bd2e6b.png) At left, you can see it’s reuploading already finished chunks (parts 1-3 are uploading after parts 4-6 have completed). At right, you can see the 403 errors in the Inspector. After a while, the upload usually fails with an “Access Denied” error. (I’ve got a screenshot of that, too, but the forum won’t let me attach as second image, so see next post.) Unfortunately, this problem seems to happen at random. A file might upload without this problem occurring at all, or it might happen early in the upload, even if it’s the same file. The same or a similar problem happens when using `cec` on the command line instead of the web browser, too, which is a huge problem for us because we’re using `cec` to try to migrate a large number of files, some of which are quite large. I haven’t experimented with changing the size of upload chunks; for now I’m sticking with the default 50 MB. I’m not inclined to mess with it to try to fix it; I suspect it will only hide the issue rather than ensure it won’t happen again. We also seem to have enough RAM free (`top` shows about 1500 GB free, which isn’t bad when your system has 4 GB RAM), and we’re not running out of disk space either. I have no idea what could be causing the problem. Any ideas? ---
**Author:** ![furrykef](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/furrykef/32/1879_2.png) [@furrykef](https://forum.pydio.com/u/furrykef)\ **Post date:** [September 28, 2022, 10:05am UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/2 "2022-09-28T10:05:37Z")
Here’s the second screenshot: ![cells-error2](https://canada1.discourse-cdn.com/flex032/uploads/pydio/original/2X/c/cce305a4fe3ba3102497148e32dcbcef10b8466b.png) You can see the 403 errors were joined by a 502 error at one point. ---
**Author:** ![furrykef](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/furrykef/32/1879_2.png) [@furrykef](https://forum.pydio.com/u/furrykef)\ **Post date:** [September 28, 2022, 9:52pm UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/3 "2022-09-28T21:52:51Z")
I’m trying it without using nginx as a reverse proxy, connecting straight to Cells in my web browser, and now there are a lot of messages like this in pydio.log (which weren’t there before): `PutObjectPart has failed {"error": "Put \"https://[redacted].com/[redacted]-pydiods1/0525dbd7-e09f-4d3a-8ea7-f289c8c47dca?partNumber=2&uploadId=2~UkbVvZJd_e5MlsMnrSSae9pmmgslLgW\": context canceled", "RemoteAddress": "[redacted]", "UserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36", "ContentType": "application/octet-stream", "HttpProtocol": "HTTP/1.1", "UserName": "admin", "UserUuid": "ab478bd6-0668-46f1-8b74-553a480049d3", "GroupPath": "/", "Profile": "admin", "Roles": "ROOT_GROUP,ADMINS,ab478bd6-0668-46f1-8b74-553a480049d3"}` That “context canceled” part is interesting. What could it mean? Despite this, the actual upload seems to be proceeding much more smoothly… (EDIT: spoke too soon. It started acting up the same way it did when it went through nginx.) ---
**Author:** ![c12simple](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/c12simple/32/90_2.png) [@c12simple](https://forum.pydio.com/u/c12simple)\ **Post date:** [September 29, 2022, 8:08am UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/4 "2022-09-29T08:08:22Z")
Hi, Could you please post all upload settings? ---
**Author:** ![furrykef](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/furrykef/32/1879_2.png) [@furrykef](https://forum.pydio.com/u/furrykef)\ **Post date:** [September 29, 2022, 1:49pm UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/5 "2022-09-29T13:49:41Z")
They’re all default settings: - 0 maximum size - 100 MB multipart threshold - 50 MB multipart - Queue size 3 - Timeout 3 When Cells is configured with the default site (0.0.0.0:8080) and access it directly in the web browser, I rarely have problems—though they still seem to occur from time to time—but when I have it set up behind nginx as a reverse proxy, the problems seem _much_ more frequent. Here is my nginx setup: ````auto server { server_name [redacted].com; # Allow any size file to be uploaded. client_max_body_size 0; # To disable buffering proxy_buffering off; # Cells 4.0.0 will give 502 Bad Gateway without this line proxy_ssl_server_name on; location / { # Uncomment this to enable gRPC and thus be able to use cells-sync #if ($http_content_type = "application/grpc") { # grpc_pass grpcs://localhost:3224; #} proxy_pass https://localhost:3224; } location /ws/ { proxy_pass https://localhost:3224; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; } error_log /var/log/nginx/cells-proxy-error.log; access_log /var/log/nginx/cells-proxy-access.log; listen 443 ssl http2; # managed by Certbot ssl_certificate /etc/letsencrypt/live/[redacted].com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/[redacted].com/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } server { if ($host = [redacted].com) { return 301 https://$host$request_uri; } # managed by Certbot server_name [redacted].com; listen 80; return 404; # managed by Certbot }``` ```` ---
**Author:** ![furrykef](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/furrykef/32/1879_2.png) [@furrykef](https://forum.pydio.com/u/furrykef)\ **Post date:** [September 30, 2022, 5:28pm UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/6 "2022-09-30T17:28:55Z")
To try to diagnose this issue, I tried having nginx listen on port 80 without SSL. Cells itself was still running in SSL mode with a self-signed cert. What happens now is, a few minutes into the upload, Cells kicks me out to the login screen and the upload fails. It wasn’t doing this before, but it’s doing it at about the same time the upload went screwy when I connected using HTTPS. At the moment the upload failed, this appeared in pydio.log: `Rest Error 401 {"error": "No refresh token", "SpanUuid": "00e9c57b-52cd-442f-b50b-44ef9c394589", "RemoteAddress": "[::1]:47174", "UserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36", "ContentType": "application/json", "HttpProtocol": "HTTP/1.0"}` Nothing appears in nginx’s error log. ---
**Author:** ![furrykef](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/furrykef/32/1879_2.png) [@furrykef](https://forum.pydio.com/u/furrykef)\ **Post date:** [October 6, 2022, 9:17am UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/7 "2022-10-06T09:17:39Z")
In a desperate attempt to work around the issue, I tried the following upload settings: - Multipart parts size: 10 MB - Queue size: 6 - Timeout: 10 minutes No dice. I tried uploading a single 4 GB file and the issue still occurred before it finished uploading. ---
**Author:** ![furrykef](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.pydio.com/furrykef/32/1879_2.png) [@furrykef](https://forum.pydio.com/u/furrykef)\ **Post date:** [October 13, 2022, 11:54am UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/8 "2022-10-13T11:54:23Z")
I’m not sure this problem has been solved, exactly, but I have found something that works well enough for my purposes. To the config [here](https://pydio.com/en/docs/kb/deployment/running-cells-behind-nginx-reverse-proxy), after `proxy_buffering off;` I added the following lines: ```auto proxy_ssl_server_name on; proxy_http_version 1.1; proxy_request_buffering off; ``` The first of these lines is to fix an issue with Cells 4.0.0 giving 502 Bad Gateway errors and is unrelated to the issue this thread is about. The other two _mostly_ fixed the issue. I uploaded a 4 GB file to test, which was divided into 84 or so 50 MB chunks. As it uploaded the last three chunks, the uploads became inexplicably slow, and eventually some of the chunks appeared to restart (maybe some kind of timeout?). Then it threw up a 403 error and it started the whole upload over again. However, I decided to try a second time with the same file and that time it uploaded without problems (I wasn’t around my machine as the upload completed, so I don’t know if the slowdown was still there, but there was no 403 this time). I decided to try migrating about 32 GB of data in 3078 files using `cec`, and it appears all the files were transferred. (During upload it kept showing some uploads being stuck at 100%, and I don’t know if this is symptomatic of something, but eventually they were cleared and new uploads took their place.) So I would say that _for my purposes_ the issue is solved, but I am not convinced that the underlying issue is really resolved. In any case, in lieu of a better fix, I really recommend adding those three lines to the nginx config in the Pydio Cells docs; they would have saved me weeks of grief. ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex032/uploads/pydio/original/2X/6/61f19fa52ec15fd4250db51aea351adba0e831fd.png) [@system](https://forum.pydio.com/u/system)\ **Post date:** [December 12, 2022, 11:54am UTC](https://forum.pydio.com/t/403-and-502-errors-when-uploading-large-files-especially-1-gb/4709/9 "2022-12-12T11:54:51Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.