403 and 502 errors when uploading large files (especially ≥ 1 GB)

We’re using Pydio 4.0.0 RC5, but we had the same issue with 3.0.9. We’re using DigitalOcean Spaces (an S3-compatible object store) as our storage, but also had the same issue with Wasabi. We’re using nginx 1.20.1 as a reverse proxy, but I think we had the same problem when connecting directly to the Cells web server without a reverse proxy.

When I upload a large file with the web client, it will first appear to upload normally. Then eventually a chunk often fails to upload with a 403 Forbidden error visible in Chrome’s Inspector. When this happens, it seems to start the whole upload over, reuploading chunks that had been successfully uploaded:

At left, you can see it’s reuploading already finished chunks (parts 1-3 are uploading after parts 4-6 have completed). At right, you can see the 403 errors in the Inspector. After a while, the upload usually fails with an “Access Denied” error. (I’ve got a screenshot of that, too, but the forum won’t let me attach as second image, so see next post.)

Unfortunately, this problem seems to happen at random. A file might upload without this problem occurring at all, or it might happen early in the upload, even if it’s the same file. The same or a similar problem happens when using cec on the command line instead of the web browser, too, which is a huge problem for us because we’re using cec to try to migrate a large number of files, some of which are quite large.

I haven’t experimented with changing the size of upload chunks; for now I’m sticking with the default 50 MB. I’m not inclined to mess with it to try to fix it; I suspect it will only hide the issue rather than ensure it won’t happen again. We also seem to have enough RAM free (top shows about 1500 GB free, which isn’t bad when your system has 4 GB RAM), and we’re not running out of disk space either. I have no idea what could be causing the problem. Any ideas?

Here’s the second screenshot:

You can see the 403 errors were joined by a 502 error at one point.

I’m trying it without using nginx as a reverse proxy, connecting straight to Cells in my web browser, and now there are a lot of messages like this in pydio.log (which weren’t there before):

PutObjectPart has failed {"error": "Put \"https://[redacted].com/[redacted]-pydiods1/0525dbd7-e09f-4d3a-8ea7-f289c8c47dca?partNumber=2&uploadId=2~UkbVvZJd_e5MlsMnrSSae9pmmgslLgW\": context canceled", "RemoteAddress": "[redacted]", "UserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36", "ContentType": "application/octet-stream", "HttpProtocol": "HTTP/1.1", "UserName": "admin", "UserUuid": "ab478bd6-0668-46f1-8b74-553a480049d3", "GroupPath": "/", "Profile": "admin", "Roles": "ROOT_GROUP,ADMINS,ab478bd6-0668-46f1-8b74-553a480049d3"}

That “context canceled” part is interesting. What could it mean?

Despite this, the actual upload seems to be proceeding much more smoothly… (EDIT: spoke too soon. It started acting up the same way it did when it went through nginx.)

Hi,
Could you please post all upload settings?

They’re all default settings:

  • 0 maximum size
  • 100 MB multipart threshold
  • 50 MB multipart
  • Queue size 3
  • Timeout 3

When Cells is configured with the default site (0.0.0.0:8080) and access it directly in the web browser, I rarely have problems—though they still seem to occur from time to time—but when I have it set up behind nginx as a reverse proxy, the problems seem much more frequent.

Here is my nginx setup:

server {
    server_name [redacted].com;

    # Allow any size file to be uploaded.
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    # Cells 4.0.0 will give 502 Bad Gateway without this line
    proxy_ssl_server_name on;

    location / {
        # Uncomment this to enable gRPC and thus be able to use cells-sync
        #if ($http_content_type = "application/grpc") {
        #    grpc_pass grpcs://localhost:3224;
        #}
        proxy_pass https://localhost:3224;
    }

    location /ws/ {
        proxy_pass https://localhost:3224;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }

    error_log /var/log/nginx/cells-proxy-error.log;
    access_log /var/log/nginx/cells-proxy-access.log;


    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/[redacted].com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/[redacted].com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = [redacted].com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    server_name [redacted].com;
    listen 80;
    return 404; # managed by Certbot


}```

To try to diagnose this issue, I tried having nginx listen on port 80 without SSL. Cells itself was still running in SSL mode with a self-signed cert.

What happens now is, a few minutes into the upload, Cells kicks me out to the login screen and the upload fails. It wasn’t doing this before, but it’s doing it at about the same time the upload went screwy when I connected using HTTPS.

At the moment the upload failed, this appeared in pydio.log:

Rest Error 401 {"error": "No refresh token", "SpanUuid": "00e9c57b-52cd-442f-b50b-44ef9c394589", "RemoteAddress": "[::1]:47174", "UserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36", "ContentType": "application/json", "HttpProtocol": "HTTP/1.0"}

Nothing appears in nginx’s error log.

In a desperate attempt to work around the issue, I tried the following upload settings:

  • Multipart parts size: 10 MB
  • Queue size: 6
  • Timeout: 10 minutes

No dice. I tried uploading a single 4 GB file and the issue still occurred before it finished uploading.

I’m not sure this problem has been solved, exactly, but I have found something that works well enough for my purposes.

To the config here, after proxy_buffering off; I added the following lines:

proxy_ssl_server_name on;
proxy_http_version 1.1;
proxy_request_buffering off;

The first of these lines is to fix an issue with Cells 4.0.0 giving 502 Bad Gateway errors and is unrelated to the issue this thread is about. The other two mostly fixed the issue. I uploaded a 4 GB file to test, which was divided into 84 or so 50 MB chunks. As it uploaded the last three chunks, the uploads became inexplicably slow, and eventually some of the chunks appeared to restart (maybe some kind of timeout?). Then it threw up a 403 error and it started the whole upload over again.

However, I decided to try a second time with the same file and that time it uploaded without problems (I wasn’t around my machine as the upload completed, so I don’t know if the slowdown was still there, but there was no 403 this time). I decided to try migrating about 32 GB of data in 3078 files using cec, and it appears all the files were transferred. (During upload it kept showing some uploads being stuck at 100%, and I don’t know if this is symptomatic of something, but eventually they were cleared and new uploads took their place.)

So I would say that for my purposes the issue is solved, but I am not convinced that the underlying issue is really resolved. In any case, in lieu of a better fix, I really recommend adding those three lines to the nginx config in the Pydio Cells docs; they would have saved me weeks of grief.