Describe your issue in detail
Since a couple weeks ago all WebDAV uploads bigger than some hundred megabytes (I think it’s about +500MB or so) started failing with multipart upload errors like this:
level: error
ts: '2024-09-20T17:59:35Z'
logger: pydio.gateway.dav
msg: '|- DAV END'
method: PUT
path: /dav/datasource-cellstest/XXXXXXXXX.mp4
error: 'multipart put object: rpc error: code = Canceled desc = context canceled'
errorVerbose: >-
rpc error: code = Canceled desc = context canceled multipart put object
github.com/pydio/cells/v4/gateway/dav.(*File).ReadFrom.func1
github.com/pydio/cells/v4/gateway/dav/file.go:73
github.com/pydio/cells/v4/gateway/dav.(*File).ReadFrom
github.com/pydio/cells/v4/gateway/dav/file.go:105 io.copyBuffer io/io.go:416
io.Copy io/io.go:389 golang.org/x/net/webdav.(*Handler).handlePut
golang.org/x/net@v0.26.0/webdav/webdav.go:275
golang.org/x/net/webdav.(*Handler).ServeHTTP
golang.org/x/net@v0.26.0/webdav/webdav.go:58
github.com/pydio/cells/v4/gateway/dav.newHandler.logRequest.func2
github.com/pydio/cells/v4/gateway/dav/handler.go:57
net/http.HandlerFunc.ServeHTTP net/http/server.go:2141
github.com/pydio/cells/v4/gateway/dav.newHandler.(*BasicAuthenticator).Wrap.func3
github.com/pydio/cells/v4/common/auth/basic.go:68
net/http.HandlerFunc.ServeHTTP net/http/server.go:2141
github.com/pydio/cells/v4/gateway/dav.newHandler.HttpWrapperMeta.func4
github.com/pydio/cells/v4/common/service/context/http-to-meta.go:126
net/http.HandlerFunc.ServeHTTP net/http/server.go:2141
github.com/pydio/cells/v4/common/server.(*ListableMux).ServeHTTP
github.com/pydio/cells/v4/common/server/mux.go:238
github.com/pydio/cells/v4/common/client/http.(*resolver).ServeHTTP
github.com/pydio/cells/v4/common/client/http/resolver.go:142
github.com/pydio/cells/v4/common/server/caddy/mux.(*Middleware).ServeHTTP
github.com/pydio/cells/v4/common/server/caddy/mux/mux.go:80
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:268
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/headers/headers.go:110
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/headers/headers.go:110
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/headers/headers.go:110
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
It leaves an empty file behind that, when I delete it, I find this message in the error log:
"Exec.DeleteNode : cannot find object in s3! Should it be removed from index?"
- Web upload works fine
- We haven’t upgraded lately, we just upgraded when the error started to see if it went away
- I tested two different storage backends: regular filesystem and a different third party S3 server (Garage) with the same result
- I’ve tested three different clients: Cyberduck, Transmit and the macOS Finder DAV client
- Smaller files (under 500 MB) don’t error
I guess it’s the Minio gateway giving an error?
What version of Cells are you using?
4.4.4, and I upgraded to 4.4.5 but I’m still having the same error
What is the server OS? Database name/version? Browser name or mobile device description (if issue appears client-side)?
- Debian 12
- Kernel 6.5.13-5
- MariaDB 11.4.3
- Traefik 3.0 as a proxy and TLS terminator
- Cyberduck 9.0.2
- Transmit 5.10.4
- macOS 14.6.1
What steps have you taken to resolve this issue already?
- I’ve tested different storage backends
- I’ve tested different DAV clients
- I’ve tested different file types and sizes
- I’ve updated and restarted the system
- I’ve reviewed Traefik logs
- I’ve tested with both regular and admin users
