Deletion fails when using Pydio Cells as S3 Storage Provider

Hi There

I’ve setup Pydio Cells using the docker container. I can access and delete files in the Web GUI without any issues.

I’m trying to connect Pydio with Directus, a headless CMS that supports S3 Storage providers. Internally Directus is using the Javascript aws-sdk to connect to S3 providers.

The used configuration is the following:

  STORAGE_PYDIO_DRIVER: 's3'
  STORAGE_PYDIO_KEY: "THE PERSONAL ACCESS TOKEN OF THE ADMIN GOES HERE"
  STORAGE_PYDIO_SECRET: "gatewaysecret" -> fixed string
  STORAGE_PYDIO_BUCKET: "io" -> fixed bucket name
  STORAGE_PYDIO_ROOT: "/directus" -> name of the workspace
  STORAGE_PYDIO_ENDPOINT: "http://dms.service.lo:8080" url to access pydio through traefik

Uploading files works perfectly, I just can’t delete files. Once I delete a file, I get the following exception from the aws-sdk inside of Directus:

E_UNKNOWN: An unknown error happened with the file directus/8d4e93c6-a2b7-452e-86ce-3218df81847e.\n\nError code: InternalError\nOriginal stack:\nInternalError: We encountered an internal error, please try again.\n at Request.extractError (/directus/node_modules/aws-sdk/lib/services/s3.js:714:35)\n at Request.callListeners (/directus/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/directus/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/directus/node_modules/aws-sdk/lib/request.js:688:14)\n at Request.transition (/directus/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/directus/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /directus/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request. (/directus/node_modules/aws-sdk/lib/request.js:38:9)\n at Request. (/directus/node_modules/aws-sdk/lib/request.js:690:12)\n at Request.callListeners (/directus/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

I’m asking here since when I connect Directus to Minio, it works without a hitch. Also if I connect Directus to a different S3 provider it also works.

I also connected to Pydio with Cyberduck. There deletion seems to work but it also behaves weirdly. If I delete a file, it still shows up until I refresh the list, then it disappears.

Is it possible that file deletion on Pydio is not 100% S3 compatible?

Is there a good way how I could debug this issue further on the Pydio side? I don’t see much output on the logs when the deletion is failing.

Thank you for your help!

I did find some more information, when the error occurs, this is Pydios output:

| 2021-08-18T09:32:58.191Z  DEBUG  pydio.gateway.data  jwt rawIdToken verify: failed  {"error": "request_unauthorized", "errorVerbose": "request_unauthorized\ngithub.com/pydio/cells/vendor/github.com/ory/fosite.(*Fosite).IntrospectToken\n\tgithub.com/pydio/cells/vendor/github.com/ory/fosite/introspect.go:69\ngithub.com/pydio/cells/common/auth.(*oryprovider).Verify\n\tgithub.com/pydio/cells/common/auth/jwt_ory.go:382\ngithub.com/pydio/cells/common/auth.(*JWTVerifier).verifyTokenWithRetry\n\tgithub.com/pydio/cells/common/auth/jwt.go:246\ngithub.com/pydio/cells/common/auth.(*JWTVerifier).Verify\n\tgithub.com/pydio/cells/common/auth/jwt.go:294\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.pydioAuthHandler.ServeHTTP\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/gateway-pydio-auth-handler.go:83\ngithub.com/pydio/cells/common/service/context.HttpSpanHandlerWrapper.func1\n\tgithub.com/pydio/cells/common/service/context/span.go:178\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.criticalErrorHandler.ServeHTTP\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/generic-handlers.go:775\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/http.(*Server).Start.func1\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/http/server.go:115\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2843\nnet/http.(*conn).serve\n\tnet/http/server.go:1925\nruntime.goexit\n\truntime/asm_amd64.s:1374"}
dms_1              | 2021-08-18T09:32:58.196Z  DEBUG  pydio.gateway.data  jwt rawIdToken verify: failed  {"error": "{\"id\":\"\",\"code\":0,\"detail\":\"request_unauthorized\",\"status\":\"\"}"}
dms_1              | 2021-08-18T09:32:58.206Z  DEBUG  Users Search Query   {"q": "SELECT `t`.`uuid`, `t`.`level`, `t`.`mpath1`, `t`.`mpath2`, `t`.`mpath3`, `t`.`mpath4`, `t`.`name`, `t`.`leaf`, `t`.`etag`, `t`.`mtime` FROM `idm_user_idx_tree` AS `t` WHERE (`t`.`uuid` = ?) ORDER BY `t`.`name` ASC", "q2 length": 1}
dms_1              | 2021-08-18T09:32:58.225Z  DEBUG  pydio.grpc.tree  No user/claims found - skipping user metas on metaStreamers init!
dms_1              | 2021-08-18T09:32:58.226Z  DEBUG  pydio.grpc.tree  ReadNode  {"uuid": "DATASOURCE:personal"}
dms_1              | 2021-08-18T09:32:58.231Z  DEBUG  pydio.grpc.data.index.personal  ReadNode  {"time": "1.182ms", "req": "Node:<Uuid:\"ROOT\" > ", "resp": "Success:true Node:<Uuid:\"ROOT\" Path:\"/\" Type:COLLECTION Size:474121 MTime:1629268303 MetaStore:<key:\"name\" value:\"\\\"\\\"\" > MetaStore:<key:\"pydio:meta-data-source-name\" value:\"\\\"personal\\\"\" > > "}
dms_1              | 2021-08-18T09:32:58.232Z  DEBUG  pydio.grpc.tree  [Look Up] Found node  {"uuid": "DATASOURCE:personal", "datasource": "personal"}
dms_1              | 2021-08-18T09:32:58.232Z  DEBUG  pydio.grpc.tree  Response after lookUp  {"path": "personal/"}
dms_1              | 2021-08-18T09:32:58.233Z  DEBUG  pydio.grpc.meta  ReadNodeStream  {"path": "personal/"}
dms_1              | 2021-08-18T09:32:58.235Z  DEBUG  pydio.grpc.tree  EnrichMetaProvider - Average time spent  {"pydio.grpc.meta": "1.8817ms"}
dms_1              | 2021-08-18T09:32:58.235Z  DEBUG  pydio.grpc.tree  ReadNode  {"time": "8.8924ms", "req": "Node:<Uuid:\"DATASOURCE:personal\" > ", "resp": "Node:<Uuid:\"DATASOURCE:personal\" Path:\"personal/\" Type:COLLECTION Size:474121 MTime:1629268303 MetaStore:<key:\"name\" value:\"\\\"\\\"\" > MetaStore:<key:\"pydio:meta-data-source-name\" value:\"\\\"personal\\\"\" > MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"\\\"\" > MetaStore:<key:\"pydio:meta-loaded-pydio.grpc.meta\" value:\"true\" > > "}
dms_1              | 2021-08-18T09:32:58.247Z  DEBUG  Users Search Query   {"q": "SELECT `t`.`uuid`, `t`.`level`, `t`.`mpath1`, `t`.`mpath2`, `t`.`mpath3`, `t`.`mpath4`, `t`.`name`, `t`.`leaf`, `t`.`etag`, `t`.`mtime` FROM `idm_user_idx_tree` AS `t` WHERE ((`t`.`name` = ?) AND (`t`.`leaf` = ?)) ORDER BY `t`.`name` ASC", "q2 length": 1}
dms_1              | 2021-08-18T09:32:58.252Z  DEBUG  pydio.grpc.data.index.personal  ReadNode  {"time": "2.0158ms", "req": "Node:<Path:\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" > ", "resp": ""}
dms_1              | 2021-08-18T09:32:58.254Z  DEBUG  pydio.grpc.tree  ReadNode  {"time": "4.9506ms", "req": "Node:<Path:\"personal/ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\\\"\" > > ", "resp": ""}
dms_1              | 2021-08-18T09:32:58.254Z  DEBUG  pydio.grpc.tree  ListNodes  {"time": "15.2313ms", "req": "Node:<Path:\"personal/ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\\\"\" > > Ancestors:true ", "resp": {}}
dms_1              | 2021-08-18T09:32:58.257Z  DEBUG  Users Search Query   {"q": "SELECT `t`.`uuid`, `t`.`level`, `t`.`mpath1`, `t`.`mpath2`, `t`.`mpath3`, `t`.`mpath4`, `t`.`name`, `t`.`leaf`, `t`.`etag`, `t`.`mtime` FROM `idm_user_idx_tree` AS `t` WHERE ((`t`.`name` = ?) AND (`t`.`leaf` = ?)) ORDER BY `t`.`name` ASC", "q2 length": 1}
dms_1              | 2021-08-18T09:32:58.310Z  DEBUG  pydio.gateway.data  jwt rawIdToken verify: failed  {"error": "request_unauthorized", "errorVerbose": "request_unauthorized\ngithub.com/pydio/cells/vendor/github.com/ory/fosite.(*Fosite).IntrospectToken\n\tgithub.com/pydio/cells/vendor/github.com/ory/fosite/introspect.go:69\ngithub.com/pydio/cells/common/auth.(*oryprovider).Verify\n\tgithub.com/pydio/cells/common/auth/jwt_ory.go:382\ngithub.com/pydio/cells/common/auth.(*JWTVerifier).verifyTokenWithRetry\n\tgithub.com/pydio/cells/common/auth/jwt.go:246\ngithub.com/pydio/cells/common/auth.(*JWTVerifier).Verify\n\tgithub.com/pydio/cells/common/auth/jwt.go:294\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.pydioAuthHandler.ServeHTTP\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/gateway-pydio-auth-handler.go:83\ngithub.com/pydio/cells/common/service/context.HttpSpanHandlerWrapper.func1\n\tgithub.com/pydio/cells/common/service/context/span.go:178\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.criticalErrorHandler.ServeHTTP\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/generic-handlers.go:775\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/http.(*Server).Start.func1\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/http/server.go:115\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2843\nnet/http.(*conn).serve\n\tnet/http/server.go:1925\nruntime.goexit\n\truntime/asm_amd64.s:1374"}
dms_1              | 2021-08-18T09:32:58.315Z  DEBUG  pydio.gateway.data  jwt rawIdToken verify: failed  {"error": "{\"id\":\"\",\"code\":0,\"detail\":\"request_unauthorized\",\"status\":\"\"}"}
dms_1              | 2021-08-18T09:32:58.343Z  DEBUG  pydio.grpc.data.index.personal  ReadNode  {"time": "894.9µs", "req": "Node:<Path:\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" > ", "resp": ""}
dms_1              | 2021-08-18T09:32:58.345Z  DEBUG  pydio.grpc.tree  ReadNode  {"time": "4.1047ms", "req": "Node:<Path:\"personal/ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\\\"\" > > ", "resp": ""}
dms_1              | 2021-08-18T09:32:58.345Z  DEBUG  pydio.grpc.tree  ListNodes  {"time": "13.7849ms", "req": "Node:<Path:\"personal/ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\\\"\" > > Ancestors:true ", "resp": {}}
dms_1              | 2021-08-18T09:32:58.487Z  DEBUG  pydio.gateway.data  jwt rawIdToken verify: failed  {"error": "request_unauthorized", "errorVerbose": "request_unauthorized\ngithub.com/pydio/cells/vendor/github.com/ory/fosite.(*Fosite).IntrospectToken\n\tgithub.com/pydio/cells/vendor/github.com/ory/fosite/introspect.go:69\ngithub.com/pydio/cells/common/auth.(*oryprovider).Verify\n\tgithub.com/pydio/cells/common/auth/jwt_ory.go:382\ngithub.com/pydio/cells/common/auth.(*JWTVerifier).verifyTokenWithRetry\n\tgithub.com/pydio/cells/common/auth/jwt.go:246\ngithub.com/pydio/cells/common/auth.(*JWTVerifier).Verify\n\tgithub.com/pydio/cells/common/auth/jwt.go:294\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.pydioAuthHandler.ServeHTTP\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/gateway-pydio-auth-handler.go:83\ngithub.com/pydio/cells/common/service/context.HttpSpanHandlerWrapper.func1\n\tgithub.com/pydio/cells/common/service/context/span.go:178\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.criticalErrorHandler.ServeHTTP\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/generic-handlers.go:775\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/http.(*Server).Start.func1\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/http/server.go:115\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2843\nnet/http.(*conn).serve\n\tnet/http/server.go:1925\nruntime.goexit\n\truntime/asm_amd64.s:1374"}
dms_1              | 2021-08-18T09:32:58.491Z  DEBUG  pydio.gateway.data  jwt rawIdToken verify: failed  {"error": "{\"id\":\"\",\"code\":0,\"detail\":\"request_unauthorized\",\"status\":\"\"}"}
dms_1              | 2021-08-18T09:32:58.525Z  DEBUG  pydio.grpc.data.index.personal  ReadNode  {"time": "909µs", "req": "Node:<Path:\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" > ", "resp": ""}
dms_1              | 2021-08-18T09:32:58.526Z  DEBUG  pydio.grpc.tree  ReadNode  {"time": "4.9094ms", "req": "Node:<Path:\"personal/ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\\\"\" > > ", "resp": ""}
dms_1              | 2021-08-18T09:32:58.527Z  DEBUG  pydio.grpc.tree  ListNodes  {"time": "17.7057ms", "req": "Node:<Path:\"personal/ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\\\"\" > > Ancestors:true ", "resp": {}}
dms_1              | 2021-08-18T09:32:58.720Z  DEBUG  pydio.gateway.data  jwt rawIdToken verify: failed  {"error": "request_unauthorized", "errorVerbose": "request_unauthorized\ngithub.com/pydio/cells/vendor/github.com/ory/fosite.(*Fosite).IntrospectToken\n\tgithub.com/pydio/cells/vendor/github.com/ory/fosite/introspect.go:69\ngithub.com/pydio/cells/common/auth.(*oryprovider).Verify\n\tgithub.com/pydio/cells/common/auth/jwt_ory.go:382\ngithub.com/pydio/cells/common/auth.(*JWTVerifier).verifyTokenWithRetry\n\tgithub.com/pydio/cells/common/auth/jwt.go:246\ngithub.com/pydio/cells/common/auth.(*JWTVerifier).Verify\n\tgithub.com/pydio/cells/common/auth/jwt.go:294\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.pydioAuthHandler.ServeHTTP\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/gateway-pydio-auth-handler.go:83\ngithub.com/pydio/cells/common/service/context.HttpSpanHandlerWrapper.func1\n\tgithub.com/pydio/cells/common/service/context/span.go:178\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.criticalErrorHandler.ServeHTTP\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/generic-handlers.go:775\ngithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/http.(*Server).Start.func1\n\tgithub.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd/http/server.go:115\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2843\nnet/http.(*conn).serve\n\tnet/http/server.go:1925\nruntime.goexit\n\truntime/asm_amd64.s:1374"}
dms_1              | 2021-08-18T09:32:58.724Z  DEBUG  pydio.gateway.data  jwt rawIdToken verify: failed  {"error": "{\"id\":\"\",\"code\":0,\"detail\":\"request_unauthorized\",\"status\":\"\"}"}
dms_1              | 2021-08-18T09:32:58.757Z  DEBUG  pydio.grpc.data.index.personal  ReadNode  {"time": "846.1µs", "req": "Node:<Path:\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" > ", "resp": ""}
dms_1              | 2021-08-18T09:32:58.757Z  DEBUG  pydio.grpc.tree  ReadNode  {"time": "3.3138ms", "req": "Node:<Path:\"personal/ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\\\"\" > > ", "resp": ""}
dms_1              | 2021-08-18T09:32:58.758Z  DEBUG  pydio.grpc.tree  ListNodes  {"time": "14.1787ms", "req": "Node:<Path:\"personal/ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\" MetaStore:<key:\"pydio:meta-data-source-path\" value:\"\\\"ad36a715-621e-4bb1-b0f4-8bc8a1863cb0\\\"\" > > Ancestors:true ", "resp": {}}

I could locate the issue, the problem appears when Prefix is called with something that does not exist, when using the aws s3 client.

				const response = await this.$driver
					.listObjectsV2({
						Bucket: this.$bucket,
						Prefix: 'directus/c619ee99-1544-4023-8abc-99c82075e8bc',
						ContinuationToken: continuationToken,
						MaxKeys: 1000,
					})
					.promise();
1 Like

Hello @nicam ,

If you think that you found a bug, would you mind opening a github issue with the same exact details.
Thank you.

I did at the same time I created this forum post :slight_smile: Prefix in listObjects throws 500 error · Issue #342 · pydio/cells · GitHub