Brand new installation from docker-compose not able to upload big file

Pydio Cells
,
1

Hi,

I just finished installing Pydio using this docker-compose configuration.

I’m using port 4443 as ${port} and my ip is 192.168.2.6

version: '3'
services:

  cells:
    image: pydio/cells:latest
    restart: always
    ports: ["${port}:${port}"]
    environment:
      - CELLS_BIND=${ip}:${port}
      - CELLS_EXTERNAL=https://${ip}:${port}
    volumes:
      - "cellsdir:/var/cells"
      - "/srv/raiddata/celldata:/var/cells/data"

  # MySQL image with a default database cells and a dedicated user pydio
  mysql:
    image: mysql:5.7
    restart: always
    ports: ["3306:3306"]
    environment:
      MYSQL_ROOT_PASSWORD: ${mysql_root_password}
      MYSQL_DATABASE: cells
      MYSQL_USER: ${mysql_user}
      MYSQL_PASSWORD: ${mysql_user_password}
    command: [mysqld, --character-set-server=utf8mb4, --collation-server=utf8mb4_unicode_ci]
    volumes:
      - "mysqldir:/var/lib/mysql"

volumes:
  cellsdir: {}
  mysqldir: {}

Everything works perfectly. But when I try to upload large files I get several errors. However I manage to upload a smaller file.

2020-03-31T00:18:02.500Z        INFO    pydio.grpc.data.sync.cellsdata  [batcher] Processing batch after timeout
2020-03-31T00:18:02.507Z        INFO    pydio.grpc.data.sync.cellsdata  Filtering TreePatch took        {"time": "7.51µs", "source": "s3://172.23.0.3:44453/cellsdata", "target": "index://cellsdata"}
2020-03-31T00:18:02.510Z        INFO    pydio.grpc.data.index.cellsdata CreateNode      {"request": "Node:<Uuid:\"befea5f5-8e8c-40fa-9611-3a003e4f3f5f\" Path:\"test\" Type:COLLECTION Size:36 MTime:1585613881 > "}
2020-03-31T00:18:02.523Z        INFO    pydio.grpc.data.sync.cellsdata  CreateNode      {"node": {"Uuid": "befea5f5-8e8c-40fa-9611-3a003e4f3f5f", "Path": "test", "MTime": "2020-03-31T00:18:01.000Z", "Size": 36}}
2020-03-31T00:18:02.523Z        INFO    pydio.grpc.data.sync.cellsdata  Created folder  {"path": "test", "target": "index://cellsdata"}
2020-03-31T00:18:02.526Z        INFO    pydio.grpc.data.index.cellsdata CreateNode      {"request": "Node:<Path:\"test/.pydio\" Type:LEAF Size:36 MTime:1585613881 Mode:511 Etag:\"79cb943a6ff8471134d9cca660d77bce\" > "}
2020-03-31T00:18:02.535Z        INFO    pydio.grpc.data.sync.cellsdata  CreateNode      {"node": {"Path": "test/.pydio", "Etag": "79cb943a6ff8471134d9cca660d77bce", "MTime": "2020-03-31T00:18:01.000Z", "Size": 36}}
2020-03-31T00:18:02.535Z        INFO    pydio.grpc.data.sync.cellsdata  Indexed file    {"path": "test/.pydio", "target": "index://cellsdata"}
2020-03-31T00:18:05.528Z        INFO    pydio.grpc.search       Flushing search batch   {"size": 1}
2020-03-31T00:18:16.830Z        INFO    pydio.grpc.tasks        Run Job internal-prune-jobs on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:03/PT10M"
2020-03-31T00:18:16.879Z        INFO    pydio.grpc.data.index.cellsdata CreateNode      {"request": "Node:<Path:\"test/VNC-Viewer-6.20.113-Windows.exe\" Type:LEAF Size:10559784 MTime:1585613896 Etag:\"temporary\" > "}
2020-03-31T00:18:18.108Z        INFO    pydio.grpc.data.sync.cellsdata  [batcher] Processing batch after timeout
2020-03-31T00:18:18.111Z        INFO    pydio.grpc.data.sync.cellsdata  Filtering TreePatch took        {"time": "11.698µs", "source": "s3://172.23.0.3:44453/cellsdata", "target": "index://cellsdata"}
2020-03-31T00:18:18.113Z        INFO    pydio.grpc.data.index.cellsdata CreateNode      {"request": "Node:<Uuid:\"ecb6ab05-2be1-41ca-a95f-998326c9bc04\" Path:\"test/VNC-Viewer-6.20.113-Windows.exe\" Type:LEAF Size:10559784 M
Time:1585613897 Mode:511 Etag:\"e9ecfbb45f941b6ca96e39782e36213a\" > UpdateIfExists:true "}
2020-03-31T00:18:18.121Z        INFO    pydio.grpc.data.sync.cellsdata  CreateNode      {"node": {"Uuid": "ecb6ab05-2be1-41ca-a95f-998326c9bc04", "Path": "test/VNC-Viewer-6.20.113-Windows.exe", "Etag": "e9ecfbb45f941b6ca96e3
9782e36213a", "MTime": "2020-03-31T00:18:17.000Z", "Size": 10559784}}
2020-03-31T00:18:18.121Z        INFO    pydio.grpc.data.sync.cellsdata  Indexed file    {"path": "test/VNC-Viewer-6.20.113-Windows.exe", "target": "index://cellsdata"}
2020-03-31T00:18:19.890Z        INFO    pydio.grpc.search       Flushing search batch   {"size": 1}
2020-03-31T00:18:53.566Z        INFO    pydio.grpc.data.index.cellsdata CreateNode      {"request": "Node:<Path:\"test/SLI_3.0.0_A00.iso\" Type:LEAF Size:1882193920 MTime:1585613933 Etag:\"temporary\" > "}
2020-03-31T00:18:53.658Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a
8f: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a8f. Retry again."}
2020-03-31T00:18:53.982Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a
8f: context canceled"}
2020-03-31T00:18:53.982Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=1&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a
8f: context canceled"}
2020-03-31T00:18:54.085Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a
8f: context canceled"}
2020-03-31T00:18:54.136Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=1&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a
8f: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=1&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a8f. Retry again."}
2020-03-31T00:18:54.174Z        INFO    pydio.gateway.data      Received MultipartAbort - Clean temporary node: {"Node": {"Uuid": "0753a3e7-abd9-4734-83b5-ea828d60dc2c", "Path": "cellsdata/test/SLI_3.0.0_A00.iso", "Etag": "t
emporary", "MTime": "2020-03-31T00:18:53.000Z", "Size": 1882193920, "MetaStore": {"name":"\"SLI_3.0.0_A00.iso\"","pydio:meta-data-source-name":"\"cellsdata\"","pydio:meta-data-source-path":"\"test/SLI_3.0.0_A00.iso\""}}}
2020-03-31T00:18:54.302Z        INFO    pydio.grpc.data.index.cellsdata CreateNode      {"request": "Node:<Path:\"test/SLI_3.0.0_A00.iso\" Type:LEAF Size:1882193920 MTime:1585613934 Etag:\"temporary\" > "}
2020-03-31T00:18:54.329Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a
8f: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a8f. Retry again."}
2020-03-31T00:18:54.407Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a
8f: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a8f. Retry again."}
2020-03-31T00:18:54.669Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=1&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: context canceled"}
2020-03-31T00:18:54.670Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: context canceled"}
2020-03-31T00:18:54.703Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a
8f: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=d5708d29-01cf-4992-a92f-7c6f52e12a8f. Retry again."}
2020-03-31T00:18:54.786Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: context canceled"}
2020-03-31T00:18:54.787Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=1&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: context canceled"}
2020-03-31T00:18:54.787Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: context canceled"}
2020-03-31T00:18:54.951Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=16aa3fdf-0879-424c-9d32-ee9863174ed0. Retry again."}
2020-03-31T00:18:55.084Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=1&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: context canceled"}
2020-03-31T00:18:55.108Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=16aa3fdf-0879-424c-9d32-ee9863174ed0. Retry again."}
2020-03-31T00:18:55.353Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=16aa3fdf-0879-424c-9d32-ee9863174ed0. Retry again."}
2020-03-31T00:18:55.626Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=1&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: context canceled"}
2020-03-31T00:18:55.648Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=3&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: context canceled"}
2020-03-31T00:18:55.675Z        ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=16aa3fdf-0879-424c-9d32-ee9863174e
d0: Connection closed by foreign host http://172.23.0.3:44453/cellsdata/test/SLI_3.0.0_A00.iso?partNumber=2&uploadId=16aa3fdf-0879-424c-9d32-ee9863174ed0. Retry again."}
2020-03-31T00:18:55.683Z        INFO    pydio.gateway.data      Received MultipartAbort - Clean temporary node: {"Node": {"Uuid": "a99a4b36-0cb7-4996-b4fc-bc4287029893", "Path": "cellsdata/test/SLI_3.0.0_A00.iso", "Etag": "t
emporary", "MTime": "2020-03-31T00:18:54.000Z", "Size": 1882193920, "MetaStore": {"name":"\"SLI_3.0.0_A00.iso\"","pydio:meta-data-source-name":"\"cellsdata\"","pydio:meta-data-source-path":"\"test/SLI_3.0.0_A00.iso\""}}}
2020-03-31T00:18:58.694Z        INFO    pydio.grpc.search       Flushing search batch   {"size": 2}
2

I also try with a simple container on docker host network.

3

I also try by installing directly on a linux box. I also see the error on ulimit and I ulimit -n 8192. I don’t know what I can do to get it work.

4

Hello @metabsd,

Could you tell me the size of the file that you are attempting to upload ?
And how do you upload the file, webui or a client ?

5

I succeeded with a 50m file. By cons it does not work with a file of 1g or 2g. I also tried with a 20g file without success. I need to use it with the web interface. Thank you!!

6

If you want to upload files via a command line tool you can make use of the cells-client, https://download.pydio.com/pub/cells-client/release/2.0.2/

7

You want me to upload file via the client ? The result is not the same as the WebUI ?

8

Any potential solution on my problem ? I’m not able to use pydio to upload file more than 500meg. Maybe I have to change a parameter or something else. Please help :slight_smile:

9

Did you by any chance, modify the settings in Cells Console >> Uploaders.

Could you please reproduce your issue and show me the console logs of your browser, that is weird that you cannot upload files bigger than 50mb (which actually is the default part size for multi part uploads).

10

image
image2217×701 45.6 KB

11

The log on the server. I see a lot of fail.

ERROR   pydio.gateway.data      PutObjectPart has failed        {"error": "Put http://172.23.0.3:40007/pydiods1/ISO/myfile.iso?partNumber=14&uploadId=fb9cd1c7
-3a35-49e8-ba6c-abd6cf2e2a33: Connection closed by foreign host http://172.23.0.3:40007/pydiods1/ISO/myfile.iso?partNumber=14&uploadId=fb9cd1c7-3a35-49e8-ba6c-abd6cf2e2a33. Retry again."}

I see this in the web console of Firefox

image
image2558×503 42.8 KB

I will install the client and try with the client to compare.

12

I confirm that with the client this is working really smooth. No error in the server log.

13

What is the difference between cli and webui ?

14

I am getting similar error on Pydio cells version 3 docker image. I am still trying to upload 125GiB file to the server and getting constant failure with PutObjectPart.

Interestingly, the error shown on browser was kind of 403 error.