Upload failures with CEC

For the past week I have made attempts to migrate from Dropbox (170GB) to Cells. Using Cells sync client has lead to multiple failures. cec was used as an alternative and seemingly works in some cases and not in others. Amongst the folders being copied over is a Cryptomator based vault (6GB+) it contains 1000’s of files and folders due to Cryptomators security architecture. cec fails to upload this folder with an error:

  0% [--------------------------------------------------------------------] Transfering 1/310344 files or folders
2022/03/17 19:44:16 Post "https://cells.matekole.com/a/tree/create": context deadline exceeded

One more time, it would help if you could give more details.

And please do not post also in github isues until you are suggested to do so by one of us, thanks.

@bsinou I cannot give you any other details other than the command that was ran:
cec scp [redacted] [redacted]

It appears this issue is related to this directory alone and the only distinguishing factor is the directory/file structure is very large due to Cryptomator’s security model. And yes Cryptomator was disabled during these attempts. I have ran similar setup with Dropbox for 6+ years and have had no issue with it.

The issue might be related to my installation but then that does not explain why some folders/files successfully upload and others(Cryptomator vault) don’t. I might remove S3 as primary storage and use disk only, I have had one issue there in that Cells is unable to communicate with Wasabi and mount the S3 stores after restarts.

There are no errors being reported on the server side at the time of running that command.
Furthermore, it appears changing the verbosity in Cells via docker swarm config has no effect.
At least there are no additional messages in the standard output.

Apologies for breaking rules regarding creation of Github issues. I did not come across any procedures/rules that suggest issues should go through triage first.

~victor

@bsinou Same error. Steps to reproduce:

  1. Fresh installation (Docker)
  2. Use local FS instead of S3 as primary storage
  3. Added encryption key to datastores(Personal/Common/Thumbnails)
  4. Authenticated cec and ran scp command to upload ‘vault’ directory.
  5. Directory structure is created in Cells, however, cec fails with the following error:
  0% [--------------------------------------------------------------------] Transfering 1/310344 files or folders
2022/03/18 11:18:02 Post "https://cells.matekole.com/a/tree/create": context deadline exceeded

Logs on server side:

baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:05:02.710Z	INFO	pydio.gateway.proxy	Restart done
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:05:05.206Z	INFO	pydio.rest.workspace	Creating a Personal workspace
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:05:05.230Z	INFO	pydio.rest.workspace	Settings ACLS for workspace
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:05:05.363Z	INFO	pydio.rest.workspace	Creating a Common Files workspace on pydiods1
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:05:05.396Z	INFO	pydio.rest.workspace	Settings ACLS for workspace
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:05:16.829Z	INFO	pydio.grpc.tasks	Run Job flush-mailer-queue on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT5M" 
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:05:29.270Z	INFO	pydio.grpc.search	Flushing search batch	{"size": 1}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:08:16.829Z	INFO	pydio.grpc.tasks	Run Job internal-prune-jobs on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:03/PT10M" 
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:10:16.829Z	INFO	pydio.grpc.tasks	Run Job users-activity-digest on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT15M" 
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:10:16.830Z	INFO	pydio.grpc.tasks	Run Job flush-mailer-queue on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT5M" 
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:13:23.611Z	ERROR	pydio.rest.config	Rest Error 500	{"error": "context canceled"}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:13:26.480Z	INFO	pydio.rest.config	Checking parent folder is writable before creating datasource	{"ds": {"Name": "personal", "Bucket": "personal", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:13:34.421Z	INFO	pydio.grpc.data.objects	Got an event on sources keys for pydio.grpc.data.objects. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:13:34.421Z	INFO	pydio.grpc.data.index	Got an event on sources keys for pydio.grpc.data.index. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:13:34.421Z	INFO	pydio.grpc.data.sync	Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:13:34.954Z	INFO	pydio.grpc.data.sync.personal	Versioning policy changed on pydio.grpc.data.sync.personal, updating internal config	{"cfg": {"Name": "personal", "Bucket": "personal", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:13:44.422Z	INFO	pydio.grpc.data.sync	Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:13:59.683Z	INFO	pydio.rest.config	Checking parent folder is writable before creating datasource	{"ds": {"Name": "thumbnails", "Bucket": "thumbs", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:04.422Z	INFO	pydio.grpc.data.objects	Got an event on sources keys for pydio.grpc.data.objects. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:04.422Z	INFO	pydio.grpc.data.index	Got an event on sources keys for pydio.grpc.data.index. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:04.422Z	INFO	pydio.grpc.data.sync	Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:04.948Z	INFO	pydio.grpc.data.sync.thumbnails	Versioning policy changed on pydio.grpc.data.sync.thumbnails, updating internal config	{"cfg": {"Name": "thumbnails", "Bucket": "thumbs", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:14.421Z	INFO	pydio.grpc.data.sync	Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:14.840Z	INFO	pydio.rest.config	Checking parent folder is writable before creating datasource	{"ds": {"Name": "pydiods1", "Bucket": "pydiods1", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:14.861Z	INFO	pydio.grpc.data.sync.pydiods1	Versioning policy changed on pydio.grpc.data.sync.pydiods1, updating internal config	{"cfg": {"Name": "pydiods1", "Bucket": "pydiods1", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:24.421Z	INFO	pydio.grpc.data.objects	Got an event on sources keys for pydio.grpc.data.objects. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:24.421Z	INFO	pydio.grpc.data.sync	Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:14:24.421Z	INFO	pydio.grpc.data.index	Got an event on sources keys for pydio.grpc.data.index. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:15:16.829Z	INFO	pydio.grpc.tasks	Run Job flush-mailer-queue on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT5M" 
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:17:32.691Z	WARN	pydio.rest.tree	Rest Error 404	{"error": "{\"id\":\"pydio.grpc.data.index.personal\",\"code\":404,\"detail\":\"Could not retrieve node /icevault/Vault\",\"status\":\"Not Found\"}"}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:18:02.772Z	ERROR	pydio.rest.tree	Rest Error 500	{"error": "{\"id\":\"go.micro.client\",\"code\":408,\"detail\":\"context canceled - Request was NodeReceiver.CreateNode on pydio.grpc.tree - Micro-registry had node(s) : [10.0.4.17:42249]\",\"status\":\"Request Timeout\"}"}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:18:05.786Z	INFO	pydio.grpc.search	Flushing search batch	{"size": 937}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:18:16.829Z	INFO	pydio.grpc.tasks	Run Job internal-prune-jobs on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:03/PT10M" 
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud    | 2022-03-18T10:20:16.829Z	INFO	pydio.grpc.tasks	Run Job flush-mailer-queue on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT5M"

Unfortunately, upload errors related to synchronisation of 5GB+ Cryptomater vault persisted and generally quite slow uploads, even after several clean re-installs. The architecture of Cells looks very promising and offers the best in class in FOSS for private cloud IMHO. Unfortunately, it probably needs some maturity before it can take on use cases such as mine:

  1. 150GB+ storage
  2. Encrypted at rest
  3. Persistence of 5GB+ Cryptomator vaults

hey @vmatekole
I guess cec is currently not optimized for such operation (creating tons of folders at once (> 100k?) ) during a recursive upload. cells-sync should probably be better at that, maybe you should have a try !

Hi Charles,

I initially used cells-sync but that too also failed. After testing several paid and opensource solutions it appears Dropbox is the most stable and performant, even TresorIt underperformed on some evaluations .

For now I will move to a NAS solution and host everything at home, not an ideal solution but convenient for now.