For the past week I have made attempts to migrate from Dropbox (170GB) to Cells. Using Cells sync client has lead to multiple failures. cec was used as an alternative and seemingly works in some cases and not in others. Amongst the folders being copied over is a Cryptomator based vault (6GB+) it contains 1000’s of files and folders due to Cryptomators security architecture. cec fails to upload this folder with an error:
0% [--------------------------------------------------------------------] Transfering 1/310344 files or folders
2022/03/17 19:44:16 Post "https://cells.matekole.com/a/tree/create": context deadline exceeded
bsinou
March 18, 2022, 7:44am
2
One more time, it would help if you could give more details.
And please do not post also in github isues until you are suggested to do so by one of us, thanks.
@bsinou I cannot give you any other details other than the command that was ran:cec scp [redacted] [redacted]
It appears this issue is related to this directory alone and the only distinguishing factor is the directory/file structure is very large due to Cryptomator’s security model. And yes Cryptomator was disabled during these attempts. I have ran similar setup with Dropbox for 6+ years and have had no issue with it.
The issue might be related to my installation but then that does not explain why some folders/files successfully upload and others(Cryptomator vault) don’t. I might remove S3 as primary storage and use disk only, I have had one issue there in that Cells is unable to communicate with Wasabi and mount the S3 stores after restarts.
There are no errors being reported on the server side at the time of running that command.
Apologies for breaking rules regarding creation of Github issues. I did not come across any procedures/rules that suggest issues should go through triage first.
~victor
@bsinou Same error. Steps to reproduce:
Fresh installation (Docker)
Use local FS instead of S3 as primary storage
Added encryption key to datastores(Personal/Common/Thumbnails)
Authenticated cec and ran scp command to upload ‘vault’ directory.
Directory structure is created in Cells, however, cec fails with the following error:
0% [--------------------------------------------------------------------] Transfering 1/310344 files or folders
2022/03/18 11:18:02 Post "https://cells.matekole.com/a/tree/create": context deadline exceeded
Logs on server side:
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:05:02.710Z INFO pydio.gateway.proxy Restart done
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:05:05.206Z INFO pydio.rest.workspace Creating a Personal workspace
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:05:05.230Z INFO pydio.rest.workspace Settings ACLS for workspace
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:05:05.363Z INFO pydio.rest.workspace Creating a Common Files workspace on pydiods1
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:05:05.396Z INFO pydio.rest.workspace Settings ACLS for workspace
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:05:16.829Z INFO pydio.grpc.tasks Run Job flush-mailer-queue on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT5M"
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:05:29.270Z INFO pydio.grpc.search Flushing search batch {"size": 1}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:08:16.829Z INFO pydio.grpc.tasks Run Job internal-prune-jobs on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:03/PT10M"
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:10:16.829Z INFO pydio.grpc.tasks Run Job users-activity-digest on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT15M"
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:10:16.830Z INFO pydio.grpc.tasks Run Job flush-mailer-queue on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT5M"
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:13:23.611Z ERROR pydio.rest.config Rest Error 500 {"error": "context canceled"}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:13:26.480Z INFO pydio.rest.config Checking parent folder is writable before creating datasource {"ds": {"Name": "personal", "Bucket": "personal", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:13:34.421Z INFO pydio.grpc.data.objects Got an event on sources keys for pydio.grpc.data.objects. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:13:34.421Z INFO pydio.grpc.data.index Got an event on sources keys for pydio.grpc.data.index. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:13:34.421Z INFO pydio.grpc.data.sync Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:13:34.954Z INFO pydio.grpc.data.sync.personal Versioning policy changed on pydio.grpc.data.sync.personal, updating internal config {"cfg": {"Name": "personal", "Bucket": "personal", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:13:44.422Z INFO pydio.grpc.data.sync Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:13:59.683Z INFO pydio.rest.config Checking parent folder is writable before creating datasource {"ds": {"Name": "thumbnails", "Bucket": "thumbs", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:04.422Z INFO pydio.grpc.data.objects Got an event on sources keys for pydio.grpc.data.objects. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:04.422Z INFO pydio.grpc.data.index Got an event on sources keys for pydio.grpc.data.index. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:04.422Z INFO pydio.grpc.data.sync Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:04.948Z INFO pydio.grpc.data.sync.thumbnails Versioning policy changed on pydio.grpc.data.sync.thumbnails, updating internal config {"cfg": {"Name": "thumbnails", "Bucket": "thumbs", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:14.421Z INFO pydio.grpc.data.sync Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:14.840Z INFO pydio.rest.config Checking parent folder is writable before creating datasource {"ds": {"Name": "pydiods1", "Bucket": "pydiods1", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:14.861Z INFO pydio.grpc.data.sync.pydiods1 Versioning policy changed on pydio.grpc.data.sync.pydiods1, updating internal config {"cfg": {"Name": "pydiods1", "Bucket": "pydiods1", "StorageType": "LOCAL", "EncryptionMode": "MASTER"}}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:24.421Z INFO pydio.grpc.data.objects Got an event on sources keys for pydio.grpc.data.objects. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:24.421Z INFO pydio.grpc.data.sync Got an event on sources keys for pydio.grpc.data.sync. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:14:24.421Z INFO pydio.grpc.data.index Got an event on sources keys for pydio.grpc.data.index. Let's start/stop services accordingly
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:15:16.829Z INFO pydio.grpc.tasks Run Job flush-mailer-queue on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT5M"
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:17:32.691Z WARN pydio.rest.tree Rest Error 404 {"error": "{\"id\":\"pydio.grpc.data.index.personal\",\"code\":404,\"detail\":\"Could not retrieve node /icevault/Vault\",\"status\":\"Not Found\"}"}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:18:02.772Z ERROR pydio.rest.tree Rest Error 500 {"error": "{\"id\":\"go.micro.client\",\"code\":408,\"detail\":\"context canceled - Request was NodeReceiver.CreateNode on pydio.grpc.tree - Micro-registry had node(s) : [10.0.4.17:42249]\",\"status\":\"Request Timeout\"}"}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:18:05.786Z INFO pydio.grpc.search Flushing search batch {"size": 937}
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:18:16.829Z INFO pydio.grpc.tasks Run Job internal-prune-jobs on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:03/PT10M"
baobab_shop_cells.1.vjlvki0cu3fu@Ubuntu-1804-bionic-64-nextcloud | 2022-03-18T10:20:16.829Z INFO pydio.grpc.tasks Run Job flush-mailer-queue on timer event Iso8601Schedule:"R/2012-06-04T19:25:16.828696-07:00/PT5M"
Unfortunately, upload errors related to synchronisation of 5GB+ Cryptomater vault persisted and generally quite slow uploads, even after several clean re-installs. The architecture of Cells looks very promising and offers the best in class in FOSS for private cloud IMHO. Unfortunately, it probably needs some maturity before it can take on use cases such as mine:
150GB+ storage
Encrypted at rest
Persistence of 5GB+ Cryptomator vaults
hey @vmatekole cec is currently not optimized for such operation (creating tons of folders at once (> 100k?) ) during a recursive upload. cells-sync should probably be better at that, maybe you should have a try !
Hi Charles,
I initially used cells-sync but that too also failed. After testing several paid and opensource solutions it appears Dropbox is the most stable and performant, even TresorIt underperformed on some evaluations .
For now I will move to a NAS solution and host everything at home, not an ideal solution but convenient for now.
system
June 17, 2022, 11:59am
8
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.