Upload failed: undefined

Hello,
I am trying to run Cells on Ubuntu 18.04.5. I’m accessing the server through SSH and have no physical access to it.
I’ve installed Cells and it’s running as a service. I see the UI from the browser and I can use it. When I try to upload a file from my computer I get

When I try to upload this line is added twice to the caddy_errors.log:
[ERROR 502 /io/common-files/Testfolder/Test_file.odt] dial tcp: lookup PENDING on x.x.x.53:53: server misbehaving (x=redacted). I thought that the DNS was giving me trouble (port 53 right?) so I switched to the google one. Any ideas?

Thanks,

Ben

Hi @benw

This indicates you have some “PENDING” directive in the caddy file that is dynamically generated to configure the internal gateway (see our glossary )

This usually means that one of the microservices cannot be found and that the internal gateway restarts in loop.

It does not give much hints about the root issue, (but most of the time this is linked to a lack of resources).

Could you give more details about your setup ?
Could you have a look at the main pydio log and tell us what you see there ?

Hi,
thanks for the response.
RE setup, this is the output of lscpu, as far as I know it should be strong enough. ulimit -n is set at 8192 both soft and hard:

Architektur:             x86_64
CPU Operationsmodus:     32-bit, 64-bit
Byte-Reihenfolge:        Little Endian
CPU(s):                  8
Liste der Online-CPU(s): 0-7
Thread(s) pro Kern:      1
Kern(e) pro Socket:      8
Sockel:                  1
Anbieterkennung:         GenuineIntel
Prozessorfamilie:        6
Modell:                  63
Modellname:              Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping:                2
CPU MHz:                 1999.816
BogoMIPS:                4994.44
Virtualisierung:         VT-x
Hypervisor-Anbieter:     Parallels
Virtualisierungstyp:     Container
L1d Cache:               32K
L1i Cache:               32K
L2 Cache:                256K
L3 Cache:                30720K

As I wrote above it’s Ubuntu 18.04.5 LTS

I looked into the log and it looks like you’re right and it’s a caddy problem. What’s the best way to approach it? If it’s relevant, I’m having problem setting up SSL and the reverse proxy, I’ll tackle that once I have the basics.

{"level":"info","ts":"2021-02-15T11:34:11+01:00","logger":"pydio.gateway.proxy","msg":"Restart done"}

{"level":"info","ts":"2021-02-15T11:34:11+01:00","logger":"pydio.gateway.proxy","msg":"Restarting proxy","caddyfile":"\n\n\n\n\n\n\n0.0.0.0:8080 {\n\t\n\n\t\n\n\tproxy /a  xx.xxx.xxx.xx:36073 {\n\t\twit$

Thanks,

Ben

Hi @benw
Re-reading your thread, upload probably fails because of the Gateway not being able to forward the PUT request to the object service that is served by the datasource.
Can you screenshot your DataSource settings page ? Do you have any “service down” here ?
Otherwise, the best info you could provide would be your “start log” : stop cells, clean your command line, then start and wait for the “Restarting Proxy” log. Copy all the log until that line.
Charles

Hi,

all the data sources services are up, but pydio.gateway.proxy, pydio.gateway.data. pydio.gateway.grpc are down. The log says that it’s because there’s no private ip configured. I looked at the troubleshooting steps here and I have an interface configured under another name and with 127.0.0.1 but I still get the error. Should I configure the internal ip address somewhere? Maybe I made a mistake in the setup?
Is this the screenshot that you mean? (I cut the right column with my external ip)
Thanks again,

Ben

Hi,
I played around a bit more with the interface and understood what you meant, so here it is. I created the buvo datasource. I guess it doesn’t look very encouraging.

Here is the start log until the restart, I removed some lines showing hex addresses, it was too long for a comment:

Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:48.394+0100	INFO	pydio.gateway.rest	started
2021-02-19T10:10:48.394+0100	INFO	pydio.grpc.update	started
2021-02-19T10:10:48.394+0100	INFO	pydio.grpc.data.objects	started
2021-02-19T10:10:48.394+0100	INFO	pydio.grpc.data.objects	Starting umbrella service pydio.grpc.data.objects. with sources	{"sources": ["local1"]}
2021-02-19T10:10:48.394+0100	ERROR	pydio.gateway.data	Could not run 	{"error": "No private IP address found, and explicit IP not provided"}
2021-02-19T10:10:48.413+0100	INFO	pydio.grpc.statics	started
2021-02-19T10:10:48.413+0100	INFO	pydio.grpc.data.sync	started
2021-02-19T10:10:48.413+0100	INFO	pydio.grpc.data.sync	Starting umbrella service pydio.grpc.data.sync. with sources	{"sources": ["buvo", "pydiods1", "personal", "cellsdata"]}
2021-02-19T10:10:48.414+0100	INFO	pydio.grpc.docstore	started
2021-02-19T10:10:48.436+0100	INFO	pydio.grpc.log	started
2021-02-19T10:10:48.438+0100	INFO	pydio.grpc.user-key	started
2021-02-19T10:10:48.450+0100	INFO	pydio.grpc.acl	started
2021-02-19T10:10:48.450+0100	INFO	pydio.grpc.meta	started
2021-02-19T10:10:48.451+0100	INFO	pydio.grpc.config	started
2021-02-19T10:10:48.452+0100	INFO	pydio.grpc.token	started
2021-02-19T10:10:48.456+0100	INFO	pydio.grpc.chat	started
2021-02-19T10:10:48.457+0100	INFO	pydio.grpc.policy	started
2021-02-19T10:10:48.462+0100	INFO	pydio.grpc.user-meta	started
2021-02-19T10:10:48.484+0100	INFO	pydio.grpc.oauth	started
Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:48.676+0100	INFO	pydio.grpc.data.objects.local1	Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:48.678+0100	INFO	pydio.grpc.data.sync.pydiods1	Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:48.696+0100	INFO	pydio.grpc.data.sync.buvo	Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:48.727+0100	INFO	pydio.grpc.data.objects.local1	started
2021-02-19T10:10:48.730+0100	INFO	pydio.grpc.data.sync.personal	Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:48.730+0100	INFO	pydio.grpc.data.objects.local1	Starting local objects service local1 on /var/cells/data
2021-02-19T10:10:48.746+0100	INFO	pydio.grpc.jobs	started
2021-02-19T10:10:48.756+0100	INFO	pydio.grpc.jobs	Setting task 0a717535-e63a-4be2-8da8-9ab8c8a0d43f in error status as it was saved as running
2021-02-19T10:10:48.758+0100	INFO	pydio.grpc.data.sync.cellsdata	Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:48.761+0100	INFO	pydio.grpc.data.sync.pydiods1	Sync pydiods1 - Try to contact Objects
2021-02-19T10:10:48.775+0100	INFO	pydio.grpc.data.sync.buvo	Sync buvo - Try to contact Objects
2021-02-19T10:10:48.779+0100	INFO	pydio.grpc.data.sync.personal	Sync personal - Try to contact Objects
2021-02-19T10:10:48.817+0100	INFO	pydio.grpc.data.sync.cellsdata	Sync cellsdata - Try to contact Objects
2021-02-19T10:10:48.865+0100	INFO	pydio.grpc.data.sync.pydiods1	Successfully listed objects from bucket pydiods1
2021-02-19T10:10:48.878+0100	INFO	pydio.grpc.data.sync.buvo	Successfully listed objects from bucket buvo
2021-02-19T10:10:48.882+0100	INFO	pydio.grpc.data.sync.personal	Successfully listed objects from bucket personal
2021-02-19T10:10:49.020+0100	INFO	pydio.grpc.data.sync.cellsdata	Successfully listed objects from bucket cellsdata
2021-02-19T10:10:49.319+0100	INFO	Activating privacy features... done.
2021-02-19T10:10:49.319+0100	INFO	https://<redacted ip>:8080
2021-02-19T10:10:49.320+0100	INFO	pydio.gateway.proxy	started
2021-02-19T10:10:50.393+0100	INFO	pydio.grpc.mailer	Starting mailer with queue 'boltdb'
2021-02-19T10:10:50.394+0100	INFO	pydio.grpc.mailer	Starting mailer with sender 'smtp'
2021-02-19T10:10:50.394+0100	INFO	pydio.grpc.mailer	started
2021-02-19T10:10:50.396+0100	INFO	pydio.grpc.tree	started
2021-02-19T10:10:50.397+0100	INFO	pydio.grpc.data-key	started
2021-02-19T10:10:50.397+0100	INFO	pydio.grpc.workspace	started
2021-02-19T10:10:50.399+0100	INFO	pydio.grpc.role	started
2021-02-19T10:10:50.417+0100	INFO	pydio.grpc.user	started
2021-02-19T10:10:50.502+0100	INFO	pydio.grpc.versions	started
2021-02-19T10:10:50.703+0100	INFO	pydio.gateway.websocket	started
2021-02-19T10:10:50.769+0100	INFO	pydio.grpc.search	started
2021-02-19T10:10:50.771+0100	INFO	pydio.grpc.tasks	started
2021-02-19T10:10:52.395+0100	INFO	pydio.grpc.activity	started
2021-02-19T10:10:52.396+0100	INFO	pydio.grpc.data.index	started
2021-02-19T10:10:52.397+0100	INFO	pydio.grpc.data.index	Starting umbrella service pydio.grpc.data.index. with sources	{"sources": ["buvo", "pydiods1", "personal", "cellsdata"]}
2021-02-19T10:10:52.398+0100	INFO	pydio.web.statics	started
2021-02-19T10:10:52.408+0100	INFO	pydio.test.objects	started
2021-02-19T10:10:52.408+0100	INFO	pydio.gateway.wopi	started
2021-02-19T10:10:52.411+0100	INFO	pydio.gateway.dav	started
2021-02-19T10:10:52.415+0100	INFO	pydio.grpc.timer	Registering Job	{"job": "actions.auth.prune.tokens"}
2021-02-19T10:10:52.416+0100	INFO	pydio.grpc.timer	Registering Job	{"job": "flush-mailer-queue"}
2021-02-19T10:10:52.416+0100	INFO	pydio.grpc.timer	Registering Job	{"job": "internal-prune-jobs"}
2021-02-19T10:10:52.416+0100	INFO	pydio.grpc.timer	Registering Job	{"job": "prune-versions-job"}
2021-02-19T10:10:52.416+0100	INFO	pydio.grpc.timer	Registering Job	{"job": "users-activity-digest"}
2021-02-19T10:10:52.417+0100	INFO	pydio.grpc.timer	started
2021-02-19T10:10:52.420+0100	INFO	pydio.web.oauth	started
2021-02-19T10:10:52.560+0100	INFO	pydio.rest.meta	started
2021-02-19T10:10:52.560+0100	INFO	pydio.rest.mailer	started
2021-02-19T10:10:52.560+0100	INFO	pydio.rest.update	started
2021-02-19T10:10:52.561+0100	INFO	pydio.rest.graph	started
2021-02-19T10:10:52.561+0100	INFO	pydio.rest.search	started
2021-02-19T10:10:52.561+0100	INFO	pydio.rest.role	started
2021-02-19T10:10:52.562+0100	INFO	pydio.rest.tree	started
2021-02-19T10:10:52.562+0100	INFO	pydio.rest.templates	started
2021-02-19T10:10:52.562+0100	INFO	pydio.rest.log	started
2021-02-19T10:10:52.563+0100	INFO	pydio.rest.share	started
2021-02-19T10:10:52.564+0100	INFO	pydio.rest.config	started
2021-02-19T10:10:52.564+0100	INFO	pydio.rest.user	started
2021-02-19T10:10:52.564+0100	INFO	pydio.rest.workspace	started
2021-02-19T10:10:52.565+0100	INFO	pydio.rest.user-meta	started
2021-02-19T10:10:52.565+0100	INFO	pydio.rest.jobs	started
2021-02-19T10:10:52.565+0100	INFO	pydio.rest.acl	started
2021-02-19T10:10:52.566+0100	INFO	pydio.rest.auth	started
2021-02-19T10:10:52.571+0100	INFO	pydio.rest.policy	started
2021-02-19T10:10:52.660+0100	ERROR	pydio.grpc.data.index.pydiods1	runtime/cgo: pthread_create failed: Resource temporarily unavailable
2021-02-19T10:10:52.661+0100	ERROR	pydio.grpc.data.index.personal	runtime/cgo: pthread_create failed: Resource temporarily unavailable
2021-02-19T10:10:52.663+0100	ERROR	pydio.grpc.data.index.personal	SIGABRT: abort
2021-02-19T10:10:52.665+0100	ERROR	pydio.grpc.data.index.personal	PC=0x7fed66694fb7 m=0 sigcode=18446744073709551610
2021-02-19T10:10:52.666+0100	ERROR	pydio.grpc.data.index.personal	
2021-02-19T10:10:52.666+0100	ERROR	pydio.grpc.data.index.personal	goroutine 0 [idle]:
2021-02-19T10:10:52.666+0100	ERROR	pydio.grpc.data.index.personal	runtime: unknown pc 0x7fed66694fb7
2021-02-19T10:10:52.666+0100	ERROR	pydio.grpc.data.index.personal	stack: frame={sp:0x7ffd01139610, fp:0x0} stack=[0x7ffd0093acb8,0x7ffd01139cf0)
<Error in personal with hex>
2021-02-19T10:10:52.669+0100	ERROR	pydio.grpc.data.index.personal	runtime: unknown pc 0x7fed66694fb7
2021-02-19T10:10:52.669+0100	ERROR	pydio.grpc.data.index.personal	stack: frame={sp:0x7ffd01139610, fp:0x0} stack=[0x7ffd0093acb8,0x7ffd01139cf0)
<Error in personal with hex>
2021-02-19T10:10:52.676+0100	ERROR	pydio.grpc.data.index.personal	
2021-02-19T10:10:52.676+0100	ERROR	pydio.grpc.data.index.personal	goroutine 1 [runnable, locked to thread]:
2021-02-19T10:10:52.676+0100	ERROR	pydio.grpc.data.index.personal	runtime.gopark(0x7d9cd18, 0xc0000560c8, 0x170e, 0x2)
2021-02-19T10:10:52.676+0100	ERROR	pydio.grpc.data.index.personal		runtime/proc.go:306 +0xe5
2021-02-19T10:10:52.677+0100	ERROR	pydio.grpc.data.index.personal	runtime.chanrecv(0xc000056070, 0x0, 0xc000000101, 0x1780101)
2021-02-19T10:10:52.677+0100	ERROR	pydio.grpc.data.index.personal		runtime
2021-02-19T10:10:52.677+0100	ERROR	pydio.grpc.data.index.pydiods1	SubProcess was not killed properly: exit status 2
2021-02-19T10:10:52.677+0100	ERROR	pydio.grpc.data.index.pydiods1	Restarting service in 3s...
2021-02-19T10:10:52.666+0100	ERROR	pydio.grpc.data.index.personal	SubProcess was not killed properly: exit status 2
2021-02-19T10:10:52.678+0100	ERROR	pydio.grpc.data.index.personal	Restarting service in 3s...
2021-02-19T10:10:52.671+0100	ERROR	pydio.grpc.data.index.pydiods1	SIGABRT: abort
2021-02-19T10:10:52.678+0100	ERROR	pydio.grpc.data.index.pydiods1	PC=0x7fec4cf1cfb7 m=5 sigcode=18446744073709551610
2021-02-19T10:10:52.678+0100	ERROR	pydio.grpc.data.index.pydiods1	
2021-02-19T10:10:52.679+0100	ERROR	pydio.grpc.data.index.pydiods1	goroutine 0 [idle]:
2021-02-19T10:10:52.679+0100	ERROR	pydio.grpc.data.index.pydiods1	runtime: unknown pc 0x7fec4cf1cfb7
2021-02-19T10:10:52.679+0100	ERROR	pydio.grpc.data.index.pydiods1	stack: frame={sp:0x7fec24bc87f0, fp:0x0} stack=[0x7fec243c9288,0x7fec24bc8e88)
2021-02-19T10:10:52.679+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc86f0:  00007fec24bc8b80  00007fecffffffff 
2021-02-19T10:10:52.679+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8700:  00000000017c2b29 <runtime.pcvalue+361>  0000000000000000 
2021-02-19T10:10:52.679+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8710:  00000000017ddac0 <runtime.goexit+0>  00000000013b496a 
<Error in pydios1 with hex>
2021-02-19T10:10:52.682+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8820:  00007fec24bc8af0  00000000017cacb5 <runtime.gentraceback+4501> 
2021-02-19T10:10:52.682+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8830:  00007fec24bc8a98  0000000000000000 
2021-02-19T10:10:52.682+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8840:  00000000017dda01 <runtime.memhash64+1>  00007fec24bc8910 
<Error in pydios1 with hex>
2021-02-19T10:10:52.683+0100	ERROR	pydio.grpc.data.index.pydiods1	runtime: unknown pc 0x7fec4cf1cfb7
2021-02-19T10:10:52.683+0100	ERROR	pydio.grpc.data.index.pydiods1	stack: frame={sp:0x7fec24bc87f0, fp:0x0} stack=[0x7fec243c9288,0x7fec24bc8e88)
2021-02-19T10:10:52.683+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc86f0:  00007fec24bc8b80  00007fecffffffff 
2021-02-19T10:10:52.683+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8700:  00000000017c2b29 <runtime.pcvalue+361>  0000000000000000 
2021-02-19T10:10:52.683+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8710:  00000000017ddac0 <runtime.goexit+0>  00000000013b496a 
<Error in pydios1 with hex>
2021-02-19T10:10:52.685+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8820:  00007fec24bc8af0  00000000017cacb5 <runtime.gentraceback+4501> 
2021-02-19T10:10:52.685+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8830:  00007fec24bc8a98  0000000000000000 
2021-02-19T10:10:52.685+0100	ERROR	pydio.grpc.data.index.pydiods1	00007fec24bc8840:  00000000017dda01 <runtime.memhash64+1>  00007fec24bc8910 
<Error in pydios1 with hex>
2021-02-19T10:10:52.687+0100	ERROR	pydio.grpc.data.index.pydiods1	goroutine
2021-02-19T10:10:52.690+0100	INFO	pydio.grpc.data.index.buvo	Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:52.743+0100	INFO	pydio.grpc.data.index.buvo	started
2021-02-19T10:10:52.759+0100	INFO	pydio.grpc.data.index.cellsdata	Warning: no private IP detected for binding broker. Will bind to <redacted ip>, which may give public access to the broker.
2021-02-19T10:10:52.826+0100	INFO	pydio.grpc.data.index.cellsdata	started
2021-02-19T10:10:54.393+0100	INFO	pydio.rest.frontend	started
2021-02-19T10:10:54.395+0100	INFO	pydio.rest.activity	started
2021-02-19T10:10:55.121+0100	INFO	pydio.grpc.data.sync.cellsdata	started
2021-02-19T10:10:55.121+0100	INFO	pydio.grpc.tasks	Run Job resync-ds-cellsdata on demand
2021-02-19T10:10:55.180+0100	INFO	pydio.grpc.data.sync.buvo	started
2021-02-19T10:10:55.180+0100	INFO	pydio.grpc.tasks	Run Job resync-ds-buvo on demand
2021-02-19T10:10:55.228+0100	INFO	pydio.grpc.data.sync.cellsdata	Got Stats for s3://<redacted ip>:39628/cellsdata	{"stats": {"HasChildrenInfo":false,"HasSizeInfo":false,"Size":0,"Folders":0,"Files":0}}
2021-02-19T10:10:55.229+0100	INFO	pydio.grpc.data.sync.cellsdata	Got Stats for index://cellsdata	{"stats": {"HasChildrenInfo":false,"HasSizeInfo":false,"Size":0,"Folders":0,"Files":0}}
2021-02-19T10:10:55.229+0100	INFO	pydio.grpc.data.sync.cellsdata	Listing all S3 objects for path	{"bucket": "cellsdata", "path": ""}
2021-02-19T10:10:55.235+0100	INFO	pydio.grpc.data.sync.cellsdata	S3 Walk Operation + Stats took	{"d": "6.29557ms"}
2021-02-19T10:10:55.235+0100	INFO	pydio.grpc.data.sync.cellsdata	Diff Stats	{"s": {"EndpointLeft":"s3://<redacted ip>:39628/cellsdata","EndpointRight":"index://cellsdata","conflicts":0,"missingLeft":0,"missingRight":0}}
2021-02-19T10:10:55.235+0100	INFO	pydio.grpc.data.sync.cellsdata	Finished analyzing nodes	{"i": 14}
2021-02-19T10:10:55.236+0100	INFO	pydio.grpc.data.sync.cellsdata	Sending unidirectional patch	{"patch": {"Source":"s3://<redacted ip>:39628/cellsdata","Target":"index://cellsdata","Type":"TreePatch"}}
2021-02-19T10:10:55.236+0100	INFO	pydio.grpc.data.sync.cellsdata	Filtering TreePatch took 2.868µs	{"duration": "2.868µs", "source": "s3://<redacted ip>:39628/cellsdata", "target": "index://cellsdata"}
2021-02-19T10:10:55.237+0100	INFO	pydio.grpc.data.sync.cellsdata	Empty Patch : nothing to do
2021-02-19T10:10:55.327+0100	INFO	pydio.grpc.data.sync.buvo	Got Stats for index://buvo	{"stats": {"HasChildrenInfo":false,"HasSizeInfo":false,"Size":0,"Folders":0,"Files":0}}
2021-02-19T10:10:55.328+0100	INFO	pydio.grpc.data.sync.buvo	Create Hidden File for folder	{"path": ".pydio"}
2021-02-19T10:10:55.893+0100	ERROR	pydio.grpc.data.index.personal	runtime/cgo: pthread_create failed: Resource temporarily unavailable
2021-02-19T10:10:55.894+0100	ERROR	pydio.grpc.data.index.pydiods1	runtime/cgo: pthread_create failed: Resource temporarily unavailable
2021-02-19T10:10:55.895+0100	ERROR	pydio.grpc.data.index.personal	SIGABRT: abort
2021-02-19T10:10:55.896+0100	ERROR	pydio.grpc.data.index.personal	PC=0x7f465ef40fb7 m=0 sigcode=18446744073709551610
2021-02-19T10:10:55.896+0100	ERROR	pydio.grpc.data.index.personal	
2021-02-19T10:10:55.896+0100	ERROR	pydio.grpc.data.index.personal	goroutine 0 [idle]:
2021-02-19T10:10:55.896+0100	ERROR	pydio.grpc.data.index.personal	runtime: unknown pc 0x7f465ef40fb7
2021-02-19T10:10:55.896+0100	ERROR	pydio.grpc.data.index.personal	stack: frame={sp:0x7fff778044c0, fp:0x0} stack=[0x7fff77005a58,0x7fff77804a90)
2021-02-19T10:10:55.896+0100	ERROR	pydio.grpc.data.index.personal	00007fff778043c0:  00007fff77804438  00000000017df85e <runtime.callCgoMmap+62> 
<Error in personal service with hex>
2021-02-19T10:10:55.899+0100	ERROR	pydio.grpc.data.index.personal	00007fff778044f0:  000000000179714d <runtime.(*pageAlloc).update+621>  00007f465f854c00 
<Error in personal service with hex>
2021-02-19T10:10:55.899+0100	ERROR	pydio.grpc.data.index.personal	00007fff77804530:  000000000179714d <runtime.(*pageAlloc).update+621>  0000000000000012 
<Error in personal service with hex>
2021-02-19T10:10:55.897+0100	ERROR	pydio.grpc.data.index.personal	SubProcess was not killed properly: exit status 2
2021-02-19T10:10:55.900+0100	ERROR	pydio.grpc.data.index.personal	Restarting service in 3s...
<Error in personal service with hex> 
2021-02-19T10:10:55.901+0100	ERROR	pydio.grpc.data.index.personal	runtime: unknown pc 0x7f465ef40fb7
2021-02-19T10:10:55.901+0100	ERROR	pydio.grpc.data.index.personal	stack: frame={sp:0x7fff778044c0, fp:0x0} stack=[0x7fff77005a58,0x7fff77804a90)
2021-02-19T10:10:55.901+0100	ERROR	pydio.grpc.data.index.personal	00007fff778043c0:  00007fff77804438  00000000017df85e <runtime.callCgoMmap+62> 
<Error in personal service with hex>
2021-02-19T10:10:55.904+0100	ERROR	pydio.grpc.data.index.personal	00007fff778044e0:  00007fff778045a8  00007fff778045b0 
2021-02-19T10:10:55.904+0100	ERROR	pydio.grpc.data.index.personal	00007fff778044f0:  000000000179714d <runtime.(*pageAlloc).update+621>  00007f465f854c00 
<Error in personal service with hex>
2021-02-19T10:10:55.905+0100	ERROR	pydio.grpc.data.index.personal	00007fff77804530:  000000000179714d <runtime.(*pageAlloc).update+621>  0000000000000012 
<Error in personal service with hex>
2021-02-19T10:10:55.907+0100	ERROR	pydio.grpc.data.index.personal	
2021-02-19T10:10:55.907+0100	ERROR	pydio.grpc.data.index.personal	gorou
2021-02-19T10:10:55.903+0100	ERROR	pydio.grpc.data.index.pydiods1	SIGABRT: abort
2021-02-19T10:10:55.907+0100	ERROR	pydio.grpc.data.index.pydiods1	PC=0x7fe8e8cecfb7 m=0 sigcode=18446744073709551610
2021-02-19T10:10:55.907+0100	ERROR	pydio.grpc.data.index.pydiods1	
2021-02-19T10:10:55.908+0100	ERROR	pydio.grpc.data.index.pydiods1	goroutine 0 [idle]:
2021-02-19T10:10:55.908+0100	ERROR	pydio.grpc.data.index.pydiods1	runtime: unknown pc 0x7fe8e8cecfb7
2021-02-19T10:10:55.908+0100	ERROR	pydio.grpc.data.index.pydiods1	stack: frame={sp:0x7ffc8674f3f0, fp:0x0} stack=[0x7ffc85f50988,0x7ffc8674f9c0)
2021-02-19T10:10:55.908+0100	ERROR	pydio.grpc.data.index.pydiods1	00007ffc8674f2f0:  00007ffc8674f368  00000000017df85e <runtime.callCgoMmap+62> 
<Error in pydios1 service with hex>
2021-02-19T10:10:55.912+0100	ERROR	pydio.grpc.data.index.pydiods1	SubProcess was not killed properly: exit status 2
2021-02-19T10:10:55.914+0100	ERROR	pydio.grpc.data.index.pydiods1	Restarting service in 3s...
2021-02-19T10:10:58.925+0100	INFO	pydio.gateway.proxy	Restarting proxy	{"caddyfile": "\n\n\n\n\n\n\n<redacted ip>:8080 {\n\t\n\n\t\n\n\tproxy /a  <redacted ip>:39999 {\n\t\twithout /a\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tproxy /oidc <redacted ip>:44706 {\n\t\tinsecure_skip_verify\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tproxy /io   PENDING {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\tproxy /data PENDING {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\tproxy /ws   <redacted ip>:35384 {\n\t\twebsocket\n\t\twithout /ws\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tproxy /dav <redacted ip>:41823 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\t\n\n\tproxy /plug/ <redacted ip>:34057 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Cache-Control \"public, max-age=31536000\"\n\t}\n\tproxy /public/ <redacted ip>:34057 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tproxy /public/plug/ <redacted ip>:34057 {\n\t\twithout /public\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Cache-Control \"public, max-age=31536000\"\n\t}\n\tproxy /user/reset-password/ <redacted ip>:34057 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\t\n\tproxy /robots.txt <redacted ip>:34057 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\t\n\tproxy /login <redacted ip>:34057/gui {\n\t\twithout /login\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\n\n\tproxy /grpc https://PENDING {\n\t\twithout /grpc\n\t\tinsecure_skip_verify\n\t}\n\t\n\trewrite {\n\t\tif {>Content-type} has \"application/grpc\"\n\t\tto /grpc/{path}\n\t}\n\n\n\tredir 302 {\n\t\tif {>Content-type} not_has \"application/grpc\"\n\t\tif {path} is /\n\t\t/ /login\n\t}\n\t\n\t\n\t\n        proxy /wopi/ <redacted ip>:44018 {\n            transparent\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n        }\n\n        proxy /loleaflet/ https://localhost:9980/loleaflet {\n            transparent\n            insecure_skip_verify\n            without /loleaflet/\n        }\n\n        proxy /hosting/discovery https://localhost:9980/hosting/discovery {\n            transparent\n            insecure_skip_verify\n            without /hosting/discovery\n        }\n\n        proxy /lool/ https://localhost:9980/lool/ {\n            transparent\n            insecure_skip_verify\n            websocket\n            without /lool/\n        }\n    \n\t\n\t\n\trewrite {\n\t\tif {path} not_starts_with \"/a/\"\n\t\tif {path} not_starts_with \"/oidc/\"\n\t\tif {path} not_starts_with \"/io\"\n\t\tif {path} not_starts_with \"/data\"\n\t\tif {path} not_starts_with \"/ws/\"\n\t\tif {path} not_starts_with \"/plug/\"\n\t\tif {path} not_starts_with \"/dav\"\n\t\t\n\t\tif {path} not_starts_with \"/wopi/\"\n\t\t\n\t\tif {path} not_starts_with \"/loleaflet/\"\n\t\t\n\t\tif {path} not_starts_with \"/hosting/discovery\"\n\t\t\n\t\tif {path} not_starts_with \"/lool/\"\n\t\t\n\t\tif {path} not_starts_with \"/public/\"\n\t\tif {path} not_starts_with \"/user/reset-password\"\n\t\tif {path} not_starts_with \"/robots.txt\"\n\t\tto {path} {path}/ /login\n\t}\n\n\troot \"/2c669532-d4ec-4fa6-9ca9-dcce68693cc7\"\n\n\t\n\ttls \"/var/cells/certs/59d712d4f41dd1273062d6068134f939.pem\" \"/var/cells/certs/59d712d4f41dd1273062d6068134f939-key.pem\"\n\terrors \"/var/cells/logs/caddy_errors.log\"\n}\n\n\n\n\n\t"}

Thanks a lot,

Ben

Hello,
I solved the binding broker problem, I realised that the 172.0.0.1 network is configured with 255.255.255.255 mask so it makes sense that it didn’t get an address. I configured a 10.0.0.1 virtual network on the same interface with the right mask and it binds to it.
This error pops up when I’m in the data source section:
image .
My UFW is not active so I don’t see what could block the connection.

Now the services related to the storage fail. I’ll post the errors from the log here (they repeat themselves several times):

2021-02-22T13:47:56.892+0100	INFO	pydio.grpc.data.sync.cellsdata	Sync cellsdata - Try to contact Objects
2021-02-22T13:47:56.937+0100	INFO	pydio.grpc.data.sync.pydiods1	Sync pydiods1 - Try to contact Objects
2021-02-22T13:47:56.960+0100	INFO	pydio.grpc.data.objects.local1	started
2021-02-22T13:47:56.962+0100	INFO	pydio.grpc.data.objects.local1	Starting local objects service local1 on /var/cells/data
2021-02-22T13:47:56.964+0100	INFO	pydio.grpc.jobs	started
2021-02-22T13:47:56.968+0100	INFO	[pydio.grpc.log] Cannot open bleve index /var/cells/services/pydio.grpc.jobs/tasklogs.bleve cannot create new index, path already exists
2021-02-22T13:47:57.016+0100	INFO	pydio.grpc.data.sync.buvo	Sync buvo - Try to contact Objects
2021-02-22T13:47:57.085+0100	INFO	pydio.grpc.data.sync.personal	Sync personal - Try to contact Objects
2021-02-22T13:47:57.220+0100	WARN	pydio.grpc.data.sync.buvo	Cannot contact s3 service (bucket buvo), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:47:57.222+0100	WARN	pydio.grpc.data.sync.personal	Cannot contact s3 service (bucket personal), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:47:57.550+0100	INFO	Activating privacy features... done.
2021-02-22T13:47:57.551+0100	INFO	https://0.0.0.0:8080
...
2021-02-22T13:48:05.095+0100	WARN	pydio.grpc.data.sync.cellsdata	Cannot contact s3 service (bucket cellsdata), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:48:05.139+0100	INFO	pydio.grpc.data.sync.pydiods1	Sync pydiods1 - Try to contact Objects
2021-02-22T13:48:05.141+0100	WARN	pydio.grpc.data.sync.pydiods1	Cannot contact s3 service (bucket pydiods1), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:48:05.220+0100	INFO	pydio.grpc.data.sync.buvo	Sync buvo - Try to contact Objects
2021-02-22T13:48:05.222+0100	WARN	pydio.grpc.data.sync.buvo	Cannot contact s3 service (bucket buvo), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:48:05.222+0100	INFO	pydio.grpc.data.sync.personal	Sync personal - Try to contact Objects
2021-02-22T13:48:05.228+0100	WARN	pydio.grpc.data.sync.personal	Cannot contact s3 service (bucket personal), will retry in 4s	{"error": "Server not initialized, please try again."}
...
2021-02-22T13:48:08.315+0100	ERROR	pydio.grpc.data.index.pydiods1	goroutine 1 [chan receive, locked to thread]:
2021-02-22T13:48:08.315+0100	ERROR	pydio.grpc.data.index.pydiods1	text/template/parse.(*lexer).nextItem(...)
2021-02-22T13:48:08.315+0100	ERROR	pydio.grpc.data.index.pydiods1		text/template/parse/lex.go:195
2021-02-22T13:48:08.315+0100	ERROR	pydio.grpc.data.index.pydiods1	t
....
2021-02-22T13:48:08.315+0100	ERROR	pydio.grpc.data.index.pydiods1	goroutine 1 [chan receive, locked to thread]:
2021-02-22T13:48:08.315+0100	ERROR	pydio.grpc.data.index.pydiods1	text/template/parse.(*lexer).nextItem(...)
2021-02-22T13:48:08.315+0100	ERROR	pydio.grpc.data.index.pydiods1		text/template/parse/lex.go:195
2021-02-22T13:48:08.315+0100	ERROR	pydio.grpc.data.index.pydiods1	t
...
2021-02-22T13:48:08.827+0100	ERROR	pydio.grpc.data.index.cellsdata	SubProcess was not killed properly: exit status 2
2021-02-22T13:48:08.832+0100	ERROR	pydio.grpc.data.index.cellsdata	Restarting service in 3s...
2021-02-22T13:48:09.092+0100	INFO	pydio.grpc.data.sync.cellsdata	Sync cellsdata - Try to contact Objects
2021-02-22T13:48:09.094+0100	WARN	pydio.grpc.data.sync.cellsdata	Cannot contact s3 service (bucket cellsdata), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:48:09.139+0100	INFO	pydio.grpc.data.sync.pydiods1	Sync pydiods1 - Try to contact Objects
2021-02-22T13:48:09.141+0100	WARN	pydio.grpc.data.sync.pydiods1	Cannot contact s3 service (bucket pydiods1), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:48:09.220+0100	INFO	pydio.grpc.data.sync.buvo	Sync buvo - Try to contact Objects
2021-02-22T13:48:09.222+0100	WARN	pydio.grpc.data.sync.buvo	Cannot contact s3 service (bucket buvo), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:48:09.222+0100	INFO	pydio.grpc.data.sync.personal	Sync personal - Try to contact Objects
2021-02-22T13:48:09.224+0100	WARN	pydio.grpc.data.sync.personal	Cannot contact s3 service (bucket personal), will retry in 4s	{"error": "Server not initialized, please try again."}
2021-02-22T13:48:09.254+0100	INFO	pydio.gateway.proxy	Restarting proxy	{"caddyfile": "\n\n\n\n\n\n\n0.0.0.0:8080 {\n\t\n\n\t\n\n\tproxy /a  10.0.0.1:38106 {\n\t\twithout /a\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tproxy /oidc 10.0.0.1:33300 {\n\t\tinsecure_skip_verify\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tproxy /io   10.0.0.1:45840 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\tproxy /data 10.0.0.1:45840 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\tproxy /ws   10.0.0.1:44757 {\n\t\twebsocket\n\t\twithout /ws\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tproxy /dav 10.0.0.1:38401 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\t\n\n\tproxy /plug/ 10.0.0.1:40817 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Cache-Control \"public, max-age=31536000\"\n\t}\n\tproxy /public/ 10.0.0.1:40817 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tproxy /public/plug/ 10.0.0.1:40817 {\n\t\twithout /public\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Cache-Control \"public, max-age=31536000\"\n\t}\n\tproxy /user/reset-password/ 10.0.0.1:40817 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\t\n\tproxy /robots.txt 10.0.0.1:40817 {\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\t\n\tproxy /login 10.0.0.1:40817/gui {\n\t\twithout /login\n\t\theader_upstream Host {host}\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\n\n\tproxy /grpc https://10.0.0.1:45495 {\n\t\twithout /grpc\n\t\tinsecure_skip_verify\n\t}\n\t\n\trewrite {\n\t\tif {>Content-type} has \"application/grpc\"\n\t\tto /grpc/{path}\n\t}\n\n\n\tredir 302 {\n\t\tif {>Content-type} not_has \"application/grpc\"\n\t\tif {path} is /\n\t\t/ /login\n\t}\n\t\n\t\n\t\n        proxy /wopi/ 10.0.0.1:45404 {\n            transparent\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n        }\n\n        proxy /loleaflet/ https://localhost:9980/loleaflet {\n            transparent\n            insecure_skip_verify\n            without /loleaflet/\n        }\n\n        proxy /hosting/discovery https://localhost:9980/hosting/discovery {\n            transparent\n            insecure_skip_verify\n            without /hosting/discovery\n        }\n\n        proxy /lool/ https://localhost:9980/lool/ {\n            transparent\n            insecure_skip_verify\n            websocket\n            without /lool/\n        }\n    \n\t\n\t\n\trewrite {\n\t\tif {path} not_starts_with \"/a/\"\n\t\tif {path} not_starts_with \"/oidc/\"\n\t\tif {path} not_starts_with \"/io\"\n\t\tif {path} not_starts_with \"/data\"\n\t\tif {path} not_starts_with \"/ws/\"\n\t\tif {path} not_starts_with \"/plug/\"\n\t\tif {path} not_starts_with \"/dav\"\n\t\t\n\t\tif {path} not_starts_with \"/wopi/\"\n\t\t\n\t\tif {path} not_starts_with \"/loleaflet/\"\n\t\t\n\t\tif {path} not_starts_with \"/hosting/discovery\"\n\t\t\n\t\tif {path} not_starts_with \"/lool/\"\n\t\t\n\t\tif {path} not_starts_with \"/public/\"\n\t\tif {path} not_starts_with \"/user/reset-password\"\n\t\tif {path} not_starts_with \"/robots.txt\"\n\t\tto {path} {path}/ /login\n\t}\n\n\troot \"/646fbc6a-36d1-44df-aef0-b0bc96d54e8b\"\n\n\t\n\ttls \"/var/cells/certs/bdb498058625f90d79c2562d22fe9918.pem\" \"/var/cells/certs/bdb498058625f90d79c2562d22fe9918-key.pem\"\n\terrors \"/var/cells/logs/caddy_errors.log\"\n}\n\n\n\n\n\t"}
2021-02-22T13:48:09.756+0100	INFO	pydio.gateway.proxy	Restart done
2021-02-22T13:48:11.564+0100	ERROR	pydio.grpc.data.index.pydiods1	runtime/cgo: pthread_create failed: Resource temporarily unavailable
2021-02-22T13:48:11.572+0100	ERROR	pydio.grpc.data.index.pydiods1	SIGABRT: abort

Thanks,

Ben

Hey @benw

We had another look at your thread, we still miss some pieces of information:

  • How much RAM do you have (pb we are speaking about are often triggered by lack of RAM, and you only gave info about your proc)?
  • Did you build the binary by yourself or retrieve it from the dowload server? Could you give the out put of the ./cells version command ?
  • Could you confirm, you made a “direct to disk” install (not via docker or so)?
  • Are you sure you don’t have some zombie processes (ps -aux | grep cells) before you start Cells.

Thus said:

I played around a bit more with the interface and understood what you meant, so here it is. I created the buvo datasource. I guess it doesn’t look very encouraging.

I don’t understand why you say “it is not encouraging”: in the screenshot list the buvo DS seems to be OK. Could you create a workspace in it and perform an upload ?

After you defined you additional private IP address, when you restarted your server, did you see some additionnal:

Warning: no private IP detected for binding broker. 
Will bind to <redacted ip>, which may give public access to the broker.

errors in your log when you restarted your server ?

If you are still stuck, could you provide your pydio.json so that we can have a look ?