Pydio Cells loops on Received Proxy Restart Event, Frontend Service Not Detected

Depau · May 27, 2019, 9:40pm

I’m trying to get this setup working:

Cells under Docker, with docker-compose
Nginx as reverse proxy
Cloudflare on top of that, serving it under my domain, let’s say cells.example.com

docker-compose.yml

version: '3'
services:
    # Cells image with two named volumes for the static and for the data
    cells:
        image: pydio/cells:latest
        restart: always
        volumes:
            - "/srv/docker/volumes/pydio/cells_home:/root/.config"
            - "mysql_socket:/var/run/mysqld/"
        ports:
            - "127.0.0.1:8080:8080"
        environment:
            - CELLS_BIND=127.0.0.1:8080
            - CELLS_EXTERNAL=127.0.0.1
            - CELLS_NO_SSL=1
        command: ['cells', 'start', '--log', 'debug']

    mysql:
         image: mariadb:latest
         restart: always

         environment:
             MYSQL_ROOT_PASSWORD: rootpwd
             MYSQL_DATABASE: cells
             MYSQL_USER: pydio
             MYSQL_PASSWORD: userpwd
         volumes:
             - "/srv/docker/volumes/pydio/mariadb:/var/lib/mysql"
             - "mysql_socket:/var/run/mysqld/"
         command: [mysqld, --character-set-server=utf8mb4, --collation-server=utf8mb4_unicode_ci]

volumes:
    mysql_socket:

Relevant part of nginx config, even though my problem is not the reverse proxy:

server {
    server_name cells.example.com;
    charset     utf-8;

    client_max_body_size 0;

    location / {
        proxy_buffering off;
        proxy_pass http://127.0.0.1:8080/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host   $host:443;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Port   443;
        proxy_set_header X-Forwarded-Proto  https;
    }
    
    location /ws {
        proxy_buffering off;
        proxy_pass http://127.0.0.1:8011/;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }

    listen 443 ssl;

    ssl_certificate /path/to/cert.pwm;
    ssl_certificate_key /path/to/privkey.pem;

    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    }
}

I tried basically every CELLS_BIND/EXTERNAL combination (including putting my domain name on both, my domain on EXTERNAL and 0.0.0.0:8080 on BIND, 127.0.0.1:8080 on EXTERNAL and 127.0.0.1 on BIND, 127.0.0.1:8080 on both, and more)

However, regardless of what I put there, the installation runs (eventually by spoofing the Host header on the client if needed, or changing the nginx config, or installing it with cells install-cli in the container), however when I start it it always loops with this output:

cells_1  | 2019-05-27T21:25:42.392Z	DEBUG	[INFO] Reloading
cells_1  | 2019-05-27T21:25:42.393Z	DEBUG	[INFO] Reloading complete
cells_1  | 2019-05-27T21:25:42.393Z	INFO	pydio.gateway.proxy	Restart done
cells_1  | 2019-05-27T21:25:42.393Z	DEBUG	Received Proxy Restart Event
cells_1  | 2019-05-27T21:25:42.394Z	DEBUG	Received Proxy Restart Event
cells_1  | 2019-05-27T21:25:42.394Z	DEBUG	Received Proxy Restart Event
cells_1  | 2019-05-27T21:25:42.394Z	DEBUG	http: Server closed
cells_1  | 2019-05-27T21:25:52.394Z	DEBUG	Restarting Proxy Now
[generated caddyfile]
[...and so on]

If I try to curl /login it responds with something like “Please wait while Pydio server is starting…” and I get this output in the logs:

cells_1  | 2019-05-27T21:35:59.387Z	DEBUG	[INFO] Reloading
cells_1  | 2019-05-27T21:35:59.388Z	DEBUG	[INFO] Reloading complete
cells_1  | 2019-05-27T21:35:59.389Z	INFO	pydio.gateway.proxy	Restart done
cells_1  | 2019-05-27T21:35:59.389Z	DEBUG	Received Proxy Restart Event
cells_1  | 2019-05-27T21:35:59.389Z	DEBUG	Received Proxy Restart Event
cells_1  | 2019-05-27T21:35:59.389Z	DEBUG	Received Proxy Restart Event
cells_1  | 2019-05-27T21:35:59.390Z	DEBUG	http: Server closed
cells_1  | 2019-05-27T21:36:02.360Z	DEBUG	Loading messages for es-es	{"m": 2425, "conf": 668}
cells_1  | 2019-05-27T21:36:02.658Z	DEBUG	Loading messages for it	{"m": 2425, "conf": 668}
cells_1  | 2019-05-27T21:36:02.971Z	DEBUG	Loading messages for pt-br	{"m": 2425, "conf": 668}
cells_1  | 2019-05-27T21:36:03.298Z	DEBUG	Loading messages for en-us	{"m": 2425, "conf": 668}
cells_1  | 2019-05-27T21:36:03.638Z	DEBUG	Loading messages for fr	{"m": 2425, "conf": 668}
cells_1  | 2019-05-27T21:36:03.987Z	DEBUG	Loading messages for de	{"m": 2425, "conf": 668}
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin editor.ckeditor (disabled)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin editor.libreoffice (disabled)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin action.avatar (disabled)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin gui.mobile (disabled)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin access.homepage (wrong access)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin access.settings (wrong access)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin access.gateway (wrong access)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin editor.infopanel (missing dependencies)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin editor.codemirror (missing dependencies)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin editor.pdfjs (missing dependencies)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin editor.diaporama (missing dependencies)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin uploader.http (missing dependencies)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin action.compression (missing dependencies)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin editor.browser (missing dependencies)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin uploader.html (missing dependencies)
cells_1  | 2019-05-27T21:36:03.988Z	DEBUG	Filtering out plugin editor.text (missing dependencies)
cells_1  | 2019-05-27T21:36:03.989Z	DEBUG	Filtering out plugin editor.video (missing dependencies)
cells_1  | 2019-05-27T21:36:03.989Z	DEBUG	Filtering out plugin meta.simple_lock (missing dependencies)
cells_1  | 2019-05-27T21:36:04.098Z	ERROR	Error while applying modifiers to registry!	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"none available\",\"status\":\"Internal Server Error\"}"}
cells_1  | github.com/pydio/cells/common/service/frontend.(*PluginsPool).RegistryForStatus
cells_1  | 	/opt/teamcity/agent/work/fd94d35d44c59350/src/github.com/pydio/cells/common/service/frontend/pool.go:148
cells_1  | github.com/pydio/cells/frontend/front-srv/web/index.(*IndexHandler).ServeHTTP
cells_1  | 	/opt/teamcity/agent/work/fd94d35d44c59350/src/github.com/pydio/cells/frontend/front-srv/web/index/handler.go:56
cells_1  | github.com/pydio/cells/vendor/github.com/gorilla/mux.(*Router).ServeHTTP
cells_1  | 	/opt/teamcity/agent/work/fd94d35d44c59350/src/github.com/pydio/cells/vendor/github.com/gorilla/mux/mux.go:159
cells_1  | net/http.(*timeoutHandler).ServeHTTP.func1
cells_1  | 	/usr/local/go/src/net/http/server.go:3188
cells_1  | 2019-05-27T21:36:05.408Z	ERROR	Frontend Service Not Detected
cells_1  | github.com/pydio/cells/frontend/front-srv/web/index.(*IndexHandler).detectFrontendService
cells_1  | 	/opt/teamcity/agent/work/fd94d35d44c59350/src/github.com/pydio/cells/frontend/front-srv/web/index/handler.go:121
cells_1  | github.com/pydio/cells/frontend/front-srv/web/index.(*IndexHandler).ServeHTTP
cells_1  | 	/opt/teamcity/agent/work/fd94d35d44c59350/src/github.com/pydio/cells/frontend/front-srv/web/index/handler.go:93
cells_1  | github.com/pydio/cells/vendor/github.com/gorilla/mux.(*Router).ServeHTTP
cells_1  | 	/opt/teamcity/agent/work/fd94d35d44c59350/src/github.com/pydio/cells/vendor/github.com/gorilla/mux/mux.go:159
cells_1  | net/http.(*timeoutHandler).ServeHTTP.func1
cells_1  | 	/usr/local/go/src/net/http/server.go:3188
cells_1  | 2019-05-27T21:36:09.390Z	DEBUG	Restarting Proxy Now

and then it continues to loop.

What the hell am I doing wrong? I tried to read the source code to see what’s happening and I can’t figure it out.

Thank you,

Depau · May 27, 2019, 9:46pm

By the way, full log: https://gist.github.com/Depau/ecd5a18799197857608a9cfadd25b451

falk.john · May 27, 2019, 10:13pm

This is caused by an invalid setting in pydio.json - for some reason peerAddress is invalid for your storage related services (see line 45-51 of the log).

Ignoring this service as peerAddress is not local {“name”: “pydio.grpc.data.index.pydiods1”, “ip”: “fe090cf102b5”}

Without storage service most of the other services will fail too. Could be specific to docker installs (not using docker in my installations).

Depau · May 28, 2019, 10:56am

What is supposed to be there instead?

Should it be the IP address of the container itself? In what format?

This looks like an IPv6 IP grep’d in a weird way.

falk.john · May 28, 2019, 11:14am

This should be a local IPv4 address (can not tell wether IPv6 is supported) or a host name that resolves to a local IPv4 address.

What is configured in your installlation?
grep -i peeraddress ~pydio/.config/pydio/cells/pydio.json

Depau · May 28, 2019, 2:33pm

It used to be “fe090cf102b5” as you can see from the logs, at all occurrences.

I tried this so now it contains the container IP address:

“PeerAddress”: “172.24.0.3”

Depau · May 28, 2019, 2:37pm

It still doesn’t work, anyway. New logs with PeerAddress = container IP: https://gist.github.com/Depau/e757d67f3c8d64fe1ccc0395d8b59d5e

Topic		Replies	Views
How to start pydio-cells docker with my own ssl certificate file? Pydio Cells	6	2613	September 14, 2018
502 error with nginx proxy Pydio Cells networking , docker	3	619	February 8, 2021
Pydio cells server error 502 after installation Pydio Cells	2	825	September 14, 2018
Dockerized Cells GRPC configuration with NGINX as reverse proxy Pydio Cells docker	12	1640	November 30, 2021
Pydio with LetsEncrypt companion on docker compose Pydio Cells	2	841	July 2, 2019

Pydio Cells loops on Received Proxy Restart Event, Frontend Service Not Detected

Related Topics