Upgrade, now nothing works properly

Pydio Cells
#1

Hi
I was running one of the older versions of cells 2.0.x in production. I decided to try the latest version as I needed the cec client to actually work. Now nothing seems to work properly
the gRPC service refuses to start.
When users are trying to upload files they are failing. And I’m getting a bunch of caddy errors
I’m running cells behind NGINX as my reverse proxy as per your instructions on your website.
I’m on centos 7

[Unit]
Description=Pydio Cells
Documentation=https://pydio.com
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/opt/pydio/cells
StartLimitInterval=0

[Service]
WorkingDirectory=/home/pydio
User=pydio
Group=pydio
PermissionsStartOnly=true
AmbientCapabilities=CAP_NET_BIND_SERVICE
ExecStartPre=/bin/sudo /sbin/setcap ‘cap_net_bind_service=+ep’ /opt/pydio/cells
ExecStart=/opt/pydio/cells start --grpc_external=33060
Restart=always
RestartSec=1
StandardOutput=journal
StandardError=inherit

LimitNOFILE=65536
TimeoutStopSec=5
KillSignal=INT
SendSIGKILL=yes
SuccessExitStatus=0

Add environment variables

Environment=CELLS_LOGS_LEVEL=debug
Environment=CELLS_GRPC_EXTERNAL=33060
Environment=CELLS_WORKING_DIR=/home/pydio/.config/pydio/cells
Environment=PYDIO_ENABLE_METRICS=false

[Install]
WantedBy=multi-user.target

I’ve explicitly set the -grpc_external=33060 and as an environment variable
But the microservice wont start and
Do you have to enable it in some other config file??

And in my caddy logs I get this error

[ERROR 502 /io/personal-files/201207-GN-PMC66-BUR-1_test_wells/A50363_201215020001_D05f07d2.C01] dial tcp: lookup PENDING on xxx.xxx.xxx.xx:53: no such host

(actual ip of dns server redacted to protect the innocent)
Why is it trying to do a reverse lookup on every file that comes in? Can it be turned off?

#2

Hey @ryan_lindsay

Could you please also share your nginx config? Do you use the synchronisation client?
We quickly tested on Debian with nginx and the latest cells but could not reproduce your problems. And by the way, the doc on the website seems to be a little bit outdated, we have to update it soon.

#3

And it would be also nice to know your exact version and the output of cells configure site command (anonymised). Thx.

#4

server {
client_max_body_size 0;
server_name pydio.mycompany.org.au;

location / {
proxy_buffering off;
proxy_pass http://localhost:8080$request_uri;
proxy_set_header X-Real-IP $remote_addr;
}
location /ws {
proxy_buffering off;
proxy_pass http://localhost:8080;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
proxy_read_timeout 86400;
}
location /dav {
proxy_buffering off;
proxy_pass http://localhost:8080/dav;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Read-IP $remote_addr;
}

listen [::]:443 ssl ipv6only=on;
listen 443 ssl;
ssl_certificate /etc/pki/ca-trust/extracted/openssl/wildcard.bundle.crt;
ssl_certificate_key /etc/pki/tls/certs/wildcard.mycompany.org.au.key;
include /etc/letsencrypt/options-ssl-nginx.conf;

This is the nginx.conf file.
We use the sync client and the cec client
Mainly the cec client. Users tend to get a lot of error 500’s from the cec client. But I’m assuming thats because it also tries to use the gRPC port to update something once it’s transferred a file.

I’m sunning on centos 7 btw

#5

[root@pydio-cells conf.d]# cells version
Warning: no private IP detected for binding broker. Will bind to 115.xxx.xxx.xxx, which may give public access to the broker.
Pydio Cells Home Edition
Version: 2.2.3
Built: 09 Mar 21 17:56 +0000
Git commit: 38059a9c24b1a2b41722b6563122de16615de59e
OS/Arch: linux/amd64
Go version: go1.15.5

#6

I think you have a winner here:

There has been a regression we have missed in 2.2.x: launching the App without a private IP is broken.
It will be fixed in 2.2.4 that should be released very soon, together with more doc on the subject (that is already written, but not yet published)

Anyway it is strongly advised to have a private IP and that will fix your setup with 2.2.3.
On CentOS7 it is quite easy:

Assuming your main interface is called eth0, adapt otherwise.

sudo tee /etc/sysconfig/network-scripts/ifcfg-eth0:0 << EOF
DEVICE="eth0:0"
ONBOOT="yes"
BOOTPROTO="static"
IPADDR="10.0.0.1"
NETMASK="255.255.255.0"
BROADCAST="10.0.0.1"
EOF

# Start the interface and check new IP
sudo ifup eth0:0
sudo ip address
1 Like
#7

Hi
ok, that helped for making the grpc service start up.
The problem is now, how are you actually supposed to pass it through nginx
From what I can gather starting cells with the external grpc environment variable and a port number is really for when cells isn’t behind a reverse proxy.
Since I am behind an nginx proxy and it’s that which is handling my SSL certs. What is the way which this is supposed to work?

It looks as if the service pydio.gateway.grpc.clear is the service which would handle the grpc connections. But this seems to be randomly assigned it’s port?