[solved] Kubernetes - 500/502 WS/XHR errors

thecurseofrng · September 17, 2022, 12:53pm

I’m trying to setup Pydio Cells in a Kubernetes cluster. The configuration phase mostly worked, but I’m now stuck with a broken setup, with XHR calls returning 500 errors and WebSockets calls 502 errors. The UI is only halfway functional, I can’t upload anything, any content-related command just stalls. Using the command-line through kubectl exec, like calling cells admin user search, also hangs indefinitely.

The Kubernetes cluster is running version 1.25, uses Traefik proxy as HTTP/2 ingress controller, Pydio is configured with TLS activated, both the standard HTTP and gRPC calls should pass through the same channel. I’m also trying to use S3 (Wasabi) as storage backend, but not sure this is really relevant at this point.

From where I stand, it seems internal cells components can’t talk to each other somehow. I’ve seen others in the forum solve similar issues by adding a private IP on their Ethernet interface, but I can’t do this in my environment. My pod range is 100.64.0.0/16 and cells seems to think it is bearing a public IPv4 even though it’s not.

Kubernetes manifests

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pydio-cells
  labels:
    app: pydio-cells
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pydio-cells
  serviceName: pydio-cells
  template:
    metadata:
      labels:
        app: pydio-cells
    spec:
      containers:
        - name: pydio-cells
          image: pydio/cells:3.0.9
          volumeMounts:
            - name: storage
              mountPath: "/var/cells"
          ports:
            - name: http
              containerPort: 8080
          envFrom:
            - configMapRef:
                name: pydio-cells-env
  volumeClaimTemplates:
    - metadata:
        name: storage
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: pydio-cells
  labels:
    app: pydio-cells
spec:
  selector:
    app: pydio-cells
  ports:
    - name: http
      port: 80
      targetPort: 8080
---
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: pydio-cells
spec:
  insecureSkipVerify: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: pydio-cells
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`pydio-cells.lordran.net`)
      kind: Rule
      services:
        - name: pydio-cells
          port: 80
          scheme: https
          serversTransport: pydio-cells
  tls:
    certResolver: default

Environment config

CELLS_BIND=:8080
CELLS_EXTERNAL=https://pydio-cells.domain.example
CELLS_LOG=info

Pydio Cells logs

### Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
Pydio Cells Home Edition
 Version: 	3.0.9
 Built: 	08 Jul 22 14:24 +0000
 Git commit: 	45829abcab9274b3127b59ab7373a57b41b12ea0
 OS/Arch: 	linux/amd64
 Go version: 	go1.15.15
### About to execute: [cells configure]
Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
**************************************************************
     Warning! A keyring is not found on this machine,
 	A Master Key has been created for cyphering secrets
   It has been stored in /var/cells/cells-vault-key
   Please make sure to secure this file and update the configs
   with its new location, under the defaults/keyPath key.
***************************************************************

Welcome to Pydio Cells Home Edition installation
Pydio Cells Home Edition (v3.0.9) will be configured to run on this machine.
Make sure to prepare access and credentials to a MySQL 5.6+ (or MariaDB equivalent) server.
Pick your installation mode when you are ready.

2022-09-17T11:45:27.412Z	INFO	pydio.gateway.rest	Started
2022-09-17T11:45:27.528Z	INFO	pydio.rest.install	Started

✅ Created a new local CA at "/var/cells/certs/rootCA.pem" 💥
✅ Created a new certificate valid for the following names 📜 - "100.64.104.39" - "127.0.0.1" - "localhost"
✅ The certificate is at "/var/cells/certs/6fa4734a10af271a992d4fd061715f46.pem"
 and the key at "/var/cells/certs/6fa4734a10af271a992d4fd061715f46-key.pem"

👉 If you are behind a reverse proxy, you can either install the RootCA on the proxy machine trust store, or configure your proxy to `insecure_skip_verify` for pointing to Cells.
👉 If you are developing locally, you may install the RootCA in your system trust store to see a green light in your browser!
🗒  To easily install the RootCA in your trust store, use https://github.com/FiloSottile/mkcert. Set the $CAROOT environment variable to the rootCA folder then use 'mkcert -install'

[DEV NOTICE] Registered directive 'pydioproxy' before 'proxy'
Activating privacy features... done.
https://0.0.0.0:8080

Installation Server is starting...
Listening to: 0.0.0.0:8080

2022-09-17T11:45:30.717Z	INFO	pydio.gateway.proxy	Restarting proxy	{"caddyfile": "\n\n0.0.0.0:8080  {\n\troot \"/var/cells/static/install\"\n\tproxy /install [::]:37729\n\n\t\n\ttls \"/var/cells/certs/6fa4734a10af271a992d4fd061715f46.pem\" \"/var/cells/certs/6fa4734a10af271a992d4fd061715f46-key.pem\"\n}\n\n\n\n\t "}
2022-09-17T11:45:31.218Z	INFO	pydio.gateway.proxy	Restart done

Opening URL https://pydio-cells.domain.example in your browser. Please copy/paste it if the browser is not on the same machine.
2022-09-17T11:48:43.337Z	INFO	pydio.rest.install	Stopping

✅ Using the local CA at "/var/cells/certs/rootCA.pem" ✨
✅ Created a new certificate valid for the following names 📜 - "pydio.gateway.grpc"
✅ The certificate is at "/var/cells/certs/1763c734e5c385445c57f58c3a8ee7a5.pem"
 and the key at "/var/cells/certs/1763c734e5c385445c57f58c3a8ee7a5-key.pem"

👉 If you are behind a reverse proxy, you can either install the RootCA on the proxy machine trust store, or configure your proxy to `insecure_skip_verify` for pointing to Cells.
👉 If you are developing locally, you may install the RootCA in your system trust store to see a green light in your browser!
🗒  To easily install the RootCA in your trust store, use https://github.com/FiloSottile/mkcert. Set the $CAROOT environment variable to the rootCA folder then use 'mkcert -install'

*****************************************************************
*  Dynamic bind flag detected, overriding any configured sites  *
*****************************************************************
2022-09-17T11:48:43.505Z	INFO	pydio.grpc.update	Started
2022-09-17T11:48:43.505Z	INFO	pydio.grpc.broker	Started
2022-09-17T11:48:43.505Z	INFO	pydio.grpc.data.sync	Started
2022-09-17T11:48:43.506Z	INFO	pydio.grpc.data.sync	Starting umbrella service pydio.grpc.data.sync. with sources	{"sources": ["pydiods1", "personal", "cellsdata", "versions", "thumbnails"]}
2022-09-17T11:48:43.506Z	INFO	pydio.grpc.data.objects	Started
2022-09-17T11:48:43.506Z	INFO	pydio.grpc.data.objects	Starting umbrella service pydio.grpc.data.objects. with sources	{"sources": ["gateway1"]}
2022-09-17T11:48:43.514Z	INFO	pydio.gateway.rest	Started
2022-09-17T11:48:43.538Z	INFO	pydio.grpc.statics	Started
2022-09-17T11:48:43.539Z	INFO	pydio.gateway.data	Started
2022-09-17T11:48:43.540Z	INFO	pydio.grpc.registry	Started
https://:8080
2022-09-17T11:48:43.546Z	INFO	pydio.gateway.proxy	Started
2022-09-17T11:48:43.602Z	INFO	pydio.grpc.config	Started
2022-09-17T11:48:43.661Z	INFO	pydio.grpc.log	Started
2022-09-17T11:48:43.688Z	INFO	pydio.grpc.chat	Started
2022-09-17T11:48:43.708Z	INFO	pydio.grpc.docstore	Started
2022-09-17T11:48:43.802Z	INFO	pydio.grpc.user-key	Started
2022-09-17T11:48:43.868Z	INFO	pydio.grpc.data.sync.pydiods1	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:44.061Z	INFO	pydio.grpc.data.objects.gateway1	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:44.067Z	INFO	pydio.grpc.data.sync.versions	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:44.115Z	INFO	pydio.grpc.meta	Started
Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:44.168Z	INFO	pydio.grpc.data.sync.thumbnails	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:44.312Z	INFO	pydio.grpc.data.sync.personal	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:44.315Z	INFO	pydio.grpc.data.sync.cellsdata	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:44.516Z	INFO	pydio.grpc.user-meta	Started
2022-09-17T11:48:44.516Z	INFO	pydio.grpc.user-meta	Inserting default namespace for metadata
2022-09-17T11:48:44.850Z	INFO	pydio.grpc.acl	Started
2022-09-17T11:48:45.053Z	INFO	pydio.grpc.token	Started
2022-09-17T11:48:45.292Z	INFO	pydio.grpc.workspace	Started
2022-09-17T11:48:45.562Z	INFO	pydio.grpc.role	Started
2022-09-17T11:48:46.828Z	INFO	pydio.grpc.policy	Started
2022-09-17T11:48:47.075Z	INFO	pydio.grpc.data-key	Started
2022-09-17T11:48:48.165Z	INFO	pydio.grpc.tree	Started
2022-09-17T11:48:48.206Z	INFO	pydio.grpc.data.index	Started
2022-09-17T11:48:48.207Z	INFO	pydio.grpc.data.index	Starting umbrella service pydio.grpc.data.index. with sources	{"sources": ["pydiods1", "personal", "cellsdata", "versions", "thumbnails"]}
2022-09-17T11:48:48.211Z	INFO	pydio.gateway.grpc	Activating self-signed configuration for gRPC gateway to allow full TLS chain.
2022-09-17T11:48:48.212Z	INFO	pydio.gateway.grpc	Started
2022-09-17T11:48:48.261Z	INFO	pydio.test.objects	Started
2022-09-17T11:48:48.445Z	INFO	pydio.grpc.policy	Inserted default policies
2022-09-17T11:48:48.520Z	INFO	pydio.grpc.data.index.thumbnails	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:48.525Z	INFO	pydio.grpc.data.index.pydiods1	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:48.531Z	INFO	pydio.grpc.data.index.versions	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:48.558Z	INFO	pydio.grpc.data.index.personal	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:48.560Z	INFO	pydio.grpc.data.index.cellsdata	Warning: no private IP detected for binding broker. Will bind to 100.64.104.39, which may give public access to the broker.
2022-09-17T11:48:48.570Z	INFO	pydio.grpc.role	Created default role Root Group
2022-09-17T11:48:48.575Z	INFO	pydio.grpc.role	 - Policies added for role Root Group
2022-09-17T11:48:48.695Z	INFO	pydio.grpc.role	 - ACLS set for role Root Group
2022-09-17T11:48:48.703Z	INFO	pydio.grpc.role	Created default role Administrators
2022-09-17T11:48:48.709Z	INFO	pydio.grpc.role	 - Policies added for role Administrators
2022-09-17T11:48:48.786Z	INFO	pydio.grpc.role	 - ACLS set for role Administrators
2022-09-17T11:48:48.791Z	INFO	pydio.grpc.role	Created default role External Users
2022-09-17T11:48:48.802Z	INFO	pydio.grpc.role	 - Policies added for role External Users
2022-09-17T11:48:48.884Z	INFO	pydio.grpc.role	 - ACLS set for role External Users
2022-09-17T11:48:48.900Z	INFO	pydio.grpc.role	Created default role Minisite Permissions
2022-09-17T11:48:48.907Z	INFO	pydio.grpc.role	 - Policies added for role Minisite Permissions
2022-09-17T11:48:48.961Z	INFO	pydio.grpc.role	 - ACLS set for role Minisite Permissions
2022-09-17T11:48:48.971Z	INFO	pydio.grpc.role	Created default role Minisite (Download Disabled)
2022-09-17T11:48:48.980Z	INFO	pydio.grpc.role	 - Policies added for role Minisite (Download Disabled)
2022-09-17T11:48:49.038Z	INFO	pydio.grpc.role	 - ACLS set for role Minisite (Download Disabled)
2022-09-17T11:48:50.510Z	INFO	pydio.grpc.user	Started
2022-09-17T11:48:50.515Z	INFO	pydio.rest.policy	Started
2022-09-17T11:48:50.515Z	INFO	pydio.rest.log	Started
2022-09-17T11:48:50.516Z	INFO	pydio.rest.acl	Started
2022-09-17T11:48:50.517Z	INFO	pydio.web.statics	Started
2022-09-17T11:48:50.526Z	INFO	pydio.rest.update	Started
2022-09-17T11:48:50.526Z	INFO	pydio.rest.config	Started
2022-09-17T11:48:50.544Z	INFO	pydio.grpc.user	Initialization: creating admin user: johann
2022-09-17T11:48:50.555Z	INFO	pydio.rest.user-meta	Started
2022-09-17T11:48:50.556Z	INFO	pydio.rest.role	Started
2022-09-17T11:48:50.556Z	INFO	pydio.rest.workspace	Started
2022-09-17T11:48:50.556Z	INFO	pydio.rest.graph	Started
2022-09-17T11:48:50.560Z	INFO	pydio.rest.user	Started
2022-09-17T11:48:50.561Z	INFO	pydio.rest.frontend	Started
2022-09-17T11:48:50.688Z	INFO	pydio.grpc.role	Role [johann role] has been created	{"MsgId": "51", "RoleUuid": "bbb59ee0-43b3-494f-bd5c-0ab0133e1ff1"}
2022-09-17T11:48:50.689Z	INFO	pydio.grpc.user	Initialization: creating s3 anonymous user
2022-09-17T11:48:50.758Z	INFO	pydio.grpc.role	Role [pydio.anon.user role] has been created	{"MsgId": "51", "RoleUuid": "ba141797-3cc4-415f-b0c6-a8fce060940f"}
2022-09-17T11:48:55.518Z	INFO	pydio.gateway.proxy	Restarting proxy	{"caddyfile": "\n\n\n\n\n\n\n:8080 {\n\t\n\n\t\n\n\tpydioproxy /a  pydio.gateway.rest {\n\t\twithout /a\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tpydioproxy /oidc pydio.web.oauth {\n\t\tinsecure_skip_verify\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tpydioproxy /io   pydio.gateway.data {\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\tpydioproxy /data pydio.gateway.data {\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\tpydioproxy /buckets pydio.gateway.data {\n\t\twithout /buckets\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\tpydioproxy /ws pydio.gateway.websocket {\n\t\twebsocket\n\t\twithout /ws\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tpydioproxy /dav pydio.gateway.dav {\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Content-Security-Policy \"script-src 'none'\"\n\t\theader_downstream X-Content-Security-Policy \"sandbox\"\n\t}\n\t\n\n\tpydioproxy /plug/ pydio.web.statics {\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Cache-Control \"public, max-age=31536000\"\n\t}\n\tpydioproxy /public/ pydio.web.statics {\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\tpydioproxy /public/plug/ pydio.web.statics {\n\t\tfail_timeout 20s\n\t\twithout /public\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\theader_downstream Cache-Control \"public, max-age=31536000\"\n\t}\n\tpydioproxy /user/reset-password/ pydio.web.statics {\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\n\tpydioproxy /robots.txt pydio.web.statics {\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\t\n\tpydioproxy /login pydio.web.statics {\n\t\twithout /login\n\t\twith /gui\n\t\tfail_timeout 20s\n\t\theader_upstream Host pydio-cells.domain.example\n\t\theader_upstream X-Real-IP {remote}\n\t\theader_upstream X-Forwarded-Proto {scheme}\n\t}\n\n\n\tpydioproxy /grpc pydio.gateway.grpc {\n\t\ttls\n\t\twithout /grpc\n\t\tinsecure_skip_verify\n\t\tfail_timeout 20s\n\t}\n\t\n\trewrite {\n\t\tif {>Content-type} has \"application/grpc\"\n\t\tto /grpc/{path}\n\t}\n\n\n\tredir 302 {\n\t\tif {>Content-type} not_has \"application/grpc\"\n\t\tif {>Authorization} not_has \"AWS4-HMAC-SHA256\"\n\t\tif {path} is /\n\t\t/ /login\n\t}\n\n\trewrite {\n\t\tif {>Authorization} has \"AWS4-HMAC-SHA256\"\n\t\tif {path} is / \n\t\tto /buckets{path}\n\t}\n\n\trewrite {\n\t\tif {>A
2022-09-17T11:48:56.021Z	INFO	pydio.gateway.proxy	Restart done
2022-09-17T11:48:58.556Z	INFO	pydio.rest.workspace	Creating a Personal workspace
2022-09-17T11:48:58.608Z	INFO	pydio.rest.workspace	Settings ACLS for workspace
2022-09-17T11:48:58.799Z	INFO	pydio.rest.workspace	Creating a Common Files workspace on pydiods1
2022-09-17T11:48:58.846Z	INFO	pydio.rest.workspace	Settings ACLS for workspace
2022-09-17T11:49:03.258Z	ERROR	pydio.rest.frontend	Rest Error 401	{"error": "{\"id\":\"login.failed\",\"code\":401,\"detail\":\"Login failed\",\"status\":\"Unauthorized\"}"}
2022-09-17T11:49:06.914Z	ERROR	pydio.rest.frontend	Rest Error 401	{"error": "{\"id\":\"login.failed\",\"code\":401,\"detail\":\"Login failed\",\"status\":\"Unauthorized\"}"}
2022-09-17T11:49:08.257Z	INFO	pydio.grpc.oauth	Started
2022-09-17T11:49:08.274Z	INFO	pydio.web.oauth	Started
2022-09-17T11:49:12.802Z	INFO	pydio.rest.frontend	[WrapWithUserLocks] Resetting user failedConnections	{"UserName": "johann"}
2022-09-17T11:49:38.264Z	ERROR	pydio.grpc.oauth	could not initialise service at version 3.0.9	{"error": "timeout"}
2022-09-17T11:49:38.265Z	ERROR	pydio.grpc.oauth	Could not run 	{"error": "timeout"}
2022-09-17T11:58:44.026Z	INFO	pydio.grpc.data.sync.pydiods1	Trying to contact object service data.objects.gateway1 (retry 1)
2022-09-17T11:58:44.228Z	INFO	pydio.grpc.data.sync.versions	Trying to contact object service data.objects.gateway1 (retry 1)
2022-09-17T11:58:44.260Z	ERROR	pydio.grpc.data.objects.gateway1	Could not run 	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"Error creating stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \\\"transport: Error while dialing dial tcp :8003: connect: connection refused\\\"\",\"status\":\"Internal Server Error\"}"}
2022-09-17T11:58:44.338Z	INFO	pydio.grpc.data.sync.thumbnails	Trying to contact object service data.objects.gateway1 (retry 1)
2022-09-17T11:58:44.404Z	ERROR	pydio.grpc.jobs	Could not run 	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"Error creating stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \\\"transport: Error while dialing dial tcp :8003: connect: connection refused\\\"\",\"status\":\"Internal Server Error\"}"}
2022-09-17T11:58:44.473Z	INFO	pydio.grpc.data.sync.personal	Trying to contact object service data.objects.gateway1 (retry 1)
2022-09-17T11:58:44.476Z	INFO	pydio.grpc.data.sync.cellsdata	Trying to contact object service data.objects.gateway1 (retry 1)
2022-09-17T11:58:54.975Z	INFO	pydio.grpc.data.index.thumbnails	2022-09-17T11:58:54.975Z	INFO		Creating root node in index
2022-09-17T11:58:54.995Z	ERROR	pydio.grpc.data.index.thumbnails	Could not run 	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"Error creating stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \\\"transport: Error while dialing dial tcp :8003: connect: connection refused\\\"\",\"status\":\"Internal Server Error\"}"}
2022-09-17T11:58:55.090Z	INFO	pydio.grpc.data.index.personal	2022-09-17T11:58:55.090Z	INFO		Creating root node in index
2022-09-17T11:58:55.106Z	ERROR	pydio.grpc.data.index.personal	Could not run 	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"Error creating stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \\\"transport: Error while dialing dial tcp :8003: connect: connection refused\\\"\",\"status\":\"Internal Server Error\"}"}
2022-09-17T11:58:55.131Z	INFO	pydio.grpc.data.index.cellsdata	2022-09-17T11:58:55.131Z	INFO		Creating root node in index
2022-09-17T11:58:55.137Z	ERROR	pydio.grpc.data.index.cellsdata	Could not run 	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"Error creating stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \\\"transport: Error while dialing dial tcp :8003: connect: connection refused\\\"\",\"status\":\"Internal Server Error\"}"}
2022-09-17T11:58:55.143Z	INFO	pydio.grpc.data.index.pydiods1	2022-09-17T11:58:55.143Z	INFO		Creating root node in index
2022-09-17T11:58:55.149Z	ERROR	pydio.grpc.data.index.pydiods1	Could not run 	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"Error creating stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \\\"transport: Error while dialing dial tcp :8003: connect: connection refused\\\"\",\"status\":\"Internal Server Error\"}"}
2022-09-17T11:58:55.263Z	INFO	pydio.grpc.data.index.versions	2022-09-17T11:58:55.263Z	INFO		Creating root node in index
2022-09-17T11:58:55.268Z	ERROR	pydio.grpc.data.index.versions	Could not run 	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"Error creating stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \\\"transport: Error while dialing dial tcp :8003: connect: connection refused\\\"\",\"status\":\"Internal Server Error\"}"}

Caddy errors

17/Sep/2022:11:48:58 +0000 [ERROR 502 /plug/gui.ajax/res/themes/common/images/hires-photos/40/04.jpg] context canceled
17/Sep/2022:11:49:13 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:13 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:15 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:16 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:20 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:22 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:27 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:31 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:38 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:49:48 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:50:02 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:50:27 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:51:00 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:51:56 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:53:10 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:54:11 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:55:11 +0000 [ERROR 502 /ws/event] no hosts available upstream
17/Sep/2022:11:56:11 +0000 [ERROR 502 /ws/event] no hosts available upstream

Web browser console errors

WebSocket Closed Connection:The connection was closed abnormally, e.g., without sending or receiving a Close control frame (code 1006) line 1 > injectedScript line 1 > eval:1:3931532
XHRPOSThttps://pydio-cells.domain.example/a/meta/bulk/get
[HTTP/2 500 Internal Server Error 20ms]

XHRPOSThttps://pydio-cells.domain.example/a/jobs/user
[HTTP/2 500 Internal Server Error 19ms]

XHRGEThttps://pydio-cells.domain.example/a/templates
[HTTP/2 500 Internal Server Error 18ms]

GETwss://pydio-cells.domain.example/ws/event
[HTTP/1.1 502 Bad Gateway 56ms]

Firefox can’t establish a connection to the server at wss://pydio-cells.domain.example/ws/event. line 1 > injectedScript line 1 > eval:1:3604473
WebSocket Closed Connection:The connection was closed abnormally, e.g., without sending or receiving a Close control frame (code 1006) line 1 > injectedScript line 1 > eval:1:3931532
Firefox can’t establish a connection to the server at wss://pydio-cells.domain.example/ws/event. line 1 > injectedScript line 1 > eval:1:3604473
The connection to wss://pydio-cells.domain.example/ws/event was interrupted while the page was loading. line 1 > injectedScript line 1 > eval:1:3604473

GETwss://pydio-cells.domain.example/ws/event
[HTTP/1.1 502 Bad Gateway 63ms]

thecurseofrng · September 17, 2022, 1:04pm

Also, it seems I can’t maintain my browser session very long, and I can only login successfully for the first ten minutes or so. At some point, the current token expires, and the OAuth backend doesn’t respond anymore, I can’t login unless I kill the pod and let it restart.

Trying to restart sometime leads to a Go panic in the log package:

2022-09-17T12:58:23.151Z    ERROR    pydio.grpc.docstore    Could not run     {"error": "timeout"}
2022-09-17T12:58:23.162Z    INFO        [pydio.grpc.log] Cannot open bleve index /var/cells/services/pydio.grpc.log/syslog.bleve cannot create new index, path already exists
2022-09-17T12:58:23.163Z    INFO    pydio.grpc.log    Started
panic: runtime error: index out of range [-1]

goroutine 1038 [running]:
github.com/pydio/cells/broker/log.(*SyslogServer).getWriteIndex(...)
    github.com/pydio/cells/broker/log/syslog.go:130
github.com/pydio/cells/broker/log.(*SyslogServer).watchInserts(0xc0011aa880)
    github.com/pydio/cells/broker/log/syslog.go:210 +0x590
created by github.com/pydio/cells/broker/log.(*SyslogServer).Open
    github.com/pydio/cells/broker/log/syslog.go:125 +0x3b4

thecurseofrng · October 1, 2022, 10:30am

Switched to Pydio Cells v4 RC5 and now everything works without any other change from my part.

bsinou · October 17, 2022, 1:44pm

Hi @thecurseofrng

That’s nice to hear, thanks for the final feedback.
The v4 is on its way and should be GA this week, don’t hesitate to let us know if you have some more feedback!

thecurseofrng · October 20, 2022, 10:05pm

Switched to release 4.0.1 without any change of note server-side. My issue is now with cells-sync, which I can’t get to work with big files (>5GB) using an S3 backend.

I opened an issue on your Github project: Cells-sync 0.9.2 doesn't trigger multipart uploads proprerly with a Cells v4 server and fails at 5GB using an S3 backend · Issue #433 · pydio/cells · GitHub

system · December 19, 2022, 10:05pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
[Help Wanted] Deploying Pydio Cells to Kubernetes Pydio Cells	1	870	September 24, 2023
Issues Running Behind Traefik Pydio Cells	0	889	January 26, 2019
502 error with nginx proxy Pydio Cells networking , docker	3	686	February 8, 2021
502 Bad gateway error after installation Pydio Cells	6	3187	August 13, 2019
Trouble Deploying Pydio Cells on AKS - Site Not Found Issue Pydio Cells	2	223	January 16, 2024