Failed to initialize OIDC server: server: no connectors specified

Soon after the installation, I’m not able to connect. Here is the error.
It’s about about a 502 to the auth’ connector internal service… but why, and how to further debug and resolve this?

As shown by the cfuid cookie in the client request, the server is behind a Cloudflare proxy.
The redacted hexadecimal IP address is the actual public address of the VPS.

[...]
Failed to initialize OIDC server: server: no connectors specified
2020-02-20T18:22:38.269Z	DEBUG	pydio.web.auth	Config 	{"config": {"dex":{"issuer":"https://my.domain.com/auth/dex","staticClients":[{"Id":"cells-front","IdTokensExpiry":"10m","Name":"cells-front","OfflineSessionsSliding":true,"RedirectURIs":["https://my.domain.com/login/callback"],"RefreshTokensExpiry":"30m","Secret":"iZxcyMidSFFuuVEM7Ku3UItR"}],"web":{"http":"https://my.domain.com/auth/dex"}}}}
Failed to initialize OIDC server: server: no connectors specified
2020-02-20T18:22:39.180Z	DEBUG	pydio.rest.frontend	JWTHttpHandler: Checking JWT
2020-02-20T18:22:39.180Z	DEBUG	pydio.rest.frontend	No Claims Found	{"ctx": "context.Background.WithValue(&http.contextKey{name:\"http-server\"}, &http.Server{Addr:\"\", Handler:(*http.ServeMux)(0xc0030a4d80), TLSConfig:(*tls.Config)(0xc0090ae780), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler){\"h2\":(func(*http.Server, *tls.Conn, http.Handler))(0x1c99f80)}, ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), disableKeepAlives:0, inShutdown:0, nextProtoOnce:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x1}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}{(*net.Listener)(0xc00a760100):struct {}{}}, activeConn:map[*http.conn]struct {}{(*http.conn)(0xc009cc2fa0):struct {}{}}, doneChan:(chan struct {})(nil), onShutdown:[]func(){(func())(0x1ca93a0)}}).WithValue(&http.contextKey{name:\"local-addr\"}, &net.TCPAddr{IP:net.IP{XXXXXX}, Port:40347, Zone:\"\"}).WithCancel.WithCancel.WithValue(4, \"pydio.rest.frontend\").WithValue(3, 0x20).WithValue(8, config.Map(nil)).WithValue(metadata.metaKey{}, metadata.Metadata{\"ClientTime\":\"2020-02-20T18:22+0000\", \"ContentType\":\"application/json\", \"CookiesString\":\"pydio=MTU4MjIyMjAyMXxEdi1CQkFFQ180SUFBUkFCRUFBQUJQLUNBQUE9fB3rZYPFLDSg290AOCgMZezFVxY7quMkvPHXqECy5kV-//__cfduid=d7cbdf51ea3d0bdca613e1df377a86b111572639992//_gcl_au=1.1.131278179.1574864160//_fbp=fb.1.1575491962426.921951848\", \"HttpMetaExtracted\":\"HttpMetaExtracted\", \"HttpProtocol\":\"HTTP/1.1\", \"RemoteAddress\":\"XXXXX\", \"RequestMethod\":\"POST\", \"RequestURI\":\"/frontend/session\", \"ServerTime\":\"2020-02-20T18:22+0000\", \"UserAgent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0\"}).WithValue(metadata.metaKey{}, metadata.Metadata{\"ClientTime\":\"2020-02-20T18:22+0000\", \"ContentType\":\"application/json\", \"CookiesString\":\"pydio=MTU4MjIyMjAyMXxEdi1CQkFFQ180SUFBUkFCRUFBQUJQLUNBQUE9fB3rZYPFLDSg290AOCgMZezFVxY7quMkvPHXqECy5kV-//__cfduid=d7cbdf51ea3d0bdca613e1df377a86b111572639992//_gcl_au=1.1.131278179.1574864160//_fbp=fb.1.1575491962426.921951848\", \"HttpMetaExtracted\":\"HttpMetaExtracted\", \"HttpProtocol\":\"HTTP/1.1\", \"RemoteAddress\":\"XXXXX\", \"RequestMethod\":\"POST\", \"RequestURI\":\"/frontend/session\", \"ServerTime\":\"2020-02-20T18:22+0000\", \"UserAgent\":\"Mozilla/5.0 Firefox/73.0\", \"x-pydio-span-id\":\"f9e116ec-540d-11ea-88a0-fa163ee72a12\"}).WithValue(servicecontext.spanContextKey{}, &servicecontext.Span{SpanId:\"f9e116ec-540d-11ea-88a0-fa163ee72a12\", ParentId:\"\", RootParentId:\"\"})"}
2020-02-20T18:22:39.281Z	DEBUG	pydio.web.auth	Config 	{"config": {"dex":{"issuer":"https://my.domain.com/auth/dex","staticClients":[{"Id":"cells-front","IdTokensExpiry":"10m","Name":"cells-front","OfflineSessionsSliding":true,"RedirectURIs":["https://my.domain.com/login/callback"],"RefreshTokensExpiry":"30m","Secret":"XXXXXXXXXXXXXX"}],"web":{"http":"https://my.domain.com/auth/dex"}}}}
Failed to initialize OIDC server: server: no connectors specified
2020-02-20T18:22:39.383Z	ERROR	pydio.rest.frontend	Got a 502 when contacting gateway - maybe it is currently restarting, wait for 11s and retry once
github.com/pydio/cells/frontend/front-srv/rest/modifiers.GrantTypeAccess
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/frontend/front-srv/rest/modifiers/authfront-login.go:171
github.com/pydio/cells/frontend/front-srv/rest/modifiers.LoginPasswordAuth.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/frontend/front-srv/rest/modifiers/authfront-login.go:42
github.com/pydio/cells/common/service/frontend.ApplyAuthMiddlewares
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/frontend/modifiers.go:128
github.com/pydio/cells/frontend/front-srv/rest.(*FrontendHandler).FrontSession
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/frontend/front-srv/rest/rest.go:202
reflect.callMethod
	/usr/local/go/src/reflect/value.go:690
reflect.methodValueCall
	/usr/local/go/src/reflect/asm_amd64.s:35
github.com/pydio/cells/vendor/github.com/emicklei/go-restful.(*Container).dispatch
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/vendor/github.com/emicklei/go-restful/container.go:282
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2375
github.com/pydio/cells/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/vendor/github.com/emicklei/go-restful/container.go:297
github.com/pydio/cells/common/service/context.NewMetricsHttpWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/context/metrics.go:50
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service.PolicyHttpWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/policy.go:96
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service.JWTHttpWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/jwt.go:99
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service/context.HttpSpanHandlerWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/context/span.go:178
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service/context.HttpMetaExtractorWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/context/http-to-meta.go:115
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service/frontend.NewSessionWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/frontend/session.go:100
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service.NewConfigHttpHandlerWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/config.go:90
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service.NewLogHttpHandlerWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/log.go:72
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/vendor/github.com/rs/cors.(*Cors).Handler.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/vendor/github.com/rs/cors/cors.go:214
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2375
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2774
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1878
[...]

The corresponding sourcecode at


shows that it’s a kind of “error catch-all”. I think clearer error message could help to tackle it.

For example : Which URL has been requested, what was the returned HTTP code? And the body?

I tried the installation yet another time. This time I’m seeing this:
jwt rawIdToken verify: failed {"error": "oidc: expected audience \"example-app\" got [\"cells-front\"]"}
that keeps me from connecting as an admin after the installation.

I don’t know if it’s tightly related, but still, worth mentioning here I think.

Helpful command:
# ./go/bin/bbolt page-item --format bytes --value-only ./.config/pydio/cells/configs-versions.db 9 1 | jq '.Data.services["pydio.grpc.auth"].dex.staticClients'
(boltbrowser is helpful too)

/edit:
Sounds like after stopping the cells install and running cells start afterwards I was finally able to connect (!)
Note: configs-versions.db (latest version!) does not contain the above example-app anymore.

I’m getting this problem again, but just for cells start this time (!)
auth/dex/ always returns 502 what keeps authentication from working.
cells boot is stuck in a loop with generated caddy configuration always showing auth/dex as PENDING.

I tried to restore previous configuration, mysql dumps, … (but I’m always getting Configs upgrades applied which forcefully set services/pydio.grpc.auth/dex/issuer and services/pydio.grpc.auth/dex/web/http to https://domainorg:443/auth/dex).

I feel like this may be somehow related to my problem. I use to have a 2.0.5 which works. I ran 2.0.7 / 2.1.0-rc1 (which failed to work), but even after trying to rollback, I can’t get cells to boot anymore.

Indeed, if you upgrade to 2.1 you cannot downgrade, you’ll have to re-install

Ouch… Reinstalling means losing configuration (users, …), it also means losing encryption keys what means loosing files. But in my case, 2.1.0-rc1 errors out some Go stacktrace which means that it was a bad idea to try it on a live system… and that I’ve to downgrade.
Why a combination of config restore+ mysql dump restoration wouldn’t make a successful downgrade?
(I tried but still have the above error)

Hopefully, I got 2.1.0-rc2 to work!