Failed to initialize OIDC server: server: no connectors specified

drzraf · February 20, 2020, 6:32pm

Soon after the installation, I’m not able to connect. Here is the error.
It’s about about a 502 to the auth’ connector internal service… but why, and how to further debug and resolve this?

As shown by the cfuid cookie in the client request, the server is behind a Cloudflare proxy.
The redacted hexadecimal IP address is the actual public address of the VPS.

[...]
Failed to initialize OIDC server: server: no connectors specified
2020-02-20T18:22:38.269Z	DEBUG	pydio.web.auth	Config 	{"config": {"dex":{"issuer":"https://my.domain.com/auth/dex","staticClients":[{"Id":"cells-front","IdTokensExpiry":"10m","Name":"cells-front","OfflineSessionsSliding":true,"RedirectURIs":["https://my.domain.com/login/callback"],"RefreshTokensExpiry":"30m","Secret":"iZxcyMidSFFuuVEM7Ku3UItR"}],"web":{"http":"https://my.domain.com/auth/dex"}}}}
Failed to initialize OIDC server: server: no connectors specified
2020-02-20T18:22:39.180Z	DEBUG	pydio.rest.frontend	JWTHttpHandler: Checking JWT
2020-02-20T18:22:39.180Z	DEBUG	pydio.rest.frontend	No Claims Found	{"ctx": "context.Background.WithValue(&http.contextKey{name:\"http-server\"}, &http.Server{Addr:\"\", Handler:(*http.ServeMux)(0xc0030a4d80), TLSConfig:(*tls.Config)(0xc0090ae780), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler){\"h2\":(func(*http.Server, *tls.Conn, http.Handler))(0x1c99f80)}, ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), disableKeepAlives:0, inShutdown:0, nextProtoOnce:sync.Once{m:sync.Mutex{state:0, sema:0x0}, done:0x1}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}{(*net.Listener)(0xc00a760100):struct {}{}}, activeConn:map[*http.conn]struct {}{(*http.conn)(0xc009cc2fa0):struct {}{}}, doneChan:(chan struct {})(nil), onShutdown:[]func(){(func())(0x1ca93a0)}}).WithValue(&http.contextKey{name:\"local-addr\"}, &net.TCPAddr{IP:net.IP{XXXXXX}, Port:40347, Zone:\"\"}).WithCancel.WithCancel.WithValue(4, \"pydio.rest.frontend\").WithValue(3, 0x20).WithValue(8, config.Map(nil)).WithValue(metadata.metaKey{}, metadata.Metadata{\"ClientTime\":\"2020-02-20T18:22+0000\", \"ContentType\":\"application/json\", \"CookiesString\":\"pydio=MTU4MjIyMjAyMXxEdi1CQkFFQ180SUFBUkFCRUFBQUJQLUNBQUE9fB3rZYPFLDSg290AOCgMZezFVxY7quMkvPHXqECy5kV-//__cfduid=d7cbdf51ea3d0bdca613e1df377a86b111572639992//_gcl_au=1.1.131278179.1574864160//_fbp=fb.1.1575491962426.921951848\", \"HttpMetaExtracted\":\"HttpMetaExtracted\", \"HttpProtocol\":\"HTTP/1.1\", \"RemoteAddress\":\"XXXXX\", \"RequestMethod\":\"POST\", \"RequestURI\":\"/frontend/session\", \"ServerTime\":\"2020-02-20T18:22+0000\", \"UserAgent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0\"}).WithValue(metadata.metaKey{}, metadata.Metadata{\"ClientTime\":\"2020-02-20T18:22+0000\", \"ContentType\":\"application/json\", \"CookiesString\":\"pydio=MTU4MjIyMjAyMXxEdi1CQkFFQ180SUFBUkFCRUFBQUJQLUNBQUE9fB3rZYPFLDSg290AOCgMZezFVxY7quMkvPHXqECy5kV-//__cfduid=d7cbdf51ea3d0bdca613e1df377a86b111572639992//_gcl_au=1.1.131278179.1574864160//_fbp=fb.1.1575491962426.921951848\", \"HttpMetaExtracted\":\"HttpMetaExtracted\", \"HttpProtocol\":\"HTTP/1.1\", \"RemoteAddress\":\"XXXXX\", \"RequestMethod\":\"POST\", \"RequestURI\":\"/frontend/session\", \"ServerTime\":\"2020-02-20T18:22+0000\", \"UserAgent\":\"Mozilla/5.0 Firefox/73.0\", \"x-pydio-span-id\":\"f9e116ec-540d-11ea-88a0-fa163ee72a12\"}).WithValue(servicecontext.spanContextKey{}, &servicecontext.Span{SpanId:\"f9e116ec-540d-11ea-88a0-fa163ee72a12\", ParentId:\"\", RootParentId:\"\"})"}
2020-02-20T18:22:39.281Z	DEBUG	pydio.web.auth	Config 	{"config": {"dex":{"issuer":"https://my.domain.com/auth/dex","staticClients":[{"Id":"cells-front","IdTokensExpiry":"10m","Name":"cells-front","OfflineSessionsSliding":true,"RedirectURIs":["https://my.domain.com/login/callback"],"RefreshTokensExpiry":"30m","Secret":"XXXXXXXXXXXXXX"}],"web":{"http":"https://my.domain.com/auth/dex"}}}}
Failed to initialize OIDC server: server: no connectors specified
2020-02-20T18:22:39.383Z	ERROR	pydio.rest.frontend	Got a 502 when contacting gateway - maybe it is currently restarting, wait for 11s and retry once
github.com/pydio/cells/frontend/front-srv/rest/modifiers.GrantTypeAccess
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/frontend/front-srv/rest/modifiers/authfront-login.go:171
github.com/pydio/cells/frontend/front-srv/rest/modifiers.LoginPasswordAuth.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/frontend/front-srv/rest/modifiers/authfront-login.go:42
github.com/pydio/cells/common/service/frontend.ApplyAuthMiddlewares
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/frontend/modifiers.go:128
github.com/pydio/cells/frontend/front-srv/rest.(*FrontendHandler).FrontSession
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/frontend/front-srv/rest/rest.go:202
reflect.callMethod
	/usr/local/go/src/reflect/value.go:690
reflect.methodValueCall
	/usr/local/go/src/reflect/asm_amd64.s:35
github.com/pydio/cells/vendor/github.com/emicklei/go-restful.(*Container).dispatch
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/vendor/github.com/emicklei/go-restful/container.go:282
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2375
github.com/pydio/cells/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/vendor/github.com/emicklei/go-restful/container.go:297
github.com/pydio/cells/common/service/context.NewMetricsHttpWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/context/metrics.go:50
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service.PolicyHttpWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/policy.go:96
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service.JWTHttpWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/jwt.go:99
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service/context.HttpSpanHandlerWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/context/span.go:178
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service/context.HttpMetaExtractorWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/context/http-to-meta.go:115
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service/frontend.NewSessionWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/frontend/session.go:100
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service.NewConfigHttpHandlerWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/config.go:90
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/common/service.NewLogHttpHandlerWrapper.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/common/service/log.go:72
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
github.com/pydio/cells/vendor/github.com/rs/cors.(*Cors).Handler.func1
	/opt/teamcity/agent/work/fb9e7e7133d45375/go/src/github.com/pydio/cells/vendor/github.com/rs/cors/cors.go:214
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:1995
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2375
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2774
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1878
[...]

drzraf · February 24, 2020, 5:24pm

The corresponding sourcecode at

github.com

pydio/cells/blob/master/idm/auth/web/dex.go#L192


	}


	if dexServer != nil {
		dexServer.UpdateStorage(s)
	} else {
		serv, err := server.NewServer(context.Background(), serverConfig)
		if err != nil {
			if strings.Contains(err.Error(), "502") {
				return nil, fmt.Errorf("Cannot to initialize OIDC server: gateway is probably not ready yet...")
			}
			return nil, fmt.Errorf("Failed to initialize OIDC server: %v", err)
		}
		dexServer = serv
	}


	wrapped := servicecontext.HttpMetaExtractorWrapper(dexServer)
	wrapped = servicecontext.HttpSpanHandlerWrapper(wrapped)
	wrapped = service.NewLogHttpHandlerWrapper(wrapped, servicecontext.GetServiceName(pydioSrvContext), servicecontext.GetServiceColor(pydioSrvContext))


	if c.Web.HTTP != "" {
		logger.Infof("Listening (http) on %s", c.Web.HTTP)

shows that it’s a kind of “error catch-all”. I think clearer error message could help to tackle it.

For example : Which URL has been requested, what was the returned HTTP code? And the body?

drzraf · February 24, 2020, 5:34pm

I tried the installation yet another time. This time I’m seeing this:
jwt rawIdToken verify: failed {"error": "oidc: expected audience \"example-app\" got [\"cells-front\"]"}
that keeps me from connecting as an admin after the installation.

I don’t know if it’s tightly related, but still, worth mentioning here I think.

Helpful command:
# ./go/bin/bbolt page-item --format bytes --value-only ./.config/pydio/cells/configs-versions.db 9 1 | jq '.Data.services["pydio.grpc.auth"].dex.staticClients'
(boltbrowser is helpful too)

/edit:
Sounds like after stopping the cells install and running cells start afterwards I was finally able to connect (!)
Note: configs-versions.db (latest version!) does not contain the above example-app anymore.

drzraf · June 23, 2020, 4:04pm

I’m getting this problem again, but just for cells start this time (!)
auth/dex/ always returns 502 what keeps authentication from working.
cells boot is stuck in a loop with generated caddy configuration always showing auth/dex as PENDING.

I tried to restore previous configuration, mysql dumps, … (but I’m always getting Configs upgrades applied which forcefully set services/pydio.grpc.auth/dex/issuer and services/pydio.grpc.auth/dex/web/http to https://domainorg:443/auth/dex).

I feel like this may be somehow related to my problem. I use to have a 2.0.5 which works. I ran 2.0.7 / 2.1.0-rc1 (which failed to work), but even after trying to rollback, I can’t get cells to boot anymore.

charles · June 24, 2020, 12:09pm

Indeed, if you upgrade to 2.1 you cannot downgrade, you’ll have to re-install

drzraf · June 24, 2020, 1:20pm

Ouch… Reinstalling means losing configuration (users, …), it also means losing encryption keys what means loosing files. But in my case, 2.1.0-rc1 errors out some Go stacktrace which means that it was a bad idea to try it on a live system… and that I’ve to downgrade.
Why a combination of config restore+ mysql dump restoration wouldn’t make a successful downgrade?
(I tried but still have the above error)

drzraf · June 26, 2020, 3:36pm

Hopefully, I got 2.1.0-rc2 to work!

rckalex · October 30, 2020, 9:22pm

@charles

Hi Charles,

I had cells configured and running in a docker container for a few months. I don’t recall if this the first restart of this container since it was set up or not but upon cells is throwing the error “Failed to initialize OIDC server: server: no connectors specified”

The first instance of the error occurs after pydio.gateway.data is started and continues to pop-up from there intermittently until this appears to be the last remaining issue. It continues to try to initialize the OIDC server with period restart of pydio.gateway.proxy

Do you have any pointers on what I can spot check to rectify?

Topic		Replies	Views
/oidc/.well-known/openid-configuration not found Sync Client	7	1124	February 7, 2022
Cells-sync Clients cannot connect to Pydio Cells Home 4.3.6 Pydio Cells	5	259	May 26, 2024
Cells-client Oauth config fails Pydio Cells authentication	4	969	October 24, 2020
Cells 2.0.9 server keeps crashing Pydio Cells	5	599	February 3, 2021
Authentication problems since upgrade Pydio Cells authentication , api	4	885	November 17, 2020

Failed to initialize OIDC server: server: no connectors specified

Related topics