Restoring Cells from backup and getting an error on logs

c12simple · April 2, 2025, 3:02pm

Hi,

If, you did a backup while cells is running, you may get a corrupted boltdb file. In this case, boltdb of pydio.grpc.docstore service is corrupted. You must clean this db before starting cells.

Once you get the login page, try to create new user using ./cells admin user create

Don’t forget to export CELLS_WORKING_DIR=/var/cells

If you are running in Fedora, please try to disable SELinux

Don’t remove .minio.sys

Amanuel_Elhanan · April 3, 2025, 7:53am

Hello @c12simple, thanks again,

In this case, boltdb of pydio.grpc.docstore service is corrupted. You must clean this db before starting cells.

Therefore, I deleted /var/cells/services/pydio.grpc.docstore as you suggested here Restoring Cells from backup and getting an error on logs - #2 by c12simple

This time, I started cells with --log=debug as you suggested and get the following errors on the log.

2025-04-03T10:26:47.016+0300    ERROR   pydio.server.manager    ✗There was an error while starting:server.Start grpc: listen tcp 127.0.0.1:8031: bind: address already in use

2025-04-03T10:26:37.696+0300    ERROR   pydio.grpc.search       Cannot start queue, using an in-memory instead  {"error": "leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000357]"}

After receiving Unauthorised error on web I got the following log

2025-04-03T10:31:19.105+0300    ERROR   pydio.rest.frontend     Policies blocked POST request at /a/frontend/session. Response: DefaultDeny:true
github.com/pydio/cells/v4/common/log.(*logger).Error
        github.com/pydio/cells/v4/common/log/logger.go:127
github.com/pydio/cells/v4/common/server/middleware.HttpWrapperPolicy.func1
        github.com/pydio/cells/v4/common/server/middleware/policy.go:96
net/http.HandlerFunc.ServeHTTP
        net/http/server.go:2171
github.com/pydio/cells/v4/common/server/middleware.HttpWrapperJWT.func1
        github.com/pydio/cells/v4/common/server/middleware/jwt.go:89
net/http.HandlerFunc.ServeHTTP
        net/http/server.go:2171
github.com/pydio/cells/v4/common/service/context.HttpWrapperSpan.func1
        github.com/pydio/cells/v4/common/service/context/span.go:178
net/http.HandlerFunc.ServeHTTP
        net/http/server.go:2171
github.com/pydio/cells/v4/common/service/context.HttpWrapperMeta.func1
        github.com/pydio/cells/v4/common/service/context/http-to-meta.go:126
net/http.HandlerFunc.ServeHTTP
        net/http/server.go:2171
github.com/pydio/cells/v4/common/service.getWebMiddlewares.func2.1
        github.com/pydio/cells/v4/common/service/web.go:95
net/http.HandlerFunc.ServeHTTP
        net/http/server.go:2171
github.com/pydio/cells/v4/common/service/frontend.NewSessionWrapper.func1
        github.com/pydio/cells/v4/common/service/frontend/session.go:67
net/http.HandlerFunc.ServeHTTP
        net/http/server.go:2171
github.com/rs/cors.(*Cors).Handler.func1
        github.com/rs/cors@v1.11.0/cors.go:289
net/http.HandlerFunc.ServeHTTP
        net/http/server.go:2171
github.com/pydio/cells/v4/common/server.(*ListableMux).ServeHTTP
        github.com/pydio/cells/v4/common/server/mux.go:238
github.com/pydio/cells/v4/common/client/http.(*resolver).ServeHTTP
        github.com/pydio/cells/v4/common/client/http/resolver.go:142
github.com/pydio/cells/v4/common/server/caddy/mux.(*Middleware).ServeHTTP
        github.com/pydio/cells/v4/common/server/caddy/mux/mux.go:80
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:268
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/headers/headers.go:110
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/headers/headers.go:110
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/headers/headers.go:110
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/subroute.go:74
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.VarsMiddleware.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/vars.go:67
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).enforcementHandler
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/server.go:447
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision.(*Server).wrapPrimaryRoute.func1
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/server.go:423
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/server.go:353
net/http.serverHandler.ServeHTTP
        net/http/server.go:3142
net/http.initALPNRequest.ServeHTTP
        net/http/server.go:3750
golang.org/x/net/http2.(*serverConn).runHandler
        golang.org/x/net@v0.35.0/http2/server.go:2475

Here is an error log from Caddy, which seems it can not connect to cells services Connection refused

Apr 03 10:48:17 Liyu-Server caddy[555]: {"level":"error","ts":1743666497.554452,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:8080: connect: connection refused","request":{"remote_ip":"172.30.16.1","remote_port":"57700","client_ip":"172.30.16.1","proto":"HTTP/2.0","method":"GET","host":"cloud.liyumfi.net","uri":"/","headers":{"Sec-Fetch-Site":["same-site"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Referer":["https://liyumfi.net/"],"Priority":["u=0, i"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:137.0) Gecko/20100101 Firefox/137.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Sec-Fetch-Dest":["document"],"Te":["trailers"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"cloud.liyumfi.net"}},"duration":0.000594708,"status":502,"err_id":"yfkdc6sh3","err_trace":"reverseproxy.statusError (reverseproxy.go:1373)"}

I have create a new admin user successfully but when I try to login with it I am still getting the same error Unauthorised

Amanuel_Elhanan · April 3, 2025, 9:08am

Hello @c12simple. I am sorry for taking your time. The whole time, the issue was with the SQL backup file. I don’t know how, but it was broken when I copied it to the new server, therefore, I just reuploaded the file and restored it. Now it is working.

Thank you so much for your time.

Amanuel_Elhanan · April 3, 2025, 7:55pm

Hello @c12simple
Just out of curiosity, doesn’t the restore include the activity of files? Like the time it created and modification activities?
After I restored it and I couldn’t see any activities on the right pane. Instead, new activities are only starting to register and be shown.

system · April 15, 2025, 1:56am

This topic was automatically closed 11 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Returning server is starting because grpc.oauth monitor is not Up Pydio Cells	9	156	September 17, 2024
Pydio Cells start issue ( permissions issues ) Pydio Cells	21	316	June 5, 2024
Yet another "Returning server is starting because grpc.oauth monitor is not Up" Pydio Cells install	5	67	October 22, 2024
Can't login on fresh install Pydio Cells authentication , install	4	875	March 29, 2022
Cells Installation Fails on Ubuntu 22.04 Server Pydio Cells install	26	240	August 18, 2024

Restoring Cells from backup and getting an error on logs

Related topics