CellsSync (and cec) fail OAuth2 authentication on macOS (gRPC proxy issues?)

I hesitated into reopening one of my own issues from a year ago which has an interesting follow-up from a similar problem — but not quite the same as mine.

In fact, it’s pretty much the issue I’m facing now, but, unfortunately, the ‘fix’ seems to be far more elusive than the simple solution found by @thimic.

To recap: I have a working installation of Pydio Cells 2.2.2, running under Ubuntu 20.04 LTS, behind nginx (1.19.6) running as a proxy (mostly[1]) configured according to the official guidelines, using a valid Let’s Encrypt certificate (not managed by Cells).

Everything works flawlessly via the Web interface, WebDAV, or the iOS app. It’s just CellsSync that fails (or, rather, anything which tries to connect via OAuth2).

CellsSync, in fact, is not very useful for deep-level debugging. When doing the first configuration (Create Task, or via Open and starting with a new connection, or even going to http://localhost:3636 on a browser), all you can see is the browser screen very briefly flickering, and then turning blank. There are a handful of frames being shown before going blank — so I made a video capture and then isolated the frames. To my disappointment, they aren’t very interesting:

Note that, although the last frame has a ‘Reconnect Now’ button, in practice it’s impossible to click — on the video I made at 30 FPS, that screen capture lasts for about 4-5 frames: there is no human way to click on the button… and my guess is that the result would be precisely the same :frowning:

The error logs on the CellsSync side, just like @thimic has found out a year ago, are not very helpful:

CellsSync logs (click to open)
2021-03-02T00:04:07.741Z        INFO     Starting runner with Parent ID 1
2021-03-02T00:04:07.749Z        INFO     systray  Starting sub-process with args systray --url http://localhost:3636
2021-03-02T00:04:07.749Z        INFO     update.service   Starting Updater Service
2021-03-02T00:04:07.749Z        INFO     profiler Exposing debug profiles for process 42546 on port 6060

2021-03-02T00:04:07.752Z        INFO     http-server      Starting HttpServer on localhost:3636
2021-03-02T00:04:11.039Z        INFO     update.service   Posting Request for update
2021-03-02T00:04:11.972Z        INFO     systray  2021-03-02T00:04:11.970Z        INFO     systray  Opened WS Connection
2021-03-02T00:04:30.668Z        INFO     update.service   Posting Request for update
2021-03-02T01:04:21.113Z        ERROR    systray  2021/03/02 01:04:20 [INFO] Scanning for stale OCSP staples
2021-03-02T01:04:21.115Z        ERROR    systray  2021/03/02 01:04:21 [INFO] Done checking OCSP staples
2021-03-02T02:04:21.637Z        ERROR    systray  2021/03/02 02:04:21 [INFO] Scanning for stale OCSP staples
2021-03-02T02:04:21.814Z        ERROR    systray  2021/03/02 02:04:21 [INFO] Done checking OCSP staples
2021-03-02T03:04:21.646Z        ERROR    systray  2021/03/02 03:04:21 [INFO] Scanning for stale OCSP staples
2021-03-02T03:04:21.646Z        ERROR    systray  2021/03/02 03:04:21 [INFO] Done checking OCSP staples
2021-03-02T12:52:38.682Z        ERROR    systray  2021/03/02 12:52:38 [INFO] Scanning for stale OCSP staples
2021-03-02T12:52:38.682Z        ERROR    systray  2021/03/02 12:52:38 [INFO] Done checking OCSP staples
2021-03-02T13:52:36.941Z        ERROR    systray  2021/03/02 13:52:36 [INFO] Scanning for stale OCSP staples
2021-03-02T13:52:36.942Z        ERROR    systray  2021/03/02 13:52:36 [INFO] Done checking OCSP staples
2021-03-02T14:52:38.423Z        ERROR    systray  2021/03/02 14:52:38 [INFO] Scanning for stale OCSP staples
2021-03-02T14:52:38.424Z        ERROR    systray  2021/03/02 14:52:38 [INFO] Done checking OCSP staples
2021-03-02T15:52:33.797Z        ERROR    systray  2021/03/02 15:52:33 [INFO] Scanning for stale OCSP staples
2021-03-02T15:52:34.251Z        ERROR    systray  2021/03/02 15:52:33 [INFO] Done checking OCSP staples
2021-03-02T17:07:48.258Z        ERROR    systray  2021/03/02 17:07:48 [INFO] Scanning for stale OCSP staples
2021-03-02T17:07:48.263Z        ERROR    systray  2021/03/02 17:07:48 [INFO] Done checking OCSP staples
2021-03-02T18:07:47.545Z        ERROR    systray  2021/03/02 18:07:47 [INFO] Scanning for stale OCSP staples
2021-03-02T18:07:47.892Z        ERROR    systray  2021/03/02 18:07:47 [INFO] Done checking OCSP staples
2021-03-02T19:07:48.910Z        ERROR    systray  2021/03/02 19:07:48 [INFO] Scanning for stale OCSP staples
2021-03-02T19:07:48.912Z        ERROR    systray  2021/03/02 19:07:48 [INFO] Done checking OCSP staples
2021-03-02T20:07:47.299Z        ERROR    systray  2021/03/02 20:07:47 [INFO] Scanning for stale OCSP staples
2021-03-02T20:07:47.650Z        ERROR    systray  2021/03/02 20:07:47 [INFO] Done checking OCSP staples
2021-03-02T21:07:51.675Z        ERROR    systray  2021/03/02 21:07:51 [INFO] Scanning for expiring certificates
2021-03-02T21:07:51.676Z        ERROR    systray  2021/03/02 21:07:51 [INFO] Done checking certificates
2021-03-02T21:07:51.676Z        ERROR    systray  2021/03/02 21:07:51 [INFO] Scanning for stale OCSP staples
2021-03-02T21:07:51.676Z        ERROR    systray  2021/03/02 21:07:51 [INFO] Done checking OCSP staples
2021-03-02T22:07:47.043Z        ERROR    systray  2021/03/02 22:07:47 [INFO] Scanning for stale OCSP staples
2021-03-02T22:07:47.408Z        ERROR    systray  2021/03/02 22:07:47 [INFO] Done checking OCSP staples
2021-03-03T00:38:05.136Z        ERROR    systray  2021/03/03 00:38:05 [INFO] Scanning for stale OCSP staples
2021-03-03T00:38:05.183Z        ERROR    systray  2021/03/03 00:38:05 [INFO] Done checking OCSP staples
2021-03-03T01:38:05.010Z        ERROR    systray  2021/03/03 01:38:05 [INFO] Scanning for stale OCSP staples
2021-03-03T01:38:05.053Z        ERROR    systray  2021/03/03 01:38:05 [INFO] Done checking OCSP staples
2021-03-03T02:38:09.408Z        ERROR    systray  2021/03/03 02:38:09 [INFO] Scanning for stale OCSP staples
2021-03-03T02:38:09.409Z        ERROR    systray  2021/03/03 02:38:09 [INFO] Done checking OCSP staples
2021-03-03T12:51:00.738Z        ERROR    systray  2021/03/03 12:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T12:51:00.739Z        ERROR    systray  2021/03/03 12:51:00 [INFO] Done checking OCSP staples
2021-03-03T13:51:01.069Z        ERROR    systray  2021/03/03 13:51:01 [INFO] Scanning for stale OCSP staples
2021-03-03T13:51:01.070Z        ERROR    systray  2021/03/03 13:51:01 [INFO] Done checking OCSP staples
2021-03-03T14:51:00.944Z        ERROR    systray  2021/03/03 14:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T14:51:00.946Z        ERROR    systray  2021/03/03 14:51:00 [INFO] Done checking OCSP staples
2021-03-03T15:50:56.338Z        ERROR    systray  2021/03/03 15:50:56 [INFO] Scanning for stale OCSP staples
2021-03-03T15:50:56.362Z        ERROR    systray  2021/03/03 15:50:56 [INFO] Done checking OCSP staples
2021-03-03T16:51:00.687Z        ERROR    systray  2021/03/03 16:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T16:51:00.689Z        ERROR    systray  2021/03/03 16:51:00 [INFO] Done checking OCSP staples
2021-03-03T17:51:00.559Z        ERROR    systray  2021/03/03 17:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T17:51:00.559Z        ERROR    systray  2021/03/03 17:51:00 [INFO] Done checking OCSP staples
2021-03-03T18:51:00.430Z        ERROR    systray  2021/03/03 18:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T18:51:00.432Z        ERROR    systray  2021/03/03 18:51:00 [INFO] Done checking OCSP staples
2021-03-03T19:50:55.807Z        ERROR    systray  2021/03/03 19:50:55 [INFO] Scanning for expiring certificates
2021-03-03T19:50:55.808Z        ERROR    systray  2021/03/03 19:50:55 [INFO] Done checking certificates
2021-03-03T19:50:55.808Z        ERROR    systray  2021/03/03 19:50:55 [INFO] Scanning for stale OCSP staples
2021-03-03T19:50:55.808Z        ERROR    systray  2021/03/03 19:50:55 [INFO] Done checking OCSP staples
2021-03-03T20:51:00.188Z        ERROR    systray  2021/03/03 20:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T20:51:00.189Z        ERROR    systray  2021/03/03 20:51:00 [INFO] Done checking OCSP staples
2021-03-03T21:07:39.237Z        INFO     systray  2021-03-03T21:07:39.236Z        INFO     systray  Closing systray now...
2021-03-03T21:07:39.239Z        INFO     systray  2021-03-03T21:07:39.238Z        INFO     systray  Sending 'quit' message to websocket
2021-03-03T21:07:39.297Z        INFO     update.service   Stopping Updater Service
2021-03-03T21:07:39.297Z        INFO     scheduler        Stopping all tickers
2021-03-03T21:07:39.298Z        INFO     scheduler        Stopping scheduler
2021-03-03T21:07:39.298Z        INFO     profiler Stopping profiler
2021-03-04T00:36:57.642Z        INFO     Starting runner with Parent ID 1
2021-03-04T00:36:57.655Z        INFO     profiler Exposing debug profiles for process 98193 on port 6060

2021-03-04T00:36:57.655Z        INFO     update.service   Starting Updater Service
2021-03-04T00:36:57.655Z        INFO     systray  Starting sub-process with args systray --url http://localhost:3636
2021-03-04T00:36:57.660Z        INFO     http-server      Starting HttpServer on localhost:3636
2021-03-04T00:37:00.790Z        INFO     update.service   Posting Request for update
2021-03-04T00:37:02.844Z        INFO     systray  2021-03-04T00:37:02.841Z        INFO     systray  Opened WS Connection
2021-03-04T00:39:50.251Z        INFO     systray  2021-03-04T00:39:50.250Z        INFO     systray  Closing systray now...
2021-03-04T00:39:50.262Z        INFO     systray  2021-03-04T00:39:50.251Z        INFO     systray  Sending 'quit' message to websocket
2021-03-04T00:39:50.610Z        INFO     scheduler        Stopping all tickers
2021-03-04T00:39:50.610Z        INFO     scheduler        Stopping scheduler
2021-03-04T00:39:50.788Z        INFO     update.service   Stopping Updater Service
2021-03-04T00:39:50.829Z        INFO     profiler Stopping profiler
2021-03-04T01:23:50.937Z        INFO     Starting runner with Parent ID 1
2021-03-04T01:23:50.953Z        INFO     update.service   Starting Updater Service
2021-03-04T01:23:50.953Z        INFO     systray  Starting sub-process with args systray --url http://localhost:3636
2021-03-04T01:23:50.953Z        INFO     profiler Exposing debug profiles for process 98818 on port 6060

2021-03-04T01:23:50.959Z        INFO     http-server      Starting HttpServer on localhost:3636
2021-03-04T01:23:53.959Z        INFO     update.service   Posting Request for update
2021-03-04T01:23:55.758Z        INFO     systray  2021-03-04T01:23:55.755Z        INFO     systray  Opened WS Connection
2021-03-04T01:24:29.037Z        ERROR    http-server      Request error :cannot find authority

The last entry is our old friend, Request error :cannot find authority. No configuration file (config.json) is ever written to the appropriate folder (under macOS, that is ~/Application Support/Pydio/cells-sync). So I’m back to where I was a year ago.

Well, not quite. This time, I delved a little deeper. Firstly, I assumed that the problem is somewhere on the nginx proxying — something is probably failing there. The official guidelines are a more condensed version of a similar configuration using HTTP/3, so I tried to tweak the bare-bones official configuration with some extra parameters. Needless to say, I wasn’t getting anywhere.

While scratching my head in despair, I learned about the existence of [Cells Client](https:(cells-client/v2 at master · pydio/cells-client · GitHub), a command-line interface to Cells, which works remotely (or on the same server), establishing a connection similar to what CellsSync does, and allows listing/copying/moving/deleting files on a remote Cells repository (if that looks familiar, bear with me for a few more paragraphs…). While it doesn’t genuinely accomplish syncing, Cells Client (or cec, as the command-line application is called) at least can communicate with Cells and do basic things with files — and for what I had in mind (more on that later, as well), it might suffice.

So I gave it a try. Because, like most of the Pydio tools written in Go, Cells Client includes some libraries in C, it’s not always easy to get it running. I’ve managed to get it working under macOS (easier) and Ubuntu Linux 20.04 LTS (very tough), but not under Linux/ARMv8 (aarch64), even with a reasonably complete GCC installation (but apparently not complete enough!).

Let’s start with macOS. I began trying out OAuth2, since I could do a local browser-based authentication (just like with Cells Sync). The URL to be opened was:

https://my.server.name/oidc/oauth2/auth?client_id=cells-client&redirect_uri=http%3A%2F%2Flocalhost%3A3000%2Fservers%2Fcallback&response_type=code&scope=openid+email+profile+pydio+offline&state=[random sequence of letters]

which, although it looks fine, produces the following output:

Notice the error at the end, The "redirect_uri" parameter does not match any of the OAuth 2.0 Client’s pre-registered redirect urls, which is pretty much what is being addressed on this related issue. However, the config.json file that I’ve got has nothing to do with what is described on that issue (more on that in a bit), which, although it was automatically closed, clearly didn’t solve the OP’s issue.

Ok, so OAuth2 was not an option. I tried with a Personal Access Token instead, following the instructions to create one token server-side.

You might remember an issue I had when trying to run Pydio Cells under FreeBSD: dealing with the keychain. Happily, under macOS, this issue does not apply; Cells Client will have no trouble writing the token to the system keychain.

So I got connected, and everything appeared to be working — always good to know! In other words, there seems to be no major issue on the server-side configuration regarding authentication, it’s just that OAuth2 is much more finicky and doesn’t appear to like my configuration…

For the sake of completeness, here goes the config.json created for Cells Client on macOS:

config.json
{
        "url": "https://my.server.name",
        "idToken": "",
        "refreshToken": "",
        "tokenExpiresAt": 0,
        "clientKey": "",
        "clientSecret": "",
        "user": "gwynethllewelyn",
        "password": "",
        "skipVerify": false,
        "useTokenCache": false,
        "CustomHeaders": null,
        "skipKeyring": false,
        "authType": "Personal Access Token"
}

Unfortunately, as you know, Cells Client has lots of useful features to copy files to and fro the backend server, but, alas, no synchronisation (that’s the purpose of Cells Sync, after all). So I thought about a devious way to get this to work: use good old rsync to keep local and remote copies in sync; and, every time rsync is launched by cron (or a systemd.timer), it would call Cells Client with cec storage resync-ds, which I assume that will trigger a re-indexing on the storage/workspace/some folder just rsync'ed.

My next step was thus to compile cec under Ubuntu Linux and get it working there. Compiling is easy; the problem is the keyring. Unless you’re running a desktop version of Linux (with Gnome, or a Gnome derivative), you won’t have a system-wide keyring that Cells ‘likes’ to use. It took me two whole days until I finally got a working keyring — worth another chapter in my very frustrating story, but I’ll spare you the gory details — only to find out that you can use cec without a keyring, passing the --skip_keyring command-line flag… duh! I wish I had seen that earlier… Oh well. At least I learned how to install a generic, global keyring under Linux; I admit I’m spoiled by how easy it is to do for SSH, and even GnuPG, and how terribly hard it is to use the Gnome/D-Bus/SecretService keyring… but allegedly there is nothing ‘better’ out there in the Linux world, so we’re stuck with it.

Since I had cec running on my remote Linux server — remember, that’s the same one where Pydio Cells is installed — I tested out OAuth2 authentication there. This works — with a gotcha. Version 0.9.2 seems to ignore --skip_keyring for some reason, so I had no other choice but to use my so-tough-to-install keyring. The main issue I had was the complete lack of documentation on how to actually use the keyring, that is, how to get Cells Sync to understand that it needs to access a local keyring… Google to the rescue: some kind soul called Kang Zhang posted detailed instructions on how to get it working. In case that fantastic page disappears from the Web for some reason, let me post here the highlights (shamelessly copied from that very same site):

  • Install the GNOME Keyring daemon.
  • Start a D-Bus session, e.g. run dbus-run-session -- sh and run the following commands inside that shell.
  • Run gnome-keyring-daemon with --unlock option. The description of that option says:
    Read a password from stdin, and use it to unlock the login keyring or create it if the login keyring does not exist.
    When that command is started, enter a password into stdin and press Ctrl+D (end of data). After that, the daemon will fork into background (use --foreground option to block).
  • Now you can use the SecretService backend of Keyring. Remember to run your application in the same D-Bus session as the daemon.

In other words (the keyring access is provided by go-keyring), here is the sequence of commands to type:

dbus-run-session -- sh
gnome-keyring-daemon --unlock
# <enter your keyring password and press Enter, then Ctrl-D>
cec configure

In my case, this most certainly works, but, as you can see, it’s not the easiest way to automate a process. It just showed me that basic OAuth2 works well (unlike what I thought) and that the problem lies elsewhere. It’s just tricky to set up.

I didn’t mention it, but all command lines which use username and password have absolutely no problem in connecting. I wish to avoid that for obvious reasons (i.e. being forced to keep the password in plaintext somewhere). And, naturally enough, everything works perfectly by logging in via the Web interface for Cells. It’s just that macOS <-> remote Cells connection will not work, either with cells-client (cec) or cells-sync.

I’ve also tried to compile the sources for cells, cells-client and cells-sync locally on my Mac — hoping that possibly an ‘unstable’ bleeding edge version might solved this issue. Alas, I can’t compile any of those, because of this standing issue with bbolt; I did spend quite a long time trying to fix this, but, just like the people posting on GitHub, I couldn’t find a solution (no, not even scrapping my whole Go installation and starting from scratch). Pydio has absolutely nothing to do with this issue, so I can only hope that this will be fixed sooner or later (probably ‘later’, since it’s been six months…) so that I can try to compile the many Cells command-line utilities from scratch, and see if I can get it working.

In the meantime, my ‘workaround’ is to use the downloaded binaries for all those utils, and to automate the connection to the remote server running Cells, I will have to use plaintext passwords to connect via cec

Any pointers to whatever I might be doing wrong would be quite welcome.

Thank you all in advance!

— Gwyn


[1] Why ‘mostly’? Well, those guidelines assume full control over the nginx configuration, and, as I mentioned last year, my own server is mainly configured via ISPConfig3. Of course, I can override anything and hack whatever needs hacking, but that defeats the purpose of having an overall management system for the whole server.

Update/clarification: at the time of writing the above post, I wasn’t able to connect via OAuth2 from macOS to Cells (running on Ubuntu Linux). I’m not sure what I did in the attempt to fix the issue, but something worked, because I’ve just tested again (with a binary download of cec) and it worked flawlessly at the first try!

There was something I did at some point — namely, running ./cells configure to fix a few errors that had popped in when trying to edit the Cells configuration manually. Even though this required some time and patience, eventually I got Cells to start again, and, this time, apparently the remote issues with OAuth2 have been fixed — for cec. CellsSync still doesn’t work.

I’ve also attempted to compile cec on my Synology NAS, running a Linux variant (DSM) on top of an ARMv8 64-bit CPU. This gave me some interesting results. First, I got a new version of cec, one that says that cec configure is deprecated (although apparently it still works) and that one ought to use cec oauth instead. So far, so good. I tried both, starting with OAuth2 first, using the following configuration:

Server Address (provide a valid URL): https://my.server.name [redacted]
OAuth APP ID (found in your server pydio.json): cells-client
OAuth APP Secret (leave empty for a public client): [a random sequence of chars, got from the only secret field under pydio.web.oauth]

Because I cannot run a browser on that device, I let cec generate the URL for logging in to my.server.name via my Mac’s browser, and, after correctly logging in with my username and password, I did get an authorisation code, which I promptly copied & pasted back into cec.

This is what I got as a result:

Now exchanging the code for a valid IdToken
✔Successfully Received Token!
⚠ Testing this configuration before saving
✗ Could not connect to server, please recheck your configuration
   Error was cannot retrieve token with config:
https://my.server.name - cells-client - [random characters for the server key] -  -  - false
error cause: could not unmarshall response with status 200: 200 OK
error cause: invalid character '<' looking for beginning of value
2021/03/25 14:12:30 test connection failed

On the server’s side, this is what I got on pydio.log:

{"level":"error","ts":"2021-03-25T14:12:19Z","logger":"pydio.rest.frontend","msg":"Rest Error 401","SpanUuid":"1c6ddd3a-8d74-11eb-a69d-e840f23dc110","RemoteAddress":"10.169.169.1","UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_3_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","ContentType":"application/json","HttpProtocol":"HTTP/1.1","error":"{\"id\":\"login.failed\",\"code\":401,\"detail\":\"Login failed\",\"status\":\"Unauthorized\"}"}
{"level":"info","ts":"2021-03-25T14:12:19Z","msg":"Login failed with  Access token is required"}

Weird.

Note that I also tried with other OAuth APP IDs configured in pydio.json under pydio.web.oauth > staticClients, namely, cells-frontend and cells-client. I got precisely the same errors.

Because I’m curious, I also used the deprecated cec configure option. This apparently bypasses OAuth2 and requires logging in with login and password. It’s strange, since cec under macOS does not work like that. In any case, it doesn’t work, either.

I guess that at this stage I’m stumped. Unless the requested secret key is not what I think it is, it’s clear that Cells is sending back some kind of response that cec cannot parse… and this may have to do with some of my configurations of all the layers on top of Cells itself, so it’ll require a lot more debugging on my part…

That browser OAUTH token thing was terrible. Luckily they seem to have improved things in 2.2.2+
where the Admin can now generate that token for the other party. So they don’t have to try and get it to work with a specific browser.
You can also set a time limit on the token which is handy
Unfortunately 2.2.3 has some other bugs which break other things.
I still have the problem that when I try and send loads of files via the cec client I get loads of error 500
And nobody can tell me why

1 Like

I still have the problem that when I try and send loads of files via the cec client I get loads of error 500

Huh. Interesting. I get a lot of those 500 Internal Server Errors too — when using the S3 API, though. Sadly, on the logs, there is no further elaboration on where those errors originate… now I’m starting to wonder if it’s related to the nginx reverse proxy configuration…?

Well I get it even when we use the cec client. So I think nginx is the likely source
but then again, I think that needs the grpc port working. So if we can’t proxy the grpc port properly then it is going to error for everything