I hesitated into reopening one of my own issues from a year ago which has an interesting follow-up from a similar problem — but not quite the same as mine.
In fact, it’s pretty much the issue I’m facing now, but, unfortunately, the ‘fix’ seems to be far more elusive than the simple solution found by @thimic.
To recap: I have a working installation of Pydio Cells 2.2.2, running under Ubuntu 20.04 LTS, behind nginx (1.19.6) running as a proxy (mostly[1]) configured according to the official guidelines, using a valid Let’s Encrypt certificate (not managed by Cells).
Everything works flawlessly via the Web interface, WebDAV, or the iOS app. It’s just CellsSync that fails (or, rather, anything which tries to connect via OAuth2).
CellsSync, in fact, is not very useful for deep-level debugging. When doing the first configuration (Create Task, or via Open and starting with a new connection, or even going to http://localhost:3636 on a browser), all you can see is the browser screen very briefly flickering, and then turning blank. There are a handful of frames being shown before going blank — so I made a video capture and then isolated the frames. To my disappointment, they aren’t very interesting:
Note that, although the last frame has a ‘Reconnect Now’ button, in practice it’s impossible to click — on the video I made at 30 FPS, that screen capture lasts for about 4-5 frames: there is no human way to click on the button… and my guess is that the result would be precisely the same 
The error logs on the CellsSync side, just like @thimic has found out a year ago, are not very helpful:
CellsSync logs (click to open)
2021-03-02T00:04:07.741Z INFO Starting runner with Parent ID 1
2021-03-02T00:04:07.749Z INFO systray Starting sub-process with args systray --url http://localhost:3636
2021-03-02T00:04:07.749Z INFO update.service Starting Updater Service
2021-03-02T00:04:07.749Z INFO profiler Exposing debug profiles for process 42546 on port 6060
2021-03-02T00:04:07.752Z INFO http-server Starting HttpServer on localhost:3636
2021-03-02T00:04:11.039Z INFO update.service Posting Request for update
2021-03-02T00:04:11.972Z INFO systray 2021-03-02T00:04:11.970Z INFO systray Opened WS Connection
2021-03-02T00:04:30.668Z INFO update.service Posting Request for update
2021-03-02T01:04:21.113Z ERROR systray 2021/03/02 01:04:20 [INFO] Scanning for stale OCSP staples
2021-03-02T01:04:21.115Z ERROR systray 2021/03/02 01:04:21 [INFO] Done checking OCSP staples
2021-03-02T02:04:21.637Z ERROR systray 2021/03/02 02:04:21 [INFO] Scanning for stale OCSP staples
2021-03-02T02:04:21.814Z ERROR systray 2021/03/02 02:04:21 [INFO] Done checking OCSP staples
2021-03-02T03:04:21.646Z ERROR systray 2021/03/02 03:04:21 [INFO] Scanning for stale OCSP staples
2021-03-02T03:04:21.646Z ERROR systray 2021/03/02 03:04:21 [INFO] Done checking OCSP staples
2021-03-02T12:52:38.682Z ERROR systray 2021/03/02 12:52:38 [INFO] Scanning for stale OCSP staples
2021-03-02T12:52:38.682Z ERROR systray 2021/03/02 12:52:38 [INFO] Done checking OCSP staples
2021-03-02T13:52:36.941Z ERROR systray 2021/03/02 13:52:36 [INFO] Scanning for stale OCSP staples
2021-03-02T13:52:36.942Z ERROR systray 2021/03/02 13:52:36 [INFO] Done checking OCSP staples
2021-03-02T14:52:38.423Z ERROR systray 2021/03/02 14:52:38 [INFO] Scanning for stale OCSP staples
2021-03-02T14:52:38.424Z ERROR systray 2021/03/02 14:52:38 [INFO] Done checking OCSP staples
2021-03-02T15:52:33.797Z ERROR systray 2021/03/02 15:52:33 [INFO] Scanning for stale OCSP staples
2021-03-02T15:52:34.251Z ERROR systray 2021/03/02 15:52:33 [INFO] Done checking OCSP staples
2021-03-02T17:07:48.258Z ERROR systray 2021/03/02 17:07:48 [INFO] Scanning for stale OCSP staples
2021-03-02T17:07:48.263Z ERROR systray 2021/03/02 17:07:48 [INFO] Done checking OCSP staples
2021-03-02T18:07:47.545Z ERROR systray 2021/03/02 18:07:47 [INFO] Scanning for stale OCSP staples
2021-03-02T18:07:47.892Z ERROR systray 2021/03/02 18:07:47 [INFO] Done checking OCSP staples
2021-03-02T19:07:48.910Z ERROR systray 2021/03/02 19:07:48 [INFO] Scanning for stale OCSP staples
2021-03-02T19:07:48.912Z ERROR systray 2021/03/02 19:07:48 [INFO] Done checking OCSP staples
2021-03-02T20:07:47.299Z ERROR systray 2021/03/02 20:07:47 [INFO] Scanning for stale OCSP staples
2021-03-02T20:07:47.650Z ERROR systray 2021/03/02 20:07:47 [INFO] Done checking OCSP staples
2021-03-02T21:07:51.675Z ERROR systray 2021/03/02 21:07:51 [INFO] Scanning for expiring certificates
2021-03-02T21:07:51.676Z ERROR systray 2021/03/02 21:07:51 [INFO] Done checking certificates
2021-03-02T21:07:51.676Z ERROR systray 2021/03/02 21:07:51 [INFO] Scanning for stale OCSP staples
2021-03-02T21:07:51.676Z ERROR systray 2021/03/02 21:07:51 [INFO] Done checking OCSP staples
2021-03-02T22:07:47.043Z ERROR systray 2021/03/02 22:07:47 [INFO] Scanning for stale OCSP staples
2021-03-02T22:07:47.408Z ERROR systray 2021/03/02 22:07:47 [INFO] Done checking OCSP staples
2021-03-03T00:38:05.136Z ERROR systray 2021/03/03 00:38:05 [INFO] Scanning for stale OCSP staples
2021-03-03T00:38:05.183Z ERROR systray 2021/03/03 00:38:05 [INFO] Done checking OCSP staples
2021-03-03T01:38:05.010Z ERROR systray 2021/03/03 01:38:05 [INFO] Scanning for stale OCSP staples
2021-03-03T01:38:05.053Z ERROR systray 2021/03/03 01:38:05 [INFO] Done checking OCSP staples
2021-03-03T02:38:09.408Z ERROR systray 2021/03/03 02:38:09 [INFO] Scanning for stale OCSP staples
2021-03-03T02:38:09.409Z ERROR systray 2021/03/03 02:38:09 [INFO] Done checking OCSP staples
2021-03-03T12:51:00.738Z ERROR systray 2021/03/03 12:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T12:51:00.739Z ERROR systray 2021/03/03 12:51:00 [INFO] Done checking OCSP staples
2021-03-03T13:51:01.069Z ERROR systray 2021/03/03 13:51:01 [INFO] Scanning for stale OCSP staples
2021-03-03T13:51:01.070Z ERROR systray 2021/03/03 13:51:01 [INFO] Done checking OCSP staples
2021-03-03T14:51:00.944Z ERROR systray 2021/03/03 14:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T14:51:00.946Z ERROR systray 2021/03/03 14:51:00 [INFO] Done checking OCSP staples
2021-03-03T15:50:56.338Z ERROR systray 2021/03/03 15:50:56 [INFO] Scanning for stale OCSP staples
2021-03-03T15:50:56.362Z ERROR systray 2021/03/03 15:50:56 [INFO] Done checking OCSP staples
2021-03-03T16:51:00.687Z ERROR systray 2021/03/03 16:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T16:51:00.689Z ERROR systray 2021/03/03 16:51:00 [INFO] Done checking OCSP staples
2021-03-03T17:51:00.559Z ERROR systray 2021/03/03 17:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T17:51:00.559Z ERROR systray 2021/03/03 17:51:00 [INFO] Done checking OCSP staples
2021-03-03T18:51:00.430Z ERROR systray 2021/03/03 18:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T18:51:00.432Z ERROR systray 2021/03/03 18:51:00 [INFO] Done checking OCSP staples
2021-03-03T19:50:55.807Z ERROR systray 2021/03/03 19:50:55 [INFO] Scanning for expiring certificates
2021-03-03T19:50:55.808Z ERROR systray 2021/03/03 19:50:55 [INFO] Done checking certificates
2021-03-03T19:50:55.808Z ERROR systray 2021/03/03 19:50:55 [INFO] Scanning for stale OCSP staples
2021-03-03T19:50:55.808Z ERROR systray 2021/03/03 19:50:55 [INFO] Done checking OCSP staples
2021-03-03T20:51:00.188Z ERROR systray 2021/03/03 20:51:00 [INFO] Scanning for stale OCSP staples
2021-03-03T20:51:00.189Z ERROR systray 2021/03/03 20:51:00 [INFO] Done checking OCSP staples
2021-03-03T21:07:39.237Z INFO systray 2021-03-03T21:07:39.236Z INFO systray Closing systray now...
2021-03-03T21:07:39.239Z INFO systray 2021-03-03T21:07:39.238Z INFO systray Sending 'quit' message to websocket
2021-03-03T21:07:39.297Z INFO update.service Stopping Updater Service
2021-03-03T21:07:39.297Z INFO scheduler Stopping all tickers
2021-03-03T21:07:39.298Z INFO scheduler Stopping scheduler
2021-03-03T21:07:39.298Z INFO profiler Stopping profiler
2021-03-04T00:36:57.642Z INFO Starting runner with Parent ID 1
2021-03-04T00:36:57.655Z INFO profiler Exposing debug profiles for process 98193 on port 6060
2021-03-04T00:36:57.655Z INFO update.service Starting Updater Service
2021-03-04T00:36:57.655Z INFO systray Starting sub-process with args systray --url http://localhost:3636
2021-03-04T00:36:57.660Z INFO http-server Starting HttpServer on localhost:3636
2021-03-04T00:37:00.790Z INFO update.service Posting Request for update
2021-03-04T00:37:02.844Z INFO systray 2021-03-04T00:37:02.841Z INFO systray Opened WS Connection
2021-03-04T00:39:50.251Z INFO systray 2021-03-04T00:39:50.250Z INFO systray Closing systray now...
2021-03-04T00:39:50.262Z INFO systray 2021-03-04T00:39:50.251Z INFO systray Sending 'quit' message to websocket
2021-03-04T00:39:50.610Z INFO scheduler Stopping all tickers
2021-03-04T00:39:50.610Z INFO scheduler Stopping scheduler
2021-03-04T00:39:50.788Z INFO update.service Stopping Updater Service
2021-03-04T00:39:50.829Z INFO profiler Stopping profiler
2021-03-04T01:23:50.937Z INFO Starting runner with Parent ID 1
2021-03-04T01:23:50.953Z INFO update.service Starting Updater Service
2021-03-04T01:23:50.953Z INFO systray Starting sub-process with args systray --url http://localhost:3636
2021-03-04T01:23:50.953Z INFO profiler Exposing debug profiles for process 98818 on port 6060
2021-03-04T01:23:50.959Z INFO http-server Starting HttpServer on localhost:3636
2021-03-04T01:23:53.959Z INFO update.service Posting Request for update
2021-03-04T01:23:55.758Z INFO systray 2021-03-04T01:23:55.755Z INFO systray Opened WS Connection
2021-03-04T01:24:29.037Z ERROR http-server Request error :cannot find authority
The last entry is our old friend, Request error :cannot find authority. No configuration file (config.json) is ever written to the appropriate folder (under macOS, that is ~/Application Support/Pydio/cells-sync). So I’m back to where I was a year ago.
Well, not quite. This time, I delved a little deeper. Firstly, I assumed that the problem is somewhere on the nginx proxying — something is probably failing there. The official guidelines are a more condensed version of a similar configuration using HTTP/3, so I tried to tweak the bare-bones official configuration with some extra parameters. Needless to say, I wasn’t getting anywhere.
While scratching my head in despair, I learned about the existence of [Cells Client](https:(https://github.com/pydio/cells-client/tree/master/v2), a command-line interface to Cells, which works remotely (or on the same server), establishing a connection similar to what CellsSync does, and allows listing/copying/moving/deleting files on a remote Cells repository (if that looks familiar, bear with me for a few more paragraphs…). While it doesn’t genuinely accomplish syncing, Cells Client (or cec, as the command-line application is called) at least can communicate with Cells and do basic things with files — and for what I had in mind (more on that later, as well), it might suffice.
So I gave it a try. Because, like most of the Pydio tools written in Go, Cells Client includes some libraries in C, it’s not always easy to get it running. I’ve managed to get it working under macOS (easier) and Ubuntu Linux 20.04 LTS (very tough), but not under Linux/ARMv8 (aarch64), even with a reasonably complete GCC installation (but apparently not complete enough!).
Let’s start with macOS. I began trying out OAuth2, since I could do a local browser-based authentication (just like with Cells Sync). The URL to be opened was:
https://my.server.name/oidc/oauth2/auth?client_id=cells-client&redirect_uri=http%3A%2F%2Flocalhost%3A3000%2Fservers%2Fcallback&response_type=code&scope=openid+email+profile+pydio+offline&state=[random sequence of letters]
which, although it looks fine, produces the following output:
Notice the error at the end, The "redirect_uri" parameter does not match any of the OAuth 2.0 Client’s pre-registered redirect urls, which is pretty much what is being addressed on this related issue. However, the config.json file that I’ve got has nothing to do with what is described on that issue (more on that in a bit), which, although it was automatically closed, clearly didn’t solve the OP’s issue.
Ok, so OAuth2 was not an option. I tried with a Personal Access Token instead, following the instructions to create one token server-side.
You might remember an issue I had when trying to run Pydio Cells under FreeBSD: dealing with the keychain. Happily, under macOS, this issue does not apply; Cells Client will have no trouble writing the token to the system keychain.
So I got connected, and everything appeared to be working — always good to know! In other words, there seems to be no major issue on the server-side configuration regarding authentication, it’s just that OAuth2 is much more finicky and doesn’t appear to like my configuration…
For the sake of completeness, here goes the config.json created for Cells Client on macOS:
config.json
{
"url": "https://my.server.name",
"idToken": "",
"refreshToken": "",
"tokenExpiresAt": 0,
"clientKey": "",
"clientSecret": "",
"user": "gwynethllewelyn",
"password": "",
"skipVerify": false,
"useTokenCache": false,
"CustomHeaders": null,
"skipKeyring": false,
"authType": "Personal Access Token"
}
Unfortunately, as you know, Cells Client has lots of useful features to copy files to and fro the backend server, but, alas, no synchronisation (that’s the purpose of Cells Sync, after all). So I thought about a devious way to get this to work: use good old rsync to keep local and remote copies in sync; and, every time rsync is launched by cron (or a systemd.timer), it would call Cells Client with cec storage resync-ds, which I assume that will trigger a re-indexing on the storage/workspace/some folder just rsync’ed.
My next step was thus to compile cec under Ubuntu Linux and get it working there. Compiling is easy; the problem is the keyring. Unless you’re running a desktop version of Linux (with Gnome, or a Gnome derivative), you won’t have a system-wide keyring that Cells ‘likes’ to use. It took me two whole days until I finally got a working keyring — worth another chapter in my very frustrating story, but I’ll spare you the gory details — only to find out that you can use cec without a keyring, passing the --skip_keyring command-line flag… duh! I wish I had seen that earlier… Oh well. At least I learned how to install a generic, global keyring under Linux; I admit I’m spoiled by how easy it is to do for SSH, and even GnuPG, and how terribly hard it is to use the Gnome/D-Bus/SecretService keyring… but allegedly there is nothing ‘better’ out there in the Linux world, so we’re stuck with it.
Since I had cec running on my remote Linux server — remember, that’s the same one where Pydio Cells is installed — I tested out OAuth2 authentication there. This works — with a gotcha. Version 0.9.2 seems to ignore --skip_keyring for some reason, so I had no other choice but to use my so-tough-to-install keyring. The main issue I had was the complete lack of documentation on how to actually use the keyring, that is, how to get Cells Sync to understand that it needs to access a local keyring… Google to the rescue: some kind soul called Kang Zhang posted detailed instructions on how to get it working. In case that fantastic page disappears from the Web for some reason, let me post here the highlights (shamelessly copied from that very same site):
- Install the GNOME Keyring daemon.
- Start a D-Bus session, e.g. run
dbus-run-session -- shand run the following commands inside that shell.- Run
gnome-keyring-daemonwith--unlockoption. The description of that option says:
Read a password fromstdin, and use it to unlock the login keyring or create it if the login keyring does not exist.
When that command is started, enter a password intostdinand press Ctrl+D (end of data). After that, the daemon will fork into background (use--foregroundoption to block).- Now you can use the SecretService backend of Keyring. Remember to run your application in the same D-Bus session as the daemon.
In other words (the keyring access is provided by go-keyring), here is the sequence of commands to type:
dbus-run-session -- sh
gnome-keyring-daemon --unlock
# <enter your keyring password and press Enter, then Ctrl-D>
cec configure
In my case, this most certainly works, but, as you can see, it’s not the easiest way to automate a process. It just showed me that basic OAuth2 works well (unlike what I thought) and that the problem lies elsewhere. It’s just tricky to set up.
I didn’t mention it, but all command lines which use username and password have absolutely no problem in connecting. I wish to avoid that for obvious reasons (i.e. being forced to keep the password in plaintext somewhere). And, naturally enough, everything works perfectly by logging in via the Web interface for Cells. It’s just that macOS ↔ remote Cells connection will not work, either with cells-client (cec) or cells-sync.
I’ve also tried to compile the sources for cells, cells-client and cells-sync locally on my Mac — hoping that possibly an ‘unstable’ bleeding edge version might solved this issue. Alas, I can’t compile any of those, because of this standing issue with bbolt; I did spend quite a long time trying to fix this, but, just like the people posting on GitHub, I couldn’t find a solution (no, not even scrapping my whole Go installation and starting from scratch). Pydio has absolutely nothing to do with this issue, so I can only hope that this will be fixed sooner or later (probably ‘later’, since it’s been six months…) so that I can try to compile the many Cells command-line utilities from scratch, and see if I can get it working.
In the meantime, my ‘workaround’ is to use the downloaded binaries for all those utils, and to automate the connection to the remote server running Cells, I will have to use plaintext passwords to connect via cec…
Any pointers to whatever I might be doing wrong would be quite welcome.
Thank you all in advance!
— Gwyn
[1] Why ‘mostly’? Well, those guidelines assume full control over the nginx configuration, and, as I mentioned last year, my own server is mainly configured via ISPConfig3. Of course, I can override anything and hack whatever needs hacking, but that defeats the purpose of having an overall management system for the whole server.