Getting Cells working on Raspberry Pi

Pydio Cells
1

Hi All,

Im trying to get Cells working with Docker on my Raspberry Pi3 and was wondering if anyone has had any luck. So far ive only found a very old version of Pydio here

My Pi is running Raspbine so vanilla Dockerfiles usually dont work, arm64 based images. Is there one availible?

Thanks!

1 Like
2

All I can say is that Pydio Cells runs on a Synology NAS which also has an ARM processor, and it definitely runs the Go release for the aarch64 architecture. Iā€™m not using Docker, though.

3

Hello,

Itā€™s possible if you use an arm64, that will require you to build a Cells binary using the following,
(Assuming that you are inside the github.com/pydio/cells folder:

env GOOS=linux GOARCH=arm64 make main CELLS_VERSION=x.x

(that is if you build on another machine, if you are going to install go inside your raspberry then you can omit the env GOOS, GOARCH)

It all depends on your Raspberry, unfortunately, it will not for 32bits system.

4

Hi Community,

Can you explain that to me in more detail? I use the FreeBox Delta from the French ISP ā€œFREEā€ It runs a VM ā€œDebian Busterā€ Go, the last version with ARM64 was installed. the command $ make dev spits a string error from the vendor viper ā€œhttps://github.com/pydio/cells/blob/master/vendor/github.com/ory/viper/viper.goā€ no matter if I am the branch master or stable

5

Try to compile with different machine (MacBook Pro)

i get following error:

benlue@Bens-MacBook-Pro cells % env GOOS=linux GOARCH=arm64 make main CELLS_VERSION=2.0.2

go build -a
-ldflags ā€œ-X github.com/pydio/cells/common.version=2.0.2
-X github.com/pydio/cells/common.BuildStamp=2019-12-20T17:19:22
-X github.com/pydio/cells/common.BuildRevision=b9c88ef45622a65ebd9f34982d0156a9e2218a4d
-X github.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.Version=b9c88ef45622a65ebd9f34982d0156a9e2218a4d
-X github.com/pydio/cells/vendor/github.com/pydio/minio-srv/cmd.ReleaseTag=b9c88ef45622a65ebd9f34982d0156a9e2218a4dā€
-o cells
.

github.com/pydio/cells/vendor/github.com/mattn/go-sqlite3

vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:18:10: undefined: SQLiteConn
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:26:10: undefined: SQLiteConn
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:27:17: undefined: namedValue
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:29:13: undefined: namedValue
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:35:10: undefined: SQLiteConn
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:36:17: undefined: namedValue
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:44:10: undefined: SQLiteConn
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:49:10: undefined: SQLiteConn
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:54:10: undefined: SQLiteStmt
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:63:10: undefined: SQLiteStmt
vendor/github.com/mattn/go-sqlite3/sqlite3_go18.go:36:17: too many errors
make: *** [main] Error 2

6

It looks like Yasuhiro Matsumoto (a.k.a. mattn)'s go-sqlite3 has some issues with cross-compilation I found an ancient link talking about that, because itā€™s not a ā€˜pureā€™ Go SQLite3 driverā€¦ it requires CGo and acts as a wrapper around whatever SQLite library is installed on the system (see also the README.md for go-sqlite3 where the dependency on CGo is explained: GitHub - mattn/go-sqlite3: sqlite3 driver for go using database/sql)ā€” there is a problem when cross-compiling the parts that are actually in C and not in Go.

But because mattn does an outstanding job in keeping his projects up to date, itā€™s no surprise that practically everyone out there uses his go-sqlite3 package ā€” by contrast, many other SQLite packages on github.com have been abandoned. Iā€™ve looked through some 60-100 different packages listed on github.com, listed by popularity, and none of them will really ā€˜replaceā€™ go-sqlite3 with a ā€˜trueā€™ non-CGo version (or at least one that will allow cross-compilation).

Ok, so Iā€™ve been reading some suggestions on Go cross-compilation (especially as related to my home laptop ā€” an old-ish MacBook Pro). For example, take a look at the last comment on this thread on go-sqlite3ā€™s GitHub issues; this, in turn, links to information on generic cross-compilation on macOS, but having both Go and Rust in mind. Note that if you wish to go via this route, youā€™ll need to use brew install FiloSottile/musl-cross/musl-cross --with-aarch64 if you wish to cross-compile to the ARM architecture.

But while these are good starting points, I still didnā€™t manage to get go-sqlite3 to compile properly. There is always one library or another with the ā€˜wrongā€™ version, at least thatā€™s what was constantly happening with meā€¦

Enter xgo. I came across this tool a year or so ago, but hadnā€™t tried it out. Basically, karalabe got tired of all the difficulties with cross-compiling to different architectures, and he went for the nuclear option: just use several Docker instances to configure different cross-compiling environments, each tailored individually for one architecture, and provide a front-end tool, available with go get github.com/karalabe/xgo, which will automatically

prevent having to remember a potentially complex Docker command every time

(in the words of karalabe himself).

And ā€” surprise, surprise! ā€” Pydio has already put some xgo configuration parameters on the Makefile for Cells! (this is true for the stable version from GitHub, as of today). All you need to do (in theory!) is to launch Docker, add the ARM architecture on the Cells Makefile(e.g. on the line XGO_TARGETS?="linux/amd64,darwin/amd64,windows/amd64,linux/arm64" ā€” the Raspberry Pi3 is supposed to use the ARM64 architecture), type make xgo on the github.com/pydio/cells folderā€¦ and be prepared to wait a long time until every architecture gets its own binary fully compiled!

I got a clean compilation, uploaded the binary to my Synology (which has a quad-core Realtek RTD1296 SoC CPU), and, sure thing, I got it to run perfectly. After upgrading MariaDB to 10.3 (previous versions will fail with weird errors), and making sure I had more open file descriptors (cells will complain if you donā€™t have at least 8192), I got ./cells install to finish, populate the database, create all necessary directories, etc. I stumbled upon some ā€˜usualā€™ configuration issues (see the many threads about caddyā€¦) and didnā€™t actually get a working configuration, but I came pretty close!

Note that Pydio, at the end of this page comparing Pydio 8 with Pydio Cells, promises ARM64 support for Q2 2020, and thus there is no surprise that the source code on GitHub is already tweaked to allow a working cross-compilation to ARM64ā€¦ which makes a lot of sense, since there are way more devices with an ARM chip than, say, an Intel/AMD x86 CPU (theoretically, you ought to use xgo to do cross-compilation of cells to Android or even iOSā€¦ allowing Cells to run inside a Smart TV, for instance, or even inside your own smartphone).

1 Like
7

So, a few years later, here is an update on my own answerā€¦

The upcoming Pydio Cells 4 (currently available as a Release Candidate) now supports modules! Woo-hoo!

There are many wonderful advantages to having modules (as opposed to ā€˜vendorsā€™): one of which is making external tools such as xgo reasonably redundant (at least in many scenarios).

So, grab the latest copy from github/pydio/cells, clone it to wherever you wish (with modules, you donā€™t need to place them on a specific location any longer), and change the Makefile by adding a new entry:

cross-go:
        $(info Cross-compile & build for target '${GOOS}', architecture: '${GOARCH}':)
        env GOOS=${GOOS} GOARCH=${GOARCH} go build -a -trimpath\
         -ldflags "-X github.com/pydio/cells/v4/common.version=${CELLS_VERSION}\
         -X github.com/pydio/cells/v4/common.BuildStamp=${TODAY}\
         -X github.com/pydio/cells/v4/common.BuildRevision=${GITREV}"\
         -o cells\
         .

Back on the shell, all you need is to do:

$ export GOOS=linux
$ export GOARCH=arm64
$ make cross-go

Sit back, wait a bit, and enjoy :slight_smile:

You should get a freshly compiled cells on the same directory. You can check that itā€™s built for the ā€˜rightā€™ architecture:

$ file cells
cells: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=<long string>, not stripped

The actual output might vary a bit across systems, but thatā€™s what you get on an Intel Mac (as well as on an AMD chip running Linux); it should be reasonably accurate, though (assuming that you have a fairly recent version of file installed in your system).

Remote-copy the executable to your Raspberry Pi, log in remotely, and run ./cells configure as usually.

Thatā€™s it!

Oh, BTWā€¦ since the last time Iā€™ve posted, I did get a Raspberry Pi Zero W, for a pet project (connecting it to an old APC Smart UPS so that I can read the battery status and push it into all devices and computers on our home network :smile_cat: ), and Iā€™ve been happily tinkering with running Go on it. But sometimes the poor Pi overheats and consumes all memory available on the small flash card when doing heavy-duty compilations, so Iā€™m happy to do some cross-compilations instead. It works so well (and so quickly!) that Iā€™m always amazed at the power of Go cross-compilation made easy.

Oh, and in my personal case, I also own a Synology NAS, which ā€” youā€™ve guessed it! ā€” also has an ARM64 chip and runs a special version of Linux. I have been consistently compiling Go apps on it as well ā€” it has substantially more ā€˜raw powerā€™ than the tiny Pi, but itā€™s also not an environment designed for heavy-duty compilation! ā€” and, naturally enough, getting Pydio Cells on a home NAS makes a lot of sense, and for ages Iā€™ve been terribly frustrated that most of my compilations would never terminate (mostly because of the issues with Cgo).

But now I can just cross-compile once on my Mac, and just push the resulting ARM64 binary to both the NAS and the Pi! Easy-peasy! :smiley:

Extra bonus: aye, the same trick applies to cells-client ā€” just make sure youā€™re using the v2 branch (you should, in any case!). Itā€™s simply brilliant ā€” I had my Pi ā€˜talkingā€™ to my externally hosted Pydio Cells in a matter of seconds (after getting a key, etc. ā€” a process that is so easy to do nowadays)!

When I think back to the hours and days of frustration in dealing with complex compilation chains using the old vendor-based approachā€¦ I can only sigh at the wasted time (and I never got any significant results).

These days, all it takes is a recent version of Go and the latest versions of Pydio Cells and CEC (aka cells-client), since both support modules, and that means that the Go toolchain has all it needs to cross-compile everything to whatever OS and CPU architecture you wish.

Nowā€¦ I need to do the same with the client-sync CLIā€¦ thatā€™s a bit tougher, at the current stage of developmentā€¦ but Iā€™m sure that itā€™s just a question of time!