[SOLVED]Cells upload/download/create issues - Mismatch Signature

Hello,
I am having a few issues with Pydio Cells version 1.3. Here is my setup:

OS: Ubuntu 18.04.1 LTS AMD64

Storage: I have the local stores on the server OS drive setup by the Cell install as well as a NAS running OMV with a ZFS ZRAID1 served out through a Minio server(because Pydio Cells refused to use the NFS share I had mounted on the server.)

MySQL Version: 14.14 Distrib 5.7.23, for Linux (x86_64) using EditLine wrapper

PHP Version: PHP 7.2.7-0ubuntu0.18.04.2

Apache Version: Apache/2.4.29 (Ubuntu)

Configuration: I created a virtual host in Apache for saturn.ooguy.com. I redirected http to https in the virtual host config file. I then added the recommended config from the admin documents to the virtual host config file created by Webmin/Virtualmin when I added the virtual host resulting in this complete config file:

<VirtualHost 192.168.1.3:80 [fe80::fdf0:a23e:a7fb:cc51]:80>
Redirect / https://saturn.ooguy.com
</VirtualHost>
<VirtualHost 192.168.1.3:443 [fe80::fdf0:a23e:a7fb:cc51]:443>
ServerName saturn.ooguy.com
AllowEncodedSlashes On
RewriteEngine On
SSLProxyEngine On
SSLProxyCheckPeerCN Off
SSLProxyCheckPeerName Off
SSLProxyVerify none
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /(.*)           wss://saturn.ooguy.com:7070/$1 [P,L]
ProxyPass "/" "https://saturn.ooguy.com:7070/"
ProxyPassReverse "/" "https://saturn.ooguy.com:7070/"
SuexecUserGroup "#1003" "#1003"
ServerAlias www.saturn.ooguy.com
ServerAlias webmail.saturn.ooguy.com
ServerAlias admin.saturn.ooguy.com
DocumentRoot /home/saturn/public_html
ErrorLog /var/log/virtualmin/saturn.ooguy.com_error_log
CustomLog /var/log/virtualmin/saturn.ooguy.com_access_log combined
ScriptAlias /cgi-bin/ /home/saturn/cgi-bin/
ScriptAlias /awstats/ /home/saturn/cgi-bin/
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/saturn/public_html>
Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted
AddType application/x-httpd-php .php
AddHandler fcgid-script .php
AddHandler fcgid-script .php7.2
FCGIWrapper /home/saturn/fcgi-bin/php7.2.fcgi .php
FCGIWrapper /home/saturn/fcgi-bin/php7.2.fcgi .php7.2
</Directory>
<Directory /home/saturn/cgi-bin>
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted
</Directory>
RewriteEngine On
RewriteCond %{HTTP_HOST} =webmail.saturn.ooguy.com
RewriteRule ^(.*) https://saturn.ooguy.com:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.saturn.ooguy.com
RewriteRule ^(.*) https://saturn.ooguy.com:10000/ [R]
RemoveHandler .php
RemoveHandler .php7.2
php_admin_value engine Off
FcgidMaxRequestLen 1073741824
SSLEngine on
SSLCertificateFile /home/saturn/ssl.cert
SSLCertificateKeyFile /home/saturn/ssl.key
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
<Files awstats.pl>
AuthName "saturn.ooguy.com statistics"
AuthType Basic
AuthUserFile /home/saturn/.awstats-htpasswd
require valid-user
</Files>
Alias /dav /home/saturn/public_html
<Location /dav>
DAV on
AuthType Basic
AuthName "saturn.ooguy.com"
AuthUserFile /home/saturn/etc/dav.digest.passwd
Require valid-user
ForceType text/plain
Satisfy All
RemoveHandler .php
RemoveHandler .php7.2
RewriteEngine off
</Location>
</VirtualHost>

I also setup php-fpm on port 9000 per the admin documents. On installation, I bound to saturn.ooguy.com:7070 internal and saturn.ooguy.com external. I pointed to my SSL cert and key, connected to my Cells database, and declined the port 80 redirect because I handled that in my Virtual Hosts config file. I confirmed the detected php-fpm on port 9000. I left other settings at default. I was initially able to log in, but after a reboot this stopped working with a cURL cert error, and I had to download the CA and LetsEncrypt intermediate certs and add them to the CACert store. I was able to log in and add my datastore from the Minio server on my NAS without issue, and all service appear to be running:

GENERIC SERVICES
 # discovery
 nats                                    [X]
 # frontend
 pydio.api.front-plugins                 [X]
 # gateway
 micro.api                               [X]
 pydio.api.websocket                     [X]
 pydio.rest.gateway.dav                  [X]
 pydio.rest.gateway.wopi                 [X]

 GRPC SERVICES
 # broker
 pydio.grpc.activity                     [X]
 pydio.grpc.chat                         [X]
 pydio.grpc.log                          [X]
 pydio.grpc.mailer                       [X]
 # data
 pydio.grpc.changes                      [X]
 pydio.grpc.data-key                     [X]
 pydio.grpc.docstore                     [X]
 pydio.grpc.meta                         [X]
 pydio.grpc.search                       [X]
 pydio.grpc.tree                         [X]
 pydio.grpc.versions                     [X]
 # datasource
 pydio.grpc.data.index                   [X]
 pydio.grpc.data.index.cellsdata         [X]
 pydio.grpc.data.index.omvemby           [X]
 pydio.grpc.data.index.omvnas            [X]
 pydio.grpc.data.index.personal          [X]
 pydio.grpc.data.index.pydiods1          [X]
 pydio.grpc.data.objects                 [X]
 pydio.grpc.data.objects.gateway1        [X]
 pydio.grpc.data.objects.local1          [X]
 pydio.grpc.data.sync                    [X]
 pydio.grpc.data.sync.cellsdata          [X]
 pydio.grpc.data.sync.omvemby            [X]
 pydio.grpc.data.sync.omvnas             [X]
 pydio.grpc.data.sync.personal           [X]
 pydio.grpc.data.sync.pydiods1           [X]
 # discovery
 pydio.grpc.config                       [X]
 pydio.grpc.update                       [X]
 # gateway
 pydio.grpc.gateway.data                 [X]
 pydio.grpc.gateway.proxy                [X]
 # idm
 pydio.grpc.acl                          [X]
 pydio.grpc.auth                         [X]
 pydio.grpc.policy                       [X]
 pydio.grpc.role                         [X]
 pydio.grpc.share                        [X]
 pydio.grpc.user                         [X]
 pydio.grpc.user-key                     [X]
 pydio.grpc.user-meta                    [X]
 pydio.grpc.workspace                    [X]
 # scheduler
 pydio.grpc.jobs                         [X]
 pydio.grpc.tasks                        [X]
 pydio.grpc.timer                        [X]

 REST SERVICES
 # broker
 pydio.rest.activity                     [X]
 pydio.rest.log                          [X]
 pydio.rest.mailer                       [X]
 # data
 pydio.rest.changes                      [X]
 pydio.rest.docstore                     [X]
 pydio.rest.meta                         [X]
 pydio.rest.search                       [X]
 pydio.rest.tree                         [X]
 # discovery
 pydio.rest.config                       [X]
 pydio.rest.update                       [X]
 # frontend
 pydio.rest.frontend                     [X]
 # idm
 pydio.rest.acl                          [X]
 pydio.rest.auth                         [X]
 pydio.rest.graph                        [X]
 pydio.rest.policy                       [X]
 pydio.rest.role                         [X]
 pydio.rest.share                        [X]
 pydio.rest.user                         [X]
 pydio.rest.user-meta                    [X]
 pydio.rest.workspace                    [X]
 # scheduler
 pydio.rest.jobs                         [X]

I created a Pydio user, and the Pydio home directory is owned by pydio:users
The SystemD service running Cells is running cells as pydio:users
The NAS share gives full read write to the users group, and the GID of that group is the same on both systems.
To rule out permissions issues, I even tried chmod –R 7777 on the pydio home directory and the shares on the NAS.

Issues Encountered:
I am able to create workspaces on the datasources and even see the files in the workspace. I cannot download, upload, or create an empty file or folder in the workspace. I am also seeing some errors when Cells is trying to add the .pydio folder to the folders in the workspaces. Some are permissions errors, some are invalid signature errors, and some are just vague, like when I try to upload a file.
Adding a file or folder results in an error stating: The request signature we calculated does not match the signature you provided. Check your key and signing method.
Uploading a file immediately errors with Upload failed: Undefined
I am able to reproduce these errors on both the local storage workspaces and on the NAS workspaces served through the Minio server.
Then I see these errors a bunch when Cells is first started: level=error msg=“Unable to save an object /ftp/certs/.pydio” cause=“We encountered an internal error, please try again.”
I have pulled and uploaded caddy_errors.log, cells.log, pydio.log, the apache access_log, and the apache error_log. They are here:
https://drive.google.com/drive/folders/1CZ92XwC0iZ2szLT-RiWbrvpEutURmanH?usp=sharing
I checked the php7.2-fpm log and it was empty except for when it is “Started and listening” after a reboot. I checked the status and it is running:

systemctl status php7.2-fpm
● php7.2-fpm.service - The PHP 7.2 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.2-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-08-13 13:43:12 EDT; 21h ago
     Docs: man:php-fpm7.2(8)
 Main PID: 1108 (php-fpm7.2)
   Status: "Processes active: 0, idle: 3, Requests: 271, slow: 0, Traffic: 0req/sec"
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/php7.2-fpm.service
           ├─ 1108 php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
           ├─ 1143 php-fpm: pool www
           ├─ 5946 php-fpm: pool www
           └─10539 php-fpm: pool www

Aug 13 13:43:12 uranus.ooguy.com systemd[1]: Starting The PHP 7.2 FastCGI Process Manager…
Aug 13 13:43:12 uranus.ooguy.com systemd[1]: Started The PHP 7.2 FastCGI Process Manager.

I added saturn.ooguy.com:7070 to the “Replace S3 Header” option under “Config Backends” and I can now create files and folders, open and download files, and upload new files. Please mark this topic as SOLVED!