Hello, hope you’re all doing well today. I recently stood up a Pydio Cells installation in AWS and I’m experiencing a few problems that I’m hoping someone can help me with. My problems/questions are as follows:
I am unable to utilize any “additional” storage on the Pydio Server. Whenever I tried to add a new disk into the software, it caused major issues. (more details below)
Is there a way to make Pydio listen on more than one URL? For example, lets say I wanted to setup the website: downloads.kevin.com but I also wanted to setup downloads.kevin.org on the same server. Is there any way to have the server listen to both? Or would I need to put some kind of proxy or URL redirection in front of the server?
Problem Description for Problem 1:
I setup Pydio in AWS according to the Pydio cells guides located on the Pydio webpage. I am using an Ubuntu VM hosted in Amazon AWS. My original plan was to have the OS on one physical disk and to have any Pydio related file storage on another physical disk. This would obviously ease backups/snapshots/moving to new servers in the future.
For those who are unaware the typical disk on AWS is called an EBS(Elastic Block Storage). For all intents and purposes, this appears to the OS as a physically connected hard drive. Nothing special there that I am aware of. When operating with just the disk that the OS installed on, everything works fine. However, when I add a new disk and try to add a new datastore and workspace location in Pydio, the server no longer works properly. Uploading files appears to work but as soon as they are uploaded, they are immediately deleted and the server tries to re-upload them again. Just to be sure, I attempted to do this a few separate ways:
I attempted to mount the new drive in /mnt/testing/pydio (or something similar to that… basically I just wanted it to be enough layers deep into the filesystem that Pydio would be okay with it).
I attempted to copy the contents of one of the existing Pydio folders (cellsdata, pydiods1, etc) into the new drive. Then I mounted that drive into the filesystem at the proper location… same type of errors.
From a permissions perspective everything looks the same, from an ownership perspective all of the files are owned by the pydio user and group. Within the OS itself, the pydio user can create, delete, move files etc without any issues. I’m pretty sure it can’t be an incompatibility with EBS because the base OS disk is also an EBS.
Additionally, I’ve tried the same steps with whats called an EFS (Elastic File System). This winds up appearing to the OS as an NFS mounted disk. This had the same issues.
This has happened across several rebuilds of the entire server(aka, starting from scratch with a new AWS Instance).
Has anyone seen issues like this or have any ideas what else I can try? I would really like to build this server “properly” and not have to worry about things like the OS disk running out of space, difficult backups, etc.
Thanks for the detailed description of your issue. At first sight, you seem to follow the right path and things should be working smoothly. The important part when pointing on mounted folders is to make sure that the datasource object storage can properly start, and it starts on the parent folder of the target location (and writes some internal stuff in it). So if /mnt/testing has the correct permissions, /mnt/testing/pydio should be correct for creating a DS. Does the DS report all services running correctly?
The “upload and immediately disappear” behavior is known for a failed upload (data is in fact never uploaded, but an entry temporarily created in the index and then delete after upload fails).
I guess the next step would be to look at your cells logs. Can you check what are the errors there when you perform an upload?
Regarding the multiple address support, unfortunately it’s not there yet, but should be soon. We will work on that soon after the 2.0 release (probably 2.1).
Mr. Charles. Thank you for your response. I’ve attached images from the device that show the setup. It definitely looks like it has problems seeing the datastore. I also attached some folder and file permissions as well as “mount” data.
One thing I noticed is that when setting up the workspace the “folder path” option spins seemingly forever. Definite sign that something isn’t connecting to storage correctly.
I also confirmed that the service does run as the pydio user. Both by checking ps -aux and by checking the systemd service file specifying the user.
Does anything jump out at you as looking wrong? Obviously it says the datastore is unavailable. I’m almost tempted to try out the same configurations in a VM and see if I have the same issues. If no one else using this software is having this issue it has to be something I’m doing. Also please note, for some reason it looks like the “…” in the ouput is showing up as “…” in the formatted version on this page but the original does look normal.
For what its worth, I also tested with Ubuntu’s AppArmor off, I’m not sure if there is any other weird stuff like extended ACLs that could be messing with permissions but its a vanilla Ubuntu 18.04 AMI from the AWS Marketplace so I doubt that stuff would even be turned on.
#“Testing” Folder Permissions
ubuntu@ip-172-31-32-130:/mnt$ ls -al
total 12
drwxr-xr-x 3 root root 4096 Nov 6 15:55 .
drwxr-xr-x 23 root root 4096 Oct 31 21:01 …
drwxr-xr-x 3 pydio pydio 4096 Nov 6 15:56 testing #Pydio Folders Folder Permissions
ubuntu@ip-172-31-32-130:/mnt$ ls -al /home/pydio/.config/pydio/cells/data/
total 40
drwxr-xr-x 10 pydio pydio 4096 Nov 1 17:00 .
drwxrwxr-x 5 pydio pydio 4096 Nov 1 15:13 …
drwxr-xr-x 6 pydio pydio 4096 Oct 31 15:08 .minio.sys
drwxr-xr-x 2 pydio pydio 4096 Oct 31 17:27 binaries
drwxr-xr-x 3 pydio pydio 4096 Oct 31 18:46 cellsdata
drwxr-xr-x 3 pydio pydio 4096 Oct 31 15:08 personal
drwxr-xr-x 2 pydio pydio 4096 Oct 31 15:13 pydiods1
drwxr-xr-x 2 pydio pydio 4096 Nov 1 15:24 thumbs
drwxr-xr-x 2 pydio pydio 4096 Oct 31 15:21 userdata
drwxr-xr-x 2 pydio pydio 4096 Oct 30 21:59 versions #Testing/Pydio Folder Permissions
ubuntu@ip-172-31-32-130:/mnt$ cd testing/
ubuntu@ip-172-31-32-130:/mnt/testing$ ls -al
total 12
drwxr-xr-x 3 pydio pydio 4096 Nov 6 15:56 .
drwxr-xr-x 3 root root 4096 Nov 6 15:55 …
drwxr-xr-x 2 pydio pydio 4096 Nov 6 16:04 pydio #Interior Pydio Folder Permissions
ubuntu@ip-172-31-32-130:/mnt/testing$ cd pydio/
ubuntu@ip-172-31-32-130:/mnt/testing/pydio$ ls -al
total 8
drwxr-xr-x 2 pydio pydio 4096 Nov 6 16:04 .
drwxr-xr-x 3 pydio pydio 4096 Nov 6 15:56 … #Mounting Options
ubuntu@ip-172-31-32-130:/mnt/testing/pydio$ mount |grep /dev/xvd
/dev/xvda1 on / type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdf1 on /mnt/testing/pydio type ext4 (rw,relatime,discard,data=ordered)
Interestingly enough, after trying several much more difficult things, I made the folder one level deeper and everything started working:
I set the datastore location to this and its all good now:
/mnt/testing/pydio/temp
As far as I can tell the permissions were the same between the two but maybe I just missed something glaring. Thank you guys for your time, it is much appreciated.
Another setup you could try can be by using symbolic links : create a symlink mount -> /mnt/test/pydio inside your .config/pydio/cells/data/ folder, then you should be able to create a DS pointing to that mount folder. Not 100% sure but i think it will work.
-c
