Pydio 8 Migration - Data Import - Pydio 8 Connection fails

[dpat]

Hello,

I am finally trying to migrate from Pydio 8.2.5 (running on Debian/stretch) to Cells 2.2.12 (running on Docker on Debian/bullseye).

After hours of debugging I think I need help. I’ve ran out of ideas how to solve the problem.

In Step 3 (Pydio 8 Connection) of the Data Import I always get the error: “Cannot connect to https://mypydio8.example.com/api/v2/admin/workspaces”.

I’ve read: Migrate Pydio 8 to Cells | Pydio

I can confirm that I have done the following:

• Enabled AllowEncodedSlashes On in my Pydio 8 apache config
• Created dedicated admin user for the import on Pydio 8
• Downloaded and installed the migration plugin on Pydio 8.

(See: Migrating from Pydio 8.2.5 -> Cells 2.0.5 - #2 by zayn)

In the Firefox Developer Tools Console I came across the following error:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote source at https://mypydio8.example.com/api/v2/admin/workspaces. (Reason: CORS header 'Access-Control-Allow-Origin' missing).
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote source at https://mypydio8.example.com/api/v2/admin/workspaces. (Reason: CORS request did not succeed).

Which lead me to play around with something like this in my Pydio 8 apache config:

Header always set Access-Control-Allow-Origin "https://pydio-cells.intern.example.com:8080"
Header always set Access-Control-Allow-Methods "GET, POST, PUT, OPTIONS"
Header always set Access-Control-Allow-Headers "Authorization, Content-Type, X-Requested-With"
Header always set Access-Control-Allow-Credentials "true"

(Please note, that there is a Cross-Origin resource sharing section in the Pydio 8 Main Options (Application Parameters → Application Core), but changing these settings somehow had no effect).

I found no documentation and/or hint in the forums about this. Is this really needed?

The Cross-Origin Request Blocked error vanished, but now I am stuck with the following error (in the Developer Console):

XHR GET https://mypydio8.example.com/api/v2/admin/workspaces [HTTP/1.1 401 Unauthorized 98ms]

Can you please help me on this? Thank you!

[dpat]

I think I finally solved this problem.

In Pydio 8 → Settings → Application Parameters → Application Core → Cross-Origin resource sharing (CORS) I had to set the following settings:

• Allowed Origin: https://pydio-cells.intern.example.com:8080
• Allowed Methods: GET, POST, OPTIONS
• Allowed Headers: Authorization, Content-Type, X-Requested-With

I actually only changed Allowed Origin, I kept the other fields unchanged. Note, that “Origin” is identified by [SCHEME]://[HOST]:[PORT] (this wasn’t obvious for me) and that I had to make sure that the Pydio Cells URL / host was reachable by the Pydio 8 host (mypydio8.example.com) and vice versa (not only by me with the browser).
This was might be obvious, but I accessed the Pydio Cells host via the internal DMZ-IP (via VPN) and the Pydio 8 host via the external IP. I had to set correct DNS / /etc/hosts/ entries to make that work.

But this was not enough. I also had to change the Pydio 8 Apache settings and added the following lines:

Header set Access-Control-Allow-Origin "https://pydio-cells.intern.example.com:8080"
Header set Access-Control-Allow-Methods "GET, POST, PUT, OPTIONS"
Header set Access-Control-Allow-Headers "Authorization, Content-Type, X-Requested-With"

Note, that the lines had to read Header set Access-[...] not Header always set Access-[...] (which makes sense, if you read the Apache Module mod_headers documentation). After reloading Apache, the Pydio Cell Importer connection finally worked.

I am still not sure why the Apache directive is needed (it should not be needed if the header is set via Pydio, or?). If someone can shed some light on it, I would be very grateful.

Also, I was very surprised that I could not find a word about Cross-Origin resource sharing or CORS in the documentation (the forum was also very silent about that subject). You might want to add a few words about that subject to help some poor people like me. Thank you!

[blwallace]

I too am struggling with migration from Pydio8.2.x to Cells. You mention a Cross-Origin in Settings->Application Parameters->Application Core, but I don’t have that option. Is that hidden somewhere?

I’ve tried various edits to my apache2 pydio.conf file

• Header set Access-Control-Allow-Origin always (this line I’ve modified from always to “*” and “https://cells.domain:8080” but no luck)
• Header set Access-Control-Allow-Headers “Origin, X-Requested-With, Content-Type, Accept, Authorization”
• Header set Access-Control-Allow-Methods “PUT, GET, POST, DELETE, OPTIONS”

I get a 200 OK when running wget from both servers to each other.

Any other ideas?

[dpat]

Hmm. I am sorry, I only can tell you that I have this Cross-Origin resource sharing (CORS) option (I have to scroll down in Application Core and it is next to Miscellaneous). And also that the setting was essential for me to be able to migrate Pydio 8.2.5 to Cells together with the Apache settings. Why is still a mystery to me. (Then also make sure that you run the Pydio Cells Data Import via the given URL).

But we can try find it together: What Pydio version are you running?

Are you sure the migration plugin is successfully installed? (Check it in Pydio 8 → Settings → All Plugins → Available Plugins → Action Plugins → Migration. Just noticed and whoever cares: the link to the documentation of the Migration Plugin is broken). Make sure the plugin exists and is enabled.

I am not aware of an “Advanced Settings” option in Pydio 8 (Pydio Cells does have a “show/hide advanced parameters” toggle switch), but I will keep looking.

I wish you luck! I am still struggling with the migration ([1], [2]). It’s much harder than expected.

Please note: I wanted to edit my post and wanted to remove PUT from Access-Control-Allow-Methods from the Apache config (because only GET, POST, OPTIONS were given in the Pydio 8 / CORS settings page) as my migration worked without PUT. Don’t know if it is harmful, though. So this is how it should look like:

Header set Access-Control-Allow-Origin "https://pydio-cells.intern.example.com:8080"
Header set Access-Control-Allow-Methods "GET, POST, OPTIONS"
Header set Access-Control-Allow-Headers "Authorization, Content-Type, X-Requested-With"

[blwallace]

I’ve been working with support the last month on various Pydio things - one of which is migration. Support suggested installing Nginx alongside Apache and running Pydio from that web server. I have done that and made a successful migration (although, not without many trial and errors).

I have notes on Nginx if you’re interested.

I’m running Pydio8.2.4 - never made it to 8.2.5.

[dpat]

So you were able to migrate successfully without the CORS option by using nginx?
(Maybe the CORS settings are missing in Pydio 8.2.4?)

And yes, although I was able to come over my CORS problem even with my Apache setup, I think it would be wonderful and helpful if you’d share your notes.

[blwallace]

Well Yes & No. The Nginx configuration is still modified for CORS, but it works better with PHP-FPM than does Apache.

When I first tried to migrate and subsequently called support, they suggested…
“Is it possible to switch to [nginx + php-fpm] or [apache + php + libapache2-mod-php] ? We do not support the combination of apache + php-fpm.
You can find an example config of nginx this link: pydio-core/nginx.conf.sample at develop · pydio/pydio-core · GitHub”

I already had PHP-FPM installed with Apache and Pydio 8.0.1 (our host server is Ubuntu 16.04.7 LTS running Pydio 8.2.4), so this is what started me down the Nginx road.

I assume you have the php-fpm (my version is 7.0) and migration plugin already installed in your Pydio8 installation.

-btw I’m not an Nginx nor Apache expert!

Here’s my notes on installing Nginx on Ubuntu

Steps to install Pydio 8 on nginx

1. sudo apt update
2. sudo apt install nginx

Modify nginx to listen on port 8080 (apache2 is already on port 80)

1. vi /etc/nginx/sites-available/default

 server {
 listen 8080 default_server;
 listen [::]:8080 default_server;

2. vi /etc/nginx/sites-available/pydio.conf

# Debian 8 with default /etc/nginx/nginx.conf
# File name: /etc/nginx/sites-available/pydio.conf
server {
     listen 8080 default_server;
     keepalive_requests    10;
     keepalive_timeout     60 60;

     client_max_body_size 800M;
     client_body_buffer_size 128k;

   # Document Root
     root /usr/share/pydio;
#if ($request_method ~* "(GET|POST)") {
#add_header "Access-Control-Allow-Origin" *;
#}
#if ($request_method = OPTIONS){
add_header "Access-Control-Allow-Origin" "*" always;
add_header "Access-Control-Allow-Headers" "Origin, X-Requested-With, Content-Type, Accept, Authorization";
add_header "Access-Control-Allow-Methods" "PUT, GET, POST, DELET, OPTIONS, HEAD";
#}

index index.php;

     if (!-e $request_filename){
             rewrite ^(.*)$ /index.php break;
     }

     # nginx configuration
     location ~ \.php$ {
             include snippets/fastcgi-php.conf;
             #fastcgi_pass unix:/var/lib/php5-fpm/www.sock;
             fastcgi_pass unix:/run/php/php7.0-fpm-pydio.sock;
     }

     # Manually deny some paths to ensure Pydio security
     location ~* ^/(?:\.|conf|core|data/(?:files|personal|logs|plugins|tmp|cache)|plugins/editor.zoho/agent/files) {
             deny all;
     }

}
types {
     application/font-woff2                 woff2;
}

3. enabled pydio site

sudo ln -s /etc/nginx/sites-available/pydio.conf /etc/nginx/sites-enabled/

4. check nginx syntax

nginx -t

5. ensure php-fpm configured for nginx

• How to Configure Nginx to Execute PHP Using PHP-FPM
• How to add PHP-FPM support for NGINX sites

6. start nginx

sudo systemctl start nginx.service

Access Pydio using https://:8080