404 Site is not served on this interface

Pydio Cells
1

Hi all,

I’ve installed latest version of Pydio Cells on clean VM with Ubuntu 18.04 LTS server and MariaDB v10.4.5 this official guides:
https://pydio.com/en/docs/cells/v1/ubuntu-systems
https://pydio.com/en/docs/cells/v1/install-pydio-cells

During install I’ve used this options:

Welcome to Pydio Cells Home Edition installation
Pydio Cells Home Edition will be configured to run on this machine. Make sure to prepare the following data
 - IPs and ports for binding the webserver to outside world
 - MySQL 5.6+ (or MariaDB equivalent) server access
Pick your installation mode when you are ready.

✔ Command line (performed in this terminal)
Cannot find vaultKeyPath, creating new one /home/pydio/.config/pydio/cells/cells-vault-key
**************************************************************
     Warning! A keyring is not found on this machine,
        A Master Key has been created for cyphering secrets
   It has been stored in /home/pydio/.config/pydio/cells/cells-vault-key
   Please make sure to secure this file and update the configs
   with its new location, under the defaults/keyPath key.
***************************************************************
✔ localhost:8080
✔ Disable SSL (not recommended)
✔ External Url, used to access application from outside world (it can differ from internal url if you are behind a proxy or inside a private network): http://10.0.20.54:8080

## Database Connection
✔ TCP
✔ Database Hostname: localhost
✔ Database Port: 3306
✔ Database Name: cells
✔ Database User: pydio
✔ Database Password (leave empty if not needed): *********
✔ Successfully connected to the database

## Frontend Configuration
✔ Admin Login (leave passwords empty if an admin is already created): admin
✔ Admin Password: ************
✔ Confirm Password: ************

## Advanced Settings
✗ There are some advanced settings for ports and initial data storage. Do you want to edit them:

## Performing Installation
2019-05-22T10:29:44.800Z        INFO    Starting installation now       {"bindUrl": "http://localhost:8080"}
✔ Created main database
✔ Created default datasources
✔ Configuration of gateway services
✔ Creation of logs directory

✔ Installation Finished: please restart with './cells start' command

Finally, when I try to access Pydio by typing http://10.0.20.54:8080 in web browser i got this response:

404 Site 10.0.20.54:8080 is not served on this interface

This is clean Ubuntu server installation without any additional software, no Apache, no PHP or anything else except MariaDB.

Does anyone have any idea what is the problem? Thx in advance.

2

Hi @UniMaksa,

Could you please share the logs when you start cells ?

Thanks in advance,
Greg

3

What happens if you change it to 0.0.0.0:8080? I had to do that to get Cells to work.

4

I have a very similar setup with ubuntu and I’m seeing the exact same problem.
I’ve tried internal_urls as:
localhost:8080
0.0.0.0:8080
192.168.2.77:8080
192.168.2.77:80

and external_urls
http://192.168.2.77:8080
http://192.168.2.77:80

Note that the server is in a DMZ with an interface for 192.168.2.77 which is what I need to use to talk to the outside world. The firewall has a one-to-one NAT for 207.34.219.77<->192.168.2.77

{"caddyfile": "\n\t\thttp://192.168.2.77:80 {\n\t\tproxy /a  192.168.2.77:37047 {\n\t\t\twithout /a\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\t\tproxy /auth/dex 192.168.2.77:37805 {\n\t\t\tinsecure_skip_verify\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\t\tproxy /io   :39949 {\n\t\t\theader_upstream Host 192.168.2.77\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\t\tproxy /data :39949 {\n\t\t\theader_upstream Host 192.168.2.77\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\t\tproxy /ws   192.168.2.77:40785 {\n\t\t\twebsocket\n\t\t\twithout /ws\n\t\t}\n\t\tproxy /plug/ 192.168.2.77:34647 {\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t\theader_downstream Cache-Control \"public, max-age=31536000\"\n\t\t}\n\t\tproxy /dav/ 192.168.2.77:34723 {\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\n\t\tproxy /public/ 192.168.2.77:34647 {\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\n\t\tproxy /user/reset-password/ 192.168.2.77:34647 {\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\n\t\tproxy /robots.txt 192.168.2.77:34647 {\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\n\t\tproxy /login 192.168.2.77:34647/gui {\n\t\t\twithout /login\n\t\t\theader_upstream Host {host}\n\t\t\theader_upstream X-Real-IP {remote}\n\t\t\theader_upstream X-Forwarded-Proto {scheme}\n\t\t}\n\n\t\tredir 302 {\n\t\t  if {path} is /\n\t\t  / /login\n\t\t}\n\n\t\t\n\t\t\n        proxy /wopi/ 192.168.2.77:46277 {\n            transparent\n        }\n\n        proxy /loleaflet/ https://localhost:9980/loleaflet {\n            transparent\n            insecure_skip_verify\n            without /loleaflet/\n        }\n\n        proxy /hosting/discovery https://localhost:9980/hosting/discovery {\n            transparent\n            insecure_skip_verify\n            without /hosting/discovery\n        }\n\n        proxy /lool/ https://localhost:9980/lool/ {\n            transparent\n            insecure_skip_verify\n            websocket\n            without /lool/\n        }\n    \n\t\t\n\n\t\trewrite {\n\t\t\tif {path} not_starts_with \"/a/\"\n\t\t\tif {path} not_starts_with \"/auth/\"\n\t\t\tif {path} not_starts_with \"/io\"\n\t\t\tif {path} not_starts_with \"/data\"\n\t\t\tif {path} not_starts_with \"/ws/\"\n\t\t\tif {path} not_starts_with \"/plug/\"\n\t\t\tif {path} not_starts_with \"/dav/\"\n\t\t\t\n\t\t\tif {path} not_starts_with \"/wopi/\"\n\t\t\t\n\t\t\tif {path} not_starts_with \"/loleaflet/\"\n\t\t\t\n\t\t\tif {path} not_starts_with \"/hosting/discovery\"\n\t\t\t\n\t\t\tif {path} not_starts_with \"/lool/\"\n\t\t\t\n\t\t\tif {path} not_starts_with \"/public/\"\n\t\t\tif {path} not_starts_with \"/user/reset-password\"\n\t\t\tif {path} not_starts_with \"/robots.txt\"\n\t\t\tto {path} {path}/ /login\n\t\t}\n\n\t\t\n\t\terrors \"/home/pydio/.config/pydio/cells/logs/caddy_errors.log\"\n\t\t}\n\n\t\t\n\t"}

Note that the response returned by Cells is this

404 Site 207.34.219.77 is not served on this interface

This says to me that the NAT is working to move packets back and forth, but somehow Caddy is seeing the destination as 207.34.219.77 even though it should see 192.168.2.77. I suspect there is a configuration for Caddy that is meant to handle NATs like these. Is there a way to configure Caddy more directly?

5

I have an update.

I learned something new about Caddy (the web server under Cells). I’m used to using Apache and I understand how it works, but Caddy is pretty new and does things very differently. I had assumed (partly because of the suggested list of options) that “internal_url” was a euphemism for internal interface, and similarly that external_url would therefore be the external interface. If that were the case, then my NAT would be easy.

Caddy is nothing like Apache. Apache does only what I tell it, and if I’m wrong, that’s my stupid fault.
Caddy, on the other hand, tries to do “intelligent” things like figure out if you’re behind a NAT or a proxy and configure the network interfaces all on its own. Apparently I have to hold its hand while it tries to figure it out for me.

Long story short, I got the main page accessible externally by setting

internal_url = https://get2.hhangus.com:443
external_url = https://get2.hhangus.com

HOWEVER, this isn’t the end because even though the main page comes up now, the login fails saying that the service is unavailable/timeout. So, something still isn’t being translated correctly somewhere. Also, I now get the following errors showing in the startup of cells:

2019-08-15T15:13:15.537-0400	ERROR	pydio.grpc.user-meta	could not initialise service at version 1.6.1	{"error": "Error 1062: Duplicate entry 'usermeta-tags' for key 'namespace'"}
2019-08-15T15:13:15.538-0400	ERROR	pydio.grpc.user-meta	Could not run 	{"error": "Error 1062: Duplicate entry 'usermeta-tags' for key 'namespace'"}

2019-08-15T15:13:23.111-0400	ERROR	pydio.grpc.data.sync.personal	Cannot contact s3 service (bucket personal), will retry in 4s	{"error": "Server not initialized, please try again."}
2019-08-15T15:13:23.127-0400	ERROR	pydio.grpc.data.sync.pydiods1	Cannot contact s3 service (bucket pydiods1), will retry in 4s	{"error": "Server not initialized, please try again."}
2019-08-15T15:13:23.184-0400	ERROR	pydio.grpc.data.sync.cellsdata	Cannot contact s3 service (bucket cellsdata), will retry in 4s	{"error": "Server not initialized, please try again."}

2019-08-15T15:13:27.045-0400	ERROR	pydio.grpc.data.sync.personal	Cannot contact s3 service (bucket personal), will retry in 4s	{"error": "Server not initialized, please try again."}
2019-08-15T15:13:27.125-0400	ERROR	pydio.grpc.data.sync.pydiods1	Cannot contact s3 service (bucket pydiods1), will retry in 4s	{"error": "Server not initialized, please try again."}
2019-08-15T15:13:27.183-0400	ERROR	pydio.grpc.data.sync.cellsdata	Cannot contact s3 service (bucket cellsdata), will retry in 4s	{"error": "Server not initialized, please try again."}
6

Oh, and I’m getting these errors when I attempt the login:

2019-08-15T15:32:00.930-0400	ERROR	Error while applying modifiers to registry!	{"error": "{\"id\":\"go.micro.client\",\"code\":500,\"detail\":\"Error creating stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \\\"transport: Error while dialing dial tcp 192.168.2.77:41299: connect: connection refused\\\"\",\"status\":\"Internal Server Error\"}"}
2019-08-15T15:34:15.634-0400	ERROR	pydio.rest.frontend	Connection timeout while trying to retrieve a JWT token.
This usually happens with wrong network configuration. Please insure that the machine that runs your Cells instance can reach itself using your public FQDN.
2019-08-15T15:34:15.634-0400	ERROR	pydio.rest.frontend	Rest Error 401	{"error": "Post https://get2.hhangus.com/auth/dex/token: dial tcp 207.34.219.77:443: connect: connection timed out"}

=============================================================
I have resolved the issues described above.

internal_url = get2.hhangus.com:443
external_url = https://get2.hhangus.com

Then add “192.168.2.77 get2.hhangus.com” to the /etc/hosts file so that the URL is mapped to the local interface IP. It seems that Caddy is using get2.hhangus.com on the external NAT IP for internal communications and this was breaking a lot of things. This is why, IMO, the concept of a web server that tries to configure itself from URLs instead of interfaces is doomed to failure every time. I suspect there are ways to configure Caddy properly, but Cells does not provide any fine-grained control of Caddy, or at least provides no documentation of such capabilities. I’m pretty sure if I could configure Caddy myself I would have resolved this much more quickly.

Further, I needed to wipe the entire install including database and .config files and re-install cleanly several times. Re-installs without wiping caused several issues from meta-data that wasn’t updated to SSL issues, to missing Ladon security policies. In regards to the Ladon problems, there were issues with the MySQL user not have full access so the database and tables were created, but no records were inserted. This was resolved by giving the pydio mysql user full permissions, equivalent to admin, before a clean install.

FWIW, this has not been a good experience. No responses here at all while I solve serious problems with your product installer. The installer itself does not do clean installs when run multiple times, which caused a number of issues. The least you could do is provide a “clean” option with your installer to do the grunt work of removing everything and deleting the database. No detection of MySQL permissions issues, you could at least check the records are created after setup. And the config of Caddy is opaque with little in the way of direction about how or why a problem might occur. You need better documentation regarding NAT configurations, I hope to see my solution here added to your documentation in the future.

Oh, and now just to top it off your forum won’t let me post a 3rd response in a row despite the fact I’m posting the solution SMH. I have instead added this to my previous response.

1 Like
7

I’m having very similar problems. Never-ever had problems with nginx or Apache, but Caddy is apparently trying too hard to be “good for everything”, but sucks at it…

I’m trying to run non-TLS instance of Pydio Cells on machine with few interfaces – eth0, eth1, eth2… Caddy is randomly picking interface for internal connections and cannot force it to use same each time…

ifconfig:

eth0      Link encap:Ethernet  HWaddr 02:42:0A:FF:00:73  
          inet addr:10.255.0.115  Bcast:10.255.255.255  Mask:255.255.0.0

eth1      Link encap:Ethernet  HWaddr 02:42:0A:00:16:21  
          inet addr:10.0.22.33  Bcast:10.0.22.255  Mask:255.255.255.0

eth2      Link encap:Ethernet  HWaddr 02:42:AC:12:00:07  
          inet addr:172.18.0.7  Bcast:172.18.255.255  Mask:255.255.0.0

/etc/hosts:

127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
10.0.22.33	c5ceda05ab6f
10.0.22.33	my.external.com

Env vars:

CELLS_NO_TLS: 1
CELLS_BIND: my.external.com:8080
CELLS_EXTERNAL: https://my.external.com

Proxy configured to work on https://my.external.com with SSL termination pointing to instance at port 8080.

And what is Caddy doing? Logs:

{"level":"info","ts":"2020-05-03T15:21:01Z","logger":"pydio.gateway.proxy","msg":"Restarting proxy","caddyfile":"
http://my.external.com:8080 {
	proxy /a  10.255.0.115:36213 {
		without /a
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	proxy /auth/dex 10.255.0.115:43851 {
		insecure_skip_verify
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	proxy /oidc 10.255.0.115:34037 {
		insecure_skip_verify
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	proxy /oidc-admin/oauth2/auth/requests 10.255.0.115:34037 {
		insecure_skip_verify
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	proxy /io   NOT_AVAILABLE {
		header_upstream Host my.external.com
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	proxy /data NOT_AVAILABLE {
		header_upstream Host my.external.com
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	proxy /ws   10.255.0.115:37513 {
		websocket
		without /ws
	}
	proxy /plug/ 10.255.0.115:46305 {
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
		header_downstream Cache-Control \"public, max-age=31536000\"
	}
	proxy /dav PENDING {
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	
	proxy /public/ 10.255.0.115:46305 {
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	
	proxy /user/reset-password/ 10.255.0.115:46305 {
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	
	proxy /robots.txt 10.255.0.115:46305 {
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}
	
	proxy /login 10.255.0.115:46305/gui {
		without /login
		header_upstream Host {host}
		header_upstream X-Real-IP {remote}
		header_upstream X-Forwarded-Proto {scheme}
	}


	redir 302 {
		
		if {path} is /
		/ /login
	}
	
	
	
        proxy /wopi/ PENDING {
            transparent
        }

        proxy /loleaflet/ https://localhost:9980/loleaflet {
            transparent
            insecure_skip_verify
            without /loleaflet/
        }

        proxy /hosting/discovery https://localhost:9980/hosting/discovery {
            transparent
            insecure_skip_verify
            without /hosting/discovery
        }

        proxy /lool/ https://localhost:9980/lool/ {
            transparent
            insecure_skip_verify
            websocket
            without /lool/
        }
    
	
	
	rewrite {
		if {path} not_starts_with \"/a/\"
		if {path} not_starts_with \"/auth/\"
		if {path} not_starts_with \"/oidc/\"
		if {path} not_starts_with \"/oidc-admin/oauth2/auth/requests/\"
		if {path} not_starts_with \"/io\"
		if {path} not_starts_with \"/data\"
		if {path} not_starts_with \"/ws/\"
		if {path} not_starts_with \"/plug/\"
		if {path} not_starts_with \"/dav\"
		
		if {path} not_starts_with \"/wopi/\"
		
		if {path} not_starts_with \"/loleaflet/\"
		
		if {path} not_starts_with \"/hosting/discovery\"
		
		if {path} not_starts_with \"/lool/\"
		
		if {path} not_starts_with \"/public/\"
		if {path} not_starts_with \"/user/reset-password\"
		if {path} not_starts_with \"/robots.txt\"
		to {path} {path}/ /login
	}
	root /45e818ed-e6ef-4b7a-876c-3163b4b23d3f
	
	
	errors \"/var/cells/logs/caddy_errors.log\"
}
"}

Why the hell is Caddy using 10.255.0.115 instead of 10.0.22.33 to bind internally?!

8

Thank’s for help into that message !