Upload Files via Cells API

What API method(s) allows you to upload a file(s) to a workspace. I’m unable to find any that referenced uploading files or viewing files. I can see many that allow you to view workspaces/directories but none that allow you to upload files to those workspaces:

https://demo.pydio.com/a/tree/admin/list
https://demo.pydio.com/a/meta/bulk/get
https://demo.pydio.com/a/config/virtualnodes

I feel like I am needing to utilize the TreeService API methods however I’m still at a loss. The description of the ‘/tree/admin/list’ says that it “List files and folders starting at the root (first level lists the datasources)”.

Each time I make a call all I get back are the data sources and their immediate children. I never get any files to appear in the list. I can see when logging into the demo using the admin user that there are plenty of files in the personal workspace which has a workspace root of my-files and is stored in the datasource called ‘personal’ however I am still unable to get a list of files returned.

The body and parameters I am submitting are:
{
“Ancestors”: true,
“Limit”: 0,
“Node”: {
“Mode”: 0,
“Path”: “personal”
},
“Offset”: 0,
“Recursive”: true
}

What am I missing? All my work here is in attempts to learn how to use the API to look for and upload files however I am not able to figure this out. If anyone could point me to more robust documentation that would be nice as well. The only documentation I can find is located at this link and it is severely limited.

https://pydio.com/en/docs/developer-guide/cells-api

Hi,
to upload files you must use s3 api, you can find the informations over here:
https://pydio.com/en/docs/developer-guide/usage

Thank you. Finally getting back to working in Pydio. For others who may find the information useful In Postman I used these settings to pull back information contained in a file I placed into the Personal Files workspace on the demo.pydio.com server.

GET https://demo.pydio.com/io/personal-files/[my-file.txt]

Authorization: AWS Signature
AccessKey: Token returned when using OpenID Connect. The “id_token” contained in the body.
SecretKey: The demo uses the key: ‘gatewaysecret’

Advanced Options:
AWS Region: Default is ‘us-east-1’. I didn’t have to enter anything here but it still worked when I set it to ‘us-west-1’.
Service Name: ‘s3’ - I found that this is Required
Session Token: I left this blank.

404 error when I try to go to this page. :frowning:

Having the same issue. I’m using the /a/frontend/session endpoint with a username and password to successfully get a session, including a JWT token I can use in the Authorization header.

With this I’m able to execute other API calls, such as `/a/tree/stats’, ‘/a/user/’, successfully as well.

There is virtually NO information, however, on how to upload or download files to a workspace or cell.

Using chrome developer panel to watch the request/response traffic when I use the web front-end to pydio, I was able to deduce the download. Using a GET request with url of the form

 <domain>/io/ws-<workspace-name>/rest-of-the-path-to-the-file-to-download>

I’m able to download a file.

Also using chrome developer panel, I’m able to see that uploading first checks to see if the path already exists (POST /a/tree/stats) and then does a PUT, with stuff in the querystring that, near as I can tell, is either duplicated in the headers (e.g. Content-Type) or irrelevant.

It also seems to include a header X-Pydio-Bearer the value of which looks like a JWT token. When I try to use the JWT I received when I logged in, I get this error in the logs:

2020-01-20T00:00:36.772Z        ERROR   pydio.rest.frontend     error retrieving token  {"token": "eyJhbGciOiJSUzf [deleted for brevity] W9WNbN5X3fxA", "error": "empty idToken"}

So, obviously I need to grok up another JWT. with a different embedded token. Here I’m stuck.

I surmise I’ll need to use the same “secret” to encode the JWT for the PUT request as pydio used to encrypt the JWT it sent me on session creation. Is this correct? If so, has pydio sent me this key? Where is it? (Cookies? header? There is a cookie named ‘pydio’ that looks a bit like a public key, but has no -----BEGIN PUBLIC KEY----- header.)

There does not seem to be ANY documentation on this. I’m happy to write some up as answers come to light.

Hello @n6151h,

I’ve updated the doc on postman (S3) for this matter (assuming that you are on Cells 2.x.x),
https://pydio.com/en/docs/developer-guide/using-postman

Thanks for the reply.

I avoid (like the plague) apps (in this case, postman) that require me to create an account. I don’t care to have my activity monitored, thank you.

I’ve looked at the S3 parts you’ve added. It is not consistent with what I’ve observed. The URL pattern seeems to be .../io/ws-personal-workspace/... – i.e., ws- is prepended to personal-workspace.

Secondly, you seem to be a big fan of “cut and paste”. We’re trying to automate the process here, which obviously rules out cut-and-paste. You did add what AccessKey, SecretKey, and ServiceName should be. The only hurdle now, it seems, is how to obtain (or create) an AccessKey without (directly) using OAuth2. I’m able to log in programatically via the frontend API (using python’s requests module) and do other API calls and even GET a file that’s already in one of the workspaces or cells.

The ONLY thing I can’t seem to get working is the PUT to upload a file to a workspace or cell.

Just noting… unless something has changed Postman does not require an account and is an amazing tool. There is also a tool by SmartBear called SoapUI. It does more than just SOAP and can also be used to test against REST services.

Maybe it’ll help if I include some code …

Here’s how I get my Authorization Token:

domain = 'http://localhost:8080'
user = 'normalusr'
password = 'gobbledygook'
    
# These will be the same for all requests/responses.
headers = {
    'Content-Type': 'application/json', 
    'Accept': 'application/json',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36'
}

# Create an http session.
client = requests.Session()
client.headers['User-Agent'] = self._headers['User-Agent']

# Login
login_data =  {'AuthInfo': {'login': user, 'password': password, 'type': 'credentials'}}
r = client.post('http://localhost:8080/a/frontend/session', json=login_data, headers=headers)

# a valid JWT token should be found in the returned JSON object ...
jwt_token = r.json()['JWT']
jwt_expire = r.json()['ExpireTime']  # in seconds

 # Add this to headers
headers['Authorization'] = 'Bearer {}'.format(jwt_token)

From this point on (until the jwt_token expires) I’m able to make API calls.

For example,

result = client.get('http://localhost:8080/a/user/normalusr', headers=headers)
if result.status_code == 200:
    print("normalusr's profile: ", result.json())

I can do POST requests to the /a/tree/stats endpoint to walk the tree, too. So far, I haven’t found an API endpoint that won’t work using this method.

After a fair bit of tracing with the developer tools panel in chrome, I was able to make file downloads work. I had a file named spock-logic.jpg in my personal workspace. I’ll spare the gory details and just say that the only difference between using an API endpoint and downloading a file involved figuring out what the URL should look like, and using application/octet-stream instead of application/json for the Content-Type. As I described in my earlier comment, the URL pattern looks like this:

download_url = "http://localhost:8080/io/ws-personal-workspace/cute-kitten.jpg"

Note the io prefix (instead of a, as used in the API endpoints), and that Personal Workspace has been sluggified and prepended with ws-.

That sums up what I’ve been able to do. What I cannot do, so far, is upload.

Again, dragging and dropping my logic-spock.jpg file into my Personal Workspace while tracing with developer panel, I see this is just using an http PUT request that looks much like the download. (Note, it seems that it first does a /a/tree/stats API call for the given path, which I presume is to check to see if there is already a file there by that name. When the result comes back with an empty JSON object, it then proceeds to do the PUT request. Here’s a screenshot of the network trace:
image

You can see the tree/stats request, which sends the personal-workspace/logic-spock.jpg in the request’s JSON payload: {"NodePaths":["personal-files/logic-spock.jpg"]}, and gets an empty JSON object in the response body.

Next, it sends a PUT to this URL:

http://10.33.9.46:8080/io/personal-files/logic-spock.jpg?AWSAccessKeyId=gateway&Content-Type=application%2Foctet-stream&Expires=1579564291&Signature=nqIzr%2Fm6VXKPsPxbTTlXomZY1EE%3D

Maybe it’s my bad eyesight, but, I don’t see S3 anywhere in that URL.

Here are the request headers and query string:

PUT /io/personal-files/logic-spock.jpg?AWSAccessKeyId=gateway&Content-Type=application%2Foctet-stream&Expires=1579564291&Signature=nqIzr%2Fm6VXKPsPxbTTlXomZY1EE%3D HTTP/1.1
Host: 10.33.9.46:8080
Connection: keep-alive
Content-Length: 54164
DNT: 1
X-Pydio-Bearer: eyJhbGciOiJSUzI1NiIsImtpZCI6IjVjOGRjNTY0ZGMwZjVlMDcyZmE4ZTIzMTM2YzI0NDUxM2MwZmYyZTYifQ.eyJpc3MiOiJodHRwOi8vMTAuMzMuOS40Njo4MDgwL2F1dGgvZGV4Iiwic3ViIjoiQ2lRMFl6VmhOV0UyTkMweFpUY3lMVFJqWWpJdE9UQXpOQzAzTURGalltRTJaVFV6TWpBU0JYQjVaR2x2IiwiYXVkIjoiY2VsbHMtZnJvbnQiLCJleHAiOjE1Nzk1NjM4ODYsImlhdCI6MTU3OTU2MzI4Niwibm9uY2UiOiI1ODhiNzQwYi1jZmQ2LTRkMDYtOGU0Ny1kNmYyYTk5M2NhMjgiLCJhdF9oYXNoIjoiSXgxNGJXQk1mRWJuWTRxQkdmTzJVQSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoibmlja3MifQ.TpFeFUEzgGHJP2g6FOcQ3UpNFA5TzM-nLg_1qBaOq87FGjAqI3ecXFzO1l9gVK_FyPI7q0xy38SZTXzx2XZqnmFnVuZFJ-qWPLYd8fMVvnWIwK54h4Cx7MCmxzA7OZUzFUh-0aVasEN2lLoMz-f-0RhaUod96Mc1xKcGowBTPn2yFiOWFbXUKAPnbL52xj1g6tz2GL2KgwXkRfQYtMEDv-vL8GQjRv6o1FpZAzDx9uXXlvY-JTcNdslnOLk6RqnoJaIqFaqK91SVME7FqIyovtLPRAAGhNI56BY-O9D8oOcpX6UUhRpDdCTLYzgJEEJv7H1A6XbGmw6oYAB0Jfw26Q
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36
Content-Type: application/octet-stream
Accept: */*
Origin: http://10.33.9.46:8080
Referer: http://10.33.9.46:8080/ws-personal-files/
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8

and the query string:

AWSAccessKeyId=gateway&Content-Type=application%2Foctet-stream&Expires=1579564291&Signature=nqIzr%2Fm6VXKPsPxbTTlXomZY1EE%3D

Again, no mention of S3 at all. There are some other differences, too. Instead of an Authorization header, we have X-Pydio-Bearer, which looks very much like it’s a JWT token. It doesn’t have the Bearer prefix that the Authorization header has, but it is otherwise identical to that value. Yet, when I try to replicate this PUT programmatically, using that token (using just the token part, with out the Bearer prefix found in the Authorization header) I see this in the server log:

2020-01-20T23:48:48.147Z        ERROR   pydio.rest.frontend     error retrieving token  {"token": "eyJhbGciOiJSUzI1NiIsImtpZCI6IjVjOGRjNTY0ZGMwZjVlMDcyZmE4ZTIzMTM2YzI0NDUxM2MwZmYyZTYifQ.eyJpc3MiOiJodHRwOi8vMTAuMzMuOS40Njo4MDgwL2F1dGgvZGV4Iiwic3ViIjoiQ2lRMFl6VmhOV0UyTkMweFpUY3lMVFJqWWpJdE9UQXpOQzAzTURGalltRTJaVFV6TWpBU0JYQjVaR2x2IiwiYXVkIjoiY2VsbHMtZnJvbnQiLCJleHAiOjE1Nzk1NjM4ODYsImlhdCI6MTU3OTU2MzI4Niwibm9uY2UiOiI1ODhiNzQwYi1jZmQ2LTRkMDYtOGU0Ny1kNmYyYTk5M2NhMjgiLCJhdF9oYXNoIjoiSXgxNGJXQk1mRWJuWTRxQkdmTzJVQSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoibmlja3MifQ.TpFeFUEzgGHJP2g6FOcQ3UpNFA5TzM-nLg_1qBaOq87FGjAqI3ecXFzO1l9gVK_FyPI7q0xy38SZTXzx2XZqnmFnVuZFJ-qWPLYd8fMVvnWIwK54h4Cx7MCmxzA7OZUzFUh-0aVasEN2lLoMz-f-0RhaUod96Mc1xKcGowBTPn2yFiOWFbXUKAPnbL52xj1g6tz2GL2KgwXkRfQYtMEDv-vL8GQjRv6o1FpZAzDx9uXXlvY-JTcNdslnOLk6RqnoJaIqFaqK91SVME7FqIyovtLPRAAGhNI56BY-O9D8oOcpX6UUhRpDdCTLYzgJEEJv7H1A6XbGmw6oYAB0Jfw26Q", "error": "empty idToken"}

At this point, I’m rather stuck. I’ll try a couple of the other things suggested in the page Zayn updated, but, it’s not clear to me that these are relevant if I’m not using postman, or the authentication method described.

SOLVED!

I was leaving the Authorization header in place and adding the X-Pydio-Bearer header. REMOVING the Authorization header from the headers resulted in successful upload.

Hello, @n6151h,

sorry if pointed you out to Postman, there is a documentation using CURL https://pydio.com/en/docs/developer-guide/using-curl
it could have better indications for your question.

1 Like

Thank you, Zayn. I had seen that earlier (before posting in this forum) and it was helpful in at least getting me to where I could do the API calls.