Unable to download files from Pydio Cells using Golang

I am currently writing a Golang program for downloading files from Pydio Cells using Golang program. (Using Minio library)

The working code (for downloading all files from a given bucket in a sample server play.min.io is:

package main

import (
	"github.com/minio/minio-go/v6"
	"log"
  "fmt"
  "context"
  "time"
  "os"
  "io"
)

func main() {
	endpoint := "play.min.io"
	accessKeyID := "Q3AM3UQ867SPQQA43P2F"
	secretAccessKey := "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
	useSSL := true

	// Initialize minio client object.
	minioClient, err := minio.New(endpoint, accessKeyID, secretAccessKey, useSSL)
	if err != nil {
		log.Fatalln(err)
	}

	// Make a new bucket called mymusic.
	bucketName := "mymusic"
	location := "us-east-1"

	err = minioClient.MakeBucket(bucketName, location)
	if err != nil {
		// Check to see if we already own this bucket (which happens if you run this twice)
		exists, errBucketExists := minioClient.BucketExists(bucketName)
		if errBucketExists == nil && exists {
			log.Printf("We already own %s\n", bucketName)
		} else {
			log.Fatalln(err)
		}
	} else {
		log.Printf("Successfully created %s\n", bucketName)
	}

  // Create a done channel to control 'ListObjects' go routine.
  doneCh := make(chan struct{})

  // Indicate to our routine to exit cleanly upon return.
  defer close(doneCh)

  isRecursive := true
  objectCh := minioClient.ListObjects("mymusic", "", isRecursive, doneCh)
  for object1 := range objectCh {
    if object1.Err != nil {
      fmt.Println(object1.Err)
      return
    }
    fmt.Println(object1.Key)

    ctx, cancel := context.WithTimeout(context.Background(), 100 * time.Second)
    defer cancel()

    //object, err := minioClient.GetObjectWithContext(ctx, "mymusic", "hello.txt", minio.GetObjectOptions{})
    object, err := minioClient.GetObjectWithContext(ctx, "mymusic", object1.Key, minio.GetObjectOptions{})
    if err != nil {
      fmt.Println(err)
      return
    }

    localFile, err := os.Create("/home/Desktop/MinioDownloads/"+object1.Key)
    if err != nil {
      fmt.Println(err)
      return
    }

    if _, err = io.Copy(localFile, object); err != nil {
      fmt.Println(err)
      return
    }


  }


}

Now, when I modify the same for Pydio Cells, this is the .go file:

package main

import 
(
   "github.com/minio/minio-go/v6"
   "log"
  "fmt"
  "context"
  "time"
  "os"
  "io"
)

func main() {
	
	endpoint := (Some URL)
	accessKeyID := "admin"
	secretAccessKey := "admin"
	useSSL := false
	
	// Initialize minio client object.
	minioClient, err := minio.New(endpoint, accessKeyID, secretAccessKey, useSSL)
	if err != nil {
		log.Fatalln(err)
	}

  // Create a done channel to control 'ListObjects' go routine.
  doneCh := make(chan struct{})

  // Indicate to our routine to exit cleanly upon return.
  defer close(doneCh)

  isRecursive := true
  objectCh := minioClient.ListObjects("personal", "", isRecursive, doneCh)
  for object1 := range objectCh {
    if object1.Err != nil {
      fmt.Println(object1.Err)
      return
    }
    fmt.Println(object1.Key)

    ctx, cancel := context.WithTimeout(context.Background(), 100 * time.Second)
    defer cancel()

    object, err := minioClient.GetObjectWithContext(ctx, "personal", object1.Key, minio.GetObjectOptions{})
    if err != nil {
      fmt.Println(err)
      return
    }

    localFile, err := os.Create("/home/Desktop/MinioDownloads/"+object1.Key)
    if err != nil {
      fmt.Println(err)
      return
    }

    if _, err = io.Copy(localFile, object); err != nil {
      fmt.Println(err)
      return
    }

  }

}

On running this, I get the following error:
XML syntax error on line 2: attribute name without = in element

I am unable to resolve this.
Please suggest

Hi,
Just for information, https://github.com/pydio/cells-client

Hi @c12simple,

Thanks for help. I am actually looking for those Go programming functions using which I can programatically:

  1. Login to my Pydio cells server
  2. Then list all the files and folders present inside it
  3. Download any of those files/folders

Since I am a beginner, I don’t know much.

I have installed cec client, but in the configuration process, I am getting error (Screenshot attached):

My details while configuring cec:
$ cec configure
Server Address (provide a valid URL): (some url)
OAuth APP ID (found in your server pydio.json): cells-client
OAuth APP Secret (leave empty for a public client):
:heavy_check_mark: Yes
Opening URL (some URL)

Hello,

The server is running in http or https? From error message, I guess it is in http.
So cec has a bug with skip certificate check if you are using self-sign cert. Please wait for update version.

Hello

I have one more doubt:

Is the Pydio Cells local storage S3 compatible?

The reason I am asking that is because Minio-go SDK functions (as mentioned in https://docs.min.io/docs/golang-client-api-reference.html)
, the functions like minioClient.ListBuckets(), minioClient.GetObject(), minioClient.PutObject() etc. are applicable for only S3 compatible storage

Thanks

Hello @sid,

I think this article will give you a better understanding of the architecture,
https://pydio.com/en/docs/cells/v2/understanding-datasources

You should be able to interact with the local storage through the S3 api, https://github.com/pydio/cells-client/blob/master/rest/files.go, you can use the minioClient to interact with Cells.

Hello @zayn Thanks for help!

It would be very helpful if you could please clear this also:

In the website https://pydio.com/en/docs/developer-guide/main-apis-presentation
it is mentioned that:

S3 API:
For manipulating the file contents in a consistent way (uploads/downloads), Cells provides an S3-compatible API. The root of your installation can be used as an s3-compatible storage in a third party software that supports such feature. There is only one bucket available, named io .

This endpoint requires the standard S3-Signature headers, which can be fairly complex to generate, so we recommend using one of the numerous libraries available out there to communicate with S3. To connect to the Cells, an S3 Client will require the following settings:

  • Custom URL : your Cells URL.
  • Data Bucket : io
  • Access key and secret : use the JWT (see Authentication section) as the API Key, and gatewaysecret as a fixed API Secret.

When I modified my parameters in the main() as follows:

bucket name: "io"
accessKeyID:  (some api key from pydio.json file)
secretAccessKey: "gatewaysecret"

I now get the error Information Error: Bucket name cannot be shorter than 3 characters

How to resolve this?

Which lib are you using, do you have an entire snippet of the code that I could test?

once you have your S3 Client you should be able to create operations, for instance here is one example that i’m currently working on

	multipartOutput, err := s3Client.CreateMultipartUpload(&s3.CreateMultipartUploadInput{
		Bucket:      aws.String(bucket),
		Key:         aws.String(path),
		ContentType: aws.String("application/octet-stream"),
	})

bucket := "io"
and path is personalf-files/file.txt

Hi @zayn

I am now using Amazon’s AWS-SDK-Go for listing the buckets in the Pydio cells deployed by me.

My Pydio Cells was deployed as http

The code is:

package main

import (
	"fmt"
    "github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/aws/aws-sdk-go/aws/credentials"
)

func main() {
	sess, err := session.NewSession(&aws.Config{
    Region:      aws.String("us-east-1"),
	Endpoint:    aws.String(...My Pydio Cells URL...),
    Credentials: credentials.NewStaticCredentials(...My API Key..., "gatewaysecret",""),
	})

	svc := s3.New(sess)
	buckets, err := svc.ListBuckets(&s3.ListBucketsInput{})
	if err != nil {
		panic(fmt.Sprintf("failed to list buckets, %v", err))
	}
	
	for _,b := range buckets.Buckets {
		bucket := aws.StringValue(b.Name)
		fmt.Println(bucket)
	}
}

The above code works fine for AWS S3 storage:

Now, coming to my Pydio Cells instance, i.e., when I run this code with my Pydio Cells URL:

  1. Without using http at the beginning, I get the error:
panic: failed to list buckets, RequestError: send request failed
caused by: Get "https://(...My Pydio Cells URL...)/": http: server gave HTTP response to HTTPS client

goroutine 1 [running]:
main.main()
        listbuckets.go:29 +0x330
exit status 2
  1. With using http at the beginning, I get the error:
panic: failed to list buckets, SerializationError: failed to decode REST XML response
        status code: 200, request id:
caused by: XML syntax error on line 2: attribute name without = in element

goroutine 1 [running]:
main.main()
        listbuckets.go:29 +0x330
exit status 2

@zayn @c12simple

Also, is there any Pydio API for downloading files which doesn’t internally use 3rd party libraries like AWS-SDK-Go & Minio-Go?

I’m not sure that it works or not :slight_smile:

  1. My code for listing buckets of demo.pydio.com using Minio-SDK library is:
package main

import (
	"log"
	"github.com/minio/minio-go"
)

func main() {
	s3Client, err := minio.New("demo.pydio.com/io", "gateway", "gatewaysecret", true)

	if err != nil {
		log.Fatalln(err)
	}

	buckets, err := s3Client.ListBuckets()
	if err != nil {
		log.Fatalln(err)
	}
	for _, bucket := range buckets {
		log.Println(bucket)
	}

The error I got is:

Endpoint: demo.pydio.com/io does not follow ip address or domain name standards.
exit status 1
  1. My other code for listing buckets of demo.pydio.com using Amazon-AWS-SDK library is:
package main

import (
	"fmt"
    "github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/aws/aws-sdk-go/aws/credentials"
)

func main() {
	sess, err := session.NewSession(&aws.Config{
    Region:      aws.String("us-east-1"),
	Endpoint:    aws.String("https://demo.pydio.com/io"),
	Credentials: credentials.NewStaticCredentials("gateway", "gatewaysecret",""),
	})
	
	svc := s3.New(sess)
	buckets, err := svc.ListBuckets(&s3.ListBucketsInput{})
	if err != nil {
		panic(fmt.Sprintf("failed to list buckets, %v", err))
	}
	
	for _,b := range buckets.Buckets {
		bucket := aws.StringValue(b.Name)
		fmt.Println(bucket)
	}
}

The output is blank on executing this 2nd code

Hello @sid,

Actually you cannot really list the buckets, you can list the workspaces and then work from that.

For your first case of downloading a node take a look at this code https://github.com/pydio/cells-client/blob/69a9b8b2c50427b92e980d5b80cab449e892014d/rest/files.go#L24

pathToFile is something like workspace/path-to-node and bucketname is always the same io(for local storage datasources).

Therefore you must either list the workspaces (look at the code below in this https://github.com/pydio/cells-client/blob/master/rest/files.go) we make use of our SDK and the S3 SDK for the s3 operations.

edit: if you want I can write you a little code snippet to show you how to downlaod a node from the demo

Thanks @zayn

It would be really helpful if you could share snippet of code to download a node from the demo pydio

@zayn

Kindly share some small simple working code to download a node from the demo pydio using golang

Hey Guys,

Just for the record, it would be much better if you choose another target to test your APIs and PoC.
I know some of the sample code in the doc are giving the demo as an example, but that was a bad idea and will be changed promptly.

I thank you for your understanding.

Bruno

@bsinou

Kindlly send that doc link please

Thanks

1 Like

I apologise to bump a thread that did not get a reply for a year. I can only assume that either:

  1. The OP has found a solution for his troubles in the past year, but he forgot to explain how he fixed it (that would be the more obvious reason);
  2. The many offers of help, links, documentation, etc. were sent privately, perhaps because they had sensitive data, and therefore cannot be published publicly here;
  3. The OP gave completely up on Pydio Cells and is using a different solution :frowning:

If I’m wrong on all the above, please accept my apologies. I, too, have been trying to figure out how to do all (or any!) of the above…

Well, at least I can answer this one: @charles has created a second bucket, pointing to the same place, with the name data. Although the ‘official’ bucket is still named io, it can be replaced by data in most situations where software enforces the 3-character-minimum limit…

(aye, I’m still looking for solutions to get flawless S3 sync to Pydio Cells myself…)