Unable to download files from Pydio Cells using Golang

I am currently writing a Golang program for downloading files from Pydio Cells using Golang program. (Using Minio library)

The working code (for downloading all files from a given bucket in a sample server play.min.io is:

package main

import (
	"github.com/minio/minio-go/v6"
	"log"
  "fmt"
  "context"
  "time"
  "os"
  "io"
)

func main() {
	endpoint := "play.min.io"
	accessKeyID := "Q3AM3UQ867SPQQA43P2F"
	secretAccessKey := "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
	useSSL := true

	// Initialize minio client object.
	minioClient, err := minio.New(endpoint, accessKeyID, secretAccessKey, useSSL)
	if err != nil {
		log.Fatalln(err)
	}

	// Make a new bucket called mymusic.
	bucketName := "mymusic"
	location := "us-east-1"

	err = minioClient.MakeBucket(bucketName, location)
	if err != nil {
		// Check to see if we already own this bucket (which happens if you run this twice)
		exists, errBucketExists := minioClient.BucketExists(bucketName)
		if errBucketExists == nil && exists {
			log.Printf("We already own %s\n", bucketName)
		} else {
			log.Fatalln(err)
		}
	} else {
		log.Printf("Successfully created %s\n", bucketName)
	}

  // Create a done channel to control 'ListObjects' go routine.
  doneCh := make(chan struct{})

  // Indicate to our routine to exit cleanly upon return.
  defer close(doneCh)

  isRecursive := true
  objectCh := minioClient.ListObjects("mymusic", "", isRecursive, doneCh)
  for object1 := range objectCh {
    if object1.Err != nil {
      fmt.Println(object1.Err)
      return
    }
    fmt.Println(object1.Key)

    ctx, cancel := context.WithTimeout(context.Background(), 100 * time.Second)
    defer cancel()

    //object, err := minioClient.GetObjectWithContext(ctx, "mymusic", "hello.txt", minio.GetObjectOptions{})
    object, err := minioClient.GetObjectWithContext(ctx, "mymusic", object1.Key, minio.GetObjectOptions{})
    if err != nil {
      fmt.Println(err)
      return
    }

    localFile, err := os.Create("/home/Desktop/MinioDownloads/"+object1.Key)
    if err != nil {
      fmt.Println(err)
      return
    }

    if _, err = io.Copy(localFile, object); err != nil {
      fmt.Println(err)
      return
    }


  }


}

Now, when I modify the same for Pydio Cells, this is the .go file:

package main

import 
(
   "github.com/minio/minio-go/v6"
   "log"
  "fmt"
  "context"
  "time"
  "os"
  "io"
)

func main() {
	
	endpoint := (Some URL)
	accessKeyID := "admin"
	secretAccessKey := "admin"
	useSSL := false
	
	// Initialize minio client object.
	minioClient, err := minio.New(endpoint, accessKeyID, secretAccessKey, useSSL)
	if err != nil {
		log.Fatalln(err)
	}

  // Create a done channel to control 'ListObjects' go routine.
  doneCh := make(chan struct{})

  // Indicate to our routine to exit cleanly upon return.
  defer close(doneCh)

  isRecursive := true
  objectCh := minioClient.ListObjects("personal", "", isRecursive, doneCh)
  for object1 := range objectCh {
    if object1.Err != nil {
      fmt.Println(object1.Err)
      return
    }
    fmt.Println(object1.Key)

    ctx, cancel := context.WithTimeout(context.Background(), 100 * time.Second)
    defer cancel()

    object, err := minioClient.GetObjectWithContext(ctx, "personal", object1.Key, minio.GetObjectOptions{})
    if err != nil {
      fmt.Println(err)
      return
    }

    localFile, err := os.Create("/home/Desktop/MinioDownloads/"+object1.Key)
    if err != nil {
      fmt.Println(err)
      return
    }

    if _, err = io.Copy(localFile, object); err != nil {
      fmt.Println(err)
      return
    }

  }

}

On running this, I get the following error:
XML syntax error on line 2: attribute name without = in element

I am unable to resolve this.
Please suggest

Hi,
Just for information, https://github.com/pydio/cells-client

Hi @c12simple,

Thanks for help. I am actually looking for those Go programming functions using which I can programatically:

  1. Login to my Pydio cells server
  2. Then list all the files and folders present inside it
  3. Download any of those files/folders

Since I am a beginner, I don’t know much.

I have installed cec client, but in the configuration process, I am getting error (Screenshot attached):

My details while configuring cec:
$ cec configure
Server Address (provide a valid URL): (some url)
OAuth APP ID (found in your server pydio.json): cells-client
OAuth APP Secret (leave empty for a public client):
:heavy_check_mark: Yes
Opening URL (some URL)

Hello,

The server is running in http or https? From error message, I guess it is in http.
So cec has a bug with skip certificate check if you are using self-sign cert. Please wait for update version.

Hello

I have one more doubt:

Is the Pydio Cells local storage S3 compatible?

The reason I am asking that is because Minio-go SDK functions (as mentioned in https://docs.min.io/docs/golang-client-api-reference.html)
, the functions like minioClient.ListBuckets(), minioClient.GetObject(), minioClient.PutObject() etc. are applicable for only S3 compatible storage

Thanks

Hello @sid,

I think this article will give you a better understanding of the architecture,
https://pydio.com/en/docs/cells/v2/understanding-datasources

You should be able to interact with the local storage through the S3 api, https://github.com/pydio/cells-client/blob/master/rest/files.go, you can use the minioClient to interact with Cells.

Hello @zayn Thanks for help!

It would be very helpful if you could please clear this also:

In the website https://pydio.com/en/docs/developer-guide/main-apis-presentation
it is mentioned that:

S3 API:
For manipulating the file contents in a consistent way (uploads/downloads), Cells provides an S3-compatible API. The root of your installation can be used as an s3-compatible storage in a third party software that supports such feature. There is only one bucket available, named io .

This endpoint requires the standard S3-Signature headers, which can be fairly complex to generate, so we recommend using one of the numerous libraries available out there to communicate with S3. To connect to the Cells, an S3 Client will require the following settings:

  • Custom URL : your Cells URL.
  • Data Bucket : io
  • Access key and secret : use the JWT (see Authentication section) as the API Key, and gatewaysecret as a fixed API Secret.

When I modified my parameters in the main() as follows:

bucket name: "io"
accessKeyID:  (some api key from pydio.json file)
secretAccessKey: "gatewaysecret"

I now get the error Information Error: Bucket name cannot be shorter than 3 characters

How to resolve this?

Which lib are you using, do you have an entire snippet of the code that I could test?

once you have your S3 Client you should be able to create operations, for instance here is one example that i’m currently working on

	multipartOutput, err := s3Client.CreateMultipartUpload(&s3.CreateMultipartUploadInput{
		Bucket:      aws.String(bucket),
		Key:         aws.String(path),
		ContentType: aws.String("application/octet-stream"),
	})

bucket := "io"
and path is personalf-files/file.txt

Hi @zayn

I am now using Amazon’s AWS-SDK-Go for listing the buckets in the Pydio cells deployed by me.

My Pydio Cells was deployed as http

The code is:

package main

import (
	"fmt"
    "github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/aws/aws-sdk-go/aws/credentials"
)

func main() {
	sess, err := session.NewSession(&aws.Config{
    Region:      aws.String("us-east-1"),
	Endpoint:    aws.String(...My Pydio Cells URL...),
    Credentials: credentials.NewStaticCredentials(...My API Key..., "gatewaysecret",""),
	})

	svc := s3.New(sess)
	buckets, err := svc.ListBuckets(&s3.ListBucketsInput{})
	if err != nil {
		panic(fmt.Sprintf("failed to list buckets, %v", err))
	}
	
	for _,b := range buckets.Buckets {
		bucket := aws.StringValue(b.Name)
		fmt.Println(bucket)
	}
}

The above code works fine for AWS S3 storage:

Now, coming to my Pydio Cells instance, i.e., when I run this code with my Pydio Cells URL:

  1. Without using http at the beginning, I get the error:
panic: failed to list buckets, RequestError: send request failed
caused by: Get "https://(...My Pydio Cells URL...)/": http: server gave HTTP response to HTTPS client

goroutine 1 [running]:
main.main()
        listbuckets.go:29 +0x330
exit status 2
  1. With using http at the beginning, I get the error:
panic: failed to list buckets, SerializationError: failed to decode REST XML response
        status code: 200, request id:
caused by: XML syntax error on line 2: attribute name without = in element

goroutine 1 [running]:
main.main()
        listbuckets.go:29 +0x330
exit status 2

@zayn @c12simple

Also, is there any Pydio API for downloading files which doesn’t internally use 3rd party libraries like AWS-SDK-Go & Minio-Go?

I’m not sure that it works or not :slight_smile: