WebDAV Chunked Upload

Development
1

I’m trying to add support for( nextcloud-like) chunked uploads, this is my code:

	dir := filepath.Dir(src)
    if strings.HasSuffix(src, "/.file") {
		fileList, err := fs.OpenFile(ctx, dir, os.O_RDONLY, 0)
		if err != nil {
			if os.IsNotExist(err) {
				return http.StatusNotFound, err
			}
			return http.StatusForbidden, err
		} else {
			if _, err := fs.Stat(ctx, dst); err == nil {
				if err := fs.RemoveAll(ctx, dst); err != nil && !os.IsNotExist(err) {
					return http.StatusForbidden, err
				}
			}
			children, err := fileList.Readdir(-1)
			if err != nil {
				return http.StatusPreconditionFailed, err
			}
			dstFile, err := fs.OpenFile(ctx, dst, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644)
			if err != nil {
				if os.IsNotExist(err) {
					return http.StatusNotFound, err
				}
				return http.StatusForbidden, err
			}
			sortFiles(children)
			readers := make([]io.Reader, 0, len(children))
			for _, item := range children {
				name := item.Name()
				source := path.Join(dir, name)
				srcFile, err := fs.OpenFile(ctx, source, os.O_RDONLY, 0)
				if err != nil {
					if os.IsNotExist(err) {
						return http.StatusNotFound, err
					}
					return http.StatusForbidden, err
				}
				readers = append(readers, srcFile)
			}
			multiSources := io.MultiReader(readers...)
			_, err := io.Copy(dstFile, multiSources)
			if err != nil {
				if os.IsNotExist(err) {
					return http.StatusNotFound, err
				}
				return http.StatusForbidden, err
			}
			dstFile.Close()
		}
		fileList.Close()
		return http.StatusCreated, nil
    }

I expected the destination file to contain the data from all file parts, but it only contains the last part.
Merging the file parts and then sending it is not a viable solution, as it takes up a lot more time.
Unfortunately Pydio/MinIO doesn’t support appending data to S3 object/file.

2

My solution:

myReader := io.TeeReader(multiSources, io.Discard)
_, err = io.Copy(dstFile, myReader)

Additionally, I have edit the Writer interface to return nil instead of error.

3

Hey @jaimedelano
can you elaborate a bit about the “nextcloud-like” webdav chunk upload? Do you use special naming for chunks? How to you trig the final operation when all chunks are uploaded ?

4

I use some rewrite rules in Nginx to emulate Nextcloud’s chunked upload, as defined here: Chunked file upload — Nextcloud latest Developer Manual latest documentation
In order to make it work with Rclone, I also had to change the URL of the WebDAV endpoint.
In addition to the changes described above, I had to add an extra workspace to Pydio.