The state of Pydio Cells + Sync?

How come you are pushing pydio cells when it’s not ready for sync and all the other things that are no longer ready yet?

Sync client is not working with cells yet and this is worrying, will the be a some what working release of that soon? Is Pydio cells ready for the new sync client or will there ever be a sync client for pydio cells?

I think you will have to make a statement of this publicly so people don’t get their hopes up too much and are relying on sync and installing cells and expect everything to work. and also don’t push cells so hard when some of the best features of pydio is not working with cells.

1 Like

I was thinking that the desktop applications have at least been updated recently (maybe as an attempt to hopefully allow syncing), but then I realized that the last update, at least on the website, was almost 2.5 years ago.

@devs, if you need more support for developing the software, testing, etc, then reach out to the community. You have a good product, but your development is way too slow. There are errors and a lack of information in your docs, there’s issues with the installation, then there’s the major issue of being able to sync files on your phone / computer with Cells (kind of the whole point of a cloud storage software). I’d be more than happy to help in testing or development, but you the devs need to provide more than “we’re working on it”, which is what you’ve been saying.

@mattish.91 @TDOG
Hi guys,

First of all, thank you for your messages. Considering Pydio Cells “not finished” seems a bit unfair to me. Using Cells as a collaboration tool for sharing documents in an organisation is a perfect match, and in fact many of our open source users as well as paying customers are super happy with that. It’s waaay smoother and much more reliable than Pydio 8. I’ve been doing open source for years now, and I am used to the standard “this is not a working project” messages, whereas tons of people all around the world use it everyday in production :wink: . It may just not fit the needs at a given point.

Also, all the codes (server and desktop client) are on Github, and we’ve put great effort in updating the admin and developer docs for Cells on the website. There’s already a lot of material for starting digging and contributing. But I totally agree that we do have to put more effort in talking to the community and being more transparent on the roadmap. Looking at other open source projects, what I am thinking of is writing some “help wanted” tickets with very defined user stories and even prototypes to have people step in on a small feature without having to understand the whole platform. Publishing the roadmap is must-do as well. Any suggestions are very welcome on that topic.

Now, regarding the sync, moving the whole application away from PHP was driven (among other reasons) by the fact that we were not happy with the quality Pydio 8 and its sync. Connecting tons of sync clients to a cloud storage application drastically raises the load on the server and php was not good at that, at all. Not to speak of the fact that syncing shared workspaces could lead to users to do wrong manipulations and eventually break some data.

This rewrite project was, as any software project, bigger than we expected. And we want to be 100% sure of how the application is behaving under the hood before reconnecting the PydioSync client to it (typically not publishing something “unfinished”). Still, it is definitely in the short-term roadmap to make PydioSync work with Cells, but it may still take some time. I would say we need 4 to 6 months to make sure we publish something really reliable. But maybe a bit less or maybe a bit more, that the life of IT projects (thus the "we’re working on it answers…). We will probably try to add some inherent limitations to avoid problematic use cases, e.g. allow sync only on the “Personal Files” workspaces or only in a uni-directional manner for shared ones, to avoid having people complain because they erased something.

Finally, another subject we are looking at is providing a “virtual drive” tool instead of a pure sync, to see the files appear in your OS without necessarily downloading the whole data locally, and selectively sync them as needed. We feel it is an important trend nowadays, where people have small laptops and do not want to sync Teras of data locally.

Anyway, I’ll be happy to continue the discussion if you wish to, or to have you on-board as contributors.

Best regards
Charles

4 Likes

That was a great read, and i agree on having the roadmap more open to make the community notice it a bit more. I have been using sync with about 40 (current syncing devices) in Pydio 8, i have had some minor issues but also i have done all the connections making sure they only sync to their Personal files or within that folder. Most of my sync clients are syncing one folder within their Personal files with an /sync folder with all of their most important files. And as you said, i think it would be a great idea to have a “virtual drive” as a option, the current sync client have been working like a charm for me, running mostly on Windows. but also some mac’s and some Linux servers even. i’s been under some heavy testing in my setup and i have only had some minor issues, some which have gotten solved by reinstalling Pydio directly on top of the old Pydio install, when the the new “fresh” install is done, all sync client’s directly reconnect with no further issues and all their files in place. which is a great “feature” ^^ not sure if that was counted for or if that might be an issue, never the less it does how ever work that way. Of course backups are made beforehand and most of the times i don’t even have to push the backup back since all files and the database is already connected to begin with. I have been using Pydio “Ajaxplore” since i got my first LG Nas back in 2008 and were amazed of how easy it was to manage files on my server and i googled it and downloaded the full package and just freshly installed the updated package without any issues. so i have been using Pydio and Ajaxplore for a long time now and can’t get out of it ^^

Soon i will migrate my current Windows server over to two newer Linux servers and hopefully the migration will go fine, im also looking for a way of syncronizing these servers so tranfers don’t have to happen two times for clients. this is to secure the redundancy and the servers have the exact same hardware.

2 Likes

Hi again, thanks for your answer
So basically

  • Stick to p8 for the moment if you rely heavily on sync
  • I’ll definitely ping you when it’s time to test sync on Cells :slight_smile:
    Maybe we should basically have a developers/testers mailing for people like you
    -c
2 Likes

Yea ill be more than willing to comply with testing if i could provide you with some useful information and proper logs, and maybe even push some fixes if easy enough.

I did install cells successfully no issues at all maybe 3 months ago and i know there have been a lot if updates since then, but at the time installing i noticed sync with cells was a no go since i got alot of syncing going on ^^

But i heavily like the idea of selective sync from a virtual drive, that would make some of the work way more easy for some of my users with laptops and small SSD’s, at the time being they mainly download and upload what they need from my Pydio web access.

Do you have a road map for the time being?
I could strongly recommend Trello.com providing free service for starters, you can also publish the road map locked in public view, change colors of parts that might be complete or that are in the works but also ideas for people to vote. basically provide a lot of user input of what features are the most important for all of your users.

You can see an example of this for Wolcen here: https://trello.com/b/xK5tEdL6/wolcen-lords-of-mayhem-roadmap

It’s easy enough to see whats going on, so less posts about features not working yet or features that are excluded. They simply got a link to it on their website so people know.

Thanks for your quick answer :slight_smile:

1 Like

I’m sorry about being so harsh, but the reply you gave is exactly what I was looking for. As an engineer, this is what I like to see. See, the dilema is that while Pydio 8 works fine and has the ability to use sync (heavily needed for my type of operations, where I might do coding on one PC and then continue on a laptop later), going that route with Cells in development isn’t what I want to do. Collaboration is nice for some of my startup and business ventures, but syncing that data is important as well. Cells is a lot cleaner, and I really love the implementation of it, based on looking at it from the server’s perspective.

Where Cells is the new product that you want to promote, you certainly want to share the roadmap as well as a list of ongoing features / items being worked on. Like @mattish.91 said above, Trello is a good option, but simply adding it to your site through your CMS would be suitable if set up correctly.

As I said, I really like Pydio, especially how Cells is set up (obviously, more features will follow with time), as compared to the other storage software options. I’ve had issues with one of the main competitors with syncing. But in an effort to get away from Google, DropBox, etc for mail, storage, and other services for my own personal ventures and for local businesses, this software here helps.

I’ve seen posts for jobs with Pydio, though I unfortunately do not live in France. However, I am more than willing to jump aboard to try to work with whomever and wherever I can to help. There’s plenty of suggestions to be made, but some insight to some of those items may be answered by actions unseen.

1 Like

Hi,
Regarding a direct discussion channel with “involved developers/contributors/power-users” (not just people that post once a year a basic “help me with config” in the forum), at one point for some of our beta’s we had set up a Slack channel. What would you think of that? Wouldn’t it be more modern than a mailing-list ? That way we coud have dedicated discussions, beta sessions when releasing important features, etc…
-c

2 Likes

I agree, great conversation, and Charles it’s really good to see you guys embracing devs and testers.

A Slack channel would be excellent; Traefik uses this for their in-depth testing / troubleshooting / dev questions, and it works really well.

Cheers,
Geoff

Yea, Slack is a great way of communication, im all up for it, that would most definitely do the job.

People like Telegram as a discussion channel, but the functionality of it is limited. Certainly, the API is somewhat useful (I’m unsure of Slack’s as I’ve never used it). However, Slack would be a much better way of organizing discussion for any matter like this.

Piping in here late, but I believe this discussion has potential :wink: so let me add my 2 cents…
There are a lot of ‘collaboration tools’ out there, either for techies/programmers, for business operations, for ticketing and tech support, and so forth; all of them, to a degree, allow multiple users to share files, sometimes with versioning, often with indexing abilities, tagging and so forth. Each, however, is tailored for a particular purpose. For instance, you could use, say, GitHub (or a clone of it, like gogs, which is my favourite) to get similar features. It would be a very creative use of git, but it would work; and, in fact, many solutions use the git protocol to deal with versioning and sync’ing, for instance.
Now, when I first started to use ownCloud and similar services what I wanted was a Dropbox replacement, and I can imagine that most people expected the same from such services. I settled on Pydio mostly because, at the time I’ve tested it, it was the one that suited my purposes best, worked quite well out of the box, had all the ‘right’ features I needed, and was based on PHP, which, at the time, I rather preferred over alternatives in Perl, Python, or — ugh! — Java. (You really need to do long-time maintenance on solutions using those kinds of platforms to truly appreciate how easy it is to do things under PHP…)
The conceptual ‘future-proof’ change to Pydio Cells was a surprise to me, and one that I actually liked a lot! By sheer coincidence, a friend of mine who works for Google mentioned Go to me — he didn’t do much more than point out its existence. I learned it on my own and was immediately in love with it — not only because of its syntax but mostly because of the neat way it all fits together. Although over time I started to find out a few faults which most Go-haters have been pointing out for eons (most of which are actually Google developers, unhappy with being ‘forced’ to move to Go), for me it’s still the programming language of the present (I still like PHP, though, although I’m getting frustrated with the swiftness that things get obsoleted by the core programmers, and the way people do code in PHP these days resembles more and more the worst practices of Java and C# programmers, but that’s another story…). In fact, I ported a 3-year-old project in PHP — which was under development and not finished — to Go in about three months, from scratch, while I was still learning: it’s certainly not an ‘idiomatically correct’ usage of Go, but I couldn’t care less (I’m not a professional programmer): the point is that the project gained a ton of new features (because Go had them by default), and, because Go is compiled (and not interpreted-cum-JIT compilation) without needing dependencies, it was several orders of magnitude faster, more stable, and more easy to understand/fix/add things to it. Even spaghetti code in Go looks great! :sweat_smile:
When I first heard that Pydio was using a ‘Go backend’ to deal with sync’ing, I was impressed. A few years ago, die-hard PHP programmers, requiring some similar functionality which PHP was not adequately providing, would (possibly) add a Java thingy. Which would then have a gazillion dependency issues and work only on very specific environments. Which in turn would leave a lot of people behind (possibly switching to another platform…). Well, that would be a worst-case scenario, of course, but I’ve seen it happening in other applications/platforms. And I’ve abandoned a few exactly because of that.
Whew, long introduction… but here is my point: moving from PHP-based Pydio to a Go implementation, rewritten from scratch (and actually with a completely different conceptual framework), is, for me, the way to go (pun intended, of course! :smile:). Sure, it means that Pydio Cells will lag behind PHP-based Go for a while, simply because it takes time to ‘catch up’ with the tons of lines of code currently in Pydio 8. I miss many of the plugins, for instance, and obviously they will not work ‘soon’ with Pydio Cells, since the original developers will need to learn Go and the new development paradigm. On the other hand, I also feel that Pydio Cells is not an unfinished product — rather, it feels like a ‘late beta’, good enough to deploy in production but missing the occasional bells & whistles, which I’m sure it will get ‘sooner or later’.
However, there is a crucial difference — unless I’ve misunderstood @charles. PHP-based Pydio was developed as a ‘Dropbox replacement’ where desktop sync’ing is crucial, not merely a nice little useful feature. Remote sync’ing is what Dropbox is for — and Pydio replicated that concept neatly. We can argue if the sync’ing was slower/less efficient/consumed more resources on the client and/or the server, compared to Dropbox, but… it worked. Or rather, it still works, and while I’m sure everything can be improved at some point, it does what it’s supposed to do.
I was startled to learn that PydioSync did not support Pydio Cells (when the iOS app has no problem whatsoever to connect to it). In fact, for days I thought I had misconfigured something — namely, the certificates — because all I got were HTTPS errors. Eventually, I found this thread and understood that there was no problem with my configuration, it’s just that Pydio Cells, unlike Pydio 8, is not meant to be a Dropbox replacement but… well, a slimmed down version of Pydio 8, I guess, one that misses crucial functionality — I would even go as far as claiming it to be the core functionality, because, well, in my case, I started using Pydio because sync’ing is all I wanted.
As said, there are plenty of ‘remote folder’ solutions out there. And technically one could simply use Unison to do remote sync’ing, and possibly wrap a GUI around it, and that would be all. But Pydio is much more than that — rather, let me rephrase that: Pydio used to be much more than that, while still catering to our basic needs, i.e. sync’ing to a Dropbox-style remote environment.
It’s been around three months since @charles mentioned that PydioSync would support Cells (or, alternatively, that a new method of sync’ing would be deployed). I wonder if there is any news on the state of the development? Is the 4-6 months timescale still feasible?

Hey @GwynethLlewelyn

Thanks for drawing attention on this thread, that I actually just renamed :slight_smile:
Regarding your (long) introduction, I think you totally caught the spirit of this move and of the state of the project. Regarding the “late-beta” state, to be totally frank, I’ve been an open-source guy for a looong time now: call something a beta and release it to the open source community (or maybe should I call them the open source “consumers”), no-one will ever dare to touch it and test it to report bugs. Call it a 1.0, you’ll get all the bugs you need + unhappy people. That’s life :slight_smile:

Now I have some good news for you : the NEW SYNC is on its way, it’s 100% GO and it’s going pretty well. As a matter of fact I made my first sync with a real remote server this morning! :sweat_smile:

After looking at the PydioSync code, we decided it would be too much of hassle to re-adapt Cells to work with it, and we started experimenting with a new solution. The beauty of this new client is that it uses the same “sync” lib as the one used in the server for syncing “objects” to “datasources”. Which was more or less exactly the point of switching to Go. The same (now expanded) lib will also be able to setup sync across datasources or across servers with ease.

Now of course, it’s not - at all - production ready, not even developers ready (need to switch 3 repos on specific branches, hack vendor…). The big points that are not even PoC’ed yet are basically the windows compilation, a clever management of conflicts, a proper error/retries handling… Not the least. But I’m really confident we could have an early-alpha before summer, and a late-beta after summer.

Hope you like the news,

Best
Charles

2 Likes

‘Like’? I’m ecstatic, to say the least!! That’s wonderful to hear :smiley: