Pydio is unable to handle multi file uploads.
It crashes when receiving a large number of files via WebDAV.
Files are uploaded by Rclone. Four files at a time.
After a few minutes the CPU usage gets very high.
Pydio Cells Home v4.2.5
Database: MariaDB 10.5
OS: AlmaLinux 9
CPU: AMD Ryzen 7 5700G with Radeon Graphics
RAM: 64GB
Disk: 1TB NVME SSD
I’m going to watch this thread with curiosity 
Like you, I have experienced the same issue, for months (years?), without a solution. The very few tweaks I managed to apply didn’t make any significant difference.
FYI, I have the same amount of RAM as you do, a bit more disk space (although it’s conventional mechanical HDDs), and an older processor (Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz, 8 cores).
The irony is that Pydio is not used much (it’s currently mostly serving as a shared archive), and, when it does, it actually responds very quickly. It’s just the heavy CPU usage that is my main concern. Note that “heavy CPU” can also mean, in some cases, “lots of blocking I/O with processes getting starved for resources”. Again, in my case, this is possibly due to the sheer amount of logging that Pydio does (I’m opening a new thread just for that) and which I cannot “turn off” (at least, I haven’t figured out how).
I have avoided using WebDAV, though; for some reason, almost all WebDAV-based solutions I’ve found would break at some point. It is active, though, for those few bold enough to try it out. This is mostly because all users (except myself) don’t want to add Yet Another Sync Tool on their own computers, and just expect to use Pydio (or any other similar system) with technologies that already exist on their personal computers, preferably without the need of configuring anything more complicated than filling in a username and password on a dialogue box. WebDAV is quite convenient in that regard, and it’s just an pity that some inherent limitations of WebDAV haven’t been fi.xed in the past decades (of course, this is not a Pydio-related issue!), forcing us to use a plethora of services/protocols to overcome the WebDAV limitations, most of them proprietary or at least specific to a single technology and unable to handle anything else…
You may want to read Uploader unsuitable for folders of numerous small files
I complained about TCP/round-trip (and Pydio processing overhead) incurred by every file causing the web UI to be unable to handle efficiently a situation of 3.6 GB, 770 folders amounting for ~ 10k files.
I was said to upload a zip which is, indeed an acceptable solution in most cases.
But by comparing with WebDAV, we may possibly pinpoint a common underlying codepath that needs optimization?
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.