Automatic purge crashes

linux
apache

#1

Hi,
Pydio 8.0.2, PHP 5.6.25, Httpd 2.4.27 on centos 7.4
Fresh install with BDD imported and updated from 6.4.2
shared imported from 6.4.2

Everything seems to be ok but the purge still doesn’t works.
When i launch purge for every users via task scheduler, php process stops and task is still “in progress”.

I had the same issue on 6.4.2 with no resolution.
I have a script in batch which purge DB and files but it’s not clean.
Have you got an clue?

here’s my script.

#!/bin/bash

# ce script recherche les fichiers vieux de 30 jours et purge la base

# On cherche les fichiers vieux de 30j mais pas les fichiers cachés
find /var/lib/pydio/personal -mtime +30 -type f -not -path ‘/.’ > /tmp/liste_ files_purge.txt

while read LIGNE
do
# On met en A uniquement le nom de fichier sans chemin
A=echo $LIGNE | awk -F "/" '{ print $NF }'
# On supprime le fichier dans le FS
rm -f “$LIGNE”
# On cherche dans la table d’index le fichier et on met l’enregistrement DB en B
# En BB, on extrait la clé primaire (id) de $B
B=mysql -N -u root --password=xxx pydiodb -e "select * from ajxp_i ndex where node_path like '%$A%'"
BB=echo $B | awk '{print $1}'
# Pareil que B mais pour ajxp_repo
C=mysql -N -u root --password=xxx pydiodb -e "select * from ajxp_r epo where display like '%$A%'"
CC=echo $C | awk '{print $1}'
# Meme chose en core pour ajxp_user_bookmarks
D=mysql -N -u root --password=xxx pydiodb -e "select * from ajxp_u ser_bookmarks where path like '%$A%'"
DD=echo $D | awk '{print $1}'
# Encore une fois pour ajxp_feed

    # On test si les variables sont non vides et on supprime les enregistrem                                                                                                                                                             ents en base.
    if [ ! -z $B ]
            then
            mysql -N -u root --password=xxx pydiodb -e "delete from ajxp_index where node_id like '$BB'                                                                                                                                                           
    fi

    if [ ! -z $C ]
            then
            mysql -N -u root --password=xxx pydiodb -e "delete from ajxp_repo where uuid like '$CC'"                                                                                                                                                             
    fi

    if [ ! -z $D ]
            then
            mysql -N -u root --password=xxx pydiodb -e "delete from ajxp_user_bookmarks where rid like '$DD'"                                                                                                                                                             >
    fi

    done < /tmp/liste_files_purge.txt

#2

Up.
Anybody has an answer?


#3

Hi,
could you tell me what values did you put in the scheduler for the purge ?


#4

Here’s in attachment.


thanks for help.


#5

Hi,
your values seems fine to me, could you put your server in this mode :

DEBUG MODE PYDIO

Go to conf/bootstrap_context.php .
Switch this line to true : 
define("AJXP_SERVER_DEBUG" , true);

and reproduce the crash, then look at the logs.


#6

Done,
But i can’t locate logs.
Where are they?
in bootstrap_context.php, it’s specified at/var/log/pydio but there’s only index.html there.


#7

you can find the logs in the database under a table named, ajxp_logs.


#9

I’m watching it. I will post errors as soon as possible.
Thanks.


#10

Here’s logs
I can’t see what’s happen.

Blockquote
| 221278 | 2018-05-24 15:25:03 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221279 | 2018-05-24 15:30:03 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221280 | 2018-05-24 15:30:04 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221281 | 2018-05-24 15:35:03 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221282 | 2018-05-24 15:35:04 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221283 | 2018-05-24 15:40:03 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221284 | 2018-05-24 15:40:04 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221285 | 2018-05-24 15:40:24 | 10.0.4.102 | NOTICE | adm_interne | Pydio\Log\Core\Logger | Deprecated | You are calling debug() with 1 arguments, please use at least 3
| 221286 | 2018-05-24 15:40:24 | 10.0.4.102 | DEBUG | adm_interne | Deprecated | Starting process and sending output dev null |
| 221287 | 2018-05-24 15:45:02 | PHP_CLI | INFO | adm_interne | Pydio\Core\Services\AuthService | Log In | context=API
| 221288 | 2018-05-24 15:45:03 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221289 | 2018-05-24 15:45:03 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221290 | 2018-05-24 15:46:32 | 10.0.4.102 | DEBUG | adm_interne | CacheDriver::Http | Clear All | namespace=shared
| 221291 | 2018-05-24 15:46:42 | 10.0.4.102 | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221292 | 2018-05-24 15:46:42 | 10.0.4.102 | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221293 | 2018-05-24 15:47:24 | 10.0.4.102 | DEBUG | adm_interne | CacheDriver::Http | Clear Key xml_registry:session:light:adm_interne:settings | namespace=shared
| 221294 | 2018-05-24 15:47:25 | 10.0.4.102 | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221295 | 2018-05-24 15:47:25 | 10.0.4.102 | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221296 | 2018-05-24 15:47:27 | 10.0.4.102 | NOTICE | adm_interne | Pydio\Log\Core\Logger | Deprecated | You are calling debug() with 1 arguments, please use at least 3
| 221297 | 2018-05-24 15:47:27 | 10.0.4.102 | DEBUG | adm_interne | Deprecated | Starting process and sending output dev null |
| 221298 | 2018-05-24 15:50:03 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221299 | 2018-05-24 15:50:04 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221300 | 2018-05-24 15:51:23 | 10.0.4.102 | DEBUG | adm_interne | CacheDriver::Http | Clear All | namespace=shared
| 221301 | 2018-05-24 15:51:27 | 10.0.4.102 | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221302 | 2018-05-24 15:51:27 | 10.0.4.102 | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221303 | 2018-05-24 15:51:40 | 10.0.4.102 | DEBUG | adm_interne | CacheDriver::Http | Clear Key xml_registry:session:light:adm_interne:settings | namespace=shared
| 221304 | 2018-05-24 15:51:41 | 10.0.4.102 | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221305 | 2018-05-24 15:51:41 | 10.0.4.102 | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository test-transfert-txt as parent cannot be correctly loaded.
| 221306 | 2018-05-24 15:55:03 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load | Disabling repository 0-17000-3branchementok as parent cannot be correctly loaded.
| 221307 | 2018-05-24 15:55:04 | PHP_CLI | ERROR | adm_interne | Pydio\Core\Model\FilteredRepositoriesList | load

Thanks.


#11

Hi,
could you update your php to atleast the version 7 ?


#12

Hello zayn,

Updated to php7.1 and same issue.
process crashes before finish.


#13

Hi,
i forgot to ask but did you enable the purging process on the workspaces that you want to purge as seen in this screenshot, the default value is 0 (which disable the feature)

and if you still want to use your script you can use the following command

sudo php /var/www/pydio/cmd.php -r=testingworkspace -u=admin -p=azertyuiop -a=purge

  • -r : it can be the id or alias of the workspace
  • -u : the user that has the rights to do the purge
  • -p : password for the user above
  • -a : the type of action here purge

maybe you could put this command in your script and execute the script each X days etc …


#14

i have set up hard limit bit soft limit is still 0
I know you command but this purge only one workspace (admin in your example).
And i don’t know password of my users.

And an error occurs, i didn’t had this error on 8.0.2

[root@srv-pydio-02 personal]# php /usr/share/pydio/cmd.php -r=my-files -u=adm_interne -p=xxx -a=purge
PHP Fatal error: Uncaught Pydio\Core\Exception\DBConnectionException: There was an error trying to connect to your database! Did you change any configuration for the core connection? Or maybe your database is down? in /usr/share/pydio/plugins/conf.sql/SqlConfDriver.php:112

I have updated to 8.2.0, with php 7.1 but still no luck :confused:


#15

Hi,
i’m going to do some testing and if i can reproduce the issue i will then ask the devs.
I will give you the infos whenever i get them.


#16

hi zayn,
there were missing dependancies.
yum install rh-php71-php-pecl-apcu rh-php71-php-mysqlnd rh-php71-php-ldap.

[root@srv-pydio-02 ~]# php /usr/share/pydio/cmd.php -r=my-files -u=adm_interne -p=xxx -a=purge


Current User is ‘adm_interne’


Applying action ‘purge’ on workspace Mes fichiers (1)

So it’s ok for purging one workspace if i know password.

Purge with scheduler still freeze…


#17

I jumping on this, sorry if I asked questions already asked…
Do you have any commands working correctly when launched from within pydio? Like repository indexation? Is only the purge failing?
-c


#18

I do not use other command, only purge.
i can test other, indexation for example. What is the name of the action to make indexation?
Thanks.


#19

Hi,
for the indexation the action is index so add -a=index to the command.


#20

Hi,
tested in cli mode with one user >> KO.
Tested on scheduled task for all users >> same issue than purge.

php process crashes.


Purge files cronjob not working