site crawler

1.3.5. Run via cron

The best option is to have a cron-script running the processing. This is done by adding the command shown in the "CLI status" view to the crontab:

So, in this case the cronjob will look like:

This will run the script every minute. Before you add it as a cron-job please check it from the command line that it executes correctly! You will have to add a user called "_cli_crawler" before it does! And you must have PHP installed as a CGI script as well in /usr/bin/

After adding the cronjob you can look at the "Last seen" time:

 

This is just after the cronjob was added - it has not been activated yet. But waiting for a minute this is what I see:

 

 

So this tells me that it is processed correctly.

 

If the cron-script stalls there is a default delay of 1 hour before a new process will announce the old one dead and run a new one. If a cron-script takes more than 1 minute and thereby overlaps the next process, the next process will NOT start if it sees that the "lock-file" exists (unless that hour has passed).