site crawler

< 1.1. Introduction

Table Of Content

1.2. Configuration >

1.1.2. Screenshots

Has a backend module which displays the queue and log and allows execution and status check of the "cronscript" from the backend for testing purposes.

Here is the CLI (Command Line Interface = shell script = cron script) status display:

Here is the crawler queue (before processing) / log (after processing)

Here is the interface for submitting a batch of URLs to be crawled. The parameter combinations are programmeble through Page TSconfig.

To top