|
|||
|
Marius Watz: Universal Digest Machine, Installation. Java, MySQL (2005) | ||
|
|||
|
||||
|
|||
|
This is a list of all web hosts encountered while spidering, listed in the order they were discovered. To encourage a wide search of the net and to stop any one site or group of sites from dominating the spider's activitity, a maximum of 5 pages will be retrieved from any single web host. After that limit has been reached the host is excluded from future spidering. | |
Robots.txt rules for all hosts are stored in order to obey access restrictions set by the owner of the web site. |