Robots.txt parser - Web Search

Description:

Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).

Review It Rate It Visit Tell A Friend

Total Hits : 89 | Category : Web Search | Ratings : rating : 0

Reviews :

No available review for this entry at this moment... Would you like to be the first one to enter a review for it? Click here to review!