Web crawling

uCrawler
Universal Crawler, a concurrent Distributed and Highly Configurable and Flexible Next Generation Crawler.
Code