Web Scraping and Crawling With Scrapy and MongoDB

django-dynamic-scraper

Django Dynamic Scraper (DDS) is an app for Django build on top of the scraping framework Scrapy. While preserving many of the features of Scrapy it lets you dynamically create and manage spiders via the Django admin interface.

Indexing web sites in Solr with Python

Scrapy at a glance

Building a Web Crawler with Scrapy