crawlspider用于定义一些规则用于提取页面符合规则的数据，然后继续爬取。

一、开始一个读书网项目

scrapy startproject 项目名称
cd 项目名称/项目名称/spiders
scrapy genspider -t crawl 爬虫名称 域名
scrapy crawl 爬虫名称

scrapy startproject dushu
cd dushu/dushu/spiders
scrapy genspider -t crawl ds www.dushu.com

二、链接提取规则

allow = ()     正则表达式 提取符合正则的链接
deny = ()     正则表达式 不提取符合正则的链接
allow_domains = ()     允许的域名
deny_domains = ()     不允许的域名
restrict_xpaths = ()     xpath 提取符合xpath规则的链接
restrict_css = ()     css 提取符合选择器规则的链接

class DsSpider(CrawlSpider):
name = 'ds'
allowed_domains = ['www.dushu.com']
start_urls = ['https://www.dushu.com/book/1163_1.html']
rules = (
Rule(LinkExtractor(allow=r'/book/1163_\d+.html'), callback='parse_item', follow=True),
)

三、修改parse_item方法用于解析数据

def parse_item(self, response):
item = {}
div_list = response.xpath('//div[@class="bookslist"]/ul/li/div')
for div in div_list:
item['src'] = div.xpath('./div/a/img/@data-original').extract_first()
item['alt'] = div.xpath('./div/a/img/@alt').extract_first()
item['author'] = div.xpath('./p[1]/a[1]/text()|./p[1]/text()').extract_first()
yield item

四、修改pipelines.py文件用于写入数据

class DushuPipeline(object):
def open_spider(self,spider):
self.fp = open('dushu.json','w',encoding='utf-8')
def process_item(self, item, spider):
# obj = json.loads(str(item))
# str = json.dumps(obj,ensure_ascii=False)
self.fp.write(str(item))
return item
def close_spider(self,spider):
self.fp.close()

五、修改UA及是否遵循robots.txt

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.0;'

Obey robots.txt rules

ROBOTSTXT_OBEY = False

六、运行scrapy项目

scrapy crawl ds