Scrapy框架: middlewares.py设置
时间:2022-07-25
本文章向大家介绍Scrapy框架: middlewares.py设置,主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项,具有一定的参考价值,需要的朋友可以参考一下。
# -*- coding: utf-8 -*-
# Define here the models for your spider middleware
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/spider-middleware.html
from scrapy import signals
class DownloadtestSpiderMiddleware(object):
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the spider middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_spider_input(self, response, spider):
# Called for each response that goes through the spider
# middleware and into the spider.
# Should return None or raise an exception.
return None
def process_spider_output(self, response, result, spider):
# Called with the results returned from the Spider, after
# it has processed the response.
# Must return an iterable of Request, dict or Item objects.
for i in result:
yield i
def process_spider_exception(self, response, exception, spider):
# Called when a spider or process_spider_input() method
# (from other spider middleware) raises an exception.
# Should return either None or an iterable of Response, dict
# or Item objects.
pass
def process_start_requests(self, start_requests, spider):
# Called with the start requests of the spider, and works
# similarly to the process_spider_output() method, except
# that it doesn’t have a response associated.
# Must return only requests (not items).
for r in start_requests:
yield r
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
class DownloadtestDownloaderMiddleware(object):
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the downloader middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_request(self, request, spider):
# Called for each request that goes through the downloader
# middleware.
# Must either:
# - return None: continue processing this request
# - or return a Response object
# - or return a Request object
# - or raise IgnoreRequest: process_exception() methods of
# installed downloader middleware will be called
return None
def process_response(self, request, response, spider):
# Called with the response returned from the downloader.
# Must either;
# - return a Response object
# - return a Request object
# - or raise IgnoreRequest
return response
def process_exception(self, request, exception, spider):
# Called when a download handler or a process_request()
# (from other downloader middleware) raises an exception.
# Must either:
# - return None: continue processing this exception
# - return a Response object: stops process_exception() chain
# - return a Request object: stops process_exception() chain
pass
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
import random
#设置请求头
class RandomUA():
def __init__(self):
self.user_agent=[
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.1 Safari/605.1.16',
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)'
]
def process_request(self, request, spider):
request.headers['User-Agent'] = random.choice(self.user_agent)
def process_response(self, request, response, spider):
response.status=201
return response
#设置代理
class ProxyMiddleware:
proxy_list=[
"http://127.0.0.1:8080"
# "http://183.91.33.41:80",
# "http://111.29.3.190:80",
# "http://124.156.108.71:82"
]
def process_request(self, request, spider):
ip=random.choice(self.proxy_list)
request.meta['proxy']=ip
- mongodb11天之屠龙宝刀(八)聚合函数与管道:sql与mongodb聚合函数对比
- Java基础-day04-基础题
- OpenCV亚像素角点cornerSubPixel()源代码分析
- mongodb11天之屠龙宝刀(十) 备份 还原 导出 导入::CSV,JSON,BOSN,解决中文乱码
- 【Go 语言社区】 golang 算法课程 第一季 第2节 洗牌算法
- 一条update语句的优化探索(r9笔记第80天)
- OpenCV角点检测源代码分析(Harris和ShiTomasi角点)
- Java基础-day03-代码题
- mongodb11天之屠龙宝刀(九)js函数入门:MongoDB基于js的数据类型修改
- Go语言社区 APP --问答模块数据存储流程及代码
- Java基础-day09-重构随机点名器
- OpenCV3.4两种立体匹配算法效果对比
- 文件操作常用函数
- Java基础-day09-对象;类;封装 学生管理系统
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- Android编程实现Toast只显示最后一条的方法
- Android 照相机的实例应用
- Android 逐帧动画创建实例详解
- 腾讯云 Severless 项目开发和灰度发布实践之路
- C/C++ Search Extension —— 一款可以快速在地址栏搜索 C/C++ 文档的浏览器插件
- 一些小众却有用的 Node.js 包
- 用Single-spa 创建基于 React 和 Vue 的微型前端
- 为 Vue 的惰性加载加一个进度条
- 50个能帮你节省时间的开发工具
- 【技术向】高可定 低维护の博客搭建指南
- 如何学习源码 | 如何高效学习一个新知识
- 我们为什么要禁用 THP
- web前端基础
- 记录一个Debug Assertion Failed
- nacos做服务注册中心