python爬虫用到的库的作用_python爬虫从入门到放弃（四）之Requests库的基本使用...

什么是Requests

Requests是用python语言基于urllib编写的&＃xff0c;采用的是Apache2 Licensed开源协议的HTTP库

如果你看过上篇文章关于urllib库的使用&＃xff0c;你会发现&＃xff0c;其实urllib还是非常不方便的&＃xff0c;而Requests它会比urllib更加方便&＃xff0c;可以节约我们大量的工作。(用了requests之后&＃xff0c;你基本都不愿意用urllib了)一句话&＃xff0c;requests是python实现的最简单易用的HTTP库&＃xff0c;建议爬虫使用requests库。

默认安装好python之后&＃xff0c;是没有安装requests模块的&＃xff0c;需要单独通过pip安装

requests功能详解

总体功能的一个演示

import requests

response &＃61; requests.get("https://www.baidu.com")

print(type(response))

print(response.status_code)

print(type(response.text))

print(response.text)

print(response.COOKIEs)

print(response.content)

print(response.content.decode("utf-8"))

我们可以看出response使用起来确实非常方便&＃xff0c;这里有个问题需要注意一下&＃xff1a;

很多情况下的网站如果直接response.text会出现乱码的问题&＃xff0c;所以这个使用response.content

这样返回的数据格式其实是二进制格式&＃xff0c;然后通过decode()转换为utf-8&＃xff0c;这样就解决了通过response.text直接返回显示乱码的问题.

请求发出后&＃xff0c;Requests 会基于 HTTP 头部对响应的编码作出有根据的推测。当你访问 response.text 之时&＃xff0c;Requests 会使用其推测的文本编码。你可以找出 Requests 使用了什么编码&＃xff0c;并且能够使用 response.encoding 属性来改变它.如&＃xff1a;

response &＃61;requests.get("http://www.baidu.com")

response.encoding&＃61;"utf-8"

print(response.text)

不管是通过response.content.decode("utf-8)的方式还是通过response.encoding&＃61;"utf-8"的方式都可以避免乱码的问题发生

各种请求方式

requests里提供个各种请求方式

import requests

requests.post("http://httpbin.org/post")

requests.put("http://httpbin.org/put")

requests.delete("http://httpbin.org/delete")

requests.head("http://httpbin.org/get")

requests.options("http://httpbin.org/get")

请求

基本GET请求

import requests

response &＃61; requests.get(&＃39;http://httpbin.org/get&＃39;)

print(response.text)

带参数的GET请求&＃xff0c;例子1

import requests

response &＃61; requests.get("http://httpbin.org/get?name&＃61;zhaofan&age&＃61;23")

print(response.text)

如果我们想要在URL查询字符串传递数据&＃xff0c;通常我们会通过httpbin.org/get?key&＃61;val方式传递。Requests模块允许使用params关键字传递参数&＃xff0c;以一个字典来传递这些参数&＃xff0c;例子如下&＃xff1a;

import requests

data &＃61; {

"name":"zhaofan",

"age":22

}

response &＃61; requests.get("http://httpbin.org/get",params&＃61;data)

print(response.url)

print(response.text)

上述两种的结果是相同的&＃xff0c;通过params参数传递一个字典内容&＃xff0c;从而直接构造url

注意&＃xff1a;第二种方式通过字典的方式的时候&＃xff0c;如果字典中的参数为None则不会添加到url上

解析json

import requests

import json

response &＃61; requests.get("http://httpbin.org/get")

print(type(response.text))

print(response.json())

print(json.loads(response.text))

print(type(response.json()))

从结果可以看出requests里面集成的json其实就是执行了json.loads()方法&＃xff0c;两者的结果是一样的

获取二进制数据

在上面提到了response.content&＃xff0c;这样获取的数据是二进制数据&＃xff0c;同样的这个方法也可以用于下载图片以及

视频资源

添加headers

和前面我们将urllib模块的时候一样&＃xff0c;我们同样可以定制headers的信息&＃xff0c;如当我们直接通过requests请求知乎网站的时候&＃xff0c;默认是无法访问的

import requests

response &＃61;requests.get("https://www.zhihu.com")

print(response.text)

这样会得到如下的错误

import requests

headers &＃61; {

"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

}

response &＃61;requests.get("https://www.zhihu.com",headers&＃61;headers)

print(response.text)

这样就可以正常的访问知乎了

基本POST请求

通过在发送post请求时添加一个data参数&＃xff0c;这个data参数可以通过字典构造成&＃xff0c;这样

对于发送post请求就非常方便

import requests

data &＃61; {

"name":"zhaofan",

"age":23

}

response &＃61; requests.post("http://httpbin.org/post",data&＃61;data)

print(response.text)

同样的在发送post请求的时候也可以和发送get请求一样通过headers参数传递一个字典类型的数据

响应

我们可以通过response获得很多属性&＃xff0c;例子如下

import requests

response &＃61; requests.get("http://www.baidu.com")

print(type(response.status_code),response.status_code)

print(type(response.headers),response.headers)

print(type(response.COOKIEs),response.COOKIEs)

print(type(response.url),response.url)

print(type(response.history),response.history)

结果如下&＃xff1a;

状态码判断

Requests还附带了一个内置的状态码查询对象

主要有如下内容&＃xff1a;

100: (&＃39;continue&＃39;,),

101: (&＃39;switching_protocols&＃39;,),

102: (&＃39;processing&＃39;,),

103: (&＃39;checkpoint&＃39;,),

122: (&＃39;uri_too_long&＃39;, &＃39;request_uri_too_long&＃39;),

200: (&＃39;ok&＃39;, &＃39;okay&＃39;, &＃39;all_ok&＃39;, &＃39;all_okay&＃39;, &＃39;all_good&＃39;, &＃39;\o/&＃39;, &＃39;✓&＃39;),

201: (&＃39;created&＃39;,),

202: (&＃39;accepted&＃39;,),

203: (&＃39;non_authoritative_info&＃39;, &＃39;non_authoritative_information&＃39;),

204: (&＃39;no_content&＃39;,),

205: (&＃39;reset_content&＃39;, &＃39;reset&＃39;),

206: (&＃39;partial_content&＃39;, &＃39;partial&＃39;),

207: (&＃39;multi_status&＃39;, &＃39;multiple_status&＃39;, &＃39;multi_stati&＃39;, &＃39;multiple_stati&＃39;),

208: (&＃39;already_reported&＃39;,),

226: (&＃39;im_used&＃39;,),

Redirection.

300: (&＃39;multiple_choices&＃39;,),

301: (&＃39;moved_permanently&＃39;, &＃39;moved&＃39;, &＃39;\o-&＃39;),

302: (&＃39;found&＃39;,),

303: (&＃39;see_other&＃39;, &＃39;other&＃39;),

304: (&＃39;not_modified&＃39;,),

305: (&＃39;use_proxy&＃39;,),

306: (&＃39;switch_proxy&＃39;,),

307: (&＃39;temporary_redirect&＃39;, &＃39;temporary_moved&＃39;, &＃39;temporary&＃39;),

308: (&＃39;permanent_redirect&＃39;,

&＃39;resume_incomplete&＃39;, &＃39;resume&＃39;,), # These 2 to be removed in 3.0

Client Error.

400: (&＃39;bad_request&＃39;, &＃39;bad&＃39;),

401: (&＃39;unauthorized&＃39;,),

402: (&＃39;payment_required&＃39;, &＃39;payment&＃39;),

403: (&＃39;forbidden&＃39;,),

404: (&＃39;not_found&＃39;, &＃39;-o-&＃39;),

405: (&＃39;method_not_allowed&＃39;, &＃39;not_allowed&＃39;),

406: (&＃39;not_acceptable&＃39;,),

407: (&＃39;proxy_authentication_required&＃39;, &＃39;proxy_auth&＃39;, &＃39;proxy_authentication&＃39;),

408: (&＃39;request_timeout&＃39;, &＃39;timeout&＃39;),

409: (&＃39;conflict&＃39;,),

410: (&＃39;gone&＃39;,),

411: (&＃39;length_required&＃39;,),

412: (&＃39;precondition_failed&＃39;, &＃39;precondition&＃39;),

413: (&＃39;request_entity_too_large&＃39;,),

414: (&＃39;request_uri_too_large&＃39;,),

415: (&＃39;unsupported_media_type&＃39;, &＃39;unsupported_media&＃39;, &＃39;media_type&＃39;),

416: (&＃39;requested_range_not_satisfiable&＃39;, &＃39;requested_range&＃39;, &＃39;range_not_satisfiable&＃39;),

417: (&＃39;expectation_failed&＃39;,),

418: (&＃39;im_a_teapot&＃39;, &＃39;teapot&＃39;, &＃39;i_am_a_teapot&＃39;),

421: (&＃39;misdirected_request&＃39;,),

422: (&＃39;unprocessable_entity&＃39;, &＃39;unprocessable&＃39;),

423: (&＃39;locked&＃39;,),

424: (&＃39;failed_dependency&＃39;, &＃39;dependency&＃39;),

425: (&＃39;unordered_collection&＃39;, &＃39;unordered&＃39;),

426: (&＃39;upgrade_required&＃39;, &＃39;upgrade&＃39;),

428: (&＃39;precondition_required&＃39;, &＃39;precondition&＃39;),

429: (&＃39;too_many_requests&＃39;, &＃39;too_many&＃39;),

431: (&＃39;header_fields_too_large&＃39;, &＃39;fields_too_large&＃39;),

444: (&＃39;no_response&＃39;, &＃39;none&＃39;),

449: (&＃39;retry_with&＃39;, &＃39;retry&＃39;),

450: (&＃39;blocked_by_windows_parental_controls&＃39;, &＃39;parental_controls&＃39;),

451: (&＃39;unavailable_for_legal_reasons&＃39;, &＃39;legal_reasons&＃39;),

499: (&＃39;client_closed_request&＃39;,),

Server Error.

500: (&＃39;internal_server_error&＃39;, &＃39;server_error&＃39;, &＃39;/o\&＃39;, &＃39;✗&＃39;),

501: (&＃39;not_implemented&＃39;,),

502: (&＃39;bad_gateway&＃39;,),

503: (&＃39;service_unavailable&＃39;, &＃39;unavailable&＃39;),

504: (&＃39;gateway_timeout&＃39;,),

505: (&＃39;http_version_not_supported&＃39;, &＃39;http_version&＃39;),

506: (&＃39;variant_also_negotiates&＃39;,),

507: (&＃39;insufficient_storage&＃39;,),

509: (&＃39;bandwidth_limit_exceeded&＃39;, &＃39;bandwidth&＃39;),

510: (&＃39;not_extended&＃39;,),

511: (&＃39;network_authentication_required&＃39;, &＃39;network_auth&＃39;, &＃39;network_authentication&＃39;),

通过下面例子测试&＃xff1a;(不过通常还是通过状态码判断更方便)

import requests

response&＃61; requests.get("http://www.baidu.com")

if response.status_code &＃61;&＃61; requests.codes.ok:

print("访问成功")

requests高级用法

文件上传

实现方法和其他参数类似&＃xff0c;也是构造一个字典然后通过files参数传递

import requests

files&＃61; {"files":open("git.jpeg?s=#34;,"rb")}

response &＃61; requests.post("http://httpbin.org/post",files&＃61;files)

print(response.text)

结果如下&＃xff1a;

获取COOKIE

import requests

response &＃61; requests.get("http://www.baidu.com")

print(response.COOKIEs)

for key,value in response.COOKIEs.items():

print(key&＃43;"&＃61;"&＃43;value)

会话维持

COOKIE的一个作用就是可以用于模拟登陆&＃xff0c;做会话维持

import requests

s &＃61; requests.Session()

s.get("http://httpbin.org/COOKIEs/set/number/123456")

response &＃61; s.get("http://httpbin.org/COOKIEs")

print(response.text)

这是正确的写法&＃xff0c;而下面的写法则是错误的

import requests

requests.get("http://httpbin.org/COOKIEs/set/number/123456")

response &＃61; requests.get("http://httpbin.org/COOKIEs")

print(response.text)

因为这种方式是两次requests请求之间是独立的&＃xff0c;而第一次则是通过创建一个session对象&＃xff0c;两次请求都通过这个对象访问

证书验证

现在的很多网站都是https的方式访问&＃xff0c;所以这个时候就涉及到证书的问题

import requests

response &＃61; requests.get("https:/www.12306.cn")

print(response.status_code)

默认的12306网站的证书是不合法的&＃xff0c;这样就会提示如下错误

为了避免这种情况的发生可以通过verify&＃61;False

但是这样是可以访问到页面&＃xff0c;但是会提示&＃xff1a;

InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)

解决方法为&＃xff1a;

import requests

from requests.packages import urllib3

urllib3.disable_warnings()

response &＃61; requests.get("https://www.12306.cn",verify&＃61;False)

print(response.status_code)

这样就不会提示警告信息&＃xff0c;当然也可以通过cert参数放入证书路径

代理设置

import requests

proxies&＃61; {

"http":"http://127.0.0.1:9999",

"https":"http://127.0.0.1:8888"

}

response &＃61; requests.get("https://www.baidu.com",proxies&＃61;proxies)

print(response.text)

如果代理需要设置账户名和密码,只需要将字典更改为如下&＃xff1a;

proxies &＃61; {

"http":"http://user:password&＃64;127.0.0.1:9999"

}

如果你的代理是通过sokces这种方式则需要pip install "requests[socks]"

proxies&＃61; {

"http":"socks5://127.0.0.1:9999",

"https":"sockes5://127.0.0.1:8888"

}

超时设置

通过timeout参数可以设置超时的时间

认证设置

如果碰到需要认证的网站可以通过requests.auth模块实现

import requests

from requests.auth import HTTPBasicAuth

response &＃61; requests.get("http://120.27.34.24:9001/",auth&＃61;HTTPBasicAuth("user","123"))

print(response.status_code)

当然这里还有一种方式

import requests

response &＃61; requests.get("http://120.27.34.24:9001/",auth&＃61;("user","123"))

print(response.status_code)

异常处理

从源码我们可以看出RequestException继承IOError,

HTTPError&＃xff0c;ConnectionError,Timeout继承RequestionException

ProxyError&＃xff0c;SSLError继承ConnectionError

ReadTimeout继承Timeout异常

这里列举了一些常用的异常继承关系&＃xff0c;详细的可以看&＃xff1a;

http://cn.python-requests.org/zh_CN/latest/_modules/requests/exceptions.html#RequestException

通过下面的例子进行简单的演示

import requests

from requests.exceptions import ReadTimeout,ConnectionError,RequestException

try:

response &＃61; requests.get("http://httpbin.org/get",timout&＃61;0.1)

print(response.status_code)

except ReadTimeout:

print("timeout")

except ConnectionError:

print("connection Error")

except RequestException:

print("error")

其实最后测试可以发现&＃xff0c;首先被捕捉的异常是timeout,当把网络断掉的haul就会捕捉到ConnectionError&＃xff0c;如果前面异常都没有捕捉到&＃xff0c;最后也可以通过RequestExctption捕捉到