作者:岁月如风晓 | 来源:互联网 | 2023-02-12 18:27
在日常的爬虫中,如果频繁访问,会被网站屏蔽,要使用代理#-*-coding:UTF-8-*-fromurllibimportrequestimportrandomi
在日常的爬虫中,如果频繁访问,会被网站屏蔽,要使用代理
from urllib import request
import random
if __name__ == "__main__":
url = 'http://www.whatismyip.com.tw/'
proxy = [{'http':'211.94.69.74:8080'},{'http':'113.128.90.252:48888'}
,{'http':'113.128.91.92:48888'}]
proxy_support = request.ProxyHandler(proxy[2
])
opener = request.build_opener(proxy_support)
opener.addheaders = [('User-Agent','Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36')]
respOnse=opener.open(url)
html = response.read().decode("utf-8")
print(html)