用python爬虫记录_python爬虫记录

爬虫是比较常用的程序&＃xff0c;用python实现起来非常简单&＃xff0c;有几个相关的库&＃xff0c;这里就记录一下python常用的爬虫代码&＃xff0c;备忘。

1 requestxs

import requests

url &＃61;&＃39;http://onevanillachecker.com/&＃39;

rep &＃61; requests.get(url)

rep.encoding &＃61; &＃39;utf-8&＃39;

print(rep.text)

一些参数的记录

import requests

url &＃61;&＃39;http://onevanillachecker.com/&＃39;

header&＃61;{

&＃39;Accept&＃39;: &＃39;text/html,application/xhtml&＃43;xml,application/xml;q&＃61;0.9,image/webp,*/*;q&＃61;0.8&＃39;,

&＃39;Accept-Encoding&＃39;: &＃39;gzip, deflate, sdch&＃39;,

&＃39;Accept-Language&＃39;: &＃39;zh-CN,zh;q&＃61;0.8&＃39;,

&＃39;Connection&＃39;: &＃39;keep-alive&＃39;,

&＃39;User-Agent&＃39;: &＃39;Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; rv:42.0) Gecko/20100101 Firefox/42.0&＃39;

}

timeout &＃61; random.choice(range(80, 180))

rep &＃61; requests.get(url,headers &＃61; header,timeout &＃61; timeout)

rep.encoding &＃61; &＃39;utf-8&＃39;

print(rep.text)

2 urllib2

import urllib2

req &＃61; urllib2.Request(&＃39;http://onevanillachecker.com/&＃39;)

response &＃61; urllib2.urlopen(req)

html &＃61; response.read()

3 beautifulsoup

beautifulsoup是用来解析页面的库&＃xff0c;使用起来非常方便

相关文档https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/

下面简单记一些常用的东西&＃xff0c;备忘。

配置安装

pip install beautifulsoup4

简单使用

from bs4 import BeautifulSoup

import urllib2

req &＃61; urllib2.Request(&＃39;http://onevanillachecker.com/&＃39;)

response &＃61; urllib2.urlopen(req)

html &＃61; response.read()

# beautifulsoup

soup &＃61; BeautifulSoup(html)

print(soup.title)

#

print(soup.title.name)

# title

print(soup.title.string)

# One Vanilla Gift Card Balance Check -Official Website

print(soup.title.parent.name)

# head

print(soup.p)

#

Life happens every day. And OneVanilla
helps make it simpler. Shop, dine, fill &＃39;er up
and more - all with one prepaid card.

# print(soup.p[&＃39;class&＃39;])

print(soup.a)

# Vanilla Gift Card

print(soup.find_all(&＃39;a&＃39;))

# Vanilla Gift Card, Check Vanilla 3 Balance

# Vanilla Gift Cards, Where to Buy # Sign In, About Vanilla Gift Card

# Using Your Vanilla Gift Card, Try Vanilla Gift

# ......

print(soup.find(alt&＃61;"2"))

#

print(soup.get_text())