seleniumwire获取百度指数

之前我在《如何用Python下载百度指数的数据》分享了如何使用接口获取百度指数&＃xff0c;但是今年百度指数已经增加了新的校验方式&＃xff0c;例如如下代码&＃xff1a;

import requests
import json
from datetime import date, timedeltaheaders &＃61; {"Connection": "keep-alive","Accept": "application/json, text/plain, */*","User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36","Sec-Fetch-Site": "same-origin","Sec-Fetch-Mode": "cors","Sec-Fetch-Dest": "empty","Referer": "https://index.baidu.com/v2/main/index.html","Accept-Language": "zh-CN,zh;q&＃61;0.9",&＃39;COOKIE&＃39;: COOKIE,
}words &＃61; &＃39;[[{"name":"python","wordType":1}],[{"name":"java","wordType":1}]]&＃39;
start, end &＃61; "2022-06-28", "2022-07-27"
url &＃61; f&＃39;https://index.baidu.com/api/SearchApi/index?area&＃61;0&word&＃61;{words}&startDate&＃61;{start}&endDate&＃61;{end}&＃39;
res &＃61; requests.get(url, headers&＃61;headers)
res.json()


{&＃39;status&＃39;: 10018,&＃39;data&＃39;: &＃39;&＃39;,&＃39;logid&＃39;: 2631899650,&＃39;message&＃39;: &＃39;您好&＃xff0c;百度指数监测到疑似存在xx访问行为&＃xff0c;如您未有类似行为&＃xff0c;可能是由于您使用公共网络或访问频次过高&＃xff0c;\n 您可以通过邮箱ext_indexfk&＃64;baidu.com联系我们&＃39;}


百度指数并未返回数据&＃xff0c;而是提示访问异常访问。经简单检查&＃xff0c;现在的请求参数header中增加了Cipher-Text参数&＃xff0c;JS逆向大佬可以直接分析js从而正确产生该参数通过校验。

不过今天我将演示一个非常简单实用的获取百度指数的方案&＃xff0c;直接使用seleniumwire来获取数据并解密。

关于seleniumwire的介绍&＃xff0c;可参考我上一篇文章&＃xff1a;《selenium对接代理与seleniumwire访问开发者工具NetWork》

由于selenium操作百度指数网页每次都需要登录比较麻烦&＃xff0c;我们可以在缓存COOKIE到本地文件后&＃xff0c;每次重启都能自动登录百度。

自动保存COOKIE代码&＃xff1a;

from selenium import webdriver
import timebrowser &＃61; webdriver.Chrome()
browser.get("https://index.baidu.com/v2/index.html")
browser.find_element_by_css_selector("span.username-text").click()
print("等待登录...")
while True:if browser.find_element_by_css_selector("span.username-text").text !&＃61; "登录":breakelse:time.sleep(3)
print("已登录&＃xff0c;现在为您保存COOKIE...")
with open(&＃39;COOKIE.txt&＃39;, &＃39;w&＃39;, encoding&＃61;&＃39;u8&＃39;) as f:json.dump(browser.get_COOKIEs(), f)
browser.close()
print("COOKIE保存完成&＃xff0c;游览器已自动退出...")


运行以上代码后&＃xff0c;会自动打开登录界面&＃xff0c;待人工登录后&＃xff0c;会自动保存COOKIE到本地并关闭游览器。

然后我们以如下方式访问百度指数&＃xff0c;即可自动登录&＃xff1a;

from seleniumwire import webdriverbrowser &＃61; webdriver.Chrome()
with open(&＃39;COOKIE.txt&＃39;, &＃39;r&＃39;, encoding&＃61;&＃39;u8&＃39;) as f:COOKIEs &＃61; json.load(f)
browser.get(&＃39;https://index.baidu.com/v2/index.html&＃39;)
for COOKIE in COOKIEs:browser.add_COOKIE(COOKIE)
browser.get(&＃39;https://index.baidu.com/v2/index.html&＃39;)


参考&＃xff1a;《提取谷歌游览器COOKIE的五重境界》

使游览器执行搜索特定关键字&＃xff0c;例如Python&＃xff1a;

from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ECwait &＃61; WebDriverWait(browser, 30)
edit &＃61; wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#search-input-form > input.search-input")))
print("清空前历史记录数&＃xff1a;", len(browser.requests))
del browser.requests # 清空历史数据
edit.send_keys(Keys.CONTROL&＃43;&＃39;a&＃39;)
edit.send_keys(Keys.DELETE)
edit.send_keys("Python")
submit &＃61; browser.find_element_by_css_selector("span.search-input-cancle")
submit.click()
print("清空后再执行搜索后的历史记录数&＃xff1a;", len(browser.requests))


清空前历史记录数&＃xff1a; 87
清空后再执行搜索后的历史记录数&＃xff1a; 3


执行完搜索操作后&＃xff0c;我们就可以从游览器缓存中获取数据了&＃xff1a;

import gzip
import zlib
import brotli
import jsondef auto_decompress(res):content_encoding &＃61; res.headers["content-encoding"]if content_encoding &＃61;&＃61; "gzip":res.body &＃61; gzip.decompress(res.body)elif content_encoding &＃61;&＃61; "deflate":res.body &＃61; zlib.decompress(res.body)elif content_encoding &＃61;&＃61; "br":res.body &＃61; brotli.decompress(res.body)def fetch_data(rule, encoding&＃61;"u8", is_json&＃61;True):result &＃61; ""for request in reversed(browser.requests):if rule in request.url:res &＃61; request.responseauto_decompress(res)result &＃61; res.body.decode(encoding)if is_json:result &＃61; json.loads(result)return resultdef decrypt(ptbk, index_data):n &＃61; len(ptbk)//2a &＃61; dict(zip(ptbk[:n], ptbk[n:]))return "".join([a[s] for s in index_data])ptbk &＃61; fetch_data("Interface/ptbk")[&＃39;data&＃39;]
data &＃61; fetch_data("api/SearchApi/index")[&＃39;data&＃39;]for userIndexe in data[&＃39;userIndexes&＃39;]:name &＃61; userIndexe[&＃39;word&＃39;][0][&＃39;name&＃39;]index_data &＃61; userIndexe[&＃39;all&＃39;][&＃39;data&＃39;]r &＃61; decrypt(ptbk, index_data)print(name, r)


python 21077,21093,21186,19643,14612,13961,21733,21411,21085,21284,18591,13211,12753,27225,20302,19772,20156,17647,12018,11745,19535,19300,20075,20136,18153,12956,12406,17098,16259,18707


对比结果后可以看到&＃xff0c;数据获取正确。这样我们就可以通过seleniumwire获取百度指数的数据了&＃xff0c;若需要获取指定日期范围或指定省份&＃xff0c;只需通过selenium模拟人工执行相应的查询操作&＃xff0c;再通过游览器后台缓存获取即可。

多客户端数据的解析可以参考之前《如何用Python下载百度指数的数据》中的代码。