正则表达式解析抓取猫眼电影Top100

猫眼电影提供实时票房数据&＃xff0c;这个以后玩

榜单规则&＃xff1a;将猫眼电影库中的经典影片&＃xff0c;按照评分和评分人数从高到低综合排序取前100名&＃xff0c;每天上午10点更新。相关数据来源于“猫眼电影库”。

第一步&＃xff0c;分析URL&＃xff0c;一共有10页&＃xff0c;每页10个&＃xff0c;观察URL得

http://maoyan.com/board/4?offset&＃61;0 最后一个数字为增量&＃xff0c;每次加10&＃xff0c;第一页为0

#构造10页的地址
base_url &＃61; &＃39;http://maoyan.com/board/4?offset&＃61;{}&＃39;
urls &＃61; []
for i in range(10):urls.append(base_url.format(10*i))

第二步&＃xff0c;分析单个页面

不加headers访问被禁止了&＃xff0c;说是恶意访问

#构造headers
headers &＃61; {&＃39;User-Agent&＃39;:&＃39;Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36&＃39;,&＃39;COOKIE&＃39;:&＃39;__mta&＃61;251008569.1536744988778.1536745503768.1536745544944.17; _lxsdk_cuid&＃61;165cd22f1a0c8-0ec18267a3a2f5-3c604504-1fa400-165cd22f1a0c8; uuid_n_v&＃61;v1; uuid&＃61;53511D10B66F11E894F593DFAB82C37F1716AA64EB0848C2BADBC75AA2E23EA6; _csrf&＃61;5f3373e2e85bd09c75c54ffbc624db254862d2ceb2dfbeff81eb9fa58e289001; __guid&＃61;17099173.3022114119644780000.1536744988414.424; _lx_utm&＃61;utm_source%3DBaidu%26utm_medium%3Dorganic; _lxsdk&＃61;53511D10B66F11E894F593DFAB82C37F1716AA64EB0848C2BADBC75AA2E23EA6; __mta&＃61;251008569.1536744988778.1536744999954.1536745003054.4; monitor_count&＃61;17; _lxsdk_s&＃61;165cd22f1a1-995-e6-849%7C%7C47&＃39;
}
def get_onepage(url):response &＃61; requests.get(url,headers&＃61;headers).textindex &＃61; re.findall(&＃39;board-index.*?>(.*?)<&＃39;,response,re.S)[1:-1]name &＃61; re.findall(&＃39;

,response,re.S)star &＃61; re.findall(&＃39;(.*?)

最后将score去掉不存&＃xff0c;因为第一次成功了&＃xff0c;但是今天尽然龙猫那部没有评分了&＃xff0c;一页9个就会报错&＃xff0c;img的查找有些麻烦&＃xff0c;试了好几次&＃xff0c;测试中发现img的属性标签会变化位置&＃xff0c;与浏览器中看到的顺序不一致。

最后完整代码&＃xff0c;爬取10页数据&＃xff0c;并将数据打印出来&＃xff0c;并写入json文件&＃xff0c;重新读取

import requests,re
import jsonbase_url &＃61; &＃39;http://maoyan.com/board/4?offset&＃61;{}&＃39;
urls &＃61; []
for i in range(10):urls.append(base_url.format(10*i))#构造headers
headers &＃61; {&＃39;User-Agent&＃39;:&＃39;Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36&＃39;,&＃39;COOKIE&＃39;:&＃39;__mta&＃61;251008569.1536744988778.1536745503768.1536745544944.17; _lxsdk_cuid&＃61;165cd22f1a0c8-0ec18267a3a2f5-3c604504-1fa400-165cd22f1a0c8; uuid_n_v&＃61;v1; uuid&＃61;53511D10B66F11E894F593DFAB82C37F1716AA64EB0848C2BADBC75AA2E23EA6; _csrf&＃61;5f3373e2e85bd09c75c54ffbc624db254862d2ceb2dfbeff81eb9fa58e289001; __guid&＃61;17099173.3022114119644780000.1536744988414.424; _lx_utm&＃61;utm_source%3DBaidu%26utm_medium%3Dorganic; _lxsdk&＃61;53511D10B66F11E894F593DFAB82C37F1716AA64EB0848C2BADBC75AA2E23EA6; __mta&＃61;251008569.1536744988778.1536744999954.1536745003054.4; monitor_count&＃61;17; _lxsdk_s&＃61;165cd22f1a1-995-e6-849%7C%7C47&＃39;
}
def get_onepage(url):response &＃61; requests.get(url,headers&＃61;headers).textindex &＃61; re.findall(&＃39;board-index.*?>(.*?)<&＃39;,response,re.S)[1:-1]name &＃61; re.findall(&＃39;

,response,re.S)star &＃61; re.findall(&＃39;(.*?)

最后&＃xff0c;打开json看是没有utf8转码的&＃xff0c;不知是编辑器的事吗&＃xff1f;

在代码里读文件正常

json在线格式解析也正常