网页图像抓取技术学习心得：从零开始掌握爬虫技巧

今天练习了用爬虫批量爬取网站文件。练习对象是一个妹子图片网站&＃xff0c;网址在代码里有哈哈哈&＃xff0c;最后实现了将妹子的大图批量下载到电脑里&＃xff0c;好爽嘿嘿嘿。收获如下&＃xff1a;

将远程文件下载到本地用的是urlretrieve方法&＃xff0c;他主要有两个参数&＃xff1a;文件的网址和要存储的文件名。其中第二个参数要特别注意&＃xff1a;要到文件名才行&＃xff0c;不能只是路径。而文件名的构造采用了如下的代码&＃xff0c;暂时还不太懂&＃xff0c;但是先学会再说&＃xff1a;

x &＃61;0
for item in imgurl:urlretrieve(item,&＃39;/Users/zengyichao/Desktop/工作零碎文件/2.21/test4/&＃39;&＃43;&＃39;%s.jpg&＃39;%x)x&＃43;&＃61;1

import requests
from bs4 import BeautifulSoup
import time
from urllib.request import urlretrieve
headers &＃61; {&＃39;User-Agent&＃39;:&＃39;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36&＃39;
}imgurl &＃61; []def get_img(url):res &＃61; requests.get(url, headers &＃61; headers)res.encoding &＃61; &＃39;utf-8&＃39;soup &＃61; BeautifulSoup(res.text,&＃39;html.parser&＃39;)imgs &＃61; soup.select(&＃39;#big-pic > p > a > img&＃39;)for img in imgs:href &＃61; img.get(&＃39;src&＃39;)imgurl.append(href)#
# url &＃61; &＃39;http://www.mmonly.cc/mmtp/xgmn/198663.html&＃39;
# get_img(url)urls &＃61; [&＃39;http://www.mmonly.cc/mmtp/xgmn/100306_{}.html&＃39;.format(str(i)) for i in range(2,31)]
for url in urls:get_img(url)x &＃61;0
for item in imgurl:urlretrieve(item,&＃39;/Users/zengyichao/Desktop/工作零碎文件/2.21/test4/&＃39;&＃43;&＃39;%s.jpg&＃39;%x)x&＃43;&＃61;1print(imgurl)