pythonxpath定位嵌套标签_python获取具有嵌套嵌套度的html元素的内容(以及内容的xpath)...

我编写了一个递归函数,它将以以下格式返回字典中标签中所有文本的XPATH&＃xff1a;

{&＃39;xpath1&＃39;: {&＃39;text&＃39;: &＃39;text1&＃39;}, &＃39;xpath2&＃39;: {&＃39;text&＃39;: &＃39;text2&＃39;}, ...}

码&＃xff1a;

from bs4 import BeautifulSoup, NavigableString

def get_xpaths_dict(soup, xpaths&＃61;{}, curr_path&＃61;&＃39;&＃39;):

curr_path &＃43;&＃61; &＃39;/{}&＃39;.format(soup.name)

for item in soup.contents:

if isinstance(item, NavigableString):

if item.strip():

try:

xpaths[curr_path][&＃39;count&＃39;] &＃43;&＃61; 1

count &＃61; xpaths[curr_path][&＃39;count&＃39;]

curr_path &＃43;&＃61; &＃39;[{}]&＃39;.format(count)

xpaths[curr_path] &＃61; {&＃39;text&＃39;: item.strip()}

except KeyError:

xpaths[curr_path] &＃61; {&＃39;text&＃39;: item.strip(), &＃39;count&＃39;: 1}

else:

xpaths &＃61; get_xpaths_dict(item, xpaths, curr_path)

return xpaths

html &＃61; &＃39;&＃39;&＃39;

text of div 1

text of span 1.1

text of span 2.1

text of span 2.2

text of span 3&＃39;&＃39;&＃39;

soup &＃61; BeautifulSoup(html, &＃39;html.parser&＃39;)

xpaths &＃61; get_xpaths_dict(soup.div)

print(xpaths)

输出&＃xff1a;

{&＃39;/div&＃39;: {&＃39;text&＃39;: &＃39;text of div 1&＃39;, &＃39;count&＃39;: 1}, &＃39;/div/span&＃39;: {&＃39;text&＃39;: &＃39;text of span 1.1&＃39;, &＃39;count&＃39;: 1}, &＃39;/div/span/span&＃39;: {&＃39;text&＃39;: &＃39;text of span 2.1&＃39;, &＃39;count&＃39;: 2}, &＃39;/div/span/span[2]&＃39;: {&＃39;text&＃39;: &＃39;text of span 2.2&＃39;}, &＃39;/div/span/span[2]/span&＃39;: {&＃39;text&＃39;: &＃39;text of span 3&＃39;, &＃39;count&＃39;: 1}}

我知道这不是您期望输出的格式.但是,您可以将其转换为所需的任何格式.例如,要将其转换为预期的输出,只需执行以下操作&＃xff1a;

expected_output &＃61; [(v[&＃39;text&＃39;], k) for k, v in xpaths.items()]

print(expected_output)

输出&＃xff1a;

[(&＃39;text of div 1&＃39;, &＃39;/div&＃39;), (&＃39;text of span 1.1&＃39;, &＃39;/div/span&＃39;), (&＃39;text of span 2.1&＃39;, &＃39;/div/span/span&＃39;), (&＃39;text of span 2.2&＃39;, &＃39;/div/span/span[2]&＃39;), (&＃39;text of span 3&＃39;, &＃39;/div/span/span[2]/span&＃39;)]

一些解释&＃xff1a;

词典中的额外键计数用于存储当前标签中具有相同名称的标签数量.使用这种格式(字典)可以优化代码.您只能访问每个标签一次.

奖金&＃xff1a;

由于该函数返回以XPATH为键的字典,因此您可以使用XPATH获取任何文本.例如&＃xff1a;

xpaths &＃61; get_xpaths_dict(soup.div)

print(xpaths[&＃39;/div/span/span[2]/span&＃39;][&＃39;text&＃39;])

# text of span 3