NLP冻手之路(4)——pipeline管道函数的使用

✅ NLP 研 0 选手的学习笔记

● 上一篇文章链接: NLP冻手之路(3)——评价及指标函数的使用(Metric&＃xff0c;以 BLEU和GLUE 为例)

● python 需要 3.7&＃43;&＃xff0c;pytorch 需要 1.10&＃43;

● 本文使用的库基于 Hugging Face Transformer&＃xff0c;官网链接&＃xff1a;https://huggingface.co/docs/transformers/index 【一个很不错的开源网站&＃xff0c;针对于 transformer 框架做了很多大集成&＃xff0c;目前 github 72.3k ⭐️】

● 安装 Hugging Face Transformer 的库只需要在终端输入 pip install transformers【这是 pip 安装方法】&＃xff1b;如果你用的是 conda&＃xff0c;则输入 conda install -c huggingface transformers

● Hugging Face 提供了一个非常轻量化、简单的工具 pipeline&＃xff0c;我们可以通过它来解决一些简单的 NLP 任务。pipeline 提供了专门用于多个任务的简单 API&＃xff0c;包括命名实体识别、Mask 语言建模、情感分析、特征提取和问题回答等等。

● 通过学习和使用 pipeline&＃xff0c;可以让我们更直观地、直接地体会到处理 NLP 任务的感觉。

3.1 情感分类

● 如果我们没有指定模型&＃xff0c;那么它会自动下载模型 distilbert-base-uncased-finetuned-sst-2-english 到 ~/.cache/torch 文件夹当中。

● 如果下载速度过慢&＃xff0c;可以先配置清华源再重新运行&＃xff1a;pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

● 任务&＃xff1a;对给定的文本进行情感二分类。

from transformers import pipelinemy_classifier &＃61; pipeline("sentiment-analysis")
result &＃61; my_classifier("This restaurant is good")
print(result)
result &＃61; my_classifier("我觉得这家餐馆不好吃")
print(result)


● 运行结果如下&＃xff0c;首先他会下载情感分析模型&＃xff0c;然后进行情感分析。

3.2 完形填空

● 如果我们没有指定模型&＃xff0c;那么它会自动下载模型 distilroberta-base 到 ~/.cache/torch 文件夹当中。

● 任务&＃xff1a;模型会对 处进行填空&＃xff0c;分数代表填这个词的概率。

from transformers import pipeline
from pprint import pprint
my_unmasker &＃61; pipeline("fill-mask")
sentence &＃61; &＃39;HuggingFace is creating a that the community uses to solve NLP tasks.&＃39;
result &＃61; my_unmasker(sentence)
pprint(result )输出&＃xff1a;
[{&＃39;sequence&＃39;: &＃39;HuggingFace is creating a tool that the community uses to solve NLP tasks.&＃39;,&＃39;score&＃39;: 0.17927534878253937,&＃39;token&＃39;: 3944,&＃39;token_str&＃39;: &＃39; tool&＃39;},{&＃39;sequence&＃39;: &＃39;HuggingFace is creating a framework that the community uses to solve NLP tasks.&＃39;,&＃39;score&＃39;: 0.11349416524171829,&＃39;token&＃39;: 7208,&＃39;token_str&＃39;: &＃39; framework&＃39;},{&＃39;sequence&＃39;: &＃39;HuggingFace is creating a library that the community uses to solve NLP tasks.&＃39;,&＃39;score&＃39;: 0.05243571847677231,&＃39;token&＃39;: 5560,&＃39;token_str&＃39;: &＃39; library&＃39;},{&＃39;sequence&＃39;: &＃39;HuggingFace is creating a database that the community uses to solve NLP tasks.&＃39;,&＃39;score&＃39;: 0.034935351461172104,&＃39;token&＃39;: 8503,&＃39;token_str&＃39;: &＃39; database&＃39;},{&＃39;sequence&＃39;: &＃39;HuggingFace is creating a prototype that the community uses to solve NLP tasks.&＃39;,&＃39;score&＃39;: 0.028602460399270058,&＃39;token&＃39;: 17715,&＃39;token_str&＃39;: &＃39; prototype&＃39;}]


3.3 文本生成

● 如果我们没有指定模型&＃xff0c;那么它会自动下载模型 gpt2 到 ~/.cache/torch 文件夹当中。

● 任务&＃xff1a;给定模型一段话/一句话&＃xff0c;模型接着生成后续的文本&＃xff0c;生成的长度由 max_length 决定。

from transformers import pipelinetext_generator &＃61; pipeline("text-generation")
result &＃61; text_generator("As far as I am concerned, I will",max_length&＃61;50,do_sample&＃61;False)
print(result)输出&＃xff1a;
[{&＃39;generated_text&＃39;: &＃39;As far as I am concerned, I will be the first to admit that I am not a fan of the idea of a "free market." I think that the idea of a free market is a bit of a stretch. I think that the idea&＃39;}]


3.4 命名实体识别

● 如果我们没有指定模型&＃xff0c;那么它会自动下载模型 dbmdz/bert-large-cased-finetuned-conll03-english 到 ~/.cache/torch 文件夹当中。

● 任务&＃xff1a;给定模型一段话&＃xff0c;模型对其中的人名、地名、城市名、公司名等等。

from transformers import pipelinener_pipe &＃61; pipeline("ner")
sequence &＃61; """Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO,
therefore very close to the Manhattan Bridge which is visible from the window."""
for entity in ner_pipe(sequence):print(entity)输出&＃xff1a;
{&＃39;entity&＃39;: &＃39;I-ORG&＃39;, &＃39;score&＃39;: 0.9995786, &＃39;index&＃39;: 1, &＃39;word&＃39;: &＃39;Hu&＃39;, &＃39;start&＃39;: 0, &＃39;end&＃39;: 2}
{&＃39;entity&＃39;: &＃39;I-ORG&＃39;, &＃39;score&＃39;: 0.9909764, &＃39;index&＃39;: 2, &＃39;word&＃39;: &＃39;##gging&＃39;, &＃39;start&＃39;: 2, &＃39;end&＃39;: 7}
{&＃39;entity&＃39;: &＃39;I-ORG&＃39;, &＃39;score&＃39;: 0.9982225, &＃39;index&＃39;: 3, &＃39;word&＃39;: &＃39;Face&＃39;, &＃39;start&＃39;: 8, &＃39;end&＃39;: 12}
{&＃39;entity&＃39;: &＃39;I-ORG&＃39;, &＃39;score&＃39;: 0.999488, &＃39;index&＃39;: 4, &＃39;word&＃39;: &＃39;Inc&＃39;, &＃39;start&＃39;: 13, &＃39;end&＃39;: 16}
{&＃39;entity&＃39;: &＃39;I-LOC&＃39;, &＃39;score&＃39;: 0.9994345, &＃39;index&＃39;: 11, &＃39;word&＃39;: &＃39;New&＃39;, &＃39;start&＃39;: 40, &＃39;end&＃39;: 43}
{&＃39;entity&＃39;: &＃39;I-LOC&＃39;, &＃39;score&＃39;: 0.9993196, &＃39;index&＃39;: 12, &＃39;word&＃39;: &＃39;York&＃39;, &＃39;start&＃39;: 44, &＃39;end&＃39;: 48}
{&＃39;entity&＃39;: &＃39;I-LOC&＃39;, &＃39;score&＃39;: 0.9993794, &＃39;index&＃39;: 13, &＃39;word&＃39;: &＃39;City&＃39;, &＃39;start&＃39;: 49, &＃39;end&＃39;: 53}
{&＃39;entity&＃39;: &＃39;I-LOC&＃39;, &＃39;score&＃39;: 0.98625815, &＃39;index&＃39;: 19, &＃39;word&＃39;: &＃39;D&＃39;, &＃39;start&＃39;: 79, &＃39;end&＃39;: 80}
{&＃39;entity&＃39;: &＃39;I-LOC&＃39;, &＃39;score&＃39;: 0.9514269, &＃39;index&＃39;: 20, &＃39;word&＃39;: &＃39;##UM&＃39;, &＃39;start&＃39;: 80, &＃39;end&＃39;: 82}
{&＃39;entity&＃39;: &＃39;I-LOC&＃39;, &＃39;score&＃39;: 0.9336589, &＃39;index&＃39;: 21, &＃39;word&＃39;: &＃39;##BO&＃39;, &＃39;start&＃39;: 82, &＃39;end&＃39;: 84}
{&＃39;entity&＃39;: &＃39;I-LOC&＃39;, &＃39;score&＃39;: 0.97616535, &＃39;index&＃39;: 28, &＃39;word&＃39;: &＃39;Manhattan&＃39;, &＃39;start&＃39;: 114, &＃39;end&＃39;: 123}
{&＃39;entity&＃39;: &＃39;I-LOC&＃39;, &＃39;score&＃39;: 0.9914629, &＃39;index&＃39;: 29, &＃39;word&＃39;: &＃39;Bridge&＃39;, &＃39;start&＃39;: 124, &＃39;end&＃39;: 130}


3.5 摘要生成

● 如果我们没有指定模型&＃xff0c;那么它会自动下载模型 sshleifer/distilbart-cnn-12-6 到 ~/.cache/torch 文件夹当中。

● 任务&＃xff1a;略。

from transformers import pipelinesummarizer &＃61; pipeline("summarization")
ARTICLE &＃61; """ New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County, New York.
A year later, she got married again in Westchester County, but to a different man and without divorcing her first husband.
Only 18 days after that marriage, she got hitched yet again. Then, Barrientos declared "I do" five more times, sometimes only within two weeks of each other.
In 2010, she married once more, this time in the Bronx. In an application for a marriage license, she stated it was her "first and only" marriage.
Barrientos, now 39, is facing two criminal counts of "offering a false instrument for filing in the first degree," referring to her false statements on the
2010 marriage license application, according to court documents.
Prosecutors said the marriages were part of an immigration scam.
On Friday, she pleaded not guilty at State Supreme Court in the Bronx, according to her attorney, Christopher Wright, who declined to comment further.
After leaving court, Barrientos was arrested and charged with theft of service and criminal trespass for allegedly sneaking into the New York subway through an emergency exit, said Detective
Annette Markowski, a police spokeswoman. In total, Barrientos has been married 10 times, with nine of her marriages occurring between 1999 and 2002.
All occurred either in Westchester County, Long Island, New Jersey or the Bronx. She is believed to still be married to four men, and at one time, she was married to eight men at once, prosecutors say.
Prosecutors said the immigration scam involved some of her husbands, who filed for permanent residence status shortly after the marriages.
Any divorces happened only after such filings were approved. It was unclear whether any of the men will be prosecuted.
The case was referred to the Bronx District Attorney\&＃39;s Office by Immigration and Customs Enforcement and the Department of Homeland Security\&＃39;s
Investigation Division. Seven of the men are from so-called "red-flagged" countries, including Egypt, Turkey, Georgia, Pakistan and Mali.
Her eighth husband, Rashid Rajput, was deported in 2006 to his native Pakistan after an investigation by the Joint Terrorism Task Force.
If convicted, Barrientos faces up to four years in prison. Her next court appearance is scheduled for May 18.
"""result &＃61; summarizer(ARTICLE, max_length&＃61;130, min_length&＃61;30, do_sample&＃61;False)
print(result)输出&＃xff1a;
[{&＃39;summary_text&＃39;: &＃39; Liana Barrientos, 39, is charged with two counts of "offering a false instrument for filing in the first degree" In total, she has been married 10 times, with nine of her marriages occurring between 1999 and 2002 . At one time, she was married to eight men at once, prosecutors say .&＃39;}]


3.6 文本翻译

● 如果我们没有指定模型&＃xff0c;那么它会自动下载模型 t5-base 到 ~/.cache/torch 文件夹当中。

● 任务&＃xff1a;略。

from transformers import pipeline#翻译
translator &＃61; pipeline("translation_en_to_de")
sentence &＃61; "Hugging Face is a technology company based in New York and Paris"
result &＃61; translator(sentence, max_length&＃61;40)
print(result)输出&＃xff1a;
[{&＃39;translation_text&＃39;: &＃39;Hugging Face ist ein Technologieunternehmen mit Sitz in New York und Paris.&＃39;}]


3.7 阅读理解

● 如果我们没有指定模型&＃xff0c;那么它会自动下载模型 t5-base 到 ~/.cache/torch 文件夹当中。

● 这段代码可能运行不成功&＃xff0c;可能是原模型的 bug。

● 任务&＃xff1a;给定一段文本&＃xff0c;然后问文本一个问题&＃xff0c;模型给出相应的答案。

from transformers import pipelinequestion_answerer &＃61; pipeline("question-answering")
# 字符串前面加 r 是为了消除转义字符对字符串的影响. 加了 r 之后, 再打印字符串就会打印出完整的字符串
context &＃61; r"""
Extractive Question Answering is the task of extracting an answer from a text given a question. An example of a
question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune
a model on a SQuAD task, you may leverage the examples/pytorch/question-answering/run_squad.py script.
"""
result &＃61; question_answerer(question&＃61;"What is extractive question answering?",context&＃61;context)
print(result)
result &＃61; question_answerer(question&＃61;"What is a good example of a question answering dataset?",context&＃61;context)
print(result)输出&＃xff1a;
{&＃39;score&＃39;: 0.6177279353141785, &＃39;start&＃39;: 34, &＃39;end&＃39;: 95, &＃39;answer&＃39;: &＃39;the task of extracting an answer from a text given a question&＃39;}
{&＃39;score&＃39;: 0.5152313113212585, &＃39;start&＃39;: 148, &＃39;end&＃39;: 161, &＃39;answer&＃39;: &＃39;SQuAD dataset&＃39;}


● 本小节不是重点&＃xff0c;但是可以让我们直观地体会到 NLP 唾手可得的魅力&＃xff0c;后续还有深入探索&＃xff0c;一起加油~&＃xff01;

● 上一篇文章链接: NLP冻手之路(3)——评价及指标函数的使用(Metric&＃xff0c;以 BLEU和GLUE 为例)

● pipeline 会依据你指定的任务&＃xff0c;去 hugging face 上面下载当前排名最高、最热门的模型下来&＃xff0c;所以可能隔个几年&＃xff0c;上面的 SOTA 模型都会有所变更。

● 若有写得不对的地方&＃xff0c;或有疑问&＃xff0c;欢迎评论交流。

● 参考视频&＃xff1a;HuggingFace简明教程,BERT中文模型实战示例.NLP预训练模型,Transformers类库,datasets类库快速入门.

⭐️ ⭐️