Azure 认知服务工具包

此工具包用于与Azure 认知服务 API交互以实现一些多模态功能。

目前，此工具包包含四个工具

• AzureCogsImageAnalysisTool：用于从图像中提取标题、对象、标签和文本。（注意：由于依赖于azure-ai-vision包，该包目前仅支持 Windows 和 Linux，因此此工具尚不可用于 Mac OS。）
• AzureCogsFormRecognizerTool：用于从文档中提取文本、表格和键值对。
• AzureCogsSpeech2TextTool：用于将语音转录为文本。
• AzureCogsText2SpeechTool：用于将文本合成语音。
• AzureCogsTextAnalyticsHealthTool：用于提取医疗保健实体。

首先，您需要设置 Azure 帐户并创建认知服务资源。您可以按照此处的说明创建资源。

然后，您需要获取资源的端点、密钥和区域，并将它们设置为环境变量。您可以在资源的“密钥和端点”页面中找到它们。

%pip install --upgrade --quiet  azure-ai-formrecognizer > /dev/null
%pip install --upgrade --quiet  azure-cognitiveservices-speech > /dev/null
%pip install --upgrade --quiet  azure-ai-textanalytics > /dev/null

# For Windows/Linux
%pip install --upgrade --quiet  azure-ai-vision > /dev/null

%pip install -qU langchain-community

import os

os.environ["OPENAI_API_KEY"] = "sk-"
os.environ["AZURE_COGS_KEY"] = ""
os.environ["AZURE_COGS_ENDPOINT"] = ""
os.environ["AZURE_COGS_REGION"] = ""

创建工具包

from langchain_community.agent_toolkits import AzureCognitiveServicesToolkit

toolkit = AzureCognitiveServicesToolkit()

API 参考：AzureCognitiveServicesToolkit

[tool.name for tool in toolkit.get_tools()]

['Azure Cognitive Services Image Analysis',
 'Azure Cognitive Services Form Recognizer',
 'Azure Cognitive Services Speech2Text',
 'Azure Cognitive Services Text2Speech']

在代理中使用

from langchain.agents import AgentType, initialize_agent
from langchain_openai import OpenAI

API 参考：AgentType | initialize_agent | OpenAI

llm = OpenAI(temperature=0)
agent = initialize_agent(
    tools=toolkit.get_tools(),
    llm=llm,
    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
)

agent.run(
    "What can I make with these ingredients?"
    "https://images.openai.com/blob/9ad5a2ab-041f-475f-ad6a-b51899c50182/ingredients.png"
)

> Entering new AgentExecutor chain...

Action:

{ "action": "Azure 认知服务图像分析", "action_input": "https://images.openai.com/blob/9ad5a2ab-041f-475f-ad6a-b51899c50182/ingredients.png" }


Observation: Caption: a group of eggs and flour in bowls
Objects: Egg, Egg, Food
Tags: dairy, ingredient, indoor, thickening agent, food, mixing bowl, powder, flour, egg, bowl
Thought: I can use the objects and tags to suggest recipes
Action:

{ "action": "最终答案", "action_input": "您可以用这些食材制作煎饼、煎蛋卷或蛋奶酥！" }

> Finished chain.

'You can make pancakes, omelettes, or quiches with these ingredients!'

audio_file = agent.run("Tell me a joke and read it out for me.")

> Entering new AgentExecutor chain...
Action:

{ "action": "Azure 认知服务文本转语音", "action_input": "为什么鸡要过操场？为了去另一边的滑梯！" }


Observation: /tmp/tmpa3uu_j6b.wav
Thought: I have the audio file of the joke
Action:

{ "action": "最终答案", "action_input": "/tmp/tmpa3uu_j6b.wav" }

> Finished chain.

'/tmp/tmpa3uu_j6b.wav'

from IPython import display

audio = display.Audio(audio_file)
display.display(audio)

agent.run(
    """The patient is a 54-year-old gentleman with a history of progressive angina over the past several months.
The patient had a cardiac catheterization in July of this year revealing total occlusion of the RCA and 50% left main disease ,
with a strong family history of coronary artery disease with a brother dying at the age of 52 from a myocardial infarction and
another brother who is status post coronary artery bypass grafting. The patient had a stress echocardiogram done on July , 2001 ,
which showed no wall motion abnormalities , but this was a difficult study due to body habitus. The patient went for six minutes with
minimal ST depressions in the anterior lateral leads , thought due to fatigue and wrist pain , his anginal equivalent. Due to the patient's
increased symptoms and family history and history left main disease with total occasional of his RCA was referred for revascularization with open heart surgery.

List all the diagnoses.
"""
)

> Entering new AgentExecutor chain...
Action:

{ "action": "azure_cognitive_services_text_analyics_health", "action_input": "患者是一位 54 岁的男性，在过去几个月里患有进行性心绞痛病史。患者今年 7 月进行了心脏导管检查，显示 RCA 完全闭塞和左主干病变 50%，有强烈的冠状动脉疾病家族史，其兄弟在 52 岁时死于心肌梗死，另一位兄弟接受过冠状动脉搭桥术。患者于 2001 年 7 月进行了压力超声心动图检查，结果显示没有心壁运动异常，但由于体格原因，这项检查很困难。患者持续 6 分钟，前外侧导联的 ST 段压低最小，认为是由于疲劳和腕部疼痛，他的心绞痛等效物。由于患者症状加重，以及家族史和左主干疾病史，加上 RCA 完全闭塞，因此建议进行经开放心脏手术的血管重建。" }


Observation: The text conatins the following healthcare entities: 54-year-old is a healthcare entity of type Age, gentleman is a healthcare entity of type Gender, progressive angina is a healthcare entity of type Diagnosis, past several months is a healthcare entity of type Time, cardiac catheterization is a healthcare entity of type ExaminationName, July of this year is a healthcare entity of type Time, total is a healthcare entity of type ConditionQualifier, occlusion is a healthcare entity of type SymptomOrSign, RCA is a healthcare entity of type BodyStructure, 50 is a healthcare entity of type MeasurementValue, % is a healthcare entity of type MeasurementUnit, left main is a healthcare entity of type BodyStructure, disease is a healthcare entity of type Diagnosis, family is a healthcare entity of type FamilyRelation, coronary artery disease is a healthcare entity of type Diagnosis, brother is a healthcare entity of type FamilyRelation, dying is a healthcare entity of type Diagnosis, 52 is a healthcare entity of type Age, myocardial infarction is a healthcare entity of type Diagnosis, brother is a healthcare entity of type FamilyRelation, coronary artery bypass grafting is a healthcare entity of type TreatmentName, stress echocardiogram is a healthcare entity of type ExaminationName, July, 2001 is a healthcare entity of type Time, wall motion abnormalities is a healthcare entity of type SymptomOrSign, body habitus is a healthcare entity of type SymptomOrSign, six minutes is a healthcare entity of type Time, minimal is a healthcare entity of type ConditionQualifier, ST depressions in the anterior lateral leads is a healthcare entity of type SymptomOrSign, fatigue is a healthcare entity of type SymptomOrSign, wrist pain is a healthcare entity of type SymptomOrSign, anginal equivalent is a healthcare entity of type SymptomOrSign, increased is a healthcare entity of type Course, symptoms is a healthcare entity of type SymptomOrSign, family is a healthcare entity of type FamilyRelation, left is a healthcare entity of type Direction, main is a healthcare entity of type BodyStructure, disease is a healthcare entity of type Diagnosis, occasional is a healthcare entity of type Course, RCA is a healthcare entity of type BodyStructure, revascularization is a healthcare entity of type TreatmentName, open heart surgery is a healthcare entity of type TreatmentName
Thought: I know what to respond
Action:

{ "action": "最终答案", "action_input": "文本包含以下诊断：进行性心绞痛、冠状动脉疾病、心肌梗死和冠状动脉搭桥术。" }

> Finished chain.

'The text contains the following diagnoses: progressive angina, coronary artery disease, myocardial infarction, and coronary artery bypass grafting.'

相关

• 工具 概念指南
• 工具 操作指南