Azure AI 服务工具包

此工具包用于与 Azure AI Services API 交互，以实现一些多模式功能。

目前，此工具包中捆绑了五个工具

• AzureAiServicesImageAnalysisTool：用于从图像中提取标题、对象、标签和文本。
• AzureAiServicesDocumentIntelligenceTool：用于从文档中提取文本、表格和键值对。
• AzureAiServicesSpeechToTextTool：用于将语音转录为文本。
• AzureAiServicesTextToSpeechTool：用于将文本合成语音。
• AzureAiServicesTextAnalyticsForHealthTool：用于提取医疗保健实体。

首先，您需要设置 Azure 帐户并创建 AI 服务资源。您可以按照 此处 的说明创建资源。

然后，您需要获取资源的端点、密钥和区域，并将它们设置为环境变量。您可以在资源的“密钥和端点”页面中找到它们。

%pip install --upgrade --quiet  azure-ai-formrecognizer > /dev/null
%pip install --upgrade --quiet  azure-cognitiveservices-speech > /dev/null
%pip install --upgrade --quiet  azure-ai-textanalytics > /dev/null
%pip install --upgrade --quiet  azure-ai-vision-imageanalysis > /dev/null
%pip install -qU langchain-community

import os

os.environ["OPENAI_API_KEY"] = "sk-"
os.environ["AZURE_AI_SERVICES_KEY"] = ""
os.environ["AZURE_AI_SERVICES_ENDPOINT"] = ""
os.environ["AZURE_AI_SERVICES_REGION"] = ""

创建工具包

from langchain_community.agent_toolkits import AzureAiServicesToolkit

toolkit = AzureAiServicesToolkit()

API 参考：AzureAiServicesToolkit

[tool.name for tool in toolkit.get_tools()]

['azure_ai_services_document_intelligence',
 'azure_ai_services_image_analysis',
 'azure_ai_services_speech_to_text',
 'azure_ai_services_text_to_speech',
 'azure_ai_services_text_analytics_for_health']

在代理中使用

from langchain import hub
from langchain.agents import AgentExecutor, create_structured_chat_agent
from langchain_openai import OpenAI

API 参考：AgentExecutor | create_structured_chat_agent | OpenAI

llm = OpenAI(temperature=0)
tools = toolkit.get_tools()
prompt = hub.pull("hwchase17/structured-chat-agent")
agent = create_structured_chat_agent(llm, tools, prompt)

agent_executor = AgentExecutor(
    agent=agent, tools=tools, verbose=True, handle_parsing_errors=True
)

agent_executor.invoke(
    {
        "input": "What can I make with these ingredients? "
        + "https://images.openai.com/blob/9ad5a2ab-041f-475f-ad6a-b51899c50182/ingredients.png"
    }
)

> Entering new AgentExecutor chain...

Thought: I need to use the azure_ai_services_image_analysis tool to analyze the image of the ingredients.
Action:

{ "action": "azure_ai_services_image_analysis", "action_input": "https://images.openai.com/blob/9ad5a2ab-041f-475f-ad6a-b51899c50182/ingredients.png" }

Caption: a group of eggs and flour in bowls
Objects: Egg, Egg, Food
Tags: dairy, ingredient, indoor, thickening agent, food, mixing bowl, powder, flour, egg, bowl
Action:

{ "action": "Final Answer", "action_input": "您可以用这些食材制作蛋糕或其他烘焙食品。" }



> Finished chain.

{'input': 'What can I make with these ingredients? https://images.openai.com/blob/9ad5a2ab-041f-475f-ad6a-b51899c50182/ingredients.png',
 'output': 'You can make a cake or other baked goods with these ingredients.'}

tts_result = agent_executor.invoke({"input": "Tell me a joke and read it out for me."})
audio_file = tts_result.get("output")

> Entering new AgentExecutor chain...

Thought: I can use the Azure AI Services Text to Speech API to convert text to speech.
Action:

{ "action": "azure_ai_services_text_to_speech", "action_input": "为什么科学家不相信原子？因为它们构成了所有事物。" }

/tmp/tmpe48vamz0.wav


> Finished chain.

from IPython import display

audio = display.Audio(data=audio_file, autoplay=True, rate=22050)
display.display(audio)

sample_input = """
The patient is a 54-year-old gentleman with a history of progressive angina over the past several months.
The patient had a cardiac catheterization in July of this year revealing total occlusion of the RCA and 50% left main disease ,
with a strong family history of coronary artery disease with a brother dying at the age of 52 from a myocardial infarction and
another brother who is status post coronary artery bypass grafting. The patient had a stress echocardiogram done on July , 2001 ,
which showed no wall motion abnormalities , but this was a difficult study due to body habitus. The patient went for six minutes with
minimal ST depressions in the anterior lateral leads , thought due to fatigue and wrist pain , his anginal equivalent. Due to the patient's
increased symptoms and family history and history left main disease with total occasional of his RCA was referred for revascularization with open heart surgery.

List all the diagnoses.
"""

agent_executor.invoke({"input": sample_input})

> Entering new AgentExecutor chain...

Thought: The patient has a history of progressive angina, a strong family history of coronary artery disease, and a previous cardiac catheterization revealing total occlusion of the RCA and 50% left main disease.
Action:

{ "action": "azure_ai_services_text_analytics_for_health", "action_input": "患者是一位 54 岁的男性，病史为过去几个月逐渐加重的胸痛。患者于今年 7 月接受了心脏导管检查，结果显示 RCA 完全闭塞，左主干病变 50%，有强烈的冠心病家族史，其兄弟 52 岁死于心肌梗塞，另一个兄弟接受了冠状动脉搭桥术。患者于 2001 年 7 月进行了心脏超声应激检查，结果显示没有心壁运动异常，但由于体格原因，这只是一项困难的研究。患者运动 6 分钟，前外侧导联 ST 段压低轻微，认为是疲劳和腕部疼痛（他的心绞痛等效物）。由于患者症状加重，家族史和左主干病变病史，以及 RCA 完全闭塞，他被转诊接受心脏外科手术进行血管再通术。" [0m [33;1m [1;3m文本包含以下医疗保健实体：54 岁是类型为年龄的医疗保健实体，男性是类型为性别的医疗保健实体，逐渐加重的胸痛是类型为诊断的医疗保健实体，过去几个月是类型为时间的医疗保健实体，心脏导管检查是类型为检查名称的医疗保健实体，今年 7 月是类型为时间的医疗保健实体，完全是类型为条件限定词的医疗保健实体，闭塞是类型为症状或体征的医疗保健实体，RCA 是类型为身体结构的医疗保健实体，50 是类型为测量值的医疗保健实体，% 是类型为测量单位的医疗保健实体，左主干病变是类型为诊断的医疗保健实体，家族是类型为家庭关系的医疗保健实体，冠心病是类型为诊断的医疗保健实体，兄弟是类型为家庭关系的医疗保健实体，死亡是类型为诊断的医疗保健实体，52 是类型为年龄的医疗保健实体，心肌梗塞是类型为诊断的医疗保健实体，兄弟是类型为家庭关系的医疗保健实体，冠状动脉搭桥术是类型为治疗名称的医疗保健实体，心脏超声应激检查是类型为检查名称的医疗保健实体，2001 年 7 月是类型为时间的医疗保健实体，心壁运动异常是类型为症状或体征的医疗保健实体，体格是类型为症状或体征的医疗保健实体，6 分钟是类型为时间的医疗保健实体，轻微是类型为条件限定词的医疗保健实体，前外侧导联 ST 段压低是类型为症状或体征的医疗保健实体，疲劳是类型为症状或体征的医疗保健实体，腕部疼痛是类型为症状或体征的医疗保健实体，心绞痛是类型为症状或体征的医疗保健实体，加重是类型为过程的医疗保健实体，症状是类型为症状或体征的医疗保健实体，家族是类型为家庭关系的医疗保健实体，左主干病变是类型为诊断的医疗保健实体，闭塞是类型为过程的医疗保健实体，RCA 是类型为身体结构的医疗保健实体，血管再通术是类型为治疗名称的医疗保健实体，心脏外科手术是类型为治疗名称的医疗保健实体 [0m [32;1m [1;3m 操作

{
  "action": "Final Answer",
  "action_input": "The patient's diagnoses include progressive angina, total occlusion of the RCA, 50% left main disease, coronary artery disease, myocardial infarction, and a family history of coronary artery disease."
}



> Finished chain.

{'input': "\nThe patient is a 54-year-old gentleman with a history of progressive angina over the past several months.\nThe patient had a cardiac catheterization in July of this year revealing total occlusion of the RCA and 50% left main disease ,\nwith a strong family history of coronary artery disease with a brother dying at the age of 52 from a myocardial infarction and\nanother brother who is status post coronary artery bypass grafting. The patient had a stress echocardiogram done on July , 2001 ,\nwhich showed no wall motion abnormalities , but this was a difficult study due to body habitus. The patient went for six minutes with\nminimal ST depressions in the anterior lateral leads , thought due to fatigue and wrist pain , his anginal equivalent. Due to the patient's\nincreased symptoms and family history and history left main disease with total occasional of his RCA was referred for revascularization with open heart surgery.\n\nList all the diagnoses.\n",
 'output': "The patient's diagnoses include progressive angina, total occlusion of the RCA, 50% left main disease, coronary artery disease, myocardial infarction, and a family history of coronary artery disease."}

相关

• 工具 概念指南
• 工具 操作指南