TextGen
GitHub:oobabooga/text-generation-webui 用于运行大型语言模型（如 LLaMA、llama.cpp、GPT-J、Pythia、OPT 和 GALACTICA）的 Gradio Web UI。
此示例介绍了如何使用 LangChain 通过text-generation-webui API 集成与 LLM 模型交互。
请确保您已配置text-generation-webui并安装了 LLM。建议通过适合您操作系统的单键安装程序进行安装。
安装text-generation-webui并通过 Web 界面确认其正常工作后，请通过 Web 模型配置选项卡或通过向启动命令添加运行时参数--api来启用api选项。
设置 model_url 并运行示例
model_url = "http://localhost:5000"
from langchain.chains import LLMChain
from langchain.globals import set_debug
from langchain_community.llms import TextGen
from langchain_core.prompts import PromptTemplate
set_debug(True)
template = """Question: {question}
Answer: Let's think step by step."""
prompt = PromptTemplate.from_template(template)
llm = TextGen(model_url=model_url)
llm_chain = LLMChain(prompt=prompt, llm=llm)
question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"
llm_chain.run(question)
流式版本
您应该安装 websocket-client 以使用此功能。pip install websocket-client
model_url = "ws://localhost:5005"
from langchain.chains import LLMChain
from langchain.globals import set_debug
from langchain_community.llms import TextGen
from langchain_core.callbacks import StreamingStdOutCallbackHandler
from langchain_core.prompts import PromptTemplate
set_debug(True)
template = """Question: {question}
Answer: Let's think step by step."""
prompt = PromptTemplate.from_template(template)
llm = TextGen(
model_url=model_url, streaming=True, callbacks=[StreamingStdOutCallbackHandler()]
)
llm_chain = LLMChain(prompt=prompt, llm=llm)
question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"
llm_chain.run(question)
llm = TextGen(model_url=model_url, streaming=True)
for chunk in llm.stream("Ask 'Hi, how are you?' like a pirate:'", stop=["'", "\n"]):
print(chunk, end="", flush=True)