Aerospike
Aerospike Vector Search (AVS) 是 Aerospike 数据库的一个扩展，它允许在存储于 Aerospike 中的海量数据集上执行搜索。这项新服务独立于 Aerospike 运行，并构建索引以执行这些搜索。
本笔记本展示了 LangChain Aerospike VectorStore 集成的功能。
安装 AVS
在使用本笔记本之前，我们需要有一个正在运行的 AVS 实例。请使用其中一种可用的安装方法。
完成后，请存储您的 AVS 实例的 IP 地址和端口，以便稍后在本演示中使用
AVS_HOST = "<avs_ip>"
AVS_PORT = 5000
安装依赖项
sentence-transformers 依赖项较大。此步骤可能需要几分钟才能完成。
!pip install --upgrade --quiet aerospike-vector-search==4.2.0 langchain-aerospike langchain-community sentence-transformers langchain
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m25.0.1[0m[39;49m -> [0m[32;49m25.1.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpip install --upgrade pip[0m
下载引用数据集
我们将下载一个包含大约 100,000 条引用的数据集，并使用其中一部分引用进行语义搜索。
!wget https://github.com/aerospike/aerospike-vector-search-examples/raw/7dfab0fccca0852a511c6803aba46578729694b5/quote-semantic-search/container-volumes/quote-search/data/quotes.csv.tgz
--2025-05-07 21:06:30-- https://github.com/aerospike/aerospike-vector-search-examples/raw/7dfab0fccca0852a511c6803aba46578729694b5/quote-semantic-search/container-volumes/quote-search/data/quotes.csv.tgz
Resolving github.com (github.com)... 140.82.116.3
Connecting to github.com (github.com)|140.82.116.3|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/aerospike/aerospike-vector/raw/7dfab0fccca0852a511c6803aba46578729694b5/quote-semantic-search/container-volumes/quote-search/data/quotes.csv.tgz [following]
--2025-05-07 21:06:30-- https://github.com/aerospike/aerospike-vector/raw/7dfab0fccca0852a511c6803aba46578729694b5/quote-semantic-search/container-volumes/quote-search/data/quotes.csv.tgz
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/aerospike/aerospike-vector/7dfab0fccca0852a511c6803aba46578729694b5/quote-semantic-search/container-volumes/quote-search/data/quotes.csv.tgz [following]
--2025-05-07 21:06:30-- https://raw.githubusercontent.com/aerospike/aerospike-vector/7dfab0fccca0852a511c6803aba46578729694b5/quote-semantic-search/container-volumes/quote-search/data/quotes.csv.tgz
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11597643 (11M) [application/octet-stream]
Saving to: ‘quotes.csv.tgz’
quotes.csv.tgz 100%[===================>] 11.06M 12.7MB/s in 0.9s
2025-05-07 21:06:32 (12.7 MB/s) - ‘quotes.csv.tgz’ saved [11597643/11597643]
将引用加载到文档中
我们将使用 CSVLoader 文档加载器加载我们的引用数据集。在这种情况下，lazy_load 返回一个迭代器，以便更有效地摄取我们的引用。在此示例中，我们只加载 5,000 条引用。
import itertools
import os
import tarfile
from langchain_community.document_loaders.csv_loader import CSVLoader
filename = "./quotes.csv"
if not os.path.exists(filename) and os.path.exists(filename + ".tgz"):
# Untar the file
with tarfile.open(filename + ".tgz", "r:gz") as tar:
tar.extractall(path=os.path.dirname(filename))
NUM_QUOTES = 5000
documents = CSVLoader(filename, metadata_columns=["author", "category"]).lazy_load()
documents = list(
itertools.islice(documents, NUM_QUOTES)
) # Allows us to slice an iterator
API 参考:CSVLoader
print(documents[0])
page_content='quote: I'm selfish, impatient and a little insecure. I make mistakes, I am out of control and at times hard to handle. But if you can't handle me at my worst, then you sure as hell don't deserve me at my best.' metadata={'source': './quotes.csv', 'row': 0, 'author': 'Marilyn Monroe', 'category': 'attributed-no-source, best, life, love, mistakes, out-of-control, truth, worst'}
创建您的嵌入器
在此步骤中，我们使用 HuggingFaceEmbeddings 和 "all-MiniLM-L6-v2" 句子转换模型来嵌入我们的文档，以便我们可以执行向量搜索。
from aerospike_vector_search.types import VectorDistanceMetric
from langchain_community.embeddings import HuggingFaceEmbeddings
MODEL_DIM = 384
MODEL_DISTANCE_CALC = VectorDistanceMetric.COSINE
embedder = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
API 参考:HuggingFaceEmbeddings
/var/folders/h5/lm2_c1xs3s32kwp11prnpftw0000gp/T/ipykernel_84638/3255399720.py:6: LangChainDeprecationWarning: The class `HuggingFaceEmbeddings` was deprecated in LangChain 0.2.2 and will be removed in 1.0. An updated version of the class exists in the :class:`~langchain-huggingface package and should be used instead. To use it run `pip install -U :class:`~langchain-huggingface` and import as `from :class:`~langchain_huggingface import HuggingFaceEmbeddings``.
embedder = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
/Users/dwelch/Desktop/everything/projects/langchain/myfork/langchain/.venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
创建 Aerospike 索引并嵌入文档
在添加文档之前，我们需要在 Aerospike 数据库中创建一个索引。在下面的示例中，我们使用了一些便捷代码来检查预期的索引是否已存在。
from aerospike_vector_search import Client, HostPort
from aerospike_vector_search.types import VectorDistanceMetric
from langchain_aerospike.vectorstores import Aerospike
# Here we are using the AVS host and port you configured earlier
seed = HostPort(host=AVS_HOST, port=AVS_PORT)
# The namespace of where to place our vectors. This should match the vector configured in your docstore.conf file.
NAMESPACE = "test"
# The name of our new index.
INDEX_NAME = "quote-miniLM-L6-v2"
# AVS needs to know which metadata key contains our vector when creating the index and inserting documents.
VECTOR_KEY = "vector"
client = Client(seeds=seed)
index_exists = False
# Check if the index already exists. If not, create it
for index in client.index_list():
if index["id"]["namespace"] == NAMESPACE and index["id"]["name"] == INDEX_NAME:
index_exists = True
print(f"{INDEX_NAME} already exists. Skipping creation")
break
if not index_exists:
print(f"{INDEX_NAME} does not exist. Creating index")
client.index_create(
namespace=NAMESPACE,
name=INDEX_NAME,
vector_field=VECTOR_KEY,
vector_distance_metric=MODEL_DISTANCE_CALC,
dimensions=MODEL_DIM,
index_labels={
"model": "miniLM-L6-v2",
"date": "05/04/2024",
"dim": str(MODEL_DIM),
"distance": "cosine",
},
)
docstore = Aerospike.from_documents(
documents,
embedder,
client=client,
namespace=NAMESPACE,
vector_key=VECTOR_KEY,
index_name=INDEX_NAME,
distance_strategy=MODEL_DISTANCE_CALC,
)
quote-miniLM-L6-v2 does not exist. Creating index
搜索文档
现在我们已经嵌入了向量，我们可以在我们的引用上使用向量搜索。
query = "A quote about the beauty of the cosmos"
docs = docstore.similarity_search(
query, k=5, index_name=INDEX_NAME, metadata_keys=["_id", "author"]
)
def print_documents(docs):
for i, doc in enumerate(docs):
print("~~~~ Document", i, "~~~~")
print("auto-generated id:", doc.metadata["_id"])
print("author: ", doc.metadata["author"])
print(doc.page_content)
print("~~~~~~~~~~~~~~~~~~~~\n")
print_documents(docs)
~~~~ Document 0 ~~~~
auto-generated id: 4984b472-8a32-4552-b3eb-f03b31b68031
author: Carl Sagan, Cosmos
quote: The Cosmos is all that is or was or ever will be. Our feeblest contemplations of the Cosmos stir us -- there is a tingling in the spine, a catch in the voice, a faint sensation, as if a distant memory, of falling from a height. We know we are approaching the greatest of mysteries.
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 1 ~~~~
auto-generated id: 486c8d87-8dd7-450d-9008-d7549e680ffb
author: Renee Ahdieh, The Rose & the Dagger
quote: From the stars, to the stars.
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 2 ~~~~
auto-generated id: 4b43b309-ce51-498c-b225-5254383b5b4a
author: Elizabeth Gilbert
quote: The love that moves the sun and the other stars.
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 3 ~~~~
auto-generated id: af784a10-f498-4570-bf81-2ffdca35440e
author: Dante Alighieri, Paradiso
quote: Love, that moves the sun and the other stars
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 4 ~~~~
auto-generated id: b45d5d5e-d818-4206-ae6b-b1d166ea3d43
author: Thich Nhat Hanh, Teachings on Love
quote: Through my love for you, I want to express my love for the whole cosmos, the whole of humanity, and all beings. By living with you, I want to learn to love everyone and all species. If I succeed in loving you, I will be able to love everyone and all species on Earth... This is the real message of love.
~~~~~~~~~~~~~~~~~~~~
将其他引用作为文本嵌入
我们可以使用 add_texts 添加更多引用。
docstore = Aerospike(
client,
embedder,
NAMESPACE,
index_name=INDEX_NAME,
vector_key=VECTOR_KEY,
distance_strategy=MODEL_DISTANCE_CALC,
)
ids = docstore.add_texts(
[
"quote: Rebellions are built on hope.",
"quote: Logic is the beginning of wisdom, not the end.",
"quote: If wishes were fishes, we’d all cast nets.",
],
metadatas=[
{"author": "Jyn Erso, Rogue One"},
{"author": "Spock, Star Trek"},
{"author": "Frank Herbert, Dune"},
],
)
print("New IDs")
print(ids)
New IDs
['adf8064e-9c0e-46e2-b193-169c36432f4c', 'cf65b5ed-a0f4-491a-86ad-dcacc23c2815', '2ef52efd-d9b7-4077-bc14-defdf0b7dd2f']
使用最大边际相关性搜索文档
我们可以使用最大边际相关性搜索来查找与我们的查询相似但彼此不相似的向量。在此示例中，我们使用 as_retriever 创建了一个检索器对象，但也可以通过直接调用 docstore.max_marginal_relevance_search 轻松完成。lambda_mult 搜索参数决定了查询响应的多样性。0 对应最大多样性，1 对应最小多样性。
query = "A quote about our favorite four-legged pets"
retriever = docstore.as_retriever(
search_type="mmr", search_kwargs={"fetch_k": 20, "lambda_mult": 0.7}
)
matched_docs = retriever.invoke(query)
print_documents(matched_docs)
~~~~ Document 0 ~~~~
auto-generated id: 91e77b39-a528-40c6-a58a-486ae85f991a
author: John Grogan, Marley and Me: Life and Love With the World's Worst Dog
quote: Such short little lives our pets have to spend with us, and they spend most of it waiting for us to come home each day. It is amazing how much love and laughter they bring into our lives and even how much closer we become with each other because of them.
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 1 ~~~~
auto-generated id: c585b4ec-92b5-4579-948c-0529373abc2a
author: John Grogan, Marley and Me: Life and Love With the World's Worst Dog
quote: Dogs are great. Bad dogs, if you can really call them that, are perhaps the greatest of them all.
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 2 ~~~~
auto-generated id: 5768b31c-fac4-4af7-84b4-fb11bbfcb590
author: Colleen Houck, Tiger's Curse
quote: He then put both hands on the door on either side of my head and leaned in close, pinning me against it. I trembled like a downy rabbit caught in the clutches of a wolf. The wolf came closer. He bent his head and began nuzzling my cheek. The problem was…I wanted the wolf to devour me.
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 3 ~~~~
auto-generated id: 94f1b9fb-ad57-4f65-b470-7f49dd6c274c
author: Ray Bradbury
quote: Stuff your eyes with wonder," he said, "live as if you'd drop dead in ten seconds. See the world. It's more fantastic than any dream made or paid for in factories. Ask no guarantees, ask for no security, there never was such an animal. And if there were, it would be related to the great sloth which hangs upside down in a tree all day every day, sleeping its life away. To hell with that," he said, "shake the tree and knock the great sloth down on his ass.
~~~~~~~~~~~~~~~~~~~~
使用相关性阈值搜索文档
另一个有用的功能是带有相关性阈值的相似性搜索。通常，我们只希望获得与查询最相似但也在一定接近度范围内的结果。相关性为 1 表示最相似，相关性为 0 表示最不相似。
query = "A quote about stormy weather"
retriever = docstore.as_retriever(
search_type="similarity_score_threshold",
search_kwargs={
"score_threshold": 0.4
}, # A greater value returns items with more relevance
)
matched_docs = retriever.invoke(query)
print_documents(matched_docs)
~~~~ Document 0 ~~~~
auto-generated id: 6d9e67a6-0427-41e6-9e24-050518120d74
author: Roy T. Bennett, The Light in the Heart
quote: Never lose hope. Storms make people stronger and never last forever.
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 1 ~~~~
auto-generated id: 7d426e59-7935-4bcf-a676-cbe8dd4860e7
author: Roy T. Bennett, The Light in the Heart
quote: Difficulties and adversities viciously force all their might on us and cause us to fall apart, but they are necessary elements of individual growth and reveal our true potential. We have got to endure and overcome them, and move forward. Never lose hope. Storms make people stronger and never last forever.
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 2 ~~~~
auto-generated id: 6ec05e48-d162-440d-8819-001d2f3712f9
author: Vincent van Gogh, The Letters of Vincent van Gogh
quote: There is peace even in the storm
~~~~~~~~~~~~~~~~~~~~
~~~~ Document 3 ~~~~
auto-generated id: d3c3de59-4da4-4ae6-8f6d-83ed905dd320
author: Edwin Morgan, A Book of Lives
quote: Valentine WeatherKiss me with rain on your eyelashes,come on, let us sway together,under the trees, and to hell with thunder.
~~~~~~~~~~~~~~~~~~~~
清理
我们需要确保关闭客户端以释放资源并清理线程。
client.close()
准备。开始。搜索！
现在您已经熟悉了 Aerospike Vector Search 的 LangChain 集成，Aerospike 数据库和 LangChain 生态系统的强大功能尽在您的指尖。祝您构建愉快！