跳到主要内容
Open In ColabOpen on GitHub

如何分割HTML

将HTML文档拆分为可管理的小块对于各种文本处理任务至关重要，例如自然语言处理、搜索索引等。本指南将探讨LangChain提供的三种不同的文本拆分器，您可以有效地使用它们来拆分HTML内容。

这些拆分器中的每一个都具有独特的功能和用例。本指南将帮助您了解它们之间的差异，为什么您可能会选择其中一个而不是其他，以及如何有效地使用它们。

%pip install -qU langchain-text-splitters

拆分器概述​

HTMLHeaderTextSplitter​

信息

当您希望根据文档的标题保留其分层结构时非常有用。

描述：根据标题标签（例如 `

`、`

`、`

` 等）拆分HTML文本，并为每个与给定块相关的标题添加元数据。

功能:

  • 在HTML元素级别拆分文本。
  • 保留文档结构中编码的上下文丰富信息。
  • 可以逐个元素返回块，或者将具有相同元数据的元素合并。

HTMLSectionSplitter​

信息

当您希望将HTML文档拆分为更大的部分时非常有用，例如 `section`、`div` 或自定义定义的部分。

描述：类似于 HTMLHeaderTextSplitter，但侧重于根据指定标签将HTML拆分为部分。

功能:

  • 使用XSLT转换来检测和拆分部分。
  • 内部对大节使用 `RecursiveCharacterTextSplitter`。
  • 考虑字体大小来确定部分。

HTMLSemanticPreservingSplitter​

信息

当您需要确保结构化元素不会跨块拆分，从而保留上下文相关性时，它是理想选择。

描述：将HTML内容拆分为可管理的小块，同时保留表格、列表和其他HTML组件等重要元素的语义结构。

功能:

  • 保留表格、列表和其他指定的HTML元素。
  • 允许为特定的HTML标签定义自定义处理程序。
  • 确保文档的语义意义得以保持。
  • 内置标准化和停用词删除

选择正确的拆分器​

  • 当您需要使用 `HTMLHeaderTextSplitter` 时：您需要根据HTML文档的标题层级拆分文档并维护有关标题的元数据。
  • 当您需要使用 `HTMLSectionSplitter` 时：您需要将文档拆分为更大、更通用的部分，可能基于自定义标签或字体大小。
  • 当您需要使用 `HTMLSemanticPreservingSplitter` 时：您需要将文档拆分为块，同时保留表格和列表等语义元素，确保它们不被拆分且其上下文得到维护。
功能HTMLHeaderTextSplitterHTMLSectionSplitterHTMLSemanticPreservingSplitter
基于标题拆分是是是
保留语义元素（表格、列表）否否是
为标题添加元数据是是是
HTML标签的自定义处理程序否否是
保留媒体（图像、视频）否否是
考虑字体大小否是否
使用XSLT转换否是否

HTML文档示例​

让我们使用以下HTML文档作为示例

html_string = """
<!DOCTYPE html>
  <html lang='en'>
  <head>
    <meta charset='UTF-8'>
    <meta name='viewport' content='width=device-width, initial-scale=1.0'>
    <title>Fancy Example HTML Page</title>
  </head>
  <body>
    <h1>Main Title</h1>
    <p>This is an introductory paragraph with some basic content.</p>
    
    <h2>Section 1: Introduction</h2>
    <p>This section introduces the topic. Below is a list:</p>
    <ul>
      <li>First item</li>
      <li>Second item</li>
      <li>Third item with <strong>bold text</strong> and <a href='#'>a link</a></li>
    </ul>
    
    <h3>Subsection 1.1: Details</h3>
    <p>This subsection provides additional details. Here's a table:</p>
    <table border='1'>
      <thead>
        <tr>
          <th>Header 1</th>
          <th>Header 2</th>
          <th>Header 3</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>Row 1, Cell 1</td>
          <td>Row 1, Cell 2</td>
          <td>Row 1, Cell 3</td>
        </tr>
        <tr>
          <td>Row 2, Cell 1</td>
          <td>Row 2, Cell 2</td>
          <td>Row 2, Cell 3</td>
        </tr>
      </tbody>
    </table>
    
    <h2>Section 2: Media Content</h2>
    <p>This section contains an image and a video:</p>
      <img src='example_image_link.mp4' alt='Example Image'>
      <video controls width='250' src='example_video_link.mp4' type='video/mp4'>
      Your browser does not support the video tag.
    </video>

    <h2>Section 3: Code Example</h2>
    <p>This section contains a code block:</p>
    <pre><code data-lang="html">
    &lt;div&gt;
      &lt;p&gt;This is a paragraph inside a div.&lt;/p&gt;
    &lt;/div&gt;
    </code></pre>

    <h2>Conclusion</h2>
    <p>This is the conclusion of the document.</p>
  </body>
  </html>
"""

使用 HTMLHeaderTextSplitter​

HTMLHeaderTextSplitter 是一种“结构感知”文本拆分器，它在HTML元素级别拆分文本，并为每个与给定块“相关”的标题添加元数据。它可以逐个元素返回块，或将具有相同元数据的元素合并，其目的是 (a) 在语义上（或多或少地）将相关文本分组，以及 (b) 保留文档结构中编码的上下文丰富信息。它可以与其他文本拆分器一起用作分块管道的一部分。

它类似于Markdown文件的MarkdownHeaderTextSplitter。

要指定要拆分的标题，请在实例化 `HTMLHeaderTextSplitter` 时指定 `headers_to_split_on`，如下所示。

from langchain_text_splitters import HTMLHeaderTextSplitter

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)
html_header_splits = html_splitter.split_text(html_string)
html_header_splits
[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a list:  \nFirst item Second item Third item with bold text and a link'),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction', 'Header 3': 'Subsection 1.1: Details'}, page_content="This subsection provides additional details. Here's a table:"),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block:'),
 Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]

要将每个元素及其关联的标题一起返回，请在实例化 `HTMLHeaderTextSplitter` 时指定 `return_each_element=True`

html_splitter = HTMLHeaderTextSplitter(
    headers_to_split_on,
    return_each_element=True,
)
html_header_splits_elements = html_splitter.split_text(html_string)

与上面按标题聚合元素的情况进行比较

for element in html_header_splits[:2]:
    print(element)
page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}
page_content='This section introduces the topic. Below is a list:  
First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}

现在每个元素都作为独立的 `Document` 返回

for element in html_header_splits_elements[:3]:
    print(element)
page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}
page_content='This section introduces the topic. Below is a list:' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}
page_content='First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}

如何从URL或HTML文件拆分：​

要直接从URL读取，请将URL字符串传递给 `split_text_from_url` 方法。

同样，本地HTML文件可以传递给 `split_text_from_file` 方法。

url = "https://plato.stanford.edu/entries/goedel/"

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
    ("h4", "Header 4"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)

# for local file use html_splitter.split_text_from_file(<path_to_file>)
html_header_splits = html_splitter.split_text_from_url(url)

如何限制块大小：​

`HTMLHeaderTextSplitter`（基于HTML标题进行拆分）可以与另一个基于字符长度限制拆分的拆分器（例如 `RecursiveCharacterTextSplitter`）组合使用。

这可以通过使用第二个拆分器的 `.split_documents` 方法来完成

from langchain_text_splitters import RecursiveCharacterTextSplitter

chunk_size = 500
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(html_header_splits)
splits[80:85]
[Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='We see that Gödel first tried to reduce the consistency problem for analysis to that of arithmetic. This seemed to require a truth definition for arithmetic, which in turn led to paradoxes, such as the Liar paradox (“This sentence is false”) and Berry’s paradox (“The least number not defined by an expression consisting of just fourteen English words”). Gödel then noticed that such paradoxes would not necessarily arise if truth were replaced by provability. But this means that arithmetic truth'),
 Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='means that arithmetic truth and arithmetic provability are not co-extensive — whence the First Incompleteness Theorem.'),
 Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='This account of Gödel’s discovery was told to Hao Wang very much after the fact; but in Gödel’s contemporary correspondence with Bernays and Zermelo, essentially the same description of his path to the theorems is given. (See Gödel 2003a and Gödel 2003b respectively.) From those accounts we see that the undefinability of truth in arithmetic, a result credited to Tarski, was likely obtained in some form by Gödel by 1931. But he neither publicized nor published the result; the biases logicians'),
 Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='result; the biases logicians had expressed at the time concerning the notion of truth, biases which came vehemently to the fore when Tarski announced his results on the undefinability of truth in formal systems 1935, may have served as a deterrent to Gödel’s publication of that theorem.'),
 Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.2 The proof of the First Incompleteness Theorem'}, page_content='We now describe the proof of the two theorems, formulating Gödel’s results in Peano arithmetic. Gödel himself used a system related to that defined in Principia Mathematica, but containing Peano arithmetic. In our presentation of the First and Second Incompleteness Theorems we refer to Peano arithmetic as P, following Gödel’s notation.')]

限制​

不同的HTML文档之间可能存在相当大的结构差异，虽然 `HTMLHeaderTextSplitter` 会尝试将所有“相关”标题附加到任何给定的块，但它有时可能会遗漏某些标题。例如，该算法假设一种信息层级结构，其中标题始终位于关联文本“上方”的节点，即先前的同级、祖先及其组合。在以下新闻文章中（截至本文档撰写时），文档的结构使得顶级标题的文本（虽然标记为“h1”）与我们预期它应位于“上方”的文本元素位于一个不同的子树中——因此我们可以观察到“h1”元素及其关联文本未出现在块元数据中（但，在适用情况下，我们确实看到了“h2”及其关联文本）。

url = "https://www.cnn.com/2023/09/25/weather/el-nino-winter-us-climate/index.html"

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)
html_header_splits = html_splitter.split_text_from_url(url)
print(html_header_splits[1].page_content[:500])
No two El Niño winters are the same, but many have temperature and precipitation trends in common.  
Average conditions during an El Niño winter across the continental US.  
One of the major reasons is the position of the jet stream, which often shifts south during an El Niño winter. This shift typically brings wetter and cooler weather to the South while the North becomes drier and warmer, according to NOAA.  
Because the jet stream is essentially a river of air that storms flow through, they c

使用 HTMLSectionSplitter​

与 HTMLHeaderTextSplitter 的概念相似，`HTMLSectionSplitter` 是一种“结构感知”文本拆分器，它在元素级别拆分文本，并为每个与给定块“相关”的标题添加元数据。它允许您按节拆分HTML。

它可以逐个元素返回块，或者将具有相同元数据的元素合并，其目的是 (a) 在语义上（或多或少地）将相关文本分组，以及 (b) 保留文档结构中编码的上下文丰富信息。

使用 `xslt_path` 提供HTML转换的绝对路径，以便它能够根据提供的标签检测部分。默认是使用 `data_connection/document_transformers` 目录中的 `converting_to_header.xslt` 文件。这是为了将HTML转换为更容易检测部分的格式/布局。例如，可以根据字体大小将 `span` 转换为标题标签，以便检测为部分。

如何拆分HTML字符串：​

from langchain_text_splitters import HTMLSectionSplitter

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
]

html_splitter = HTMLSectionSplitter(headers_to_split_on)
html_header_splits = html_splitter.split_text(html_string)
html_header_splits
API 参考：HTMLSectionSplitter
[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title \n This is an introductory paragraph with some basic content.'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content="Section 1: Introduction \n This section introduces the topic. Below is a list: \n \n First item \n Second item \n Third item with  bold text  and  a link \n \n \n Subsection 1.1: Details \n This subsection provides additional details. Here's a table: \n \n \n \n Header 1 \n Header 2 \n Header 3 \n \n \n \n \n Row 1, Cell 1 \n Row 1, Cell 2 \n Row 1, Cell 3 \n \n \n Row 2, Cell 1 \n Row 2, Cell 2 \n Row 2, Cell 3"),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content \n This section contains an image and a video: \n \n \n      Your browser does not support the video tag.'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example \n This section contains a code block: \n \n    <div>\n      <p>This is a paragraph inside a div.</p>\n    </div>'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion \n This is the conclusion of the document.')]

如何限制块大小：​

`HTMLSectionSplitter` 可以与其他文本拆分器一起用作分块管道的一部分。在内部，当节大小大于块大小时，它会使用 `RecursiveCharacterTextSplitter`。它还会考虑文本的字体大小，根据确定的字体大小阈值来判断它是否是一个节。

from langchain_text_splitters import RecursiveCharacterTextSplitter

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
]

html_splitter = HTMLSectionSplitter(headers_to_split_on)

html_header_splits = html_splitter.split_text(html_string)

chunk_size = 50
chunk_overlap = 5
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(html_header_splits)
splits
[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title'),
 Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some'),
 Document(metadata={'Header 1': 'Main Title'}, page_content='some basic content.'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Section 1: Introduction'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='is a list:'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='First item \n Second item'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Third item with  bold text  and  a link'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Subsection 1.1: Details'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='This subsection provides additional details.'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content="Here's a table:"),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Header 1 \n Header 2 \n Header 3'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 1 \n Row 1, Cell 2'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 3 \n \n \n Row 2, Cell 1'),
 Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 2, Cell 2 \n Row 2, Cell 3'),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content'),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Your browser does not support the video'),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='tag.'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: \n \n    <div>'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='<p>This is a paragraph inside a div.</p>'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='</div>'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]

使用 HTMLSemanticPreservingSplitter​

`HTMLSemanticPreservingSplitter` 旨在将HTML内容拆分为可管理的小块，同时保留表格、列表和其他HTML组件等重要元素的语义结构。这确保了此类元素不会跨块拆分，从而避免了上下文相关性（如表格标题、列表标题等）的丢失。

这个拆分器从本质上旨在创建上下文相关的块。使用 `HTMLHeaderTextSplitter` 进行通用递归拆分可能会导致表格、列表和其他结构化元素在中间被拆分，从而丢失大量上下文并产生不良块。

`HTMLSemanticPreservingSplitter` 对于拆分包含表格和列表等结构化元素的HTML内容至关重要，特别是当完整保留这些元素至关重要时。此外，它能够为特定HTML标签定义自定义处理程序，使其成为处理复杂HTML文档的多功能工具。

重要提示：`max_chunk_size` 不是块的明确最大大小，最大大小的计算发生在被保留内容不作为块一部分时，以确保其不被拆分。当我们将保留的数据添加回块中时，块大小有可能超出 `max_chunk_size`。这对于确保我们保持原始文档的结构至关重要。

信息

备注

  1. 我们定义了一个自定义处理程序来重新格式化代码块的内容
  2. 我们为特定的HTML元素定义了一个拒绝列表，用于在预处理阶段分解它们及其内容
  3. 我们有意设置了一个小的块大小，以演示元素的不拆分特性
# BeautifulSoup is required to use the custom handlers
from bs4 import Tag
from langchain_text_splitters import HTMLSemanticPreservingSplitter

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
]


def code_handler(element: Tag) -> str:
    data_lang = element.get("data-lang")
    code_format = f"<code:{data_lang}>{element.get_text()}</code>"

    return code_format


splitter = HTMLSemanticPreservingSplitter(
    headers_to_split_on=headers_to_split_on,
    separators=["\n\n", "\n", ". ", "! ", "? "],
    max_chunk_size=50,
    preserve_images=True,
    preserve_videos=True,
    elements_to_preserve=["table", "ul", "ol", "code"],
    denylist_tags=["script", "style", "head"],
    custom_handlers={"code": code_handler},
)

documents = splitter.split_text(html_string)
documents
[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='. Below is a list: First item Second item Third item with bold text and a link Subsection 1.1: Details This subsection provides additional details'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=". Here's a table: Header 1 Header 2 Header 3 Row 1, Cell 1 Row 1, Cell 2 Row 1, Cell 3 Row 2, Cell 1 Row 2, Cell 2 Row 2, Cell 3"),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video: ![image:example_image_link.mp4](example_image_link.mp4) ![video:example_video_link.mp4](example_video_link.mp4)'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: <code:html> <div> <p>This is a paragraph inside a div.</p> </div> </code>'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]

保留表格和列表​

在此示例中，我们将演示 `HTMLSemanticPreservingSplitter` 如何在HTML文档中保留表格和大型列表。块大小将设置为50个字符，以说明拆分器如何确保这些元素不被拆分，即使它们超过定义的最大块大小。

from langchain_text_splitters import HTMLSemanticPreservingSplitter

html_string = """
<!DOCTYPE html>
<html>
    <body>
        <div>
            <h1>Section 1</h1>
            <p>This section contains an important table and list that should not be split across chunks.</p>
            <table>
                <tr>
                    <th>Item</th>
                    <th>Quantity</th>
                    <th>Price</th>
                </tr>
                <tr>
                    <td>Apples</td>
                    <td>10</td>
                    <td>$1.00</td>
                </tr>
                <tr>
                    <td>Oranges</td>
                    <td>5</td>
                    <td>$0.50</td>
                </tr>
                <tr>
                    <td>Bananas</td>
                    <td>50</td>
                    <td>$1.50</td>
                </tr>
            </table>
            <h2>Subsection 1.1</h2>
            <p>Additional text in subsection 1.1 that is separated from the table and list.</p>
            <p>Here is a detailed list:</p>
            <ul>
                <li>Item 1: Description of item 1, which is quite detailed and important.</li>
                <li>Item 2: Description of item 2, which also contains significant information.</li>
                <li>Item 3: Description of item 3, another item that we don't want to split across chunks.</li>
            </ul>
        </div>
    </body>
</html>
"""

headers_to_split_on = [("h1", "Header 1"), ("h2", "Header 2")]

splitter = HTMLSemanticPreservingSplitter(
    headers_to_split_on=headers_to_split_on,
    max_chunk_size=50,
    elements_to_preserve=["table", "ul"],
)

documents = splitter.split_text(html_string)
print(documents)
[Document(metadata={'Header 1': 'Section 1'}, page_content='This section contains an important table and list'), Document(metadata={'Header 1': 'Section 1'}, page_content='that should not be split across chunks.'), Document(metadata={'Header 1': 'Section 1'}, page_content='Item Quantity Price Apples 10 $1.00 Oranges 5 $0.50 Bananas 50 $1.50'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='Additional text in subsection 1.1 that is'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='separated from the table and list. Here is a'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content="detailed list: Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.")]

解释​

在此示例中，`HTMLSemanticPreservingSplitter` 确保整个表格和无序列表（`

    `）在其各自的块中得到保留。即使块大小设置为50个字符，拆分器也认识到这些元素不应被拆分并保持其完整性。

    这在处理数据表格或列表时尤为重要，因为拆分内容可能会导致上下文丢失或混淆。生成的 `Document` 对象保留了这些元素的完整结构，确保了信息的上下文相关性得以维持。

    使用自定义处理程序​

    `HTMLSemanticPreservingSplitter` 允许您为特定的HTML元素定义自定义处理程序。某些平台具有 `BeautifulSoup` 无法原生解析的自定义HTML标签，在这种情况下，您可以利用自定义处理程序轻松添加格式化逻辑。

    这对于需要特殊处理的元素特别有用，例如 `