LumberChunker: Lengthy-Type Narrative Doc Segmentation – Machine Studying Weblog | ML@CMU

Hyperlinks:
Paper | Code | Knowledge

LumberChunker lets an LLM determine the place a protracted story must be cut up, creating extra pure chunks that assist Retrieval Augmented Era (RAG) techniques retrieve the proper info.

Introduction

Lengthy-form narrative paperwork often have an specific construction, reminiscent of chapters or sections, however these models are sometimes too broad for retrieval duties. At a decrease stage, vital semantic shifts occur inside these bigger segments with none seen structural break. After we cut up textual content solely by formatting cues, like paragraphs or mounted token home windows, passages that belong to the identical narrative unit could also be separated, whereas unrelated content material could be grouped collectively. This misalignment between construction and which means produces chunks that include incomplete or combined context, which reduces retrieval high quality and impacts downstream RAG efficiency. Because of this, segmentation ought to goal to create chunks which might be semantically impartial, moderately than relying solely on doc construction.

So how will we protect the story’s movement and nonetheless preserve chunking sensible?

In lots of circumstances, a reader can simply acknowledge the place the narrative begins to shift—for instance, when the textual content strikes to a distinct scene, introduces a brand new entity, or modifications its goal. The issue is that the majority automated chunking strategies don’t think about this semantic sign and as an alternative rely solely on floor construction. Consequently, they might produce segmentations that look cheap from a formatting perspective however break the underlying narrative coherence.

To make this concrete, think about the quick passage beneath and determine the optimum chunking boundary!

The LumberChunker Technique

Within the instance above, Choice C supplies probably the most coherent segmentation. The boundary aligns with the purpose the place the narrative turns into semantically impartial from the previous context.

Our objective is to make this sort of segmentation resolution sensible at scale. The problem is that human-quality boundary detection requires understanding narrative context, which is pricey to use throughout 1000’s of paragraphs in long-form paperwork.

LumberChunker approaches this by treating segmentation as a boundary-finding downside: given a brief sequence of consecutive paragraphs, we ask a language mannequin to establish the earliest level the place the content material clearly shifts. This formulation permits segments to differ in size whereas remaining aligned with the underlying narrative construction. In apply, LumberChunker consists of those steps:

1) Doc Paragraph Extraction

Cleanly cut up the guide into paragraphs and assign secure IDs (ID:1, ID:2, …). This preserves the doc’s pure discourse models and offers us secure candidate boundaries.

Instance: From a novel, we extract:

ID:1 “The morning solar filtered by means of the dusty home windows…”
ID:2 “She walked slowly to the door, hesitating…”
ID:3 “In the meantime, throughout city, Detective Morrison reviewed the case information…”
ID:4 “The earlier evening’s occasions had left him puzzled…”

Every paragraph will get a singular ID for monitoring boundaries.

2) IDs Grouping for LLM

Construct a bunch G_i by appending paragraphs till the group’s size reaches a token finances θ. This supplies sufficient context for the mannequin to evaluate when a subject/scene truly shifts.

Instance: With θ = 550 tokens, we construct, per instance:

G_1 = [ID:1, ID:2, ID:3, ID:4, ID:5, ID:6]

This window, by spanning a number of paragraphs, will increase the prospect that at the very least one significant narrative shift is current inside the context.

3) LLM Question

Immediate the mannequin with the paragraphs in G_i and ask it to return the first paragraph the place content material clearly modifications relative to what got here earlier than. Use that returned ID because the chunk boundary; begin the following group at that paragraph and repeat to the tip of the guide.

Instance: Given G_1 = [p1, p2, p3, p4, p5, p6], the LLM responds: p3

Reply Extraction:
We extract p3 because the boundary. This creates:

• Chunk 1: [p1, p2]
• Subsequent group (G_2) begins at p3

GutenQA: A Benchmark for Lengthy-Type Narrative Retrieval

To guage our chunking method, we introduce GutenQA, a benchmark of 100 rigorously cleaned public-domain books paired with 3,000 needle-in-a-haystack sort of questions. This enables us to measure retrieval high quality immediately after which observe how higher retrieval results in extra correct solutions in a RAG system.

@inproceedings{duarte-etal-2024-lumberchunker,
    title = "{L}umber{C}hunker: Lengthy-Type Narrative Doc Segmentation",
    creator = "Duarte, Andr{'e} V.  and Marques, Jo{~a}o DS  and Gra{c{c}}a, Miguel  and Freire, Miguel  and Li, Lei  and Oliveira, Arlindo L.",
    editor = "Al-Onaizan, Yaser  and Bansal, Mohit  and Chen, Yun-Nung",
    booktitle = "Findings of the Affiliation for Computational Linguistics: EMNLP 2024",
    month = nov,
    yr = "2024",
    deal with = "Miami, Florida, USA",
    writer = "Affiliation for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.377/",
    doi = "10.18653/v1/2024.findings-emnlp.377",
    pages = "6473--6486",
    summary = "LumberChunker reframes doc chunking as a semantic boundary detection downside..."
}