Creating Documents

Creating Documents#

This guide explains how to create and configure Document instances to process textual and visual content for analysis.

Documents serve as the container for the content from which information (aspects and concepts) can be extracted.

⚙️ Configuration Parameters#

The minimum configuration for a document requires either raw_text, paragraphs, or images:

Document creation#

from pathlib import Path

from contextgem import Document, Paragraph, create_image


# Create a document with raw text content
contract_document = Document(
    raw_text=(
        "...This agreement is effective as of January 1, 2025.\n\n"
        "All parties must comply with the terms outlined herein. The terms include "
        "monthly reporting requirements and quarterly performance reviews.\n\n"
        "Failure to adhere to these terms may result in termination of the agreement. "
        "Additionally, any breach of confidentiality will be subject to penalties as "
        "described in this agreement.\n\n"
        "This agreement shall remain in force for a period of three (3) years unless "
        "otherwise terminated according to the provisions stated above..."
    ),
    paragraph_segmentation_mode="newlines",  # Default mode, splits on newlines
)

# Create a document with more advanced paragraph segmentation using a SaT model
report_document = Document(
    raw_text=(
        "Executive Summary "
        "This report outlines our quarterly performance. "
        "Revenue increased by [15%] compared to the previous quarter.\n\n"
        "Customer satisfaction metrics show positive trends across all regions..."
    ),
    paragraph_segmentation_mode="sat",  # Use SaT model for intelligent paragraph segmentation
    sat_model_id="sat-3l-sm",  # Specify which SaT model to use
)

# Create a document with predefined paragraphs, e.g. when you use a custom
# paragraph segmentation tool
document_from_paragraphs = Document(
    paragraphs=[
        Paragraph(raw_text="This is the first paragraph."),
        Paragraph(raw_text="This is the second paragraph with more content."),
        Paragraph(raw_text="Final paragraph concluding the document."),
        # ...
    ]
)

# Create document with images

# Path is adapted for doc tests
current_file = Path(__file__).resolve()
root_path = current_file.parents[4]
image_path = root_path / "tests" / "images" / "invoices" / "invoice.png"

# Create a document with only images (no text)
image_document = Document(
    images=[
        create_image(image_path),  # contextgem.Image instance
        # ...
    ]
)

# Create a document with both text and images
mixed_document = Document(
    raw_text="This document contains both text and visual elements.",
    images=[
        create_image(image_path),  # contextgem.Image instance
        # ...
    ],
)

The Document class accepts the following parameters:

Parameter	Type	Default Value	Description
`raw_text`	`str \| None`	`None`	The main text of the document as a single string.
`paragraphs`	`list[Paragraph]`	`[]`	List of `Paragraph` instances in consecutive order as they appear in the document. Normally auto-populated from `raw_text`. See 🧩 Providing Additional Context for Paragraphs for providing your own paragraphs with additional context.
`images`	`list[Image]`	`[]`	List of `Image` instances attached to or representing the document. Used for visual content analysis.
`aspects`	`list[Aspect]`	`[]`	List of `Aspect` instances associated with the document for focused analysis. Must have unique names and descriptions. See Aspect Extraction for more details.
`concepts`	`list[_Concept]`	`[]`	List of `_Concept` instances associated with the document for information extraction. Must have unique names and descriptions. See supported concept types in Supported Concepts.
`paragraph_segmentation_mode`	`Literal["newlines", "sat"]`	`"newlines"`	Mode for paragraph segmentation. `"newlines"` splits on newline characters, `"sat"` uses a SaT (Segment Any Text) model for intelligent segmentation.
`sat_model_id`	`SaTModelId`	`"sat-3l-sm"`	SaT model ID for paragraph/sentence segmentation or a local path to a folder with a SaT model. See wtpsplit models for available options. A model given by its ID is downloaded from Hugging Face (`huggingface.co`, `hf.co` and their subdomains) when it is first used, and cached. See 🔌 Using SaT Models Without Network Access for using a model where Hugging Face cannot be reached.
`pre_segment_sentences`	`bool`	`False`	Whether to pre-segment sentences during Document initialization. When `False`, sentence segmentation is deferred until sentences are actually needed, improving initialization performance.
`custom_data`	`dict`	`{}`	Optional. Dictionary for storing any additional data that you want to associate with the document. This data must be JSON-serializable. It is not used for extraction.

🎯 Adding Aspects and Concepts for Extraction#

Before extracting information from a document with a model, you must define and add aspects and concepts to your document instance.

Aspects define the text segments (sections, topics, themes) to be extracted from the document. They can be combined with concepts for comprehensive analysis.

Concepts define specific data points to be extracted or inferred from the document content: entities, insights, structured objects, classifications, numerical calculations, dates, ratings, and assessments.

For detailed guidance on creating and configuring these components, see:

Aspect Extraction - Complete guide to defining and using aspects
Supported Concepts - All available concept types and how to use them

🧩 Providing Additional Context for Paragraphs#

When you create the paragraphs yourself, e.g. from the structured output of a document conversion tool, you can describe each paragraph in its additional_context: its structural role, formatting, or position in the document. This lets the model take the structure of the document into account, while the text of the paragraphs, which is returned in references, stays clean:

Paragraphs with additional context#

from contextgem import Paragraph


# Create a paragraph with raw text content
contract_paragraph = Paragraph(
    raw_text=(
        "This agreement is effective as of January 1, 2025. "
        "All parties must comply with the terms outlined herein. "
        "Failure to adhere to these terms may result in termination of the agreement."
    )
)

# Create paragraphs with additional context, e.g. the structure or formatting
# details provided by your document conversion tool. The additional context is
# passed to the model together with the text of the paragraph.
heading_paragraph = Paragraph(
    raw_text="Term and Termination",
    additional_context="Heading, level 2",
)
table_cell_paragraph = Paragraph(
    raw_text="Net 30 days",
    additional_context="Table: Payment terms, row 2, column 3",
)

Pass such paragraphs to the document via paragraphs. The additional context:

should be a single-line string: it is cleaned before it is sent to a model, as the text of the paragraph is, i.e. control characters are removed, and whitespace, line breaks included, is collapsed to single spaces. A warning is logged if it has line breaks, as they are not kept. Additional context of which nothing is left is not sent;
is passed to the LLM together with the paragraph text in the prompts that list paragraphs, i.e. when aspects are extracted, or when concepts are extracted with references (add_references=True). It is not used when concepts are extracted from the document text as a whole;
is sent to a decision model together with the paragraph text when aspects are extracted, also with each sentence of the paragraph that the model is asked about, for an aspect with reference_depth="sentences" (see 📄 Extracting Aspects);
is inherited by the sentences segmented from the paragraph, and is passed to the LLM with each sentence when sentence-level references are used (reference_depth="sentences");
is preserved when the document is serialized.

📍 Locating Paragraphs and Sentences#

Extraction results reference the document’s own Paragraph and Sentence objects (see e.g. Aspect Extraction for details on references): a paragraph in reference_paragraphs of an aspect, of a sub-aspect or of an extracted item is the same object as the paragraph in document.paragraphs, and a sentence in reference_sentences is the same object as the sentence in the sentences of its paragraph. This holds after each extraction method, and in a document restored from its serialized form (see Serializing objects and results). What you change in such an object, e.g. its custom_data, is therefore changed in the document, and the other way round.

To find the position of such an object in the document, use get_paragraph_index() and get_sentence_index():

# 0-based position of a paragraph in document.paragraphs
para_index = document.get_paragraph_index(paragraph)

# (paragraph_index, sentence_index) position of a sentence,
# where sentence_index is 0-based within the paragraph's sentences
para_index, sent_index = document.get_sentence_index(sentence)

Lookups are keyed by each object’s unique ID rather than text equality, so paragraphs or sentences with identical text (e.g. duplicate clauses in a contract) resolve to their specific occurrences. Unique IDs are preserved by serialization, so lookups also work across to_dict()/from_dict() round-trips.

This makes the methods suitable for working with extraction references, e.g. citing where in the document extracted information was found, ordering references combined from multiple extracted items, or retrieving surrounding context:

# Human-readable citation for a reference paragraph
ref_para = concept.extracted_items[0].reference_paragraphs[0]
citation = f"Found in paragraph {document.get_paragraph_index(ref_para) + 1}"

# Order references combined from multiple extracted items
# (references within a single extracted item are already in document order)
combined_refs = [
    para
    for item in concept.extracted_items
    for para in item.reference_paragraphs
]
ordered_refs = sorted(combined_refs, key=document.get_paragraph_index)

# Surrounding context of a reference paragraph
ref_index = document.get_paragraph_index(ref_para)
context_window = document.paragraphs[max(0, ref_index - 2) : ref_index + 3]

🔌 Using SaT Models Without Network Access#

A SaT model splits the text of a document into sentences, and into paragraphs with paragraph_segmentation_mode="sat". Sentences are segmented when they are needed: when an aspect or a concept is extracted with references to sentences (reference_depth="sentences"), or when the document is created with pre_segment_sentences=True. A document whose paragraphs are split on newlines (the default) and that needs no sentences uses no SaT model.

A model given by its ID, e.g. the default "sat-3l-sm", is downloaded from Hugging Face the first time it is used, together with its tokenizer: several hundred megabytes in all. huggingface_hub keeps the files in its cache (~/.cache/huggingface, or the folder set in the HF_HOME environment variable), and reads them from there afterwards. Where Hugging Face cannot be reached when the application runs, e.g. in a container or on a server without internet access, prepare the model in one of these ways:

Fill the cache in advance, where Hugging Face can be reached, e.g. when the container image is built, and keep the cache folder with the application. If the application uses another model than the default one, add its sat_model_id to the Document in the command:
```
# Download the default model and its tokenizer into the cache folder
export HF_HOME=/opt/huggingface
uv run python -c "from contextgem import Document; Document(raw_text='Text.', pre_segment_sentences=True)"
```
When the application runs, set the same HF_HOME, and HF_HUB_OFFLINE=1, so that the files are read from the cache without asking Hugging Face whether they have changed. See the environment variables of huggingface_hub.
Keep the model in a folder of its own, and pass the path of the folder as sat_model_id. The folder holds the files of the model (config.json and model_optimized.onnx) and its tokenizer (tokenizer.json), which you can download with the hf command of huggingface_hub, installed with ContextGem:
```
uv run hf download segment-any-text/sat-3l-sm config.json model_optimized.onnx --local-dir models/sat-3l-sm
uv run hf download facebookAI/xlm-roberta-base tokenizer.json --local-dir models/sat-3l-sm
```
```
document = Document(raw_text=text, sat_model_id="models/sat-3l-sm")
```
A model in such a folder is loaded without network access, and without the cache. If the folder has no tokenizer.json, the tokenizer is downloaded from Hugging Face, or read from the cache, as for a model given by its ID.