Optimizing for Speed

For large-scale processing or time-sensitive applications, optimize your pipeline for speed:

• 🚀 Enable and Configure Concurrency: Process multiple extractions concurrently. Adjust the async limiter to adapt to your model API setup.

• 📦 Use Smaller Models: Select smaller/distilled LLMs that perform faster. (See Choosing the Right LLM(s) for guidance on choosing the right model.)

• ⚖️ Consider a Decision Model for Closed Questions: Boolean, label and rating concepts, as well as aspects, can be extracted with a decision model, which generates no text. For aspects, it sends a request for each paragraph: pass use_concurrency=True, set max_items_per_call to the number of the aspects or concepts, so that each text is sent once, and set the async_limiter of the model to the rate limits of your provider account. A sentence-level reference depth adds a request for each sentence of the paragraphs that belong to the aspect. (See 📄 Extracting Aspects.)

• 🔄 Use a Fallback Model: Configure a fallback model to retry extractions that failed due to rate limits.

• ⚙️ Use Default Parameters: Without concurrency, all the extractions are processed in as few model calls as possible. With concurrency enabled, each aspect/concept is processed in a call of its own by default.

• 📉 Enable Justifications Only When Necessary: Do not use justifications for simple aspects or concepts. This will reduce the number of tokens generated.

• ⚠️ Use Sentence-Level Reference Depth Sparingly: Only use sentence-level reference depth for aspects or concepts when absolutely necessary, as it requires loading a SaT model and running sentence segmentation on text, which can be slow for long documents.

Example of optimizing extraction for speed

# Example of optimizing extraction for speed

import os

from aiolimiter import AsyncLimiter

from contextgem import Document, DocumentLLM


# Define document
document = Document(
    raw_text="document_text",
    # aspects=[Aspect(...), ...],
    # concepts=[Concept(...), ...],
)

# Define LLM with a fallback model
llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key=os.environ.get("CONTEXTGEM_OPENAI_API_KEY"),
    fallback_models=[
        DocumentLLM(
            model="openai/gpt-5.6-luna",
            api_key=os.environ.get("CONTEXTGEM_OPENAI_API_KEY"),
            is_fallback=True,
        ),
    ],
)
llm.async_limiter = AsyncLimiter(
    10, 5
)  # e.g. 10 acquisitions per 5-second period; adjust to your LLM API setup
llm.fallback_models[0].async_limiter = AsyncLimiter(
    20, 5
)  # e.g. 20 acquisitions per 5-second period; adjust to your LLM API setup


# Use the LLM for extraction with concurrency enabled
llm.extract_all(document, use_concurrency=True)

# ... use the extracted data ...