Configuring LLM(s)

Configuring LLM(s)#

This guide explains how to configure DocumentLLM instances to process documents using various LLM providers. ContextGem uses LiteLLM under the hood, providing uniform access to a wide range of models. For more information on supported LLMs, see Supported LLMs.

🚀 Basic Configuration#

The minimum configuration for a cloud-based LLM requires the model parameter and an api_key:

Using a cloud-based LLM#

from contextgem import DocumentLLM


# Pattern for using any cloud LLM provider
llm = DocumentLLM(
    model="<provider>/<model_name>",
    api_key="<api_key>",
)

# Example - Using OpenAI LLM
llm_openai = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<api_key>",
    # see DocumentLLM API reference for all configuration options
)

# Example - Using Azure OpenAI LLM
llm_azure_openai = DocumentLLM(
    model="azure/gpt-6-luna",
    api_key="<api_key>",
    api_version="<api_version>",
    api_base="<api_base>",
    # see DocumentLLM API reference for all configuration options
)

For local models, usually you need to specify the api_base instead of the API key:

Using a local LLM#

from contextgem import DocumentLLM


local_llm = DocumentLLM(
    model="ollama_chat/<model_name>",
    api_base="http://localhost:11434",  # Default Ollama endpoint
)

# Example - Using Llama 3.3 LLM via Ollama
llm_llama = DocumentLLM(
    model="ollama_chat/llama3.3:70b",
    api_base="http://localhost:11434",
    # see DocumentLLM API reference for all configuration options
)

# Example - Using DeepSeek R1 reasoning model via Ollama
llm_deepseek = DocumentLLM(
    model="ollama_chat/deepseek-r1:32b",
    api_base="http://localhost:11434",
    # see DocumentLLM API reference for all configuration options
)

# Example - Using Mistral Small 3.2 via LM Studio (no API key is needed)
llm_mistral = DocumentLLM(
    model="lm_studio/mistralai/mistral-small-3.2",
    api_base="http://localhost:1234/v1",  # Default LM Studio endpoint
    # see DocumentLLM API reference for all configuration options
)

📝 Configuration Parameters#

The DocumentLLM class accepts the following parameters:

Parameter	Type	Default Value	Description
`model`	`str`	(Required)	Model identifier in format `<provider>/<model_name>`. See LiteLLM Providers for all supported providers.
`api_key`	`str \| None`	`None`	API key for authentication. Required for most cloud providers but not for local models.
`api_base`	`str \| None`	`None`	Base URL of the API endpoint. Required for local models and some cloud providers (e.g. Azure OpenAI).
`deployment_id`	`str \| None`	`None`	Deployment ID for the model. Primarily used with Azure OpenAI.
`api_version`	`str \| None`	`None`	API version. Primarily used with Azure OpenAI.
`role`	`str \| None`	`None`	Free-form role label of the LLM (up to 64 printable characters), used for routing in a model group. The same label can be explicitly assigned to extraction components (aspects and concepts) in the pipeline, using their `model_role` parameter. ContextGem then routes extraction tasks based on these assigned roles, matching components with the model of the same role, while the components without a role are routed to the model without a role. This allows you to structure your pipeline with different models for different tasks (e.g., using simpler models for basic extractions and more powerful models for complex reasoning). The role implies no capabilities of the model, and is ignored when the LLM is used on its own (not in a group), as a single LLM extracts all the aspects and concepts it is given. For more details, see 🏷️ Model Roles.
`system_message`	`str \| None`	`None`	If not provided (or set to None), ContextGem automatically sets a default system message optimized for extraction tasks, rendered based on the configured `output_language`. The default system message follows the output language: when `output_language` is changed after the LLM is created, the default system message is rendered for the new one, while a custom system message is kept as it is. Assigning `None` to `system_message` restores the default system message. This default system message template can be found here in the source code. Overriding this is typically only necessary for advanced use cases, such as custom priming or non-extraction tasks. For simple chat interactions, consider setting `system_message=''` to disable the default entirely (meaning no system message will be sent).
`max_tokens`	`int \| None`	`None`	Maximum output tokens allowed per request. For reasoning (CoT-capable) models, the reasoning tokens count towards this limit. If not set (default), no limit is sent with the requests, and the default of the model applies, which for most providers is the maximum output of the model. A value above the maximum output of the model, as known to `litellm`, raises `ValueError` before a request is sent to the model: in a chat, the fallback LLMs are checked before any request is sent, and in extraction, a fallback LLM is checked when it is tried. See 📏 Limiting the Output Tokens for how a response cut off at the limit is handled.
`reasoning_effort`	`str \| None`	`None`	Reasoning effort for reasoning (CoT-capable) models. Values: `"none"`, `"minimal"`, `"low"`, `"medium"`, `"high"`, `"xhigh"`, `"max"`. The parameter is sent only to the models detected as supporting reasoning. Support for specific values varies by model and provider: if `litellm` reports a value as not supported for the model, the request is sent without the parameter, with a warning logged, as for the sampling parameters (see 🎲 Setting the Sampling Parameters), while a value that is rejected otherwise, e.g. by the provider API, fails the request. `"none"` explicitly disables reasoning on models that support this value, which differs from the default `None` (parameter not sent, provider default applies).
`timeout`	`int`	`600`	Timeout in seconds for LLM API calls. Responses are not streamed, so the whole response, including the reasoning of the model, has to be received within the timeout. A request that times out is retried as a transient error (see 🔁 How Failed Requests Are Retried).
`num_retries_failed_request`	`int`	`3`	Number of retries when LLM request fails with a transient error (request timeout, rate limit, server or connection error). Retries are performed with exponential backoff and jitter. Non-transient errors (e.g. invalid request, failed authentication, exceeded context window) are not retried. See 🔁 How Failed Requests Are Retried.
`max_retries_failed_request`	`int`	`0`	LLM provider-specific retry count for failed requests, i.e. retries performed by the provider’s SDK, where supported. Applies to the first attempt of a request only.
`max_retries_invalid_data`	`int`	`3`	Number of retries when LLM request succeeds but returns invalid data (JSON parsing and validation fails). Each retry is a new request, i.e. the LLM is asked for the same data up to `1 + max_retries_invalid_data` times. A fallback LLM applies its own setting in the same way: it is asked once, and then again up to its `max_retries_invalid_data` times. See 🔁 How Failed Requests Are Retried.
`pricing_details`	`ModelPricing \| None`	`None`	`ModelPricing` object with pricing details for cost calculation: the prices of input and output tokens and, optionally, of the input tokens read from and written to the prompt cache of the provider.
`auto_pricing`	`bool`	`False`	Enable automatic cost calculation using `genai-prices` based on the configured `model`. Mutually exclusive with `pricing_details`. Not supported for local models (e.g., `ollama/`, `ollama_chat/`, `lm_studio/`).
`auto_pricing_refresh`	`bool`	`False`	When `auto_pricing` is enabled, allow `genai-prices` to auto-refresh its cached pricing data at runtime. Enable it if the pricing data of a recently released model is not found, as the pricing data bundled with `genai-prices` may not include such a model yet. The data is fetched from `raw.githubusercontent.com`.
`is_fallback`	`bool`	`False`	Indicates whether the LLM is a fallback model. Fallback LLMs are optionally assigned to the primary model instance and are used when the primary model fails.
`fallback_models`	`list[DocumentLLM \| DocumentDecisionModel]`	`[]`	Models to use as fallbacks if the current one fails. Fallback models are tried in the given order until one of them succeeds. Each fallback LLM must have `is_fallback=True` and the same role, output language, system message, tools, `tool_choice` and `parallel_tool_calls` as the primary LLM, and, if it has tools, the same support for tool calls (see Tools and Fallback LLMs). A fallback can also be a decision model, if it can extract everything that the LLM is to extract (see 🔄 Fallback Chains).
`output_language`	`str`	`"en"`	Language for output text. Values: `"en"` or `"adapt"` (adapts to document language). Setting value to `"adapt"` ensures that the text output (e.g. justifications, conclusions, explanations) is in the same language as the document. This is particularly useful when working with non-English documents. For example, if you’re extracting anomalies from a contract in Spanish, setting `output_language="adapt"` ensures that anomaly justifications are also in Spanish, making them immediately understandable by local end users reviewing the document. The parameter applies to the extraction prompts, and to the default system message. With a custom system message, it applies to the extraction prompts only, i.e. it has no effect in a chat, and a warning says so.
`temperature`	`float \| None`	`None`	Sampling temperature (0.0 or higher; the upper limit depends on the model) controlling response creativity. Lower values produce more predictable outputs, higher values generate more varied responses. If not set (default), the parameter is not sent, and the default of the model applies. See 🎲 Setting the Sampling Parameters for when to set it.
`top_p`	`float \| None`	`None`	Nucleus sampling value (0.0 to 1.0) controlling output focus/randomness. Lower values make output more deterministic, higher values produce more diverse outputs. If not set (default), the parameter is not sent. Some models (e.g. Anthropic’s Claude 4.x) reject requests that set both `temperature` and `top_p`; set only one of them when using those models.
`seed`	`int \| None`	`None`	Seed for random number generation to help produce more consistent outputs across multiple runs. When set to a specific integer value, the LLM will attempt to use this seed for sampling operations. However, deterministic output is still not guaranteed even with the same seed, as other factors may influence the model’s response. If not set (default), the parameter is not sent. Support depends on the LLM provider: a seed is mostly of use with local and open-weight models, while some providers (e.g. Anthropic) do not support it (see 🎲 Setting the Sampling Parameters).
`tools`	`Sequence[Callable] \| None`	`None`	Tools for chat via `DocumentLLM.chat(...)`/`.chat_async(...)`: functions decorated with `@register_tool`. The tool schemas sent to the LLM are generated from the names, docstrings and type hints of the functions. Handlers must return a non-empty string; for structured data, serialize it (e.g., with `json.dumps`) before returning. Ignored by extraction methods. For more details, see Chat with Tools.
`tool_choice`	`str \| dict \| None`	`None`	Tool choice control passed through to the provider during chat. Ignored by extraction methods.
`parallel_tool_calls`	`bool \| None`	`None`	Enable parallel tool calls during chat tool usage, if supported by the model/provider. Ignored by extraction methods.
`tool_max_rounds`	`int`	`10`	Safety limit on the number of tool-execution rounds per chat request to prevent infinite loops.
`async_limiter`	`AsyncLimiter`	`AsyncLimiter(10, 10)`	Relevant when concurrency is enabled for extraction tasks. Controls frequency of async LLM API requests for concurrent tasks. Defaults to allowing 10 acquisitions per 10-second period to prevent rate limit issues. Adjust it to the rate limits of your LLM provider account, e.g. lower it if you send long documents on a low usage tier. The rate applies to all the requests to the LLM that are sent concurrently, i.e. the extraction calls that run at the same time, e.g. for the concepts of different aspects, share it. Requests to a fallback model are controlled by the `async_limiter` of that fallback model. See aiolimiter documentation for AsyncLimiter configuration details. See Optimizing for Speed for an example of how to easily set up concurrency for extraction.

Warning

Auto-pricing accuracy

When using auto_pricing=True, cost estimates are approximate. These prices will not be 100% accurate. The price data cannot be exactly correct because model providers do not provide exact price information for their APIs in a format which can be reliably processed. See Pydantic’s genai-prices for more details.

💡 Advanced Configuration Examples#

🔄 Configuring Fallback LLMs#

You can set up one or more fallback LLMs that will be used if the primary LLM fails. Fallback LLMs are tried in the order they are listed in fallback_models, until one of them succeeds:

Configuring fallback LLMs#

from contextgem import DocumentLLM


# Primary LLM
primary_llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<your-openai-api-key>",
)

# First fallback LLM
fallback_llm_1 = DocumentLLM(
    model="anthropic/claude-haiku-4-5",
    api_key="<your-anthropic-api-key>",
    is_fallback=True,  # Must have the same role as the primary LLM (here: no role)
)

# Second fallback LLM (optional)
fallback_llm_2 = DocumentLLM(
    model="gemini/gemini-3.8-flash",
    api_key="<your-gemini-api-key>",
    is_fallback=True,  # Must have the same role as the primary LLM (here: no role)
)

# Assign fallback LLMs to primary, in the order they should be tried
primary_llm.fallback_models = [fallback_llm_1, fallback_llm_2]

# Then use the primary LLM as usual
# document = primary_llm.extract_all(document)

Each fallback LLM must be created with is_fallback=True, have the same role, output_language and system_message as the primary LLM, and cannot have fallback models of its own. An LLM created without system_message has the default system message, so if you set system_message for the primary LLM, e.g. to an empty string to disable it for a chat, set the same value for each fallback LLM. Each fallback LLM must also have the same tools as the primary LLM, in any order, the same tool_choice and parallel_tool_calls, and the same support for tool calls (see Tools and Fallback LLMs). Finally, a fallback LLM must differ from the primary LLM, and from the other fallback LLMs, in at least one configuration parameter, e.g. the model, the API key or the API base, as an exact copy would only repeat the same request. This is checked when the fallback LLMs are set, i.e. when the primary LLM is created with them or when fallback_models is assigned. If the primary LLM is used to extract concepts from images, or is passed images in a chat, each of its fallback LLMs must support vision as well, which is checked before any requests are sent. In a chat, the request parameters of the fallback LLMs are checked before any requests are sent as well, e.g. that a fallback LLM with parallel_tool_calls set supports parallel tool calls. Usage and cost are tracked separately for the primary LLM and for each of its fallback models.

The next fallback model is tried when the current LLM fails for good, i.e. when its request fails with a non-transient error, when the retries of its request are exhausted, when the prompt exceeds its context window, when it keeps returning invalid data during extraction, when its response is cut off at the max output tokens during extraction, or when it returns an empty response in a chat. See 🔁 How Failed Requests Are Retried for details.

To change the system_message of an LLM that already has fallback LLMs, change it for the fallback LLMs first, and then for the primary LLM.

A fallback of an LLM can also be a decision model, and an LLM can be a fallback of a decision model. A decision model extracts only aspects and the concepts whose answers come from a closed set, so an LLM can fall back to it only if it can extract everything that the LLM is to extract: this is checked when an extraction method is called, before any requests are sent. The requirements above for the output language, system message and tools apply between two LLMs only. In a chat, a fallback decision model is skipped. See 🔄 Fallback Chains.

Note

Each model applies its own retry settings and timeout before the next fallback model is tried, so the more fallback models you configure, the longer a request can take to fail when all of the models are unavailable.

🔁 How Failed Requests Are Retried#

Retries are configured per model, which also applies to each fallback model, and depend on why the request failed:

Transient API errors (request timeout, rate limit, server or connection error) are retried up to num_retries_failed_request times. The delay between the retries grows exponentially (about 1, 2, 4, 8 seconds, capped at 10 seconds) and is randomly shortened by up to a half (jitter), so that concurrent requests that failed together are not all retried at the same moment.
Non-transient API errors (e.g. invalid request, failed authentication, exceeded context window, content policy violation) are not retried, as the same request would fail again in the same way. A prompt that is found to exceed the context window of the model before the request is sent is handled in the same way: the request is not sent.
Invalid data returned by the LLM during extraction (the response cannot be parsed or validated, e.g. it has an ID that the prompt does not show or a value of another type than the concept has, it leaves out an aspect or a concept, which it is to list also without items, or it has more than one item for a concept with singular_occurrence=True) is requested again up to max_retries_invalid_data times, without a delay. A fallback LLM is asked again in the same way: after it is asked for the first time, up to its own max_retries_invalid_data times, i.e. as many times in total as a primary LLM with the same setting. A fallback LLM with max_retries_invalid_data=0 is asked once. When the content of a call is sent to the LLM in parts, i.e. with max_paragraphs_to_analyze_per_call or max_images_to_analyze_per_call set, the LLM is asked again only for the part that it returned invalid data for, and the call goes on with the parts after it: the parts that were answered with valid data are not sent again. The retries are those of the call, not of each part. A fallback LLM extracts from all the parts.
A response cut off at the max output tokens during extraction is not requested again, as the same request would be cut off in the same way (see 📏 Limiting the Output Tokens).

When a request to a model fails for good, it is not sent to that model again: the next fallback model is tried, if configured. If there are no more fallback models to try, ModelAPIError is raised for a failed API request, for a response cut off at the max output tokens during extraction, or for an empty response in a chat, or ModelExtractionError for invalid data. Both exceptions can be imported from contextgem, and except ContextGemError catches either of them (ContextGemError). The error is the one of the last model that was tried: the retry_count of ModelExtractionError is the number of times that this model was asked again for invalid data, i.e. it does not count the first time the model was asked. During extraction, a warning is logged instead if raise_exception_on_extraction_error=False is passed to the extraction method, with the same number of retries in its text. A prompt that exceeds the context window of the last LLM to try raises ValueError.

With use_concurrency=True, the aspects or concepts are extracted in several calls at the same time. When ModelAPIError or ModelExtractionError is to be raised for one of the calls, the other calls are stopped first: their requests in flight are cancelled, and no requests are sent afterwards. The aspects and concepts extracted by then keep their items, and the requests completed by then are recorded in the usage.

Note

max_retries_failed_request sets the number of retries performed by the LLM provider’s SDK, where supported (e.g. for OpenAI), in addition to the retries described above. It applies to the first attempt of a request only.

📏 Limiting the Output Tokens#

By default, the output tokens of a response are not limited by ContextGem: max_tokens is not set, no limit is sent with the requests, and the default of the model applies, which for most providers is the maximum output of the model. Set max_tokens to limit the output of each request, e.g. to cap the cost of a single request:

from contextgem import DocumentLLM

llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<your-openai-api-key>",
    max_tokens=16000,  # for reasoning models, includes the reasoning tokens
)

The same parameter applies to all models. For reasoning (CoT-capable) models, the reasoning tokens count towards the limit, so a limit that is too low for the reasoning effort of the model can leave no tokens for the response itself.

A response that reaches the limit is cut off by the LLM provider, whether the limit is set with max_tokens or is the one of the model:

During extraction, a cut-off response cannot be used, and is not requested again, as the same request would be cut off in the same way. The next fallback model is tried, if configured. If there are no more fallback models to try, ModelAPIError is raised, or a warning is logged if raise_exception_on_extraction_error=False is passed to the extraction method. To get a complete response, increase max_tokens or leave it unset, lower reasoning_effort, or get less output per request by setting max_items_per_call or max_paragraphs_to_analyze_per_call (see Dealing with Long Documents).
In a chat, a cut-off response is returned as it is, with a warning logged, as the limit can be set on purpose to get a short answer. A cut-off response without a text, e.g. when all the tokens are used up by the reasoning of the model, is handled as an empty response.

Note

Some providers (e.g. Anthropic) require a limit in each request. If max_tokens is not set, litellm sends the maximum output of the model for such a provider. For a model that litellm has no data on, e.g. a model released after the pinned litellm version, it sends a low default limit instead, so set max_tokens explicitly for such a model.

🎲 Setting the Sampling Parameters#

By default, temperature and top_p are not set: they are not sent with the requests, and the defaults of the model apply. If you set them, they are sent to any model:

from contextgem import DocumentLLM

llm = DocumentLLM(
    model="ollama_chat/llama3.3:70b",
    api_base="http://localhost:11434",
    temperature=0.2,  # more consistent output for structured extraction
)

A low temperature (e.g. 0.0 to 0.3) usually makes the output of structured extraction more consistent. This matters most for small and local models, whose default temperature is often set for conversational use (see Troubleshooting Issues with Small Models).

To make the output of such a model more reproducible across runs, you can also set seed. As with the sampling parameters, it is sent only if set, and to any model.

Whether a value is supported depends on the model, which is left to litellm and the LLM provider. Some reasoning (CoT-capable) models accept only their default sampling parameters, e.g. OpenAI’s GPT-5 and GPT-6 models. If litellm reports a parameter as not supported for the model, the request is sent without it, and a warning is logged for each such request, so it is better to leave the parameter unset for such a model. A value that is rejected only by the API of the provider fails the request as any other invalid request.

💰 Setting Up Cost Tracking#

You can configure pricing parameters to track costs:

Setting up LLM cost tracking#

from contextgem import DocumentLLM, ModelPricing


# Option 1: Set up a LLM with pricing details
llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<your-openai-api-key>",
    pricing_details=ModelPricing(  # example values: set the prices of your provider
        input_per_1m_tokens=1.00,  # Cost per 1M input tokens
        output_per_1m_tokens=2.00,  # Cost per 1M output tokens
        # Optional: cost per 1M input tokens read from the provider's prompt cache
        cache_read_per_1m_tokens=0.10,
    ),
)

# Option 2: Set up a LLM with auto-pricing
llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<your-openai-api-key>",
    auto_pricing=True,
)

# Perform some extraction tasks

# Later, you can check the cost
cost_info = llm.get_cost()

💾 How Cached Tokens Are Priced#

Some LLM providers cache the beginning of a prompt and reuse it across requests (prompt caching), in which case they report how many input tokens of a request were read from the cache, and some of them also how many were written to it. Tokens read from the cache are usually priced lower than regular input tokens, while tokens written to the cache may be priced higher. ContextGem does not configure prompt caching, but takes such tokens into account in the calculated costs whenever the provider reports them:

With auto_pricing=True, they are priced at the cache prices of the model, if genai-prices has such prices, and as regular input tokens otherwise.
With pricing_details, they are priced at the optional cache_read_per_1m_tokens and cache_write_per_1m_tokens of ModelPricing. If a cache price is not set, the corresponding tokens are priced as regular input tokens, i.e. at input_per_1m_tokens.

Note

Tokens written to the cache are priced at the standard cache write price of the model. A longer cache lifetime that a provider may offer at a higher price (e.g. 1-hour cache writes of Anthropic models) is not priced separately. Such a lifetime has to be requested explicitly, which ContextGem does not do.

🧠 Setting the Reasoning Effort#

For reasoning (CoT-capable) models (such as OpenAI’s GPT-5 and GPT-6 models), you can set the reasoning effort. The reasoning tokens count towards max_tokens, if set (see 📏 Limiting the Output Tokens):

Setting the reasoning effort#

from contextgem import DocumentLLM


llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<your-openai-api-key>",
    reasoning_effort="medium",  # Specific to reasoning (CoT-capable) models
    max_tokens=16000,  # Optional; the reasoning tokens count towards this limit
)

⚙️ Explicit Capability Declaration#

Model vision capabilities are automatically detected using litellm.supports_vision(). If this function does not correctly identify your model’s capabilities, an error will be raised when the model is to be sent images (e.g. when extracting concepts with the "image" or "multimodal" modality), and you can explicitly declare the capability by setting _supports_vision=True on the LLM instance:

from contextgem import DocumentLLM

# Example: Explicitly declare vision capability
llm = DocumentLLM(
    model="some_provider/custom_vision_model",
    api_base="http://localhost:3456/v1",
)
# Declare capability if automatic detection fails
# (an error is raised when the model is to be sent images)
llm._supports_vision = True

Warning

Explicit capability declarations should only be used when automatic capability detection fails. Incorrectly setting this flag may lead to unexpected behavior or API errors.

🤖🤖 Model Groups#

For complex document processing, you can organize multiple LLMs with different roles into a group:

Using model group#

from contextgem import DocumentLLM, DocumentModelGroup


# Create LLMs with different roles

# LLM without a role: extracts the aspects/concepts that have no `model_role`
extractor = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<your-openai-api-key>",
    output_language="adapt",
)

# LLM with a role: extracts the aspects/concepts that have `model_role="reasoner"`
reasoner = DocumentLLM(
    model="openai/gpt-6.1-sol",
    api_key="<your-openai-api-key>",
    role="reasoner",  # a label of your choice
    reasoning_effort="high",
    output_language="adapt",
)

# Create a group
model_group = DocumentModelGroup(
    models=[extractor, reasoner],
    output_language="adapt",  # All LLMs in the group must share the same output language setting
)

# Then use the group as usual
# document = model_group.extract_all(document)

A group has at least two models. Each model in the group must have a unique role, and only one model in the group can have no role. An aspect/concept is routed to the model with the role equal to the model_role of the aspect/concept, and an aspect/concept without model_role is routed to the model without role. If an aspect/concept cannot be routed to any model in the group, an error is raised. For more details, see 🏷️ Model Roles.

See a practical example of using a model group in 🔄 Using a Multi-LLM Pipeline to Extract Data from Several Documents.

A model group can also include decision models, to which aspects/concepts are routed by their roles in the same way. For example, a decision model can answer the yes/no questions and the classifications, while an LLM extracts the rest. The output_language of a group applies to its LLMs: a decision model has no output language. See 🤖🤖 Using a Decision Model in a Model Group.

📊 Accessing Usage and Cost Statistics#

You can track input/output token usage and costs:

Tracking usage and cost#

from contextgem import DocumentLLM


llm = DocumentLLM(
    model="anthropic/claude-haiku-4-5",
    api_key="<your-anthropic-api-key>",
    auto_pricing=True,  # or set `pricing_details=ModelPricing(...)` manually
)

# Perform some extraction tasks

# Get usage statistics
usage_info = llm.get_usage()
usage = usage_info[0].usage  # usage of the primary LLM
print(f"Input tokens: {usage.input}")
print(f"Output tokens: {usage.output}")
# Input tokens read from / written to the provider's prompt cache (part of `input`)
print(f"Cache read tokens: {usage.cache_read}")
print(f"Cache write tokens: {usage.cache_write}")

# Get cost statistics
cost_info = llm.get_cost()
cost = cost_info[0].cost  # cost of the primary LLM
print(f"Input cost: {cost.input}")  # includes the cost of the cache tokens
print(f"Output cost: {cost.output}")
print(f"Total cost: {cost.total}")

# Reset usage and cost statistics
llm.reset_usage_and_cost()

# The same methods are available for model groups, with optional filtering by model role
# usage_info = model_group.get_usage(model_role="reasoner")
# cost_info = model_group.get_cost(model_role="reasoner")
# model_group.reset_usage_and_cost(model_role="reasoner")

Token usage is tracked for every LLM, while costs are calculated only if pricing is configured for the LLM, i.e. pricing_details or auto_pricing is set.

The input token count includes the tokens that the LLM provider read from, or wrote to, its prompt cache. Such tokens are also counted separately, as cache_read and cache_write, which are zero if the provider does not report them. Accordingly, the input cost includes the cost of such tokens (see 💾 How Cached Tokens Are Priced).

The usage statistics include not only token counts but also detailed information about each individual call made to the LLM. You can access the call history, including prompts, responses, and timestamps:

Accessing detailed usage information#

from contextgem import DocumentLLM


llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<your-openai-api-key>",
)

# Perform some extraction tasks

usage_info = llm.get_usage()

# Access the first usage container in the list (for the primary LLM)
model_usage = usage_info[0]

# Get detailed call information
for call in model_usage.usage.calls:
    print(f"Prompt: {call.prompt}")
    print(f"Response: {call.response}")  # original, unprocessed response
    print(f"Sent at: {call.timestamp_sent}")
    print(f"Received at: {call.timestamp_received}")

Configuring LLM(s)

Contents

Configuring LLM(s)#

🚀 Basic Configuration#

📝 Configuration Parameters#

💡 Advanced Configuration Examples#

🔄 Configuring Fallback LLMs#

🔁 How Failed Requests Are Retried#

📏 Limiting the Output Tokens#

🎲 Setting the Sampling Parameters#

💰 Setting Up Cost Tracking#

💾 How Cached Tokens Are Priced#

🧠 Setting the Reasoning Effort#

⚙️ Explicit Capability Declaration#

🤖🤖 Model Groups#

📊 Accessing Usage and Cost Statistics#