LLMs

Contents

LLMs#

Module for handling processing logic using LLMs.

This module provides classes and utilities for interacting with LLMs in document processing workflows. It includes functionality for managing LLM configurations, handling API calls, processing text and image inputs, tracking token usage and costs, and managing rate limits for LLM requests.

The module supports various LLM providers through the litellm library, enabling both text-only and multimodal (vision) capabilities. It implements efficient asynchronous processing patterns and provides detailed usage statistics for monitoring and cost management.

class contextgem.public.llms.DocumentLLM(**data)[source]#

Bases: _DocumentLLM

Handles processing documents with a specific LLM.

This class serves as an abstraction for interacting with an LLM. It provides functionality for querying the LLM with text or image inputs, and manages prompt preparation and token usage tracking. An LLM can be used on its own or, with a role assigned, as a part of a model group.

Variables:

  • model (str) – Model identifier in format {model_provider}/{model_name}. See https://docs.litellm.ai/docs/providers for supported providers.

  • deployment_id (str | None) – Deployment ID for the LLM. Primarily used with Azure OpenAI.

  • api_key (str | None) – API key for LLM authentication. Not required for local models (e.g., Ollama). Not included in the serialized LLM data, i.e. to be set again after deserialization. Not shown when the LLM is printed, nor in the text of a validation error of the LLM, which names the invalid parameter and the reason without showing the values passed. The values are still returned by the errors() and json() methods of the error, unless include_input=False is passed, with '**********' in place of a key that is passed as a value, e.g. a rejected key or the key among the parameters that an LLM is created with, and in place of the value of any parameter that an LLM does not have, as it may be the key passed under a misspelled name. An LLM that is among the values is returned by errors() as it is, i.e. with the key as its attribute, and is written out by json() without its key. A key that is passed as the value of another parameter, e.g. of model, is returned as any other value.

  • api_base (str | None) – Base URL of the API endpoint. Not included in the serialized LLM data, i.e. to be set again after deserialization.

  • api_version (str | None) – API version. Primarily used with Azure OpenAI.

  • role (str | None) – Free-form role label of the LLM (up to 64 printable characters), used for routing in a DocumentModelGroup: the LLM extracts the aspects/concepts with the same model_role. Defaults to None, in which case the LLM extracts the aspects/concepts without model_role. Ignored when the LLM is used on its own, as it then extracts all aspects/concepts.

  • system_message (str | None) – Preparatory system-level message to set context for LLM responses. Defaults to None, in which case the default system message for extraction is set, rendered for the output language, also when the output language is changed later. Assigning None restores the default system message, while an empty string disables the system message, i.e. none is sent.

  • temperature (float | None) – Sampling temperature (0.0 or higher; the upper limit depends on the model) controlling response creativity. Lower values produce more predictable outputs, higher values generate more varied responses. Defaults to None, in which case the parameter is not sent, and the default of the model applies. A value is sent to any model; if litellm reports it as not supported for the model, it is dropped from the request, with a warning logged.

  • max_tokens (int | None) – Maximum output tokens allowed per request. For reasoning (CoT-capable) models, the reasoning tokens count towards this limit. Defaults to None, in which case no limit is sent with the requests, and the default of the model applies (for most providers, the maximum output of the model). A response cut off at the limit fails the extraction, and is returned with a warning logged in a chat.

  • reasoning_effort (ReasoningEffort | None) – The effort level for the LLM to reason about the input. Can be set to "none", "minimal", "low", "medium", "high", "xhigh", or "max". The parameter is sent only to the models detected as supporting reasoning. Support for specific values varies by model and provider: if litellm reports a value as not supported for the model, it is dropped from the request, with a warning logged, while a value that is rejected otherwise, e.g. by the provider API, fails the request. Note that "none" explicitly disables reasoning on models that support this value, which differs from the default None (parameter not sent, provider default applies). Relevant for reasoning (CoT-capable) models. Defaults to None.

  • top_p (float | None) – Nucleus sampling value (0.0 to 1.0) controlling output focus/randomness. Lower values make output more deterministic, higher values produce more diverse outputs. Defaults to None (the parameter is not sent). Some providers (e.g. Claude 4.x) reject requests that set both temperature and top_p.

  • num_retries_failed_request (int) – Number of retries when LLM request fails with a transient error (request timeout, rate limit, server or connection error). Retries are performed with exponential backoff and jitter. Non-transient errors (e.g. invalid request, failed authentication, exceeded context window) are not retried. Defaults to 3.

  • max_retries_failed_request (int) – LLM provider-specific retry count for failed requests, i.e. retries performed by the provider’s SDK, where supported. Applies to the first attempt of a request only. Defaults to 0.

  • max_retries_invalid_data (int) – Number of retries when LLM request succeeds but LLM returns invalid data. Each retry is a new request, i.e. the LLM is asked for the same data up to 1 + max_retries_invalid_data times. A fallback LLM applies its own setting in the same way: it is asked once, and then again up to its max_retries_invalid_data times. If the content of a call is sent in parts, a retry is a request for the part that the invalid data was returned for, i.e. the parts answered with valid data are not sent again. Defaults to 3.

  • timeout (int) – Timeout in seconds for LLM API calls. As responses are not streamed, the whole response has to be received within the timeout. Defaults to 600 seconds.

  • pricing_details (ModelPricing | None) – ModelPricing object with pricing details for cost calculation. Defaults to None.

  • auto_pricing (bool) – Enable automatic LLM cost calculation using genai-prices. Cannot be set together with pricing_details, and is not supported for local models (e.g. ollama/, ollama_chat/, lm_studio/): both raise ValueError. Defaults to False.

  • auto_pricing_refresh (bool) – Whether genai-prices should auto-refresh its cached pricing data, e.g. to price a recently released model that the bundled pricing data does not include yet. Defaults to False.

  • is_fallback (bool) – Indicates whether the LLM is a fallback model. Defaults to False.

  • fallback_models (list[DocumentLLM | DocumentDecisionModel]) – DocumentLLMs to use as fallbacks if the current one fails, tried in the given order until one of them succeeds. Each must have is_fallback=True and the same role, output language, system message, tools, tool_choice and parallel_tool_calls as the current LLM, must differ from the current LLM and from each other in at least one config param, and must support vision if the current LLM is used to extract concepts from images. A decision model (DocumentDecisionModel) with is_fallback=True and the same role can also be a fallback: it replaces the LLM only in the extraction, provided that a decision model can extract everything the LLM is to extract, and is skipped in a chat. Each fallback applies its own retry settings: a fallback LLM that returns invalid data during extraction is asked again up to its max_retries_invalid_data times, as the current LLM is, and a fallback decision model is asked once. Defaults to an empty list.

  • output_language (LanguageRequirement) – Language for produced output text (justifications, explanations). Can be “en” (English) or “adapt” (adapts to document/image language). Defaults to “en”. Applies to the extraction prompts, and to the default system message: with a custom system message, it applies to the extraction prompts only.

  • async_limiter (AsyncLimiter) – Controls frequency of async LLM API requests for concurrent tasks. Defaults to allowing 10 acquisitions per 10-second period to prevent rate limit issues. The rate applies to all the requests to the LLM that are sent concurrently, i.e. the extraction calls that run at the same time share it. Requests to a fallback model are controlled by the limiter of that fallback model. See mjpieters/aiolimiter for configuration details.

  • seed (int | None) – Seed for random number generation to help produce more consistent outputs across multiple runs. When set to a specific integer value, the LLM will attempt to use this seed for sampling operations. However, deterministic output is still not guaranteed even with the same seed, as other factors may influence the model’s response. Support depends on the model provider: a seed is mostly of use with local and open-weight models, while some providers (e.g. Anthropic) do not support it. If litellm reports the seed as not supported for the model, it is dropped from the request, with a warning logged. Defaults to None (the parameter is not sent).

Parameters:

Note:

  • Model groups

    Refer to the DocumentModelGroup class for more information on constructing model groups, which are a collection of models with unique roles, used for complex document processing tasks.

  • Model role

    The role of an LLM is a free-form label used to route aspects/concepts to the models of a DocumentModelGroup. For example, if an aspect/concept is assigned model_role="complex", it means that the aspect/concept is extracted from the document using the model in the group with the role set to “complex”, while the aspects/concepts without model_role are extracted using the model without role. This helps to channel different tasks to different models, ensuring that the task is handled by the most appropriate model. Usually, domain expertise is required to determine the most appropriate model for a specific aspect/concept. A role is just a label and implies no capabilities of the model. When an LLM is used on its own, i.e. not in a group, you can skip the role assignment completely: the LLM extracts all aspects/concepts, regardless of their model_role.

  • Explicit capability declaration

    Model vision capabilities are automatically detected using litellm.supports_vision(). If this function does not correctly identify your model’s capabilities, an error will be raised when the model is to be sent images, and you can explicitly declare the capability by setting _supports_vision=True on the LLM instance.

Example:
LLM definition#
from contextgem import DocumentLLM


# Create a single LLM, which extracts all aspects and concepts it is given
# (a role is needed only for routing in a model group)
llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="your-api-key",  # Replace with your actual API key
    auto_pricing=True,  # optional, to calculate the costs with `genai-prices`
)

# Create a fallback LLM in case the primary model fails
fallback_llm = DocumentLLM(
    model="anthropic/claude-sonnet-5-5",
    api_key="your-anthropic-api-key",  # Replace with your actual API key
    is_fallback=True,  # must have the same role as the primary LLM (here: no role)
    auto_pricing=True,  # optional, to calculate the costs with `genai-prices`
)
# Assign the fallback LLM to the primary LLM
# (several fallback models can be assigned, to be tried in the given order)
llm.fallback_models = [fallback_llm]

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

_eq_deserialized_llm_config(other)#

Custom config equality method to compare this _DocumentLLM with a deserialized instance.

Compares the __dict__ of both instances and performs specific checks for certain attributes that require special handling.

Note that, by default, the reconstructed deserialized _DocumentLLM will be only partially equal (==) to the original one, as the api credentials are redacted, and the async limiter and the async lock are not serialized and point to different objects in memory post-initialization. The attached prompt templates are not serialized either: the deserialized _DocumentLLM has the built-in ones, which the LLMs share, i.e. a custom prompt has to be set again. Also, usage and cost are reset by default pre-serialization.

Parameters:

other (contextgem.internal.base.llms._DocumentLLM) – Another _DocumentLLM instance to compare with

Returns:

True if the instances are equal, False otherwise

Return type:

bool

_update_default_prompt(prompt_path, prompt_type)#

For advanced users only!

Update the default Jinja2 prompt template for the LLM.

This method allows you to replace the built-in prompt templates with custom ones for specific extraction types. The framework uses these templates to guide the LLM in extracting structured information from documents.

The custom prompt must be a valid Jinja2 template and include all the necessary variables that are present in the default prompt. Otherwise, the extraction may fail. Default prompts are located under contextgem/internal/prompts/

IMPORTANT NOTES:

The default prompts are complex and specifically designed for various steps of LLM extraction with the framework. Such prompts include the necessary instructions, template variables, nested structures and loops, etc.

Only use custom prompts if you MUST have a deeper customization and adaptation of the default prompts to your specific use case. Otherwise, the default prompts should be sufficient for most use cases.

Use at your own risk!

Parameters:

  • prompt_path (str | pathlib.Path) – Path to the Jinja2 template file (.j2 extension required)

  • prompt_type (typing.Literal['aspect', 'concept']) – Type of prompt to update (“aspect” or “concept”)

Returns:

None

Return type:

None

property async_limiter: AsyncLimiter#

Gets the async rate limiter for this model.

Returns:

The AsyncLimiter instance controlling request rate limits.

Return type:

AsyncLimiter

chat(prompt, *, images=None, chat_session=None)#

Synchronously sends a prompt to the LLM and gets a response. For models supporting vision, attach images to the prompt if needed.

This method allows direct interaction with the LLM by submitting your own prompt.

Parameters:

  • prompt (str) – The input prompt to send to the LLM

  • images (list[contextgem.internal.base.images._Image] | None) – Optional list of Image instances for vision queries

  • chat_session (contextgem.internal.base.llms._ChatSession | None) – Optional stateful chat session to preserve and use history.

Returns:

The LLM’s response. A response cut off at the max output tokens is returned incomplete, with a warning logged.

Return type:

str

Raises:

  • ValueError – If the prompt is empty or not a string

  • ValueError – If images parameter is not a list of Image instances

  • ValueError – If images are provided but the model doesn’t support vision

  • ValueError – If the request params of a fallback LLM are invalid, e.g. parallel tool calls are set for a fallback LLM that does not support them

  • ValueError – If the prompt exceeds the context window of the LLM, and no fallback LLM is left to try

  • ModelAPIError – If the request to the LLM fails, or the LLM returns an empty response, and no fallback LLM is left to try

  • LLMToolLoopLimitError – If the LLM keeps calling tools after tool_max_rounds rounds of tool calls

async chat_async(prompt, *, images=None, chat_session=None)#

Asynchronously sends a prompt to the LLM and gets a response. For models supporting vision, attach images to the prompt if needed.

This method allows direct interaction with the LLM by submitting your own prompt.

Parameters:

  • prompt (str) – The input prompt to send to the LLM

  • images (list[contextgem.internal.base.images._Image] | None) – Optional list of Image instances for vision queries

  • chat_session (contextgem.internal.base.llms._ChatSession | None) – Optional stateful chat session to preserve and use history.

Returns:

The LLM’s response. A response cut off at the max output tokens is returned incomplete, with a warning logged.

Return type:

str

Raises:

  • ValueError – If the prompt is empty or not a string

  • ValueError – If images parameter is not a list of Image instances

  • ValueError – If images are provided but the model doesn’t support vision

  • ValueError – If the request params of a fallback LLM are invalid, e.g. parallel tool calls are set for a fallback LLM that does not support them

  • ValueError – If the prompt exceeds the context window of the LLM, and no fallback LLM is left to try

  • ModelAPIError – If the request to the LLM fails, or the LLM returns an empty response, and no fallback LLM is left to try

  • LLMToolLoopLimitError – If the LLM keeps calling tools after tool_max_rounds rounds of tool calls

extract_all(document, *, overwrite_existing=False, max_items_per_call=0, use_concurrency=False, max_paragraphs_to_analyze_per_call=0, max_images_to_analyze_per_call=0, raise_exception_on_extraction_error=True)#

Extracts all aspects and concepts from a document and its aspects.

This method performs comprehensive extraction by processing the document for aspects and concepts, then extracting concepts from each aspect. The operation can be configured for concurrent processing and customized extraction parameters.

This is the synchronous version of extract_all_async().

Parameters:

  • document (contextgem.internal.base.documents._Document) – The document to analyze.

  • overwrite_existing (bool) – Whether to overwrite already processed aspects and concepts with newly extracted information. Defaults to False.

  • max_items_per_call (int) – Maximum number of items with the same extraction params to process in each call. Defaults to 0 (all items in one call). If concurrency is enabled, defaults to 1. For complex tasks, you should not set a high value, in order to avoid prompt overloading.

  • use_concurrency (bool) – If True, enables concurrent processing of multiple items. Concurrency can considerably reduce processing time, but may cause rate limit errors with model providers. Use this option when API rate limits allow for multiple concurrent requests. Defaults to False.

  • max_paragraphs_to_analyze_per_call (int) – Maximum paragraphs to include in a single LLM prompt. A decision model is sent each paragraph in a request of its own for aspects, whatever the value. For concepts, it is sent a text with more paragraphs in parts of at most that many paragraphs, each in a request of its own: a concept then gets an extracted item for each part, and a concept with singular_occurrence=True is rejected with ValueError. Defaults to 0 (all paragraphs). Cannot be negative.

  • max_images_to_analyze_per_call (int) – Maximum images to include in a single LLM prompt. Defaults to 0 (all images). Cannot be negative. Relevant only for document-level concepts.

  • raise_exception_on_extraction_error (bool) – Whether to raise an exception if the extraction fails due to invalid data returned by a model or an error in the model API. If False, a warning will be issued instead, and no extracted items will be returned. Defaults to True.

Returns:

The document with extracted aspects and concepts.

Return type:

contextgem.internal.base.documents._Document

async extract_all_async(document, *, overwrite_existing=False, max_items_per_call=0, use_concurrency=False, max_paragraphs_to_analyze_per_call=0, max_images_to_analyze_per_call=0, raise_exception_on_extraction_error=True)#

Asynchronously extracts all aspects and concepts from a document and its aspects.

This method performs comprehensive extraction by processing the document for aspects and concepts, then extracting concepts from each aspect. The operation can be configured for concurrent processing and customized extraction parameters.

Parameters:

  • document (contextgem.internal.base.documents._Document) – The document to analyze.

  • overwrite_existing (bool) – Whether to overwrite already processed aspects and concepts with newly extracted information. Defaults to False.

  • max_items_per_call (int) – Maximum number of items with the same extraction params to process in each call. Defaults to 0 (all items in one call). If concurrency is enabled, defaults to 1. For complex tasks, you should not set a high value, in order to avoid prompt overloading.

  • use_concurrency (bool) – If True, enables concurrent processing of multiple items. Concurrency can considerably reduce processing time, but may cause rate limit errors with model providers. Use this option when API rate limits allow for multiple concurrent requests. Defaults to False.

  • max_paragraphs_to_analyze_per_call (int) – Maximum paragraphs to include in a single LLM prompt. A decision model is sent each paragraph in a request of its own for aspects, whatever the value. For concepts, it is sent a text with more paragraphs in parts of at most that many paragraphs, each in a request of its own: a concept then gets an extracted item for each part, and a concept with singular_occurrence=True is rejected with ValueError. Defaults to 0 (all paragraphs). Cannot be negative.

  • max_images_to_analyze_per_call (int) – Maximum images to include in a single LLM prompt. Defaults to 0 (all images). Cannot be negative. Relevant only for document-level concepts.

  • raise_exception_on_extraction_error (bool) – Whether to raise an exception if the extraction fails due to invalid data returned by a model or an error in the model API. If False, a warning will be issued instead, and no extracted items will be returned. Defaults to True.

Returns:

The document with extracted aspects and concepts.

Return type:

contextgem.internal.base.documents._Document

extract_aspects_from_document(document, *, from_aspects=None, overwrite_existing=False, max_items_per_call=0, use_concurrency=False, max_paragraphs_to_analyze_per_call=0, raise_exception_on_extraction_error=True)#

Extracts aspects from the provided document using predefined models.

If an aspect instance has extracted_items populated, the reference_paragraphs field will be automatically populated from these items. For an aspect with the “sentences” reference depth, the reference_sentences field is populated from them as well: with the extracted sentences, and not with the other sentences of the reference paragraphs.

This is the synchronous version of extract_aspects_from_document_async().

Parameters:

  • document (contextgem.internal.base.documents._Document) – The document from which aspects are to be extracted.

  • from_aspects (collections.abc.Sequence[contextgem.internal.base.aspects._Aspect] | None) – Existing aspects to use as a base for extraction. If None, uses all document’s aspects. An aspect that is given more than once is extracted once.

  • overwrite_existing (bool) – Whether to overwrite already processed aspects with newly extracted information. Defaults to False.

  • max_items_per_call (int) – Maximum items with the same extraction params to process per call. Defaults to 0 (all items in single call). For complex tasks, you should not set a value, to avoid prompt overloading. If concurrency is enabled, defaults to 1 (each item processed separately).

  • use_concurrency (bool) – If True, enables concurrent processing of multiple items. Concurrency can considerably reduce processing time, but may cause rate limit errors with model providers. Use this option when API rate limits allow for multiple concurrent requests. Defaults to False.

  • max_paragraphs_to_analyze_per_call (int) – Maximum paragraphs to analyze in a single LLM prompt. Changes nothing for a decision model, which is sent each paragraph in a request of its own. Defaults to 0 (all paragraphs). Cannot be negative.

  • raise_exception_on_extraction_error (bool) – Whether to raise an exception if the extraction fails due to invalid data returned by a model or an error in the model API. If False, a warning will be issued instead, and no extracted items will be returned. Defaults to True.

Returns:

List of processed _Aspect objects with extracted items.

Return type:

list[contextgem.internal.base.aspects._Aspect]

async extract_aspects_from_document_async(document, *, from_aspects=None, overwrite_existing=False, max_items_per_call=0, use_concurrency=False, max_paragraphs_to_analyze_per_call=0, raise_exception_on_extraction_error=True)#

Extracts aspects from the provided document using predefined models asynchronously.

If an aspect instance has extracted_items populated, the reference_paragraphs field will be automatically populated from these items. For an aspect with the “sentences” reference depth, the reference_sentences field is populated from them as well: with the extracted sentences, and not with the other sentences of the reference paragraphs.

Parameters:

  • document (contextgem.internal.base.documents._Document) – The document from which aspects are to be extracted.

  • from_aspects (collections.abc.Sequence[contextgem.internal.base.aspects._Aspect] | None) – Existing aspects to use as a base for extraction. If None, uses all document’s aspects. An aspect that is given more than once is extracted once.

  • overwrite_existing (bool) – Whether to overwrite already processed aspects with newly extracted information. Defaults to False.

  • max_items_per_call (int) – Maximum number of items with the same extraction params to process per call. Defaults to 0 (all items in one call). If concurrency is enabled, defaults to 1. For complex tasks, you should not set a high value, in order to avoid prompt overloading.

  • use_concurrency (bool) – If True, enables concurrent processing of multiple items. Concurrency can considerably reduce processing time, but may cause rate limit errors with model providers. Use this option when API rate limits allow for multiple concurrent requests. Defaults to False.

  • max_paragraphs_to_analyze_per_call (int) – Maximum paragraphs to analyze in a single LLM prompt. Changes nothing for a decision model, which is sent each paragraph in a request of its own. Defaults to 0 (all paragraphs). Cannot be negative.

  • raise_exception_on_extraction_error (bool) – Whether to raise an exception if the extraction fails due to invalid data returned by a model or an error in the model API. If False, a warning will be issued instead, and no extracted items will be returned. Defaults to True.

Returns:

List of processed _Aspect objects with extracted items.

Return type:

list[contextgem.internal.base.aspects._Aspect]

extract_concepts_from_aspect(aspect, document, *, from_concepts=None, overwrite_existing=False, max_items_per_call=0, use_concurrency=False, max_paragraphs_to_analyze_per_call=0, raise_exception_on_extraction_error=True)#

Extracts concepts associated with a given aspect in a document.

This method processes an aspect to extract related concepts using models. If the aspect has not been previously processed, a ValueError is raised.

This is the synchronous version of extract_concepts_from_aspect_async().

Parameters:

  • aspect (contextgem.internal.base.aspects._Aspect) – The aspect from which to extract concepts.

  • document (contextgem.internal.base.documents._Document) – The document that contains the aspect.

  • from_concepts (collections.abc.Sequence[contextgem.internal.base.concepts._Concept] | None) – List of existing concepts to process. Defaults to None. A concept that is given more than once is extracted once.

  • overwrite_existing (bool) – Whether to overwrite already processed concepts with newly extracted information. Defaults to False.

  • max_items_per_call (int) – Maximum number of items with the same extraction params to process in each call. Defaults to 0 (all items in one call). If concurrency is enabled, defaults to 1. For complex tasks, you should not set a high value, in order to avoid prompt overloading.

  • use_concurrency (bool) – If True, enables concurrent processing of multiple items. Concurrency can considerably reduce processing time, but may cause rate limit errors with model providers. Use this option when API rate limits allow for multiple concurrent requests. Defaults to False.

  • max_paragraphs_to_analyze_per_call (int) – Maximum paragraphs to include in a single LLM prompt. A decision model is sent a text with more paragraphs in parts of at most that many paragraphs, each in a request of its own: a concept then gets an extracted item for each part, and a concept with singular_occurrence=True is rejected with ValueError. Defaults to 0 (all paragraphs). Cannot be negative.

  • raise_exception_on_extraction_error (bool) – Whether to raise an exception if the extraction fails due to invalid data returned by a model or an error in the model API. If False, a warning will be issued instead, and no extracted items will be returned. Defaults to True.

Returns:

List of processed concept objects.

Return type:

list[contextgem.internal.base.concepts._Concept]

async extract_concepts_from_aspect_async(aspect, document, *, from_concepts=None, overwrite_existing=False, max_items_per_call=0, use_concurrency=False, max_paragraphs_to_analyze_per_call=0, raise_exception_on_extraction_error=True)#

Asynchronously extracts concepts from a specified aspect using models.

This method processes an aspect to extract related concepts using models. If the aspect has not been previously processed, a ValueError is raised.

Parameters:

  • aspect (contextgem.internal.base.aspects._Aspect) – The aspect from which to extract concepts.

  • document (contextgem.internal.base.documents._Document) – The document that contains the aspect.

  • from_concepts (collections.abc.Sequence[contextgem.internal.base.concepts._Concept] | None) – List of existing concepts to process. Defaults to None. A concept that is given more than once is extracted once.

  • overwrite_existing (bool) – Whether to overwrite already processed concepts with newly extracted information. Defaults to False.

  • max_items_per_call (int) – Maximum number of items with the same extraction params to process in each call. Defaults to 0 (all items in one call). If concurrency is enabled, defaults to 1. For complex tasks, you should not set a high value, in order to avoid prompt overloading.

  • use_concurrency (bool) – If True, enables concurrent processing of multiple items. Concurrency can considerably reduce processing time, but may cause rate limit errors with model providers. Use this option when API rate limits allow for multiple concurrent requests. Defaults to False.

  • max_paragraphs_to_analyze_per_call (int) – Maximum paragraphs to include in a single LLM prompt. A decision model is sent a text with more paragraphs in parts of at most that many paragraphs, each in a request of its own: a concept then gets an extracted item for each part, and a concept with singular_occurrence=True is rejected with ValueError. Defaults to 0 (all paragraphs). Cannot be negative.

  • raise_exception_on_extraction_error (bool) – Whether to raise an exception if the extraction fails due to invalid data returned by a model or an error in the model API. If False, a warning will be issued instead, and no extracted items will be returned. Defaults to True.

Returns:

List of processed concept objects.

Return type:

list[contextgem.internal.base.concepts._Concept]

extract_concepts_from_document(document, *, from_concepts=None, overwrite_existing=False, max_items_per_call=0, use_concurrency=False, max_paragraphs_to_analyze_per_call=0, max_images_to_analyze_per_call=0, raise_exception_on_extraction_error=True)#

Extracts concepts from the provided document using predefined models.

This is the synchronous version of extract_concepts_from_document_async().

Parameters:

  • document (contextgem.internal.base.documents._Document) – The document from which concepts are to be extracted.

  • from_concepts (collections.abc.Sequence[contextgem.internal.base.concepts._Concept] | None) – Existing concepts to use as a base for extraction. If None, uses all document’s concepts. A concept that is given more than once is extracted once.

  • overwrite_existing (bool) – Whether to overwrite already processed concepts with newly extracted information. Defaults to False.

  • max_items_per_call (int) – Maximum items with the same extraction params to process per call. Defaults to 0 (all items in single call). For complex tasks, you should not set a value, to avoid prompt overloading. If concurrency is enabled, defaults to 1 (each item processed separately).

  • use_concurrency (bool) – If True, enables concurrent processing of multiple items. Concurrency can considerably reduce processing time, but may cause rate limit errors with model providers. Use this option when API rate limits allow for multiple concurrent requests. Defaults to False.

  • max_paragraphs_to_analyze_per_call (int) – Maximum paragraphs to analyze in a single LLM prompt. A decision model is sent a text with more paragraphs in parts of at most that many paragraphs, each in a request of its own: a concept then gets an extracted item for each part, and a concept with singular_occurrence=True is rejected with ValueError. Defaults to 0 (all paragraphs). Cannot be negative.

  • max_images_to_analyze_per_call (int) – Maximum images to include in a single LLM prompt. Defaults to 0 (all images). Cannot be negative.

  • raise_exception_on_extraction_error (bool) – Whether to raise an exception if the extraction fails due to invalid data returned by a model or an error in the model API. If False, a warning will be issued instead, and no extracted items will be returned. Defaults to True.

Returns:

List of processed Concept objects with extracted items.

Return type:

list[contextgem.internal.base.concepts._Concept]

async extract_concepts_from_document_async(document, *, from_concepts=None, overwrite_existing=False, max_items_per_call=0, use_concurrency=False, max_paragraphs_to_analyze_per_call=0, max_images_to_analyze_per_call=0, raise_exception_on_extraction_error=True)#

Extracts concepts from the provided document using predefined models asynchronously.

This method processes a document to extract concepts using configured models.

Parameters:

  • document (contextgem.internal.base.documents._Document) – The document from which concepts are to be extracted.

  • from_concepts (collections.abc.Sequence[contextgem.internal.base.concepts._Concept] | None) – Existing concepts to use as a base for extraction. If None, uses all document’s concepts. A concept that is given more than once is extracted once.

  • overwrite_existing (bool) – Whether to overwrite already processed concepts with newly extracted information. Defaults to False.

  • max_items_per_call (int) – Maximum number of items with the same extraction params to process per call. Defaults to 0 (all items in one call). If concurrency is enabled, defaults to 1. For complex tasks, you should not set a high value, in order to avoid prompt overloading.

  • use_concurrency (bool) – If True, enables concurrent processing of multiple items. Concurrency can considerably reduce processing time, but may cause rate limit errors with model providers. Use this option when API rate limits allow for multiple concurrent requests. Defaults to False.

  • max_paragraphs_to_analyze_per_call (int) – Maximum paragraphs to analyze in a single LLM prompt. A decision model is sent a text with more paragraphs in parts of at most that many paragraphs, each in a request of its own: a concept then gets an extracted item for each part, and a concept with singular_occurrence=True is rejected with ValueError. Defaults to 0 (all paragraphs). Cannot be negative.

  • max_images_to_analyze_per_call (int) – Maximum images to include in a single LLM prompt. Defaults to 0 (all images). Cannot be negative.

  • raise_exception_on_extraction_error (bool) – Whether to raise an exception if the extraction fails due to invalid data returned by a model or an error in the model API. If False, a warning will be issued instead, and no extracted items will be returned. Defaults to True.

Returns:

List of processed Concept objects with extracted items.

Return type:

list[contextgem.internal.base.concepts._Concept]

classmethod from_dict(obj_dict)#

Reconstructs an instance of the class from a dictionary representation.

This method deserializes a dictionary containing the object’s attributes and values into a new instance of the class. It handles complex nested structures like aspects, concepts, and extracted items, properly reconstructing each component.

A deserialized model or model group has no API credentials, as they are not included in the serialized data, i.e. they are to be set again.

Parameters:

obj_dict (dict[str, typing.Any]) – Dictionary containing the serialized object data.

Returns:

A new instance of the class with restored attributes.

Return type:

typing.Self

classmethod from_disk(file_path)#

Loads an instance of the class from a JSON file stored on disk.

This method reads the JSON content from the specified file path and deserializes it into an instance of the class using the from_json method.

Parameters:

file_path (str | pathlib.Path) – Path to the JSON file to load (must end with ‘.json’). Can be a string or a Path object.

Returns:

An instance of the class populated with the data from the file.

Return type:

typing.Self

Raises:

  • ValueError – If the file path doesn’t end with ‘.json’, or if the file does not hold valid serialized data, e.g. it is not a valid UTF-8 text (UnicodeDecodeError) or JSON (json.JSONDecodeError), or the data fails validation (pydantic’s ValidationError). These errors are subclasses of ValueError.

  • TypeError – If the serialized data is of another class.

  • OSError – If the file cannot be read, e.g. it does not exist. The error is the one raised by the file operation, e.g. FileNotFoundError or PermissionError.

classmethod from_json(json_string)#

Creates an instance of the class from a JSON string representation.

This method deserializes the provided JSON string into a dictionary and uses the from_dict method to construct the class instance. It validates that the class name in the serialized data matches the current class.

Parameters:

json_string (str) – JSON string containing the serialized object data.

Returns:

A new instance of the class with restored state.

Return type:

typing.Self

Raises:

  • json.JSONDecodeError – If the string is not a valid JSON. The error gives the position of what is invalid, and does not hold the string.

  • ValueError – If the JSON value is not an object with a class name, i.e. not serialized data.

  • TypeError – If the class name in the serialized data doesn’t match.

get_cost()#

Retrieves the accumulated cost information of the model and of its fallback models, if configured.

This method collects cost statistics for the current model instance and for its fallback models (if configured), providing insights into API usage expenses.

Returns:

A list of cost statistics containers for the model and its fallback models.

Return type:

list[contextgem.internal.data_models._ModelCostOutputContainer]

get_usage()#

Retrieves the usage information of the model and of its fallback models, if configured.

This method collects token usage statistics for the current model instance and for its fallback models (if configured), providing insights into API consumption.

Returns:

A list of usage statistics containers for the model and its fallback models.

Return type:

list[contextgem.internal.data_models._ModelUsageOutputContainer]

property is_group: bool#

Returns False indicating this is a single model, not a group.

Returns:

Always False for single models.

Return type:

bool

property list_roles: list[str | None]#

Returns a list containing the role of this model.

(For a single model, this returns a list with just one element - the model’s role, or None if the model has no role. For model groups, the method implementation returns roles of all models in the group.)

Returns:

A list containing the role of this model.

Return type:

list[str | None]

reset_usage_and_cost()#

Resets the usage and cost statistics for the model and its fallback models (if configured).

This method clears accumulated usage and cost data, which is useful when processing multiple documents sequentially and tracking metrics for each document separately.

Returns:

None

Return type:

None

to_dict()#

Transforms the current object into a dictionary representation.

Converts the object to a dictionary that includes: - All public attributes - Special handling for specific public and private attributes

When a model or model group is serialized, its API credentials and usage/cost stats are removed.

Returns:

A dictionary representation of the current object with all necessary data for serialization

Return type:

dict[str, typing.Any]

to_disk(file_path)#

Saves the serialized instance to a JSON file at the specified path.

This method converts the instance to a dictionary representation using to_dict(), then writes it to disk as a formatted JSON file with UTF-8 encoding.

Parameters:

file_path (str | pathlib.Path) – Path where the JSON file should be saved (must end with ‘.json’). Can be a string or a Path object.

Return type:

None

Returns:

None

Raises:

  • ValueError – If the file path doesn’t end with ‘.json’.

  • OSError – If the file cannot be written, e.g. its directory does not exist. The error is the one raised by the file operation, e.g. FileNotFoundError or PermissionError.

to_json()#

Converts the object to its JSON string representation.

Serializes the object into a JSON-formatted string using the dictionary representation provided by the to_dict() method.

Returns:

A JSON string representation of the object.

Return type:

str

deployment_id: Annotated[str, Strict(strict=True), StringConstraints(strip_whitespace=True, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=None, ascii_only=None)] | None#
api_version: Annotated[str, Strict(strict=True), StringConstraints(strip_whitespace=True, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=None, ascii_only=None)] | None#
system_message: str | None#
max_tokens: Annotated[int, Strict(strict=True)] | None#
reasoning_effort: Literal['none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max'] | None#
max_retries_failed_request: Annotated[int, Strict(strict=True)]#
max_retries_invalid_data: Annotated[int, Strict(strict=True)]#
output_language: Literal['en', 'adapt']#
temperature: Annotated[float, Strict(strict=True)] | None#
top_p: Annotated[float, Strict(strict=True)] | None#
seed: Annotated[int, Strict(strict=True)] | None#
tools: Sequence[Callable[[...], str] | Callable[[...], Coroutine[Any, Any, str]]] | None#
tool_choice: str | Annotated[dict[str, Any], BeforeValidator(func=_validate_is_json_dict, json_schema_input_type=PydanticUndefined)] | None#
parallel_tool_calls: bool | None#
tool_max_rounds: Annotated[int, Strict(strict=True)]#
model: Annotated[str, Strict(strict=True), StringConstraints(strip_whitespace=True, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=None, ascii_only=None)]#
api_key: Annotated[str, Strict(strict=True), StringConstraints(strip_whitespace=True, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=None, ascii_only=None)] | None#
api_base: Annotated[str, Strict(strict=True), StringConstraints(strip_whitespace=True, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=None, ascii_only=None)] | None#
role: Annotated[str, Strict(strict=True), AfterValidator(func=_validate_model_role)] | None#
num_retries_failed_request: Annotated[int, Strict(strict=True)]#
timeout: Annotated[int, Strict(strict=True)]#
pricing_details: _ModelPricing | None#
is_fallback: Annotated[bool, Strict(strict=True)]#
fallback_models: list[_DocumentModel]#
auto_pricing: Annotated[bool, Strict(strict=True)]#
auto_pricing_refresh: Annotated[bool, Strict(strict=True)]#
class contextgem.public.llms.ChatSession(**data)[source]#

Bases: _ChatSession

Stateful chat session that preserves message history across turns.

To be used as chat_session=... parameter for DocumentLLM.chat(...) or DocumentLLM.chat_async(...).

Note:

A chat session keeps the first system message it is given: the system message of an LLM is added to the session only if the session has none yet. The system message of the session is sent with each request of the session, also by any other LLM that the session is used with, instead of the system message of that LLM.

In a chat with tools, the history of the session keeps the tool calls of each answered turn and their results, which are sent to the LLM with the next turns. The results include the errors of the tools, which the LLM receives as it calls them. A turn that ends without an answer, e.g. on an error of the LLM API, is started over when it is sent again, i.e. its tool calls are not kept. messages lists the messages of the user and the answers of the LLM, without the tool calls.

The history also keeps the thinking blocks that an LLM returns with its tool calls and answers, e.g. those of Claude when reasoning_effort is set. They are sent back unchanged, so that the LLM keeps its reasoning across tool rounds and turns, but only to the same model with the same tools and system message: a provider may reject them otherwise.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

custom_data (Annotated[dict[str, Any], BeforeValidator(func=~contextgem.internal.typings.validators._validate_is_json_dict, json_schema_input_type=PydanticUndefined)])

clone()#

Creates and returns a deep copy of the current instance.

Return type:

typing.Self

Returns:

A deep copy of the current instance.

classmethod from_dict(obj_dict)#

Reconstructs an instance of the class from a dictionary representation.

This method deserializes a dictionary containing the object’s attributes and values into a new instance of the class. It handles complex nested structures like aspects, concepts, and extracted items, properly reconstructing each component.

A deserialized model or model group has no API credentials, as they are not included in the serialized data, i.e. they are to be set again.

Parameters:

obj_dict (dict[str, typing.Any]) – Dictionary containing the serialized object data.

Returns:

A new instance of the class with restored attributes.

Return type:

typing.Self

classmethod from_disk(file_path)#

Loads an instance of the class from a JSON file stored on disk.

This method reads the JSON content from the specified file path and deserializes it into an instance of the class using the from_json method.

Parameters:

file_path (str | pathlib.Path) – Path to the JSON file to load (must end with ‘.json’). Can be a string or a Path object.

Returns:

An instance of the class populated with the data from the file.

Return type:

typing.Self

Raises:

  • ValueError – If the file path doesn’t end with ‘.json’, or if the file does not hold valid serialized data, e.g. it is not a valid UTF-8 text (UnicodeDecodeError) or JSON (json.JSONDecodeError), or the data fails validation (pydantic’s ValidationError). These errors are subclasses of ValueError.

  • TypeError – If the serialized data is of another class.

  • OSError – If the file cannot be read, e.g. it does not exist. The error is the one raised by the file operation, e.g. FileNotFoundError or PermissionError.

classmethod from_json(json_string)#

Creates an instance of the class from a JSON string representation.

This method deserializes the provided JSON string into a dictionary and uses the from_dict method to construct the class instance. It validates that the class name in the serialized data matches the current class.

Parameters:

json_string (str) – JSON string containing the serialized object data.

Returns:

A new instance of the class with restored state.

Return type:

typing.Self

Raises:

  • json.JSONDecodeError – If the string is not a valid JSON. The error gives the position of what is invalid, and does not hold the string.

  • ValueError – If the JSON value is not an object with a class name, i.e. not serialized data.

  • TypeError – If the class name in the serialized data doesn’t match.

property messages: list[_Message]#

Returns the list of messages in the session: the system message, if any, and the messages of the user and the answers of the LLM. The tool calls of the LLM and their results are not listed, although they are kept in the history of the chat that is sent to the LLM.

Returns:

The list of messages in the session.

Return type:

list[_Message]

reset()#

Clears conversation history by removing all messages.

Returns:

None

Return type:

None

to_dict()#

Transforms the current object into a dictionary representation.

Converts the object to a dictionary that includes: - All public attributes - Special handling for specific public and private attributes

When a model or model group is serialized, its API credentials and usage/cost stats are removed.

Returns:

A dictionary representation of the current object with all necessary data for serialization

Return type:

dict[str, typing.Any]

to_disk(file_path)#

Saves the serialized instance to a JSON file at the specified path.

This method converts the instance to a dictionary representation using to_dict(), then writes it to disk as a formatted JSON file with UTF-8 encoding.

Parameters:

file_path (str | pathlib.Path) – Path where the JSON file should be saved (must end with ‘.json’). Can be a string or a Path object.

Return type:

None

Returns:

None

Raises:

  • ValueError – If the file path doesn’t end with ‘.json’.

  • OSError – If the file cannot be written, e.g. its directory does not exist. The error is the one raised by the file operation, e.g. FileNotFoundError or PermissionError.

to_json()#

Converts the object to its JSON string representation.

Serializes the object into a JSON-formatted string using the dictionary representation provided by the to_dict() method.

Returns:

A JSON string representation of the object.

Return type:

str

property unique_id: str#

Returns the ULID of the instance.

custom_data: Annotated[dict[str, Any], BeforeValidator(func=_validate_is_json_dict, json_schema_input_type=PydanticUndefined)]#
Contents