Supported LLMs

ContextGem supports all LLM providers and models available through the LiteLLM integration. This means you can use models from major cloud providers like OpenAI, Anthropic, Google, Azure, and xAI, as well as run local models through providers like Ollama and LM Studio.

Both models that support reasoning (chain of thought) and models that do not are supported.

For a complete list of supported providers, see the LiteLLM Providers documentation.

☁️ Cloud-based LLMs

You can initialize cloud-based LLMs by specifying the provider and model name in the format <provider>/<model_name>:

Using cloud LLM providers

from contextgem import DocumentLLM


# Pattern for using any cloud LLM provider
llm = DocumentLLM(
    model="<provider>/<model_name>",
    api_key="<api_key>",
)

# Example - Using OpenAI LLM
llm_openai = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key="<api_key>",
    # see DocumentLLM API reference for all configuration options
)

# Example - Using Azure OpenAI LLM
llm_azure_openai = DocumentLLM(
    model="azure/gpt-6-luna",
    api_key="<api_key>",
    api_version="<api_version>",
    api_base="<api_base>",
    # see DocumentLLM API reference for all configuration options
)

💻 Local LLMs

For local LLMs, you’ll need to specify the provider, model name, and the appropriate API base URL:

Using local LLM providers

from contextgem import DocumentLLM


local_llm = DocumentLLM(
    model="ollama_chat/<model_name>",
    api_base="http://localhost:11434",  # Default Ollama endpoint
)

# Example - Using Llama 3.3 LLM via Ollama
llm_llama = DocumentLLM(
    model="ollama_chat/llama3.3:70b",
    api_base="http://localhost:11434",
    # see DocumentLLM API reference for all configuration options
)

# Example - Using DeepSeek R1 reasoning model via Ollama
llm_deepseek = DocumentLLM(
    model="ollama_chat/deepseek-r1:32b",
    api_base="http://localhost:11434",
    # see DocumentLLM API reference for all configuration options
)

# Example - Using Mistral Small 3.2 via LM Studio (no API key is needed)
llm_mistral = DocumentLLM(
    model="lm_studio/mistralai/mistral-small-3.2",
    api_base="http://localhost:1234/v1",  # Default LM Studio endpoint
    # see DocumentLLM API reference for all configuration options
)

Note

Choosing a model: ContextGem is tested with hosted LLMs such as gpt-6-luna (OpenAI) and claude-haiku-5-5 (Anthropic). A smaller LLM, such as many that run locally, may not follow the detailed extraction instructions: check it on a sample of your documents, and see the troubleshooting guide if it falls short.

Note

Vision Models with Ollama: For local vision models that process images, use the ollama/ prefix instead of ollama_chat/, as the latter does not yet support image inputs. For more details, see the relevant Ollama GitHub issues #10255 and #6451.

For a complete list of configuration options available when initializing DocumentLLM instances, see the next section Configuring LLM(s).

See also

Decision models are not LLMs: they return the probabilities of the answers to typed questions and generate no text. They are configured with a separate class, DocumentDecisionModel, and have their own list of supported providers. See Decision Models.