Chat with Tools

Chat with Tools#

This guide explains how to use LLM tool calling (function calling) with ContextGem’s chat interface. Tools allow the LLM to invoke Python functions during a conversation to perform actions or retrieve information.

Note

Tool support is only available in chat(...) and chat_async(...) methods. It is not used by extraction methods: the tools of an LLM are not sent with the requests of an extraction.

Basic Usage#

A tool is a Python function decorated with @register_tool. The function defines the tool: its name is the name of the tool, its docstring describes the tool to the LLM, and its parameters, with their type hints and default values, define the parameters of the tool.

import os
from typing import TypedDict

from contextgem import ChatSession, DocumentLLM, register_tool


# Define a TypedDict for structured tool parameters.
# This ensures the auto-generated schema includes proper types for each field.
class InvoiceItem(TypedDict):
    qty: float
    price: float


# Define tool handlers with @register_tool decorator.
# The schema is auto-generated from the function signature and docstring.
@register_tool
def compute_invoice_total(items: list[InvoiceItem]) -> str:
    """
    Compute invoice total as sum(qty*price) over items.

    :param items: List of invoice items with quantity and price
    """
    total = 0
    for it in items:
        qty = float(it.get("qty", 0))
        price = float(it.get("price", 0))
        total += qty * price
    return str(total)


# Configure an LLM that supports tool use.
# Simply pass the decorated function(s) directly.
llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key=os.getenv("CONTEXTGEM_OPENAI_API_KEY"),
    system_message="You are a helpful assistant.",  # override default system message for chat
    tools=[compute_invoice_total],  # Pass the function directly
)

# Maintain history across turns
session = ChatSession()

prompt = (
    "What's the invoice total for the items "
    "[{'qty':2.0,'price':3.5},{'qty':1.0,'price':3.0}]?"
)

answer = llm.chat(prompt, chat_session=session)
print("Answer:", answer)

Key points:

Decorate a function with @register_tool, and pass the function itself to tools=[...] when creating the LLM, or assign the functions to llm.tools later. This is the only way to define and assign tools
A tool function must have a docstring and type hints for all its parameters, and must not have positional-only parameters, *args or **kwargs
A parameter that has a default value, or is typed as Optional, is optional for the LLM
Tool handlers must return a non-empty string (serialize structured data with json.dumps if needed)
Use ChatSession to maintain conversation history across turns. The history keeps the tool calls of each turn and their results, so the LLM is sent them with the next turns and does not need to call the tools again for what it already found. ChatSession.messages lists the messages of the user and the answers of the LLM, without the tool calls
The thinking blocks that an LLM returns with its tool calls and answers, e.g. those of Claude when reasoning_effort is set, are sent back to it unchanged in the next tool rounds and, in a chat session, in the next turns, so that it keeps its reasoning. They are sent back only to the same model with the same tools and system message, as a provider may reject them otherwise

Docstring Formats#

The schema generator extracts parameter descriptions from docstrings. Multiple formats are supported:

Sphinx/reST

@register_tool
def my_tool(query: str) -> str:
    """
    Search for information.

    :param query: The search query
    """
    return "results"

Google style:

@register_tool
def my_tool(query: str) -> str:
    """Search for information.

    Args:
        query: The search query
    """
    return "results"

NumPy style:

@register_tool
def my_tool(query: str) -> str:
    """
    Search for information.

    Parameters
    ----------
    query : str
        The search query
    """
    return "results"

Note

A docstring in any other format works as well: the whole docstring describes the tool, and the parameters are sent to the LLM without descriptions, unless they are described with Annotated (see Parameter Descriptions and Constraints).

Schema Generation Best Practices#

Use TypedDict for Object Parameters#

Plain dict generates a generic {"type": "object"} schema without property definitions. Use TypedDict to specify field names and types explicitly:

from typing import TypedDict

class InvoiceItem(TypedDict):
    qty: float
    price: float

@register_tool
def compute_total(items: list[InvoiceItem]) -> str:
    """
    Compute invoice total.

    :param items: List of invoice items
    """
    total = sum(it["qty"] * it["price"] for it in items)
    return str(total)

This generates a schema with typed properties for the items parameter:

{
  "items": {
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "qty": {"type": "number"},
        "price": {"type": "number"}
      },
      "required": ["price", "qty"]
    },
    "description": "List of invoice items"
  }
}

Note

If a module uses from __future__ import annotations, define the types used in the type hints of a tool (e.g. a TypedDict) at module level, not inside a function. Otherwise, they cannot be found when the tool schema is generated, and TypeError is raised.

Supported Type Hints#

The schema generator supports the following Python type hints:

Primitives: str, int, float, bool, None
Collections: list[T], dict[K, V], tuple[T, ...], set[T]
Optionals: Optional[T], T | None
Unions: Union[X, Y, ...], X | Y
Literals: Literal["a", "b", "c"]
Structured: TypedDict
Annotated: Annotated[T, ...] with a description and/or constraints (see Parameter Descriptions and Constraints)

Note

JSON serialization of collections: Since tool arguments are transmitted as JSON, tuple and set types are received as Python list at runtime (JSON only has arrays). If you need specific collection behavior, convert inside your function: items = set(items) or items = tuple(items).

Examples:

from typing import Literal, TypedDict

class SearchFilters(TypedDict):
    category: str
    max_price: float

@register_tool
def search(
    query: str,
    limit: int = 10,
    sort: Literal["relevance", "date", "price"] = "relevance",
    filters: SearchFilters | None = None,
) -> str:
    """
    Search products.

    :param query: Search query
    :param limit: Maximum results to return
    :param sort: Sort order
    :param filters: Optional filters
    """
    return "results"

Parameter Descriptions and Constraints#

Besides the docstring, a parameter can be described with Annotated, next to its type. Annotated can also set constraints on the values of the parameter, with pydantic’s Field:

from typing import Annotated, Literal

from pydantic import Field

@register_tool
def search_database(
    query: Annotated[str, Field(description="Search query", min_length=1)],
    category: Annotated[
        Literal["electronics", "clothing", "books"], "Product category filter"
    ] = "books",
    limit: Annotated[int, Field(description="Maximum results", ge=1, le=50)] = 10,
) -> str:
    """
    Search the product database.
    """
    return f"Results for '{query}' in {category}"

A string in Annotated, or Field(description=...), is the description of the parameter. It takes precedence over the description of the parameter in the docstring
The constraints are added to the tool schema sent to the LLM, and the arguments of each tool call are validated against them. If the LLM passes an invalid value, the tool is not called: the LLM receives the validation error and can correct the call

Constraints	Apply to
`gt`, `ge`, `lt`, `le`, `multiple_of`	`int`, `float`
`min_length`, `max_length`	`str`, `list`, `tuple`, `set`, `dict`
`pattern`	`str`

Annotated can also be used for the items of a collection (e.g. list[Annotated[int, Field(ge=0)]]) and for the fields of a TypedDict. A constraint that does not apply to the type of the parameter raises TypeError when the tools are set.

Note

There is no way to pass a hand-written tool schema: the schema is always generated from the function, so that the two cannot go out of sync.

Serializing an LLM with Tools#

An LLM with tools is serialized as usual (to_dict(), to_json(), to_disk()). The functions themselves are not stored: the serialized data holds the tool schemas generated from them. When the LLM is loaded, each tool is matched by its name to a function decorated with @register_tool, so the functions of the tools have to be defined or imported, under their own names, in the code that loads the LLM.

Tool Configuration Options#

The DocumentLLM class accepts several tool-related parameters:

Parameter	Default	Description
`tools`	`None`	Functions decorated with `@register_tool`, as a list or a tuple.
`tool_choice`	`None`	Controls how the model uses tools. Options: `"none"` (model will not call tools), `"auto"` (model decides whether to call tools or respond with text), `"required"` (model must call at least one tool), or `{"type": "function", "function": {"name": "..."}}` to force a specific tool. When `None`, defers to the API default (equivalent to `"auto"` when tools are defined). Note: `"required"` or a forced function is automatically relaxed to `"auto"` in the follow-up rounds, i.e. after the results of the tools are returned to the model, so that the model can either call more tools or produce the final text response, instead of being forced to call tools in every round until `tool_max_rounds` is reached.
`parallel_tool_calls`	`None`	Whether to enable parallel tool execution. When `None`, defers to the API/model default. Set to `True` to explicitly enable or `False` to disable (if supported by the model).
`tool_max_rounds`	`10`	Maximum number of tool execution rounds per request. Prevents infinite or excessively long tool chains.

Return Value Requirements#

Tool handlers must return a non-empty string. For structured data, serialize it before returning:

import json

@register_tool
def get_user_data(user_id: str) -> str:
    """
    Retrieve user data.

    :param user_id: The user ID
    """
    data = {"name": "John", "email": "john@example.com"}
    return json.dumps(data)

The LLM will interpret the returned string and incorporate it into its response.

If a tool returns a value of another type, TypeError is raised from chat(...), and if it returns an empty string, ValueError is raised. No further requests are sent in such cases.

Errors in Tool Calls#

When the LLM can correct a failed tool call, the chat is not stopped: the error is sent to the LLM as the result of the tool call, and the LLM can call the tool again in the next round.

What happens	How it is handled
The LLM calls a tool that does not exist	The LLM receives an error with the names of the available tools
The LLM passes arguments that are not valid JSON, or that do not match the parameters of the tool or their constraints	The tool is not called, and the LLM receives the validation error
The tool function raises an exception	The LLM receives the message of the exception
The LLM returns a tool call that cannot be processed, e.g. without the name of the tool or the ID of the call	No tool is called. This is handled as a failed request: a fallback LLM is tried, or `ModelAPIError` is raised
The tool function returns a value other than a non-empty string	`TypeError` or `ValueError` is raised from `chat(...)`
The LLM still calls tools after `tool_max_rounds` rounds of tool calls	`LLMToolLoopLimitError` is raised from `chat(...)`

ModelAPIError and LLMToolLoopLimitError can be imported from contextgem, and except ContextGemError catches either of them (ContextGemError).

When chat(...) raises an error, or is cancelled (e.g. on a timeout set by the calling code), the requests completed before that are still counted in the usage and cost of the LLM.

Note

The message of an exception raised by a tool function is sent to the LLM. Make sure that it does not contain anything that the LLM should not receive.

Tools and Fallback LLMs#

An LLM with tools can have fallback LLMs (see 🔄 Configuring Fallback LLMs). Each fallback LLM must have the same tools as the primary LLM, in any order, the same tool_choice and parallel_tool_calls, and the same support for tool calls, as detected by litellm with litellm.supports_function_calling() or set manually with _supports_tools=True on the LLM instance. This is checked when the fallback LLMs are set. Before any request of a chat is sent, it is also checked that each fallback LLM with parallel_tool_calls set supports parallel tool calls.

A fallback LLM takes over when a request to the current LLM fails, at any point of the chat, or when the current LLM returns an empty response. The fallback LLM starts the chat over: it is sent the prompt, without the tool calls made by the LLM that failed and without their results.

Warning

If an LLM fails after it has called tools, the fallback LLM may call the same tools again, i.e. a tool can be called more than once in a single chat(...) call. Make sure that the tools that change something (e.g. send a message or create a record) can be safely called again with the same arguments.

A fallback LLM is not tried when the chat is stopped for a reason other than a failure of the LLM, i.e. when a tool function returns an invalid value, or when tool_max_rounds is reached (see Errors in Tool Calls).

An empty response is handled as a failure of the LLM: the next fallback LLM is tried, and ModelAPIError is raised if there is none left to try. A response cut off at the max output tokens is returned as it is, with a warning logged (see 📏 Limiting the Output Tokens).

Chat with Tools

Contents

Chat with Tools#

Basic Usage#

Docstring Formats#

Schema Generation Best Practices#

Use TypedDict for Object Parameters#

Supported Type Hints#

Parameter Descriptions and Constraints#

Serializing an LLM with Tools#

Tool Configuration Options#

Return Value Requirements#

Errors in Tool Calls#

Tools and Fallback LLMs#