Testing Without API Calls

Testing Without API Calls#

The code that you build on ContextGem can be tested without sending a request to a model: no API key, no cost, the same result in each run, e.g. in CI. ContextGem sends each request to a model through one of two functions of litellm, which a test replaces with a stand-in that returns the response.

🔌 Where to Patch#

Function	Sends
`litellm.acompletion`	Each request to an LLM (`DocumentLLM`): in the extraction methods and in a chat.
`litellm.adecisions`	Each request to a decision model (`DocumentDecisionModel`).

ContextGem looks these functions up on the litellm module when it sends each request, so unittest.mock.patch.object(litellm, "acompletion", stand_in), or monkeypatch.setattr(litellm, "acompletion", stand_in) with pytest, applies to every model, also to one created before the patch, and to the sync and async methods alike. They are the supported points for replacing the requests in tests. The stand-in is an async function, called with keyword arguments.

What ContextGem makes of the response of a stand-in is validated as a response of the model is: an answer that does not have the format that the prompt asks for is invalid data, which ContextGem asks for again up to max_retries_invalid_data times, and then raises ModelExtractionError (see 🔁 How Failed Requests Are Retried). A stand-in can also raise an error of litellm, e.g. to test the handling of a failed request. An error without an HTTP status code, e.g. an exception of your own, or an error in the stand-in itself, is not retried, unless it is a TimeoutError or a ConnectionError of Python, e.g. to simulate a failure of the network: the next fallback model is tried, and the error is the original_error of ModelAPIError (see 🔁 How Failed Requests Are Retried).

🤖 Stubbing an LLM#

An extraction with a stand-in for the LLM#

# Testing your own code without sending requests to the LLM: the stand-in
# replaces litellm.acompletion, which ContextGem calls for each request
# to an LLM

from unittest.mock import patch

import litellm

from contextgem import Document, DocumentLLM, StringConcept


# The code to test: gets the number of an invoice from its text
def get_invoice_number(text: str, llm: DocumentLLM) -> str | None:
    document = Document(raw_text=text)
    document.add_concepts(
        [
            StringConcept(
                name="Invoice number",
                description="The number of the invoice",
                singular_occurrence=True,
            )
        ]
    )
    llm.extract_concepts_from_document(document)
    items = document.concepts[0].extracted_items
    return items[0].value if items else None


# The answer of the LLM, in the format that the extraction prompt asks for:
# a JSON array with an entry for each concept, by its ID in the prompt ("C0")
answer = '[{"concept_id": "C0", "extracted_items": ["INV-001"]}]'

# The requests that ContextGem sends, e.g. to check their prompts
requests = []


async def fake_acompletion(**kwargs):
    requests.append(kwargs)
    # litellm builds a response with this answer, and sends no request whatever
    # it is given: the real litellm.acompletion is not called
    return litellm.mock_completion(**kwargs, mock_response=answer)


# No API key, neither here nor in the environment of the tests: a request that
# is not stubbed then fails instead of being billed
llm = DocumentLLM(model="openai/gpt-6-luna")

with patch.object(litellm, "acompletion", fake_acompletion):
    invoice_number = get_invoice_number(
        "Invoice INV-001 is due on 30 November 2026.", llm
    )

assert invoice_number == "INV-001"
assert len(requests) == 1
assert "Invoice INV-001" in requests[0]["messages"][-1]["content"]

The stand-in receives the keyword arguments of litellm.acompletion, among them model and messages, whose last message has the prompt, and returns a litellm.ModelResponse. ContextGem reads from it:

the answer: the message.content of the last choice that has a text, or of the first choice;
in a chat, the tool calls of the choices (message.tool_calls);
the finish_reason of the choices;
the token counts of usage: prompt_tokens, completion_tokens and, where present, the cached tokens of prompt_tokens_details, which the costs are calculated from.

litellm.mock_completion, given the keyword arguments of the request and mock_response, builds such a response, and sends no request whatever it is given: the answer as its text, finish_reason="stop", and a usage of 10 prompt tokens and 20 completion tokens whatever the request (litellm 1.105.2). mock_response="" gives an empty answer. Do not call the real litellm.acompletion from a stand-in, also not with mock_response: it takes mock_response="" as no mock, and sends the request.

The answer of an extraction is the JSON that the prompt asks for in its answer format: an array with an entry for each concept, or aspect, of the request, by its ID in the prompt ("C0", "C1", … for concepts, "A0", … for aspects), with its extracted items. The form of an item depends on the request, e.g. on whether justifications or references are asked for, and on the concept, e.g. {"labels": ["Contract"]} for a label concept, whose extracted item has the list of labels as its value: the prompt of a request shows it. The answer of a chat is its text.

To test a refusal, set finish_reason to "content_filter" on the choice of the response, e.g. of the one that litellm.mock_completion builds: litellm gives this finish reason for the refusal of a Claude model and for the content filters of the providers (see 🚫 Refusals). For the refusal of a model of OpenAI, set provider_specific_fields={"refusal": "..."} on the message, with no content: litellm keeps the refusal field of the answer of OpenAI there, where ContextGem reads it, and setting refusal on the message has no effect. For a request or an answer that the provider blocks, raise litellm.ContentPolicyViolationError.

To test a chat with tools, return the tool calls of the model with mock_tool_calls, a list of tool calls as the OpenAI API gives them, and mock_response="". ContextGem then calls the tools, and sends the next request of the turn with their results, as messages with the "tool" role, to which the stand-in answers with the text:

import json

import litellm


async def acompletion_stand_in(**kwargs):
    # The first request of the turn: the model calls a tool
    if not any(message.get("role") == "tool" for message in kwargs["messages"]):
        return litellm.mock_completion(
            **kwargs,
            mock_response="",
            mock_tool_calls=[
                {
                    "id": "call_1",
                    "type": "function",
                    "function": {
                        "name": "get_order_status",
                        "arguments": json.dumps({"order_id": "ORD-1"}),
                    },
                }
            ],
        )
    # The next request, with the result of the tool: the answer
    return litellm.mock_completion(**kwargs, mock_response="The order is shipped.")

🎯 Stubbing a Decision Model#

An extraction with a stand-in for the decision model#

# Testing your own code without sending requests to the decision model:
# the stand-in replaces litellm.adecisions, which ContextGem calls for each
# request to a decision model

from unittest.mock import patch

import litellm
from litellm.types.decisions import (
    OpenAIDecisionResponse,
    OpenAIDecisionUsage,
    OpenAIPredicateAnswer,
)

from contextgem import BooleanConcept, Document, DocumentDecisionModel


async def fake_adecisions(**kwargs):
    # One answer for each question of the request, in the order of
    # the questions: here "yes", with a probability of 0.9, to each
    # yes/no question ("predicate")
    answers = tuple(
        OpenAIPredicateAnswer(name=question["name"], probability=0.9)
        for question in kwargs["questions"]
    )
    usage = OpenAIDecisionUsage(
        input_tokens=100,
        output_tokens=0,
        total_tokens=100,
    )
    return OpenAIDecisionResponse(
        model=kwargs["model"],
        answers=answers,
        usage=usage,
    )


text = "The supplier may end the agreement on 30 days' notice."
document = Document(raw_text=text)
document.add_concepts(
    [
        BooleanConcept(
            name="Supplier termination right",
            description="Whether the supplier may terminate the agreement",
        )
    ]
)

# No API key: a request that is not stubbed fails instead of being billed
decision_model = DocumentDecisionModel(model="typesafe/jev-latest")

with patch.object(litellm, "adecisions", fake_adecisions):
    decision_model.extract_concepts_from_document(document)

item = document.concepts[0].extracted_items[0]
assert item.value is True
assert item.probabilities[True] == 0.9

The stand-in receives the keyword arguments of litellm.adecisions: model, input, the text, and questions, each a dict with its name and type: "predicate" for a yes/no question, "choice" with the options in "choices", each a dict with its "value", or "score" with the levels in "levels", each a dict with its "label". It returns a litellm.types.decisions.OpenAIDecisionResponse with one answer for each question, in the order of the questions, and its usage. An answer is an OpenAIPredicateAnswer, OpenAIChoiceAnswer or OpenAIScoreAnswer of the same module, with the probabilities of the options, or an OpenAIRefusalAnswer.

🔑 No API Keys in Tests#

Run the tests with no API key, neither passed to the models nor in their environment (e.g. OPENAI_API_KEY). A request that a test does not replace then fails for the missing key, with ModelAPIError, instead of being sent and billed.

Replacing the two functions stops the requests to the models only. Other downloads are not affected: a SaT model, which segments the text into sentences when sentences are needed, is downloaded when it is first used (see 🔌 Using SaT Models Without Network Access), and litellm fetches its model data from GitHub when it is imported, unless LITELLM_LOCAL_MODEL_COST_MAP=True is set, with which it uses the copy that it is installed with. ContextGem also counts the input tokens of each request to an LLM with litellm, to check them against the context window of the model, and tiktoken, which counts them, downloads its encodings from openaipublic.blob.core.windows.net where the copies that litellm is installed with are rejected, as on Windows with litellm 1.105.2, whose copies have Windows line endings. Without the download, the check is skipped, with a message in the log, for the requests of that model until the process is restarted: each count would try the download again, which can take several seconds where the network blocks it, e.g. behind a proxy that refuses it. To count the tokens without the download, prepare the encodings in a folder of your own where the network can be reached, and keep the folder with your tests: litellm uses the folder that CUSTOM_TIKTOKEN_CACHE_DIR names, when it is set before litellm is imported.

# Download the encodings into the folder once, e.g. when the CI image is built
export CUSTOM_TIKTOKEN_CACHE_DIR=/opt/tiktoken
uv run python -c "import litellm; litellm.token_counter(model='openai/gpt-6-luna', text='Text.')"

Run the tests with the same CUSTOM_TIKTOKEN_CACHE_DIR. Count with a model of your tests in the command: a model may use another encoding.

📼 Stand-ins and Recorded Responses#

A stand-in tests your code around ContextGem: what it does with the extracted items, errors and costs. It does not depend on the text of the prompts, so an upgrade of ContextGem that changes a prompt leaves such a test working.

To check how a model answers your documents and descriptions, record its real responses once and replay them, e.g. with pytest-recording or VCR.py. A recording matches the requests, prompts included: an upgrade of ContextGem that changes a prompt makes the recordings of that prompt stale, and they are recorded again.