# Testing Without API Calls

The code that you build on ContextGem can be tested without sending a request to a model: no API key, no cost, the same result in each run, e.g. in CI. ContextGem sends each request to a model through one of two functions of `litellm`, which a test replaces with a stand-in that returns the response.

## 🔌 Where to Patch

| Function              | Sends                                                                                                                                                              |
|-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `litellm.acompletion` | Each request to an LLM ([`DocumentLLM`](https://contextgem.shcherbak.ai/api/llms/index.html.md#contextgem.public.llms.DocumentLLM)): in the extraction methods and in a chat. |
| `litellm.adecisions`  | Each request to a decision model ([`DocumentDecisionModel`](https://contextgem.shcherbak.ai/api/decision_models/index.html.md#contextgem.public.decision_models.DocumentDecisionModel)). |

ContextGem looks these functions up on the `litellm` module when it sends each request, so `unittest.mock.patch.object(litellm, "acompletion", stand_in)`, or `monkeypatch.setattr(litellm, "acompletion", stand_in)` with pytest, applies to every model, also to one created before the patch, and to the sync and async methods alike. They are the supported points for replacing the requests in tests. The stand-in is an async function, called with keyword arguments.

What ContextGem makes of the response of a stand-in is validated as a response of the model is: an answer that does not have the format that the prompt asks for is invalid data, which ContextGem asks for again up to `max_retries_invalid_data` times, and then raises [`ModelExtractionError`](https://contextgem.shcherbak.ai/api/exceptions/index.html.md#contextgem.public.exceptions.ModelExtractionError) (see [🔁 How Failed Requests Are Retried](https://contextgem.shcherbak.ai/llms/llm_config/index.html.md#llm-retries-label)). A stand-in can also raise an error of `litellm`, e.g. to test the handling of a failed request. An error without an HTTP status code, e.g. an exception of your own, or an error in the stand-in itself, is not retried, unless it is a `TimeoutError` or a `ConnectionError` of Python, e.g. to simulate a failure of the network: the next fallback model is tried, and the error is the `original_error` of [`ModelAPIError`](https://contextgem.shcherbak.ai/api/exceptions/index.html.md#contextgem.public.exceptions.ModelAPIError) (see [🔁 How Failed Requests Are Retried](https://contextgem.shcherbak.ai/llms/llm_config/index.html.md#llm-retries-label)).

## 🤖 Stubbing an LLM

```python
# Testing your own code without sending requests to the LLM: the stand-in
# replaces litellm.acompletion, which ContextGem calls for each request
# to an LLM

from unittest.mock import patch

import litellm

from contextgem import Document, DocumentLLM, StringConcept


# The code to test: gets the number of an invoice from its text
def get_invoice_number(text: str, llm: DocumentLLM) -> str | None:
    document = Document(raw_text=text)
    document.add_concepts(
        [
            StringConcept(
                name="Invoice number",
                description="The number of the invoice",
                singular_occurrence=True,
            )
        ]
    )
    llm.extract_concepts_from_document(document)
    items = document.concepts[0].extracted_items
    return items[0].value if items else None


# The answer of the LLM, in the format that the extraction prompt asks for:
# a JSON array with an entry for each concept, by its ID in the prompt ("C0")
answer = '[{"concept_id": "C0", "extracted_items": ["INV-001"]}]'

# The requests that ContextGem sends, e.g. to check their prompts
requests = []


async def fake_acompletion(**kwargs):
    requests.append(kwargs)
    # litellm builds a response with this answer, and sends no request whatever
    # it is given: the real litellm.acompletion is not called
    return litellm.mock_completion(**kwargs, mock_response=answer)


# No API key, neither here nor in the environment of the tests: a request that
# is not stubbed then fails instead of being billed
llm = DocumentLLM(model="openai/gpt-6-luna")

with patch.object(litellm, "acompletion", fake_acompletion):
    invoice_number = get_invoice_number(
        "Invoice INV-001 is due on 30 November 2026.", llm
    )

assert invoice_number == "INV-001"
assert len(requests) == 1
assert "Invoice INV-001" in requests[0]["messages"][-1]["content"]
```

The stand-in receives the keyword arguments of `litellm.acompletion`, among them `model` and `messages`, whose last message has the prompt, and returns a `litellm.ModelResponse`. ContextGem reads from it:

- the answer: the `message.content` of the last choice that has a text, or of the first choice;
- in a chat, the tool calls of the choices (`message.tool_calls`);
- the `finish_reason` of the choices;
- the token counts of `usage`: `prompt_tokens`, `completion_tokens` and, where present, the cached tokens of `prompt_tokens_details`, which the costs are calculated from.

`litellm.mock_completion`, given the keyword arguments of the request and `mock_response`, builds such a response, and sends no request whatever it is given: the answer as its text, `finish_reason="stop"`, and a usage of 10 prompt tokens and 20 completion tokens whatever the request (`litellm` 1.105.2). `mock_response=""` gives an empty answer. Do not call the real `litellm.acompletion` from a stand-in, also not with `mock_response`: it takes `mock_response=""` as no mock, and sends the request.

The answer of an extraction is the JSON that the prompt asks for in its answer format: an array with an entry for each concept, or aspect, of the request, by its ID in the prompt (`"C0"`, `"C1"`, … for concepts, `"A0"`, … for aspects), with its extracted items. The form of an item depends on the request, e.g. on whether justifications or references are asked for, and on the concept, e.g. `{"labels": ["Contract"]}` for a label concept, whose extracted item has the list of labels as its value: the prompt of a request shows it. The answer of a chat is its text.

To test a refusal, set `finish_reason` to `"content_filter"` on the choice of the response, e.g. of the one that `litellm.mock_completion` builds: `litellm` gives this finish reason for the refusal of a Claude model and for the content filters of the providers (see [🚫 Refusals](https://contextgem.shcherbak.ai/llms/llm_config/index.html.md#llm-refusals-label)). For the refusal of a model of OpenAI, set `provider_specific_fields={"refusal": "..."}` on the message, with no `content`: `litellm` keeps the `refusal` field of the answer of OpenAI there, where ContextGem reads it, and setting `refusal` on the message has no effect. For a request or an answer that the provider blocks, raise `litellm.ContentPolicyViolationError`.

To test a chat with tools, return the tool calls of the model with `mock_tool_calls`, a list of tool calls as the OpenAI API gives them, and `mock_response=""`. ContextGem then calls the tools, and sends the next request of the turn with their results, as messages with the `"tool"` role, to which the stand-in answers with the text:

```python
import json

import litellm


async def acompletion_stand_in(**kwargs):
    # The first request of the turn: the model calls a tool
    if not any(message.get("role") == "tool" for message in kwargs["messages"]):
        return litellm.mock_completion(
            **kwargs,
            mock_response="",
            mock_tool_calls=[
                {
                    "id": "call_1",
                    "type": "function",
                    "function": {
                        "name": "get_order_status",
                        "arguments": json.dumps({"order_id": "ORD-1"}),
                    },
                }
            ],
        )
    # The next request, with the result of the tool: the answer
    return litellm.mock_completion(**kwargs, mock_response="The order is shipped.")
```

## 🎯 Stubbing a Decision Model

```python
# Testing your own code without sending requests to the decision model:
# the stand-in replaces litellm.adecisions, which ContextGem calls for each
# request to a decision model

from unittest.mock import patch

import litellm
from litellm.types.decisions import (
    OpenAIDecisionResponse,
    OpenAIDecisionUsage,
    OpenAIPredicateAnswer,
)

from contextgem import BooleanConcept, Document, DocumentDecisionModel


async def fake_adecisions(**kwargs):
    # One answer for each question of the request, in the order of
    # the questions: here "yes", with a probability of 0.9, to each
    # yes/no question ("predicate")
    answers = tuple(
        OpenAIPredicateAnswer(name=question["name"], probability=0.9)
        for question in kwargs["questions"]
    )
    usage = OpenAIDecisionUsage(
        input_tokens=100,
        output_tokens=0,
        total_tokens=100,
    )
    return OpenAIDecisionResponse(
        model=kwargs["model"],
        answers=answers,
        usage=usage,
    )


text = "The supplier may end the agreement on 30 days' notice."
document = Document(raw_text=text)
document.add_concepts(
    [
        BooleanConcept(
            name="Supplier termination right",
            description="Whether the supplier may terminate the agreement",
        )
    ]
)

# No API key: a request that is not stubbed fails instead of being billed
decision_model = DocumentDecisionModel(model="typesafe/jev-latest")

with patch.object(litellm, "adecisions", fake_adecisions):
    decision_model.extract_concepts_from_document(document)

item = document.concepts[0].extracted_items[0]
assert item.value is True
assert item.probabilities[True] == 0.9
```

The stand-in receives the keyword arguments of `litellm.adecisions`: `model`, `input`, the text, and `questions`, each a dict with its `name` and `type`: `"predicate"` for a yes/no question, `"choice"` with the options in `"choices"`, each a dict with its `"value"`, or `"score"` with the levels in `"levels"`, each a dict with its `"label"`. It returns a `litellm.types.decisions.OpenAIDecisionResponse` with one answer for each question, in the order of the questions, and its `usage`. An answer is an `OpenAIPredicateAnswer`, `OpenAIChoiceAnswer` or `OpenAIScoreAnswer` of the same module, with the probabilities of the options, or an `OpenAIRefusalAnswer`.

## 🔑 No API Keys in Tests

Run the tests with no API key, neither passed to the models nor in their environment (e.g. `OPENAI_API_KEY`). A request that a test does not replace then fails for the missing key, with [`ModelAPIError`](https://contextgem.shcherbak.ai/api/exceptions/index.html.md#contextgem.public.exceptions.ModelAPIError), instead of being sent and billed.

Replacing the two functions stops the requests to the models only. Other downloads are not affected: a SaT model, which segments the text into sentences when sentences are needed, is downloaded when it is first used (see [🔌 Using SaT Models Without Network Access](https://contextgem.shcherbak.ai/documents/document_config/index.html.md#sat-models-offline)), and `litellm` fetches its model data from GitHub when it is imported, unless `LITELLM_LOCAL_MODEL_COST_MAP=True` is set, with which it uses the copy that it is installed with. ContextGem also counts the input tokens of each request to an LLM with `litellm`, to check them against the context window of the model, and `tiktoken`, which counts them, downloads its encodings from `openaipublic.blob.core.windows.net` where the copies that `litellm` is installed with are rejected, as on Windows with `litellm` 1.105.2, whose copies have Windows line endings. Without the download, the check is skipped, with a message in the log, for the requests of that model until the process is restarted: each count would try the download again, which can take several seconds where the network blocks it, e.g. behind a proxy that refuses it. To count the tokens without the download, prepare the encodings in a folder of your own where the network can be reached, and keep the folder with your tests: `litellm` uses the folder that `CUSTOM_TIKTOKEN_CACHE_DIR` names, when it is set before `litellm` is imported.

```bash
# Download the encodings into the folder once, e.g. when the CI image is built
export CUSTOM_TIKTOKEN_CACHE_DIR=/opt/tiktoken
uv run python -c "import litellm; litellm.token_counter(model='openai/gpt-6-luna', text='Text.')"
```

Run the tests with the same `CUSTOM_TIKTOKEN_CACHE_DIR`. Count with a model of your tests in the command: a model may use another encoding.

## 📼 Stand-ins and Recorded Responses

A stand-in tests your code around ContextGem: what it does with the extracted items, errors and costs. It does not depend on the text of the prompts, so an upgrade of ContextGem that changes a prompt leaves such a test working.

To check how a model answers your documents and descriptions, record its real responses once and replay them, e.g. with [pytest-recording](https://github.com/kiwicom/pytest-recording) or [VCR.py](https://github.com/kevin1024/vcrpy). A recording matches the requests, prompts included: an upgrade of ContextGem that changes a prompt makes the recordings of that prompt stale, and they are recorded again.