Serializing objects and results

Serializing objects and results#

ContextGem provides multiple serialization methods to preserve your document processing pipeline components and results. These methods enable you to save your work, transfer data between systems, or integrate with other applications.

💾 Serialization Methods#

The following ContextGem objects support serialization:

Each object supports three serialization methods:

  • to_json() - Converts the object to a JSON string for cross-platform compatibility

  • to_dict() - Converts the object to a Python dictionary for in-memory operations

  • to_disk(file_path) - Saves the object to a JSON file at the specified path, which must end with .json

🔄 Deserialization Methods#

To reconstruct objects from their serialized forms, use the corresponding class methods:

  • from_json(json_string) - Creates an object from a JSON string

  • from_dict(dict_object) - Creates an object from a Python dictionary

  • from_disk(file_path) - Loads an object from a file on disk

to_disk() and from_disk() raise the error of a file that cannot be written or read as it is, e.g. FileNotFoundError or PermissionError (subclasses of OSError). from_disk() raises the error of what the file holds as from_json() raises it, with a note that names the file: ValueError for data that is not valid JSON or fails validation (json.JSONDecodeError and pydantic’s ValidationError are subclasses of ValueError), and TypeError for the data of another class, e.g. an aspect loaded with Document.from_disk().

Note

Deserialization reconstructs the full state of the original document or pipeline: all fields, extracted data, and unique IDs are preserved, so a round-tripped object compares equal to the original (e.g. Document.from_dict(document.to_dict()) == document). The serialized data stores each reference paragraph and reference sentence of the aspects and of the extracted items on its own, while in the restored document they are the paragraphs and sentences of the document itself, i.e. the same objects as in document.paragraphs, as after an extraction (see 📍 Locating Paragraphs and Sentences). For models, see 🔑 API Credentials of Models.

🔑 API Credentials of Models#

The API credentials of a model, i.e. its api_key and api_base, are never included in the serialized data, so that they do not end up in the files, logs or other systems that the serialized data is passed to. The usage and cost statistics of a model are not included either. The other configuration parameters of the model are preserved, together with its fallback models.

A deserialized model therefore has no API credentials. Set them again before using the model, both for the model and for each of its fallback models:

Setting the API credentials of a deserialized LLM#
from contextgem import DocumentLLM

llm = DocumentLLM.from_json(llm_config_json)  # or from_dict() / from_disk()

llm.api_key = "<your-api-key>"
llm.api_base = "<your-api-base>"  # if applicable
for fallback_llm in llm.fallback_models:
    fallback_llm.api_key = "<your-fallback-api-key>"
    fallback_llm.api_base = "<your-fallback-api-base>"  # if applicable

For a deserialized DocumentModelGroup, do the same for each model in its models.

📝 Example Usage#

# Example of serializing and deserializing ContextGem document,
# extraction pipeline, and LLM config.

import os
from pathlib import Path

from contextgem import (
    Aspect,
    BooleanConcept,
    Document,
    DocumentLLM,
    ExtractionPipeline,
    StringConcept,
)


# Create a document object
text_path = (
    Path(__file__).resolve().parents[4] / "tests" / "ndas" / "en_nda_with_anomalies.txt"
)  # your file path here (Path adapted for testing)
doc = Document(raw_text=text_path.read_text(encoding="utf-8"))

# Create an extraction pipeline
extraction_pipeline = ExtractionPipeline(
    aspects=[
        Aspect(
            name="Categories of confidential information",
            description="Clauses describing confidential information covered by the NDA",
            concepts=[
                StringConcept(
                    name="Types of disclosure",
                    description="Types of disclosure of confidential information",
                ),
                # ...
            ],
        ),
        # ...
    ],
    concepts=[
        BooleanConcept(
            name="Is mutual",
            description="Whether the NDA is mutual (both parties act as discloser/recipient)",
            add_justifications=True,
        ),
        # ...
    ],
)

# Attach the pipeline to the document
doc.assign_pipeline(extraction_pipeline)

# Configure a document LLM with your API parameters
llm = DocumentLLM(
    model="openai/gpt-6-luna",
    api_key=os.getenv("CONTEXTGEM_OPENAI_API_KEY"),
)

# Extract data from the document
doc = llm.extract_all(doc)

# Serialize the LLM config, pipeline and document
llm_config_json = llm.to_json()  # or to_dict() / to_disk()
extraction_pipeline_json = extraction_pipeline.to_json()  # or to_dict() / to_disk()
processed_doc_json = doc.to_json()  # or to_dict() / to_disk()

# Deserialize the LLM config, pipeline and document
llm_deserialized = DocumentLLM.from_json(
    llm_config_json
)  # or from_dict() / from_disk()
extraction_pipeline_deserialized = ExtractionPipeline.from_json(
    extraction_pipeline_json
)  # or from_dict() / from_disk()
processed_doc_deserialized = Document.from_json(
    processed_doc_json
)  # or from_dict() / from_disk()

# All extracted data is preserved!
assert processed_doc_deserialized.aspects[0].concepts[0].extracted_items

# API credentials (API key and API base) are not included in the serialized
# LLM config, so set them again before using the deserialized LLM
llm_deserialized.api_key = os.getenv("CONTEXTGEM_OPENAI_API_KEY")

🚀 Use Cases#

  • Caching Results: Save processed documents to avoid repeating expensive model calls

  • Transfer Between Systems: Export results from one environment and import in another

  • API Integration: Convert objects to JSON for API responses

  • Workflow Persistence: Save pipeline configurations for later reuse