# Pipelines

Module for handling document processing pipelines.

This module provides the ExtractionPipeline class, which represents a reusable collection
of pre-defined aspects and concepts that can be assigned to documents. Pipelines enable
standardized document analysis by packaging common extraction patterns into reusable units.

Pipelines serve as templates for document processing, allowing consistent application of
the same analysis approach across multiple documents. They encapsulate both the structural
organization (aspects) and the specific information to extract (concepts) in a single,
assignable object.

### *class* contextgem.public.pipelines.ExtractionPipeline(\*\*data)

Bases: `_ExtractionPipeline`

Represents a reusable collection of predefined aspects and concepts for document analysis.

Extraction pipelines serve as templates that can be assigned to multiple documents,
ensuring consistent application of the same analysis criteria. They package common
extraction patterns into reusable units, allowing for standardized document processing.

* **Variables:**
  * **aspects** ([*list*](https://docs.python.org/3/builtins/stdtypes.html#list) *[* *\_Aspect* *]*) – A list of aspects to extract from documents. Aspects represent structural
    categories of information. Defaults to an empty list.
  * **concepts** ([*list*](https://docs.python.org/3/builtins/stdtypes.html#list) *[* *\_Concept* *]*) – A list of concepts to identify within documents. Concepts represent
    specific information elements to extract. Defaults to an empty list.
* **Parameters:**
  * **custom_data** ([*Annotated*](https://docs.python.org/3/library/typing.html#typing.Annotated) *[*[*dict*](https://docs.python.org/3/builtins/stdtypes.html#dict) *[*[*str*](https://docs.python.org/3/builtins/stdtypes.html#str) *,* [*Any*](https://docs.python.org/3/library/typing.html#typing.Any) *]* *,* [*BeforeValidator*](https://pydantic.dev/docs/validation/latest/api/pydantic/functional_validators/#pydantic.functional_validators.BeforeValidator) *(**func=~contextgem.internal.typings.validators._validate_is_json_dict* *,* *json_schema_input_type=PydanticUndefined* *)* *]*)
  * **aspects** ([*Annotated*](https://docs.python.org/3/library/typing.html#typing.Annotated) *[*[*Sequence*](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence) *[* *\_Aspect* *]* *,* [*BeforeValidator*](https://pydantic.dev/docs/validation/latest/api/pydantic/functional_validators/#pydantic.functional_validators.BeforeValidator) *(**func=~contextgem.internal.typings.validators._validate_sequence_is_list* *,* *json_schema_input_type=PydanticUndefined* *)* *]*)
  * **concepts** ([*Annotated*](https://docs.python.org/3/library/typing.html#typing.Annotated) *[*[*Sequence*](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence) *[* *\_Concept* *]* *,* [*BeforeValidator*](https://pydantic.dev/docs/validation/latest/api/pydantic/functional_validators/#pydantic.functional_validators.BeforeValidator) *(**func=~contextgem.internal.typings.validators._validate_sequence_is_list* *,* *json_schema_input_type=PydanticUndefined* *)* *]*)

Note:
: A pipeline is a reusable configuration of extraction steps. You can use the same pipeline
  to extract data from multiple documents.

Example:
: ```python
  from contextgem import (
      Aspect,
      BooleanConcept,
      DateConcept,
      Document,
      ExtractionPipeline,
      StringConcept,
  )
  <br/>
  <br/>
  # Create a pipeline for NDA (Non-Disclosure Agreement) review
  nda_pipeline = ExtractionPipeline(
      aspects=[
          Aspect(
              name="Confidential information",
              description="Clauses defining the confidential information",
          ),
          Aspect(
              name="Exclusions",
              description="Clauses defining exclusions from confidential information",
          ),
          Aspect(
              name="Obligations",
              description="Clauses defining confidentiality obligations",
          ),
          Aspect(
              name="Liability",
              description="Clauses defining liability for breach of the agreement",
          ),
          # ... Add more aspects as needed
      ],
      concepts=[
          StringConcept(
              name="Anomaly",
              description="Anomaly in the contract, e.g. out-of-context or nonsensical clauses",
              model_role="reasoner",  # Route this concept to the model with the "reasoner" role
              add_references=True,  # Add references to the source text
              reference_depth="sentences",  # Reference to the sentence level
              add_justifications=True,  # Add justifications for the anomaly
              justification_depth="balanced",  # Justification with a balanced level of detail
              justification_max_sents=5,  # Maximum number of sentences in the justification
          ),
          BooleanConcept(
              name="Is mutual",
              description="Whether the NDA is mutual (bidirectional) or one-way",
              singular_occurrence=True,
              model_role="reasoner",  # Route this concept to the model with the "reasoner" role
          ),
          DateConcept(
              name="Effective date",
              description="The date when the NDA agreement becomes effective",
              singular_occurrence=True,
          ),
          StringConcept(
              name="Term",
              description="The term of the NDA",
          ),
          StringConcept(
              name="Governing law",
              description="The governing law of the agreement",
              singular_occurrence=True,
          ),
          # ... Add more concepts as needed
      ],
  )
  <br/>
  # Assign the pipeline to the NDA document
  nda_document = Document(raw_text="[NDA text]")
  nda_document.assign_pipeline(nda_pipeline)
  <br/>
  # Now the document is ready for processing with the NDA review pipeline!
  # The document can be processed to extract the defined aspects and concepts
  <br/>
  # Extract all aspects and concepts from the NDA using a model group that has
  # a model with the role "reasoner" and a model without a role (for all other tasks).
  # model_group.extract_all(nda_document)
  ```

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be
validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

#### add_aspects(aspects)

Adds aspects to the existing aspects list of an instance and returns the
updated instance. This method ensures that the provided aspects are deeply
copied to avoid any unintended state modification of the original reusable
aspects.

* **Parameters:**
  **aspects** ([`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[`contextgem.internal.base.aspects._Aspect`]) – A list of aspects to be added. Each aspect is deeply copied
  to ensure the original list remains unaltered.
* **Returns:**
  Updated instance containing the newly added aspects.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)

#### add_concepts(concepts)

Adds a list of new concepts to the existing concepts attribute of the instance. This method ensures
that the provided list of concepts is deep-copied to prevent unintended side effects from modifying
the input list outside of this method.

* **Parameters:**
  **concepts** ([`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[`contextgem.internal.base.concepts._Concept`]) – A list of concepts to be added. It will be deep-copied before being added
  to the instance’s concepts attribute.
* **Returns:**
  Returns the instance itself after the modification.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)

#### clone()

Creates and returns a deep copy of the current instance.

* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)
* **Returns:**
  A deep copy of the current instance.

#### *classmethod* from_dict(obj_dict)

Reconstructs an instance of the class from a dictionary representation.

This method deserializes a dictionary containing the object’s attributes and values
into a new instance of the class. It handles complex nested structures like aspects,
concepts, and extracted items, properly reconstructing each component.

A deserialized model or model group has no API credentials, as they are not
included in the serialized data, i.e. they are to be set again.

* **Parameters:**
  **obj_dict** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`typing.Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Dictionary containing the serialized object data.
* **Returns:**
  A new instance of the class with restored attributes.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)

#### *classmethod* from_disk(file_path)

Loads an instance of the class from a JSON file stored on disk.

This method reads the JSON content from the specified file path and
deserializes it into an instance of the class using the from_json
method.

* **Parameters:**
  **file_path** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path)) – Path to the JSON file to load (must end with ‘.json’).
  Can be a string or a Path object.
* **Returns:**
  An instance of the class populated with the data from the file.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)
* **Raises:**
  * [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – If the file path doesn’t end with ‘.json’, or if the file
    does not hold valid serialized data, e.g. it is not a valid UTF-8 text
    (`UnicodeDecodeError`) or JSON (`json.JSONDecodeError`), or the data
    fails validation (pydantic’s `ValidationError`). These errors are
    subclasses of `ValueError`.
  * [**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – If the serialized data is of another class.
  * [**OSError**](https://docs.python.org/3/builtins/exceptions.html#OSError) – If the file cannot be read, e.g. it does not exist.
    The error is the one raised by the file operation, e.g.
    `FileNotFoundError` or `PermissionError`.

#### *classmethod* from_json(json_string)

Creates an instance of the class from a JSON string representation.

This method deserializes the provided JSON string into a dictionary and uses
the from_dict method to construct the class instance. It validates that the
class name in the serialized data matches the current class.

* **Parameters:**
  **json_string** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – JSON string containing the serialized object data.
* **Returns:**
  A new instance of the class with restored state.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)
* **Raises:**
  * [**json.JSONDecodeError**](https://docs.python.org/3/library/json.html#json.JSONDecodeError) – If the string is not a valid JSON. The error
    gives the position of what is invalid, and does not hold the string.
  * [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – If the JSON value is not an object with a class name,
    i.e. not serialized data.
  * [**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – If the class name in the serialized data doesn’t match.

#### get_aspect_by_name(name)

Finds and returns an aspect with the specified name from the list of available aspects,
if the instance has aspects attribute.

* **Parameters:**
  **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the aspect to find.
* **Returns:**
  The aspect with the specified name.
* **Return type:**
  `contextgem.internal.base.aspects._Aspect`
* **Raises:**
  [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – If no aspect with the specified name is found.

#### get_aspects_by_names(names)

Retrieve a list of \_Aspect objects corresponding to the provided list of names.

* **Parameters:**
  **names** ([`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – List of aspect names to retrieve. The names must be provided
  as a list of strings.
* **Returns:**
  A list of \_Aspect objects corresponding to provided names.
* **Return type:**
  [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[`contextgem.internal.base.aspects._Aspect`]

#### get_concept_by_name(name)

Retrieves a concept from the list of concepts based on the provided name,
if the instance has concepts attribute.

* **Parameters:**
  **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the concept to search for.
* **Returns:**
  The \_Concept object with the specified name.
* **Return type:**
  `contextgem.internal.base.concepts._Concept`
* **Raises:**
  [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – If no concept with the specified name is found.

#### get_concepts_by_names(names)

Retrieve a list of \_Concept objects corresponding to the provided list of names.

* **Parameters:**
  **names** ([`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – List of concept names to retrieve. The names must be provided
  as a list of strings.
* **Returns:**
  A list of \_Concept objects corresponding to provided names.
* **Return type:**
  [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[`contextgem.internal.base.concepts._Concept`]

#### remove_all_aspects()

Removes all aspects from the instance and returns the updated instance.

This method clears the aspects attribute of the instance by resetting it to
an empty list. It returns the same instance, allowing for method chaining.

* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)
* **Returns:**
  The updated instance with all aspects removed

#### remove_all_concepts()

Removes all concepts from the instance and returns the updated instance.

This method clears the concepts attribute of the instance by resetting it to
an empty list. It returns the same instance, allowing for method chaining.

* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)
* **Returns:**
  The updated instance with all concepts removed

#### remove_all_instances()

Removes all assigned instances from the object and resets
them as empty lists. Returns the modified instance.

* **Returns:**
  The modified object with all assigned instances removed.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)

#### remove_aspect_by_name(name)

Removes an aspect from the assigned aspects by its name.

* **Parameters:**
  **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the aspect to be removed
* **Returns:**
  Updated instance with the aspect removed.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)

#### remove_aspects_by_names(names)

Removes multiple aspects from an object based on the provided list of names.

* **Parameters:**
  **names** ([`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – A list of names identifying the aspects to be removed.
* **Returns:**
  The updated object after the specified aspects have been removed.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)

#### remove_concept_by_name(name)

Removes a concept from the assigned concepts by its name.

* **Parameters:**
  **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The name of the concept to be removed
* **Returns:**
  Updated instance with the concept removed.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)

#### remove_concepts_by_names(names)

Removes concepts from the object by their names.

* **Parameters:**
  **names** ([`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – A list of concept names to be removed.
* **Returns:**
  Returns the updated instance after removing the specified concepts.
* **Return type:**
  [`typing.Self`](https://docs.python.org/3/library/typing.html#typing.Self)

#### to_dict()

Transforms the current object into a dictionary representation.

Converts the object to a dictionary that includes:
- All public attributes
- Special handling for specific public and private attributes

When a model or model group is serialized, its API credentials and usage/cost stats are removed.

* **Returns:**
  A dictionary representation of the current object with all necessary data for serialization
* **Return type:**
  [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`typing.Any`](https://docs.python.org/3/library/typing.html#typing.Any)]

#### to_disk(file_path)

Saves the serialized instance to a JSON file at the specified path.

This method converts the instance to a dictionary representation using to_dict(),
then writes it to disk as a formatted JSON file with UTF-8 encoding.

* **Parameters:**
  **file_path** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path)) – Path where the JSON file should be saved (must end with ‘.json’).
  Can be a string or a Path object.
* **Return type:**
  [`None`](https://docs.python.org/3/builtins/constants.html#None)
* **Returns:**
  None
* **Raises:**
  * [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – If the file path doesn’t end with ‘.json’.
  * [**OSError**](https://docs.python.org/3/builtins/exceptions.html#OSError) – If the file cannot be written, e.g. its directory does not
    exist. The error is the one raised by the file operation, e.g.
    `FileNotFoundError` or `PermissionError`.

#### to_json()

Converts the object to its JSON string representation.

Serializes the object into a JSON-formatted string using the dictionary
representation provided by the to_dict() method.

* **Returns:**
  A JSON string representation of the object.
* **Return type:**
  [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)

#### *property* unique_id *: [str](https://docs.python.org/3/builtins/stdtypes.html#str)*

Returns the ULID of the instance.

#### aspects *: [Annotated](https://docs.python.org/3/library/typing.html#typing.Annotated)[[Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[\_Aspect], [BeforeValidator](https://pydantic.dev/docs/validation/latest/api/pydantic/functional_validators/#pydantic.functional_validators.BeforeValidator)(func=\_validate_sequence_is_list, json_schema_input_type=PydanticUndefined)]*

#### concepts *: [Annotated](https://docs.python.org/3/library/typing.html#typing.Annotated)[[Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[\_Concept], [BeforeValidator](https://pydantic.dev/docs/validation/latest/api/pydantic/functional_validators/#pydantic.functional_validators.BeforeValidator)(func=\_validate_sequence_is_list, json_schema_input_type=PydanticUndefined)]*

#### custom_data *: [Annotated](https://docs.python.org/3/library/typing.html#typing.Annotated)[[dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)], [BeforeValidator](https://pydantic.dev/docs/validation/latest/api/pydantic/functional_validators/#pydantic.functional_validators.BeforeValidator)(func=\_validate_is_json_dict, json_schema_input_type=PydanticUndefined)]*