Chunk

The Chunk model is closely related to the File model. It is used to store the actual data of a file in chunks. This is done to allow the Large Language Models to process the data in smaller sections.

The embedding field is used to store the text embedding of the chunk, which is crucial to the vector search functionality.

Each chunk references the File it belongs to using the parent_file_uuid field.

uuid = Field(default_factory=uuid4)

created_datetime = Field(default_factory=utcnow)

creator_user_uuid

model_type

parent_file_uuid = Field(description='id of the original file which this text came from')

index = Field(description='relative position of this chunk in the original file')

text = Field(description='chunk of the original text')

metadata = Field(description='subset of the unstructured Element.Metadata object', default=None)

embedding = Field(description='the vector representation of the text', default=None)

text_hash

token_count

ChunkStatus

The Chunk model also has a companion ChunkStatus model that helps track the status of the chunk processing. This includes information about the embedding process.

chunk_uuid

embedded