dlt.destinations.impl.lance.utils

create_in_filter

def create_in_filter(field_name: str, array: Union[pa.Array,
                                                   pa.ChunkedArray]) -> str

View source on GitHub

Filters all rows where field_name is one of the distinct values in the array.

set_remove_orphans_hint

def set_remove_orphans_hint(
        table: PreparedTableSchema,
        schema_tables: TSchemaTables) -> PreparedTableSchema

View source on GitHub

Resolves REMOVE_ORPHANS_HINT on table, inheriting it from the parent table chain and otherwise following the merge key, which names the document a reload replaces.

verify_lance_tables

def verify_lance_tables(loaded_tables: Sequence[PreparedTableSchema],
                        has_embeddings: bool,
                        destination_name: str = "lance") -> None

View source on GitHub

Raises if a table cannot be loaded into a lance format destination.

get_orphan_scope_key_col

def get_orphan_scope_key_col(load_table: PreparedTableSchema,
                             dataset_name: str) -> str

View source on GitHub

Returns the column identifying the documents a load owns in load_table: the root key for a nested table, the canonical doc id merge key for a root table.

build_orphan_scope_filter

def build_orphan_scope_filter(load_table: PreparedTableSchema,
                              file_paths: Sequence[str],
                              dataset_name: str) -> str

View source on GitHub

Builds a SQL filter scoping orphan deletion to the documents present in file_paths.

file_paths must be all job files of load_table: a subset scopes the delete to itself and removes what the remaining files wrote.