universal_transfer_operator.datasets.table

Module Contents

Classes

Metadata

Contains additional information to access a SQL Table, which is very likely optional and, in some cases, may

Table

Repersents all Table datasets.

Attributes

MAX_TABLE_NAME_LENGTH

TEMP_PREFIX

universal_transfer_operator.datasets.table.MAX_TABLE_NAME_LENGTH = 62
universal_transfer_operator.datasets.table.TEMP_PREFIX = '_tmp'
class universal_transfer_operator.datasets.table.Metadata

Contains additional information to access a SQL Table, which is very likely optional and, in some cases, may be database-specific.

Parameters:

  • schema – A schema name

  • database – A database name

schema: str | None
database: str | None
is_empty()

Check if all the fields are None.

Return type:

bool

class universal_transfer_operator.datasets.table.Table

Bases: universal_transfer_operator.datasets.base.Dataset

Repersents all Table datasets. Intended to be used within library.

Parameters:

  • name – The name of the database table. If name not provided then it would create a temporary name

  • conn_id – The Airflow connection id. This will be used to identify the right database type at the runtime

  • metadata – A metadata object which will have database or schema name

  • columns – columns which define the database table schema.

property sqlalchemy_metadata: sqlalchemy.MetaData

Return the Sqlalchemy metadata for the given table.

Return type:

sqlalchemy.MetaData

property row_count: Any

Return the row count of table.

Return type:

Any

property sql_type: Any
Return type:

Any

name: str
conn_id: str
metadata: Metadata
columns: list[sqlalchemy.Column]
uri: str
extra: dict
abstract exists()

Check if the table exists or not

create_similar_table()

Create a new table with a unique name but with the same metadata.

Return type:

Table

to_json()
classmethod from_json(obj)
Parameters:

obj (dict) –

openlineage_dataset_name()

Returns the open lineage dataset name as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

openlineage_dataset_namespace()

Returns the open lineage dataset namespace as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

openlineage_dataset_uri()

Returns the open lineage dataset uri as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str