universal_transfer_operator.data_providers.database.sqlite

Module Contents

Classes

SqliteDataProvider

SqliteDataProvider represent all the DataProviders interactions with Sqlite Databases.
class universal_transfer_operator.data_providers.database.sqlite.SqliteDataProvider(dataset, transfer_mode, transfer_params=attr.field(factory=TransferIntegrationOptions, converter=lambda val: ...))

Bases: universal_transfer_operator.data_providers.database.base.DatabaseDataProvider

SqliteDataProvider represent all the DataProviders interactions with Sqlite Databases.

Parameters:
property sql_type: str
Return type:

str

property hook: airflow.providers.sqlite.hooks.sqlite.SqliteHook

Retrieve Airflow hook to interface with the Sqlite database.

Return type:

airflow.providers.sqlite.hooks.sqlite.SqliteHook

property sqlalchemy_engine: sqlalchemy.engine.base.Engine

Return SQAlchemy engine.

Return type:

sqlalchemy.engine.base.Engine

property default_metadata: universal_transfer_operator.datasets.table.Metadata

Since Sqlite does not use Metadata, we return an empty Metadata instances.

Return type:

universal_transfer_operator.datasets.table.Metadata

property openlineage_dataset_name: str

Returns the open lineage dataset name as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md Example: /tmp/local.db.table_name

Return type:

str

property openlineage_dataset_namespace: str

Returns the open lineage dataset namespace as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md Example: file://127.0.0.1:22

Return type:

str

property openlineage_dataset_uri: str

Returns the open lineage dataset uri as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

static get_table_qualified_name(table)

Return the table qualified name.

Parameters:

table (universal_transfer_operator.datasets.table.Table) – The table we want to retrieve the qualified name for.

Return type:

str

populate_metadata()

Since SQLite does not have a concept of databases or schemas, we just return the table as is, without any modifications.

create_schema_if_needed(schema)

Since SQLite does not have schemas, we do not need to set a schema here.

Parameters:

schema (str | None) –

Return type:

None

schema_exists(schema)

Check if a schema exists. We return false for sqlite since sqlite does not have schemas

Parameters:

schema (str) –

Return type:

bool

get_sqla_table(table)

Return SQLAlchemy table instance

Parameters:

table (universal_transfer_operator.datasets.table.Table) – Astro Table to be converted to SQLAlchemy table instance

Return type:

sqlalchemy.sql.schema.Table