universal_transfer_operator.data_providers.filesystem.local

Module Contents

Classes

LocalDataProvider

Handler Local file path operations

class universal_transfer_operator.data_providers.filesystem.local.LocalDataProvider(dataset, transfer_mode, transfer_params=attr.field(factory=TransferIntegrationOptions, converter=lambda val: ...))

Bases: universal_transfer_operator.data_providers.filesystem.base.BaseFilesystemProviders

Handler Local file path operations

Parameters:

• dataset (universal_transfer_operator.datasets.file.base.File) –

• transfer_params (universal_transfer_operator.universal_transfer_operator.TransferIntegrationOptions) –

property paths: list[str]

Resolve local filepath

Return type:

list[str]

property size: int

Return the size in bytes of the given file. :return: File size in bytes

Return type:

int

property openlineage_dataset_namespace: str

Returns the open lineage dataset namespace as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

property openlineage_dataset_name: str

Returns the open lineage dataset name as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md

Return type:

str

abstract property hook: airflow.hooks.base.BaseHook

Return an instance of the Airflow hook.

Return type:

airflow.hooks.base.BaseHook

validate_conn()

Override as conn_id is not always required for local location.

delete(path=None)

Delete a file/object if they exists

Parameters:

path (str | None) –

check_if_exists(path=None)

Return true if the dataset exists

Parameters:

path (str | None) –

Return type:

bool

write_using_smart_open(source_ref)

Write the source data from remote object i/o buffer to the dataset using smart open

Parameters:

source_ref (DataStream | pd.DataFrame) –

Return type:

str