universal_transfer_operator.universal_transfer_operator

Module Contents

Classes

UniversalTransferOperator

Transfers all the data that could be read from the source Dataset into the destination Dataset. From a DAG author

class universal_transfer_operator.universal_transfer_operator.UniversalTransferOperator(*, source_dataset, destination_dataset, transfer_params=attr.field(factory=TransferIntegrationOptions, converter=lambda val: ...), transfer_mode=TransferMode.NONNATIVE, **kwargs)

Bases: airflow.models.BaseOperator

Transfers all the data that could be read from the source Dataset into the destination Dataset. From a DAG author standpoint, all transfers would be performed through the invocation of only the Universal Transfer Operator.

Parameters:

• source_dataset (Table | File) – Source dataset to be transferred.

• destination_dataset (Table | File) – Destination dataset to be transferred to.

• transfer_params (universal_transfer_operator.integrations.base.TransferIntegrationOptions) – kwargs to be used by method involved in transfer flow.

• transfer_mode (universal_transfer_operator.constants.TransferMode) – Use transfer_mode TransferMode; native, non-native or thirdparty.

• if_exists – Overwrite file if exists. Default False.

Returns:

returns the destination dataset

execute(context)

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Parameters:

context (airflow.utils.context.Context) –

Return type:

Any