universal_transfer_operator.integrations.fivetran

Module Contents

Classes

Group

Fivetran group details.

Destination

Fivetran destination details

Connector

Fivetran connector details

FiveTranOptions

FiveTran load options.

FivetranIntegration

Fivetran integration to transfer datasets using Fivetran APIs.
class universal_transfer_operator.integrations.fivetran.Group

Fivetran group details.

Parameters:

  • name – The name of the group within Fivetran account.

  • group_id – Group id in fivetran system

name: str
group_id: str | None
to_dict()

Convert options class to dict

Return type:

dict

class universal_transfer_operator.integrations.fivetran.Destination

Fivetran destination details

Parameters:

  • service – services as per https://fivetran.com/docs/rest-api/destinations/config

  • config – Configuration as per destination specified at https://fivetran.com/docs/rest-api/destinations/config

  • destination_id – The unique identifier for the destination within the Fivetran system

  • time_zone_offset – Determines the time zone for the Fivetran sync schedule.

  • region – Data processing location. This is where Fivetran will operate and run computation on data.

  • run_setup_tests – Specifies whether setup tests should be run automatically.

service: str
config: dict
destination_id: str | None
time_zone_offset: str | None = '-5'
region: str | None = 'GCP_US_EAST4'
run_setup_tests: bool | None = True
to_dict()

Convert options class to dict

Return type:

dict

class universal_transfer_operator.integrations.fivetran.Connector

Fivetran connector details

Parameters:

  • connector_id – The unique identifier for the connector within the Fivetran system

  • service – services as per https://fivetran.com/docs/rest-api/destinations/config

  • config – Configuration as per destination specified at https://fivetran.com/docs/rest-api/connectors/config

  • paused – Specifies whether the connector is paused. Defaults to True.

  • pause_after_trial – Specifies whether the connector should be paused after the free trial period has ended. Defaults to True.

  • sync_frequency – The connector sync frequency in minutes Enum: “5” “15” “30” “60” “120” “180” “360” “480” “720” “1440”. Default to “5”

  • daily_sync_time – Defines the sync start time when the sync frequency is already set or being set by the current request to 1440.

  • schedule_type – Define the schedule type

  • connect_card_config – Connector card configuration

  • trust_certificates – Specifies whether we should trust the certificate automatically. The default value is FALSE. If a certificate is not trusted automatically,

  • trust_fingerprints – Specifies whether we should trust the SSH fingerprint automatically. The default value is FALSE.

  • run_setup_tests – Specifies whether the setup tests should be run automatically. The default value is TRUE.

connector_id: str | None
service: str
config: dict
connect_card_config: dict
paused: bool = True
pause_after_trial: bool = True
sync_frequency: str = '5'
daily_sync_time: str = '00:00'
schedule_type: str = ''
trust_certificates: bool = False
trust_fingerprints: bool = False
run_setup_tests: bool = True
to_dict()

Convert options class to dict

Return type:

dict

class universal_transfer_operator.integrations.fivetran.FiveTranOptions

Bases: universal_transfer_operator.integrations.base.TransferIntegrationOptions

FiveTran load options.

Parameters:

  • conn_id – Connection id of Fivetran

  • connector_id – The unique identifier for the connector within the Fivetran system

  • retry_limit – Retry limit. Defaults to 3

  • retry_delay – Retry delay

  • poll_interval – Polling interval. Defaults to 15 seconds

  • schedule_type – Define the schedule type

  • connector – Connector in FiveTran

  • group – Group in FiveTran

  • destination – Destination in Fivetran

conn_id: str
connector_id: str | None
retry_limit: int = 3
retry_delay: int = 1
poll_interval: int = 15
schedule_type: str = 'manual'
connector: Connector | None
group: Group | None
destination: Destination | None
to_dict()

Convert options class to dict

Return type:

dict

class universal_transfer_operator.integrations.fivetran.FivetranIntegration(transfer_params=attr.field(factory=FiveTranOptions, converter=lambda val: ...))

Bases: universal_transfer_operator.integrations.base.TransferIntegration

Fivetran integration to transfer datasets using Fivetran APIs.

Parameters:

transfer_params (FiveTranOptions) –

api_user_agent = 'airflow_provider_fivetran/1.1.3'
api_protocol = 'https'
api_host = 'api.fivetran.com'
api_path_connectors = 'v1/connectors/'
api_path_groups = 'v1/groups/'
api_path_destinations = 'v1/destinations/'
hook()

Return an instance of the database-specific Airflow hook.

Return type:

fivetran_provider.hooks.fivetran.FivetranHook

transfer_job(source_dataset, destination_dataset)

Loads data from source dataset to the destination using ingestion config

Parameters:

  • source_dataset (universal_transfer_operator.datasets.base.Dataset) – Source dataset

  • destination_dataset (universal_transfer_operator.datasets.base.Dataset) – Destination dataset

Return type:

Any

check_for_connector_id(fivetran_hook)

Ensures connector configuration has been completed successfully and is in a functional state.

Parameters:

fivetran_hook (fivetran_provider.hooks.fivetran.FivetranHook) – Fivetran hook

Return type:

bool

check_group_details(fivetran_hook, group_id)

Check if group_id is exists.

Parameters:

  • fivetran_hook (fivetran_provider.hooks.fivetran.FivetranHook) – Fivetran hook

  • group_id (str | None) – Group id in fivetran system

Return type:

bool

create_group(fivetran_hook)

Creates the group based on group name passed

Parameters:

fivetran_hook (fivetran_provider.hooks.fivetran.FivetranHook) – Fivetran hook

Return type:

str

check_destination_details(fivetran_hook, destination_id)

Check if destination_id is exists.

Parameters:

  • fivetran_hook (fivetran_provider.hooks.fivetran.FivetranHook) – Fivetran hook

  • destination_id (str | None) – The unique identifier for the destination within the Fivetran system

Return type:

bool

create_destination(fivetran_hook, group_id)

Creates the destination based on destination configuration passed

Parameters:

  • fivetran_hook (fivetran_provider.hooks.fivetran.FivetranHook) – Fivetran hook

  • group_id (str) – Group id in fivetran system

Return type:

dict

create_connector(fivetran_hook, group_id)

Creates the connector based on connector configuration passed

Parameters:

  • fivetran_hook (fivetran_provider.hooks.fivetran.FivetranHook) – Fivetran hook

  • group_id (str) – Group id in fivetran system

Return type:

str

run_connector_setup_tests(fivetran_hook, connector_id)

Runs the setup tests for an existing connector within your Fivetran account.

Parameters:

  • fivetran_hook (fivetran_provider.hooks.fivetran.FivetranHook) – Fivetran hook

  • connector_id (str) – The unique identifier for the connector within the Fivetran system