universal_transfer_operator.data_providers.filesystem

Subpackages

• universal_transfer_operator.data_providers.filesystem.aws
  • universal_transfer_operator.data_providers.filesystem.aws.s3
• universal_transfer_operator.data_providers.filesystem.google
  • universal_transfer_operator.data_providers.filesystem.google.cloud
    • universal_transfer_operator.data_providers.filesystem.google.cloud.gcs

Submodules

• universal_transfer_operator.data_providers.filesystem.base
• universal_transfer_operator.data_providers.filesystem.local
• universal_transfer_operator.data_providers.filesystem.sftp

Package Contents

Functions

resolve_file_path_pattern(file[, filetype, ...])

get file objects by resolving path_pattern from local/object stores

universal_transfer_operator.data_providers.filesystem.resolve_file_path_pattern(file, filetype=None, normalize_config=None, transfer_params=None, transfer_mode=TransferMode.NONNATIVE)

get file objects by resolving path_pattern from local/object stores path_pattern can be 1. local location - glob pattern 2. s3/gcs location - prefix

Parameters:

• file (universal_transfer_operator.datasets.file.base.File) – File dataset object

• filetype (FileType | None) – constant to provide an explicit file type

• normalize_config (dict | None) – parameters in dict format of pandas json_normalize() function

• transfer_params (TransferParameters | None) – kwargs to be used by method involved in transfer flow.

• transfer_mode (universal_transfer_operator.constants.TransferMode) – Use transfer_mode TransferMode; native, non-native or thirdparty.

Return type:

list[universal_transfer_operator.datasets.file.base.File]