Google Cloud storage

Transfer to Google Cloud storage as destination dataset

User can transfer data to Google cloud storage as destination as from following sources dataset:

1. Tables

       SQLITE = "sqlite"
       BIGQUERY = "bigquery"
       SNOWFLAKE = "snowflake"
   

2. Files

       LOCAL = "local"
       GS = "gs"  # Google Cloud Storage
       S3 = "s3"  # Amazon S3
       SFTP = "sftp"
   

Following transfer modes are supported:

1. Non-native transfer

   Following is an example of non-native transfers between AWS S3 and Google cloud storage using non-native transfer:

       transfer_non_native_s3_to_gs = UniversalTransferOperator(
           task_id="transfer_non_native_s3_to_gs",
           # [START dataset_individual_file]
           source_dataset=File(path=f"{s3_bucket}/example_uto/", conn_id="aws_default"),
           # [END dataset_individual_file]
           destination_dataset=File(
               path=f"{gcs_bucket}/example_uto/",
               conn_id="google_cloud_default",
           ),
       )
   

Examples

1. S3 to GCS transfers

   • Non-native transfer

     Following is an example of non-native transfers between AWS S3 to Google cloud storage using non-native transfer:

         transfer_non_native_s3_to_gs = UniversalTransferOperator(
             task_id="transfer_non_native_s3_to_gs",
             # [START dataset_individual_file]
             source_dataset=File(path=f"{s3_bucket}/example_uto/", conn_id="aws_default"),
             # [END dataset_individual_file]
             destination_dataset=File(
                 path=f"{gcs_bucket}/example_uto/",
                 conn_id="google_cloud_default",
             ),
         )
     

Transfer from Google Cloud storage as source dataset

User can transfer data from Google cloud storage to the following destination dataset:

1. Tables

       SQLITE = "sqlite"
       BIGQUERY = "bigquery"
       SNOWFLAKE = "snowflake"
   

2. Files

       LOCAL = "local"
       GS = "gs"  # Google Cloud Storage
       S3 = "s3"  # Amazon S3
       SFTP = "sftp"
   

Following transfer modes are supported:

1. Non-native transfer

   Following is an example of non-native transfers between Google cloud storage and AWS S3 using non-native transfer:

       transfer_non_native_gs_to_s3 = UniversalTransferOperator(
           task_id="transfer_non_native_gs_to_s3",
           source_dataset=input_file,
           destination_dataset=File(path=f"{s3_bucket}/example_uto/", conn_id="aws_default"),
       )
   

2. Transfer using third-party tool

Examples

1. GCS to AWS S3 transfers

   • Non-native transfer

     Following is an example of non-native transfers between Google cloud storage to AWS S3 using non-native transfer:

         transfer_non_native_gs_to_s3 = UniversalTransferOperator(
             task_id="transfer_non_native_gs_to_s3",
             source_dataset=input_file,
             destination_dataset=File(path=f"{s3_bucket}/example_uto/", conn_id="aws_default"),
         )
     

2. GCS to Sqlite transfers

   • Non-native transfer

     Following is an example of non-native transfers between Google cloud storage to Sqlite using non-native transfer:

         transfer_non_native_gs_to_sqlite = UniversalTransferOperator(
             task_id="transfer_non_native_gs_to_sqlite",
             source_dataset=File(
                 path=f"{gcs_bucket}/example_uto/csv_files/", conn_id="google_cloud_default", filetype=FileType.CSV
             ),
             destination_dataset=Table(name="uto_gs_to_sqlite_table", conn_id="sqlite_default"),
         )
     

3. GCS to Snowflake transfers

   • Non-native transfer

     Following is an example of non-native transfers between Google cloud storage to Snowflake using non-native transfer:

         transfer_non_native_gs_to_snowflake = UniversalTransferOperator(
             task_id="transfer_non_native_gs_to_snowflake",
             source_dataset=File(
                 path="gs://uto-test/example_uto/csv_files/", conn_id="google_cloud_default", filetype=FileType.CSV
             ),
             destination_dataset=Table(name="uto_gs_to_snowflake_table", conn_id="snowflake_conn"),
         )
     

4. GCS to Google Bigquery transfers

   • Non-native transfer

     Following is an example of non-native transfers between Google cloud storage to Google Bigquery using non-native transfer:

         transfer_non_native_gs_to_bigquery = UniversalTransferOperator(
             task_id="transfer_non_native_gs_to_bigquery",
             source_dataset=File(path="gs://uto-test/example_uto/homes_main.csv", conn_id="google_cloud_default"),
             destination_dataset=output_table,
         )