Snowflake

Transfer to Snowflake as destination dataset

User can transfer data to Snowflake as destination as from following sources dataset:

1. Tables

       SQLITE = "sqlite"
       BIGQUERY = "bigquery"
       SNOWFLAKE = "snowflake"
   

2. Files

       LOCAL = "local"
       GS = "gs"  # Google Cloud Storage
       S3 = "s3"  # Amazon S3
       SFTP = "sftp"
   

Following transfer modes are supported:

1. Non-native transfer

   Following is an example of non-native transfers between AWS S3 to Snowflake using non-native transfer:

       transfer_non_native_s3_to_snowflake = UniversalTransferOperator(
           task_id="transfer_non_native_s3_to_snowflake",
           source_dataset=File(
               path="s3://astro-sdk-test/example_uto/csv_files/", conn_id="aws_default", filetype=FileType.CSV
           ),
           destination_dataset=Table(name="uto_s3_table_to_snowflake", conn_id="snowflake_conn"),
       )
   

2. Transfer using third-party tool

Examples

1. AWS S3 to Snowflake transfers

   • Non-native transfer

     Following is an example of non-native transfers between AWS S3 to Snowflake using non-native transfer:

         transfer_non_native_s3_to_snowflake = UniversalTransferOperator(
             task_id="transfer_non_native_s3_to_snowflake",
             source_dataset=File(
                 path="s3://astro-sdk-test/example_uto/csv_files/", conn_id="aws_default", filetype=FileType.CSV
             ),
             destination_dataset=Table(name="uto_s3_table_to_snowflake", conn_id="snowflake_conn"),
         )
     

   • Transfer using third-party tool

     Following is an example of transfers between AWS S3 to Snowflake using Fivetran with connector passed:

         transfer_fivetran_with_connector_id = UniversalTransferOperator(
             task_id="transfer_fivetran_with_connector_id",
             source_dataset=File(path=f"{s3_bucket}/uto/", conn_id="aws_default"),
             destination_dataset=Table(name="fivetran_test", conn_id="snowflake_default"),
             transfer_mode=TransferMode.THIRDPARTY,
             transfer_params=FiveTranOptions(conn_id="fivetran_default", connector_id="filing_muppet"),
         )
     

     Following is an example of transfers between AWS S3 to Snowflake using Fivetran without connector passed:

         transfer_fivetran_without_connector_id = UniversalTransferOperator(
             task_id="transfer_fivetran_without_connector_id",
             source_dataset=File(path=f"{s3_bucket}/uto/", conn_id="aws_default"),
             destination_dataset=Table(
                 name="fivetran_test",
                 conn_id="snowflake_conn",
                 metadata=Metadata(database=snowflake_database, schema=snowflake_schema),
             ),
             transfer_mode=TransferMode.THIRDPARTY,
             transfer_params=FiveTranOptions(
                 conn_id="fivetran_default",
                 connector_id="filing_muppet",
                 group=Group(name="test_group"),
                 connector=Connector(
                     service="s3",
                     config=connector_config,
                     connector_id=None,
                     connect_card_config={"connector_val": "test_connector"},
                 ),
                 destination=Destination(
                     service="snowflake",
                     time_zone_offset="-5",
                     region="GCP_US_EAST4",
                     config=destination_config,
                 ),
             ),
         )
     

2. GCS to Snowflake transfers

   • Non-native transfer

     Following is an example of non-native transfers between GCS to Snowflake using non-native transfer:

         transfer_non_native_gs_to_snowflake = UniversalTransferOperator(
             task_id="transfer_non_native_gs_to_snowflake",
             source_dataset=File(
                 path="gs://uto-test/example_uto/csv_files/", conn_id="google_cloud_default", filetype=FileType.CSV
             ),
             destination_dataset=Table(name="uto_gs_to_snowflake_table", conn_id="snowflake_conn"),
         )
     

2. Bigquery to Snowflake transfers

   • Non-native transfer

     Following is an example of non-native transfers between Bigquery to Snowflake using non-native transfer:

         transfer_non_native_bigquery_to_snowflake = UniversalTransferOperator(
             task_id="transfer_non_native_bigquery_to_snowflake",
             source_dataset=Table(
                 name="uto_s3_to_bigquery_table",
                 conn_id="google_cloud_default",
                 metadata=Metadata(schema="astro"),
             ),
             destination_dataset=Table(
                 name="uto_bigquery_to_snowflake_table",
                 conn_id="snowflake_conn",
             ),
         )
     

Transfer from Snowflake as source dataset

User can transfer data from Snowflake to the following destination dataset:

1. Tables

       SQLITE = "sqlite"
       BIGQUERY = "bigquery"
       SNOWFLAKE = "snowflake"
   

2. Files

       LOCAL = "local"
       GS = "gs"  # Google Cloud Storage
       S3 = "s3"  # Amazon S3
       SFTP = "sftp"
   

Following transfer modes are supported:

1. Non-native transfer