Google Bigquery

Transfer to Google Bigquery as destination dataset

User can transfer data to Google Bigquery from following sources:

1. Tables

       SQLITE = "sqlite"
       BIGQUERY = "bigquery"
       SNOWFLAKE = "snowflake"
   

2. Files

       LOCAL = "local"
       GS = "gs"  # Google Cloud Storage
       S3 = "s3"  # Amazon S3
       SFTP = "sftp"
   

Following transfer modes are supported:

1. Non-native transfer

   Following is an example of a non-native transfer between GCS to BigQuery using non-native transfer:

       transfer_non_native_gs_to_bigquery = UniversalTransferOperator(
           task_id="transfer_non_native_gs_to_bigquery",
           source_dataset=File(path="gs://uto-test/example_uto/homes_main.csv", conn_id="google_cloud_default"),
           destination_dataset=output_table,
       )
   

Examples

1. AWS S3 to Google Bigquery transfers

   • Non-native transfer

     Following is an example of a non-native transfer between AWS S3 to BigQuery using non-native transfer:

         transfer_non_native_s3_to_bigquery = UniversalTransferOperator(
             task_id="transfer_non_native_s3_to_bigquery",
             source_dataset=File(
                 path="s3://astro-sdk-test/example_uto/csv_files/", conn_id="aws_default", filetype=FileType.CSV
             ),
             destination_dataset=Table(
                 name="uto_s3_to_bigquery_destination_table",
                 conn_id="google_cloud_default",
                 metadata=Metadata(schema="astro"),
             ),
         )
     

2. GCS to Google Bigquery transfers

   • Non-native transfer

     Following is an example of a non-native transfer between GCS to BigQuery using non-native transfer:

         transfer_non_native_gs_to_bigquery = UniversalTransferOperator(
             task_id="transfer_non_native_gs_to_bigquery",
             source_dataset=File(path="gs://uto-test/example_uto/homes_main.csv", conn_id="google_cloud_default"),
             destination_dataset=output_table,
         )
     

Transfer from Google Bigquery as source dataset

User can transfer data from Google BigQuery to the following destination dataset:

1. Tables

       SQLITE = "sqlite"
       BIGQUERY = "bigquery"
       SNOWFLAKE = "snowflake"
   

2. Files

       LOCAL = "local"
       GS = "gs"  # Google Cloud Storage
       S3 = "s3"  # Amazon S3
       SFTP = "sftp"
   

Following transfer modes are supported:

1. Transfer using non-native approach

   Following is an example of non-native transfers between Bigquery to Snowflake using non-native transfer:

       transfer_non_native_bigquery_to_snowflake = UniversalTransferOperator(
           task_id="transfer_non_native_bigquery_to_snowflake",
           source_dataset=Table(
               name="uto_s3_to_bigquery_table",
               conn_id="google_cloud_default",
               metadata=Metadata(schema="astro"),
           ),
           destination_dataset=Table(
               name="uto_bigquery_to_snowflake_table",
               conn_id="snowflake_conn",
           ),
       )
   

2. Transfer using third-party platform

Examples

1. Bigquery to Snowflake transfers

   • Non-native transfer

     Following is an example of non-native transfers between Bigquery to Snowflake using non-native transfer:

         transfer_non_native_bigquery_to_snowflake = UniversalTransferOperator(
             task_id="transfer_non_native_bigquery_to_snowflake",
             source_dataset=Table(
                 name="uto_s3_to_bigquery_table",
                 conn_id="google_cloud_default",
                 metadata=Metadata(schema="astro"),
             ),
             destination_dataset=Table(
                 name="uto_bigquery_to_snowflake_table",
                 conn_id="snowflake_conn",
             ),
         )
     

2. Bigquery to Sqlite transfers

   • Non-native transfer

     Following is an example of non-native transfers between Bigquery to Sqlite using non-native transfer:

         transfer_non_native_bigquery_to_sqlite = UniversalTransferOperator(
             task_id="transfer_non_native_bigquery_to_sqlite",
             source_dataset=Table(
                 name="uto_s3_to_bigquery_table", conn_id="google_cloud_default", metadata=Metadata(schema="astro")
             ),
             destination_dataset=Table(name="uto_bigquery_to_sqlite_table", conn_id="sqlite_default"),
         )