Sqlite

Transfer to Sqlite as destination dataset

User can transfer data to Sqlite as destination as from following sources dataset:

1. Tables

       SQLITE = "sqlite"
       BIGQUERY = "bigquery"
       SNOWFLAKE = "snowflake"
   

2. Files

       LOCAL = "local"
       GS = "gs"  # Google Cloud Storage
       S3 = "s3"  # Amazon S3
       SFTP = "sftp"
   

Following transfer modes are supported:

1. Non-native transfer

   Following is an example of non-native transfers between google bigquery to Sqlite using non-native transfer:

       transfer_non_native_bigquery_to_sqlite = UniversalTransferOperator(
           task_id="transfer_non_native_bigquery_to_sqlite",
           source_dataset=Table(
               name="uto_s3_to_bigquery_table", conn_id="google_cloud_default", metadata=Metadata(schema="astro")
           ),
           destination_dataset=Table(name="uto_bigquery_to_sqlite_table", conn_id="sqlite_default"),
       )
   

2. Transfer using third-party tool

Examples

1. Google Bigquery to Sqlite transfers

   • Non-native transfer

     Following is an example of non-native transfers between Google Bigquery to Sqlite using non-native transfer:

         transfer_non_native_bigquery_to_sqlite = UniversalTransferOperator(
             task_id="transfer_non_native_bigquery_to_sqlite",
             source_dataset=Table(
                 name="uto_s3_to_bigquery_table", conn_id="google_cloud_default", metadata=Metadata(schema="astro")
             ),
             destination_dataset=Table(name="uto_bigquery_to_sqlite_table", conn_id="sqlite_default"),
         )
     

2. GCS to Sqlite transfers

   • Non-native transfer

     Following is an example of non-native transfers between GCS to Sqlite using non-native transfer:

         transfer_non_native_gs_to_sqlite = UniversalTransferOperator(
             task_id="transfer_non_native_gs_to_sqlite",
             source_dataset=File(
                 path=f"{gcs_bucket}/example_uto/csv_files/", conn_id="google_cloud_default", filetype=FileType.CSV
             ),
             destination_dataset=Table(name="uto_gs_to_sqlite_table", conn_id="sqlite_default"),
         )
     

2. AWS S3 to Sqlite transfers

   • Non-native transfer

     Following is an example of non-native transfers between AWS S3 to Sqlite using non-native transfer:

         transfer_non_native_s3_to_sqlite = UniversalTransferOperator(
             task_id="transfer_non_native_s3_to_sqlite",
             source_dataset=File(
                 path=f"{s3_bucket}/example_uto/csv_files/", conn_id="aws_default", filetype=FileType.CSV
             ),
             destination_dataset=Table(name="uto_s3_to_sqlite_table", conn_id="sqlite_default"),
         )
     

Transfer from Sqlite as source dataset

User can transfer data from Sqlite to the following destination dataset:

1. Tables

       SQLITE = "sqlite"
       BIGQUERY = "bigquery"
       SNOWFLAKE = "snowflake"
   

2. Files

       LOCAL = "local"
       GS = "gs"  # Google Cloud Storage
       S3 = "s3"  # Amazon S3
       SFTP = "sftp"
   

Following transfer modes are supported:

1. Non-native transfer

2. Transfer using third-party tool