Azure SQL Data Warehouse is a cloud-based, scale-out database capable of processing massive volumes of data, both relational and non-relational. Built on our massively parallel processing (MPP) architecture, SQL Data Warehouse can handle your enterprise workload. Azure Data Lake Store is an enterprise-wide hyper-scale repository for big data analytic workloads. Azure Data Lake enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics. Microsoft yesterday announced that you can now directly import or export your data from Azure Data Lake Store (ADLS) into Azure SQL Data Warehouse (SQL DW) using External Tables.
With SQL DW PolyBase support for ADLS, you can now load data directly into your SQL DW instance at nearly 3 TB per hour. Because SQL DW can now ingest data directly from Windows Azure Storage Blob and ADLS, you can now load data from any storage service in Azure.
Microsoft offered the following guidelines for loading data into SQL DW from Azure Data Lake Store,
- Co-locate the services in the same data center for better performance and no data egress charges.
- Split large compressed files into at least 60 smaller compressed files.
- Use a large resource class in SQL DW to load the data.
- Ensure that your AAD Application has read access from your chosen ADLS Directory.
- Scale up your DW SLO when importing a large data set.
- Use a medium resource class for loading data into SQL DW.
Learn more about it -.
