This is a forked version from dbt-fabric adapter, which is officially maintained by Microsoft. Since the official adapter does not support Serverless Pool, this fork aims to do it so.
Disclaimer - I cannot ensure this adapter would work 100% correct for all cases as I have not tested all features. If you mainly work with view creations and unit testing, it should work well. If you need to create snapshots or create tables from seeds, you will need to wait as I have not changed those materializations to conform the Serverless Pools' capability yet.
Since Serverless Pool does not allow to create tables, this leads to many features have to use "view" alternatively.
- Test materialization uses a view as the medium containing testing result instead of a temp table.
- External materialization is supported in this fork by utilizing CETAS (Create External Table As Select) feature in the Serverless Pool. To create an external, you need to declare these properties:
- materialized: 'external'
- location: '<Relative Azure Blob Storage/Data Lake location>', e.g., 'Folder1/Folder2'
- data_source: '', e.g., 'AzureDataLakeSource'
- file_format: '', e.g., 'SynapseParquetSnappyFormat'
You should predefine those resources by following these steps
-- Create credential for the target blob container
IF NOT EXISTS (SELECT * FROM sys.credentials WHERE name = 'https://<storage_account>.dfs.core.windows.net/<blob_container>') BEGIN CREATE CREDENTIAL [https://<storage_account>.dfs.core.windows.net/<blob_container>] WITH IDENTITY = 'Managed Identity' END
-- Create Database Credential
CREATE DATABASE SCOPED CREDENTIAL SynapseIdentity WITH IDENTITY = 'Managed Identity';
-- Create External Data Source
CREATE EXTERNAL DATA SOURCE [AzureDataLakeSource] WITH ( LOCATION = 'abfss://@<storage_account>.dfs.core.windows.net' , CREDENTIAL = SynapseIdentity )
-- Create External File Format: Parquet with Snappy Compression
IF NOT EXISTS (SELECT * FROM sys.external_file_formats WHERE name = 'SynapseParquetSnappyFormat') CREATE EXTERNAL FILE FORMAT [SynapseParquetSnappyFormat] WITH ( FORMAT_TYPE = PARQUET, DATA_COMPRESSION = 'org.apache.hadoop.io.compress.SnappyCodec' )
Setup remains the same as the official adapter
We've bundled all documentation on the dbt docs site
This adapter requires the Microsoft ODBC driver to be installed: Windows | macOS | Linux
Debian/Ubuntu
Make sure to install the ODBC headers as well as the driver linked above:
sudo apt-get install -y unixodbc-devpip install git+https://github.com/daihuynh/dbt-fabric-serverlessSee the changelog from the official adapter
This adapter is Microsoft-maintained. You are welcome to contribute by creating issues, opening or reviewing pull requests. If you're unsure how to get started, check out our contributing guide.
This project and everyone involved is expected to follow the Microsoft Code of Conduct.