This tutorial shows you how to connect Flyway to Databricks Lakehouse (Unity Catalog), using a service principal, a personal access token or browser sign-in.

For Databricks Lakebase, which is a managed PostgreSQL service, see the Databricks Lakebase driver reference instead. 

Prerequisites:
- A Databricks workspace with Unity Catalog enabled. Flyway doesn't support the legacy `hive_metastore` catalog.
- A SQL warehouse that Flyway can use.
- A catalog to deploy to and, if you'll generate migrations, a second catalog for your shadow environment.
- Flyway Desktop 9.10.10 or later, if you're connecting from Flyway Desktop.
- To generate migrations, view changes or check for drift: Flyway Teams or Enterprise in Flyway Desktop, or Flyway Enterprise for the `generate` and `check -drift` commands.

The Databricks JDBC driver ships with Flyway, so there's nothing extra to install.

Getting your connection details

1. In your Databricks workspace, open "SQL Warehouses" and select your warehouse.
2. Open the "Connection details" tab.
3. Note the "Server hostname" and "HTTP path". The port is 443.
4. Choose the catalog Flyway will deploy to, for example `main`.

Flyway connects with a JDBC URL like this:

jdbc:databricks://<server-hostname>:443/default;transportMode=http;ssl=1;httpPath=<http-path>;EnableArrow=0;<auth-parameters>;ConnCatalog=<catalog>

`ssl=1` and `ConnCatalog` are required. `default` is the schema Flyway uses, and creates its schema history table in, if you don't set `schemas` or `defaultSchema`. See the Databricks driver reference for the other parameters.

It is recommended to not write secrets directly into `flyway.toml` if you intend to check it into version control. The examples below read them from environment variables, but you can also use any of Flyway's secret resolvers.

Connecting with a service principal

OAuth machine-to-machine (M2M) is the best fit for pipelines. Flyway signs in as a Databricks service principal, so deployments don't depend on a person's token.

Setting up the service principal

1. Add a service principal to your Databricks account and assign it to your workspace.
2. Give it the "Can use" permission on your SQL warehouse.
3. Click your username, then "Settings" > "Identity and access". Next to "Service principals", click "Manage", select your service principal, and open the "Secrets" tab.
4. Click "Generate secret" and set its lifetime, up to 730 days.
5. Copy the secret and the client ID. Databricks shows the secret only once.

Connecting using the service principal

Add an environment to `flyway.toml`. Set `OAuth2ClientId` to the service principal's application ID and `OAuth2Secret` to its secret:

[environments.production]
url = "jdbc:databricks://adb-1234567890123456.7.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=/sql/1.0/warehouses/1a2b3c4d5e6f7g8h;EnableArrow=0;AuthMech=11;Auth_Flow=1;ConnCatalog=main"

[environments.production.jdbcProperties]
OAuth2ClientId = "${env.DATABRICKS_CLIENT_ID}"
OAuth2Secret = "${env.DATABRICKS_CLIENT_SECRET}"

The Flyway Desktop connection dialog doesn't offer service principal sign-in yet, so add this environment to `flyway.toml` directly.

Connecting with a personal access token

Personal access tokens are the quickest to set up, but Databricks recommends OAuth over them where you can.

Creating the token

1. Click your username, then "Settings" > "Developer". Next to "Access tokens", click "Manage", then "Generate new token".
2. Copy the token. Databricks shows it only once.

Connecting using the token

[environments.development]
url = "jdbc:databricks://adb-1234567890123456.7.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=/sql/1.0/warehouses/1a2b3c4d5e6f7g8h;EnableArrow=0;AuthMech=3;ConnCatalog=main"
password = "${env.DATABRICKS_TOKEN}"

If you use a token in a pipeline, use one that belongs to a service principal rather than a named user. A workspace admin creates the first one with `databricks token-management create-obo-token`. Tokens don't rotate automatically, and Databricks revokes any that go unused for 90 days.

Connecting with browser sign-in

OAuth user-to-machine (U2M) opens a browser so you sign in with your own Databricks identity. Interactive use only.

[environments.development]
url = "jdbc:databricks://adb-1234567890123456.7.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=/sql/1.0/warehouses/1a2b3c4d5e6f7g8h;EnableArrow=0;AuthMech=11;Auth_Flow=2;ConnCatalog=main"

[environments.development.jdbcProperties]
TokenCachePassPhrase = "${databricks.tokenCachePassPhrase}"

Granting permissions

To connect, your user or service principal needs the **Can use** permission on the SQL warehouse. To work with anything in the catalog, it also needs `USE CATALOG` and `USE SCHEMA` on the catalog and schemas Flyway works with. Replace `<principal>` with the user's email address or the service principal's application ID:

GRANT USE CATALOG ON CATALOG main TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA main.default TO `<principal>`; -- this is required for all schemas in the catalog

Deploying, drift checks and generating migrations need more privileges. See Database permissions. If privileges are missing, Databricks hides objects from drift checks and generated migrations rather than reporting an error.

Testing the connection

From the command line, run:

flyway testConnection -environment=production

This opens a connection to Databricks without changing anything. With Flyway Enterprise, it also checks that the comparison engine can connect. In Flyway Desktop, use "Test connection" in the connection dialog.

Reference

- Databricks driver reference
- Databricks Resolver
- Advanced capabilities for Databricks
- testConnection