I have deployed a Databricks App in Azure Databricks. I am now trying to connect it to the Databricks SQLWarehouse using the service principal ID and secret from the app itself (these are automatically generated when deploying a DB App and accessible in the environment). I have verified that the app has access to the SQLWarehouse as its SP ID has been added with 'Can use' permissions.

I am using the python package databricks-sql-connector to make the connection. Herewith the code that I use:

from contextlib import contextmanager
from typing import Generator

import databricks.sql as dbsql
from databricks.sdk.core import Config, oauth_service_principal
from databricks.sql.client import Cursor


@contextmanager
def get_cursor() -> Generator[Cursor, None, None]:
    with dbsql.connect(
        server_hostname=os.getenv("DATABRICKS_HOST"),
        http_path=os.getenv("DATABRICKS_SQLWAREHOUSE_HTTP_PATH"),
        credential_provider=oauth_service_principal(
            Config(
                host=os.getenv("DATABRICKS_HOST"),
                client_id=os.getenv("DATABRICKS_CLIENT_ID"),
                client_secret=os.getenv("DATABRICKS_CLIENT_SECRET"),
            )
        ),
    ) as connection:
        cursor = connection.cursor()
        yield cursor
        cursor.close()
        connection.close()

Side note: this code works fine when the instead of the credential_provider argument I give it the access_token argument with a PAT token for access to SQL Warehouse. I get returned a cursor to execute queries.

It is my (limited) understanding that the above code would start a machine-2-machine oauth flow, but this does not seem to happen. What gets returned when the function is called is the message

INFO:databricks.sql.auth.oauth:Listening for OAuth authorization callback at http://localhost:8020

Looking into the Databricks functions this seems to be logged when doing User authentication, at which point there would be a browser login window. It is waiting for a user input it seems, because the app won't continue here.

What am I doing wrong?

I have deployed a Databricks App in Azure Databricks. I am now trying to connect it to the Databricks SQLWarehouse using the service principal ID and secret from the app itself (these are automatically generated when deploying a DB App and accessible in the environment). I have verified that the app has access to the SQLWarehouse as its SP ID has been added with 'Can use' permissions.

I am using the python package databricks-sql-connector to make the connection. Herewith the code that I use:

from contextlib import contextmanager
from typing import Generator

import databricks.sql as dbsql
from databricks.sdk.core import Config, oauth_service_principal
from databricks.sql.client import Cursor


@contextmanager
def get_cursor() -> Generator[Cursor, None, None]:
    with dbsql.connect(
        server_hostname=os.getenv("DATABRICKS_HOST"),
        http_path=os.getenv("DATABRICKS_SQLWAREHOUSE_HTTP_PATH"),
        credential_provider=oauth_service_principal(
            Config(
                host=os.getenv("DATABRICKS_HOST"),
                client_id=os.getenv("DATABRICKS_CLIENT_ID"),
                client_secret=os.getenv("DATABRICKS_CLIENT_SECRET"),
            )
        ),
    ) as connection:
        cursor = connection.cursor()
        yield cursor
        cursor.close()
        connection.close()

Side note: this code works fine when the instead of the credential_provider argument I give it the access_token argument with a PAT token for access to SQL Warehouse. I get returned a cursor to execute queries.

It is my (limited) understanding that the above code would start a machine-2-machine oauth flow, but this does not seem to happen. What gets returned when the function is called is the message

INFO:databricks.sql.auth.oauth:Listening for OAuth authorization callback at http://localhost:8020

Looking into the Databricks functions this seems to be logged when doing User authentication, at which point there would be a browser login window. It is waiting for a user input it seems, because the app won't continue here.

What am I doing wrong?

• databricks
• databricks-sql

• are you using Azure databricks? – Dileep Raj Narayan Thumula Commented Mar 26 at 16:49

1 Answer 1

In order for this user authentication to work, a workspace admin must enable this feature and define access scope.

Please check the Authorization page in the app setting.