In Databricks Python notebook I can easily use dbutils module. Now I also would like to use it within plain Python file which I import into Databricks notebook

Here is an example.

Here is content of some_python_module.py

secret_value = dbutils.secrets.get("some_location", "some_secret")

Later on I am importing it in Databricks notebook

import some_python_module.py

But I get error message: NameError: name 'dbutils' is not defined

I tried to add import statement into my some_python_module.py

import dbutils

but it returns: ModuleNotFoundError: No module named 'dbutils'

Aslo dbutils.secrets.get("some_location", "some_secret") works fine in Databricks notebook

In Databricks Python notebook I can easily use dbutils module. Now I also would like to use it within plain Python file which I import into Databricks notebook

Here is an example.

Here is content of some_python_module.py

secret_value = dbutils.secrets.get("some_location", "some_secret")

Later on I am importing it in Databricks notebook

import some_python_module.py

But I get error message: NameError: name 'dbutils' is not defined

I tried to add import statement into my some_python_module.py

import dbutils

but it returns: ModuleNotFoundError: No module named 'dbutils'

Aslo dbutils.secrets.get("some_location", "some_secret") works fine in Databricks notebook

• python
• azure
• databricks
• dbutils

• 1 any chance you are using the Azure databricks? – Dileep Raj Narayan Thumula Commented Nov 18, 2024 at 2:25
• @DileepRajNarayanThumula, Yes, I am – user1700890 Commented Nov 18, 2024 at 16:12
• 1 have you tried 'from pyspark import dbutils' ? or from pyspark.dbutils import DBUtils spark = SparkSession.builder.getOrCreate() dbutils = DBUtils(spark) ? – AdForte Commented Nov 18, 2024 at 16:30
• @AdForte, thank you. Now I need to import SparkSession – user1700890 Commented Nov 18, 2024 at 17:11
• 1 @user1700890 and with this "from pyspark.sql.session import SparkSession" ? – AdForte Commented Nov 19, 2024 at 9:46

2 Answers 2

As you want to import the python file into Azure databricks workspace.

I have created the below python file and uploaded the file to my filestore dbfs path:

try:
    import IPython
    
    dbutils = IPython.get_ipython().user_ns["dbutils"]
except (ImportError, KeyError):
    
    dbutils = None
if dbutils:
    secret_value = dbutils.secrets.get("Key-vault-secret-dbx", "secretkv")
else:
    secret_value = "dbutils not available"

In the above code I have used

import IPython
dbutils = IPython.get_ipython().user_ns["dbutils"]

It is used to access the dbutils object in a Databricks notebook's environment when it is not directly available in the imported module. So basically Accessing dbutils from the Databricks notebook's IPython environment

I have tried to retrieve a secret using the below:

import sys
sys.path.append("/dbfs/FileStore/tables/")
import dilip_module
print(f"Retrieved secret value: {dilip_module.secret_value}")

Results:

Retrieved secret value: [REDACTED]

dbutils is a variable/instance-of-DBUtils-class.

You can create your own instance of DBUtils if you like.

dbutils = DBUtils(SparkSession.builder.getOrCreate())

From my conftest.py for pytest

from pyspark.sql import SparkSession
from unittest.mock import MagicMock


@pytest.fixture(scope='session')
def spark_session():
  yield SparkSession.builder.getOrCreate()

@pytest.fixture(scope='session')
def is_running_on_databricks_cluster(spark_session: SparkSession):
  return spark_session.conf.get('spark.app.name') == 'Databricks Shell'

@pytest.fixture(scope='session')
def dbutils(spark_session: SparkSession, is_running_on_databricks_cluster):
  if is_running_on_databricks_cluster:
    from pyspark.dbutils import DBUtils

    yield DBUtils(spark)
  else:
    yield MagicMock()