I am trying to create a connection betweeen R (Shiny) and Databricks using dbpool and ODBC with simba 64 bit as the driver. using dbGetQuery(conn, "select current_catalog()")) i can see that its hitting the correct catalog. When I do dblisttables(conn), i can see the table i want to query

but when i do dbGetQuery(conn, "select * from table_name") it gives me an error saying table not found even though the table exists.

In the error message it has added a "SPARK" keyword in front of the table: Table or view not found: SPARK..table_name From 'nanodbc/nanodbc.cpp:1722'

Im setting the environment using a dockerfile. As im new to doing this could someone point out what Im doing wrong? This is how im downloading the Simba driver and configuring the ODBC details.

# Download the Simba Spark ODBC Driver
RUN wget -O /tmp/SimbaSparkODBC.zip ".6.15/SimbaSparkODBC-2.6.15.1018-LinuxRPM-64bit.zip"

# Unzip and install the driver
RUN unzip /tmp/SimbaSparkODBC.zip -d /tmp/ && \
    rpm -ivh /tmp/SimbaSparkODBC-2.6.15.1018-LinuxRPM-64bit/simbaspark-2.6.15.1018-1.x86_64.rpm && \
    rm -rf /tmp/SimbaSparkODBC-2.6.15.1018-LinuxRPM-64bit/*

# Configure the ODBC driver in odbcinst.ini
RUN echo "" >> /etc/odbcinst.ini && \
    echo "[Simba Spark ODBC Driver 64-bit]" >> /etc/odbcinst.ini && \
    echo "Description=Simba Spark ODBC Driver" >> /etc/odbcinst.ini && \
    echo "Driver=/opt/simba/spark/lib/64/libsparkodbc_sb64.so" >> /etc/odbcinst.ini && \
    echo "Setup=/opt/simba/spark/lib/64/libsparkodbc_sb64.so" >> /etc/odbcinst.ini

Im confused as to why the conn works in terms of hitting the right catalog and schema, but fails to run select queries, even though I can see the table in dblistTables(conn)

Note: I have tried passing the catalog and schema to the select query, without any luck. Also, I'm using service principal credentials ODBC connection string, so im guessing it cannot be not an access issue.