I have two GCP projects such as proj-A & proj-B. And, proj-A has a dataset under that there a view created (proj-A.proj_A_dataset.test_table).

I am trying to access this view from proj-B. proj-B has a ServiceAccount and this SA has provided with proj-B dataset "BigQuery Data Viewer" permissions already.

I am getting an error like Error creating destination table using the following query:< And the entire Query> Also, User does not have permission to query table. Access Denied

but access is granted. I am not getting the same error when I try with PySpark. Its only happening with Java. Looks like its trying to create a temp table in the proj-A dataset but only we have Bigdata viewer permissions.

How to resolve this error as this is happening only in Java BigQuery.

        exclude group: 'commons-io', module: 'commons-io'
        exclude group: 'com.google.guava', module: 'guava'
    }
    implementation '.apache.spark:spark-sql_2.12:3.4.3'
    implementation '.projectlombok:lombok:1.18.24'
    implementation 'com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.42.1'

                .appName(“test”)
                .getOrCreate();

        sparkSession.conf().set("viewsEnabled", "true");
        sparkSession.conf().set("materializationProject”,”proj-B”);
        sparkSession.conf().set("materializationDataset”,”test”_dataset);

        String projectId = “proj-A”;
        String dataset = “proj_A_dataset“;
        String table = "test_table";

        String bigQueryTable = projectId + "." + dataset + "." + table ;

        Dataset<Row> bigQueryData  = sparkSession.read().format("bigquery").option("table",bigQueryTable).load();

I even tried without materialization project options. but nothing seems working. Please help. Thanks