最新消息:雨落星辰是一个专注网站SEO优化、网站SEO诊断、搜索引擎研究、网络营销推广、网站策划运营及站长类的自媒体原创博客

unable to write to gcs bucket using spark 3.5.2 - Stack Overflow

programmeradmin3浏览0评论

I am using spark 3.5.2 and I added the dependency google-auth-library-oauth2-http-1.33.1.jar but this doesnt contain com/google/api/client/googleapis/auth/oauth2/GoogleCredential so I found a jar that contains which is google-api-client-1.33.2.jar but when I add this I get another NoClassDefError for com/google/api/client/auth/oauth2/Credential so I am kind of stuck what jars is needed? my app simply writes from kafka to gcs bucket.

The question here is what are all the jars that are required to write to a GCS bucket?

25/03/19 20:56:01 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/api/client/googleapis/auth/oauth2/GoogleCredential
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:467)
    at .apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2625)
    at .apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2590)
    at .apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
    at .apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
    at .apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
   
发布评论

评论列表(0)

  1. 暂无评论