I have a Cloudera cluster under knox gateway with basic authentication (username/password). I want to access HDFS (SWebHDFS) through SSL (https) using Java Apache Hadoop client library (Apache Hadoop Client Aggregator » 3.4.1). Using swebhdfs scheme I can see that https is used, but I can't find a way to use basic authentication.

I can successfully use HTTP requests or Java HttpClient to access HDFS, but is it possible to apply basic authentication with the Java Hadoop client library?

Sample code

        Configuration conf = new Configuration();

        conf.set("fs.defaultFS", "swebhdfs://...");
        conf.setBoolean("dfs.webhdfs.enabled", true);
        conf.set("fs.webhdfs.impl", ".apache.hadoop.hdfs.web.SWebHdfsFileSystem");
        conf.set("hadoop.security.authentication", "simple");
        conf.set("dfs.http.policy", "HTTPS_ONLY");
        
        FileSystem fs = FileSystem.get(conf);
        fs.listStatus(new Path("/"));

With the above code I get the error Unexpected HTTP response: code=302 != 200, op=LISTSTATUS, message=Found This is because since basic authentication is not present and hadoop-jwt cokie doesn't exist, Knox tries to redirect to a Sign in form