Query Engine: Impala
When we try to do a select query on a iceberg table which has TimestampNTZ column (Timestamp with timezone), it displays the results.
But when I try to insert a value to the table in impala, I am getting the error saying,
AnalysisException: The Iceberg Table has a TIMESTAMPNTZ column that impala cannot write
When I searched Cloudera says, it is an unsupported data type.
Reference: .2.18/iceberg-how-to/topics/iceberg-data-types.html#:~:text=Impala%20is%20unable%20to%20write%20to%20Iceberg%20tables,all%20the%20engines%20must%20be%20running%20in%20UTC.
Because of this, I have gone to other options like Timestamp without timezone data type.
But when I create that data type table from spark and query it in impala discrepancies occur
Spark displaying current time zone, while impala displaying UTC time.
Any suggestions are welcome!
Note: I have used this spark property while creating,
spark.sql.iceberg.handle-timestamp-without-timezone to true