I face the attached problem when reading an orc file:
Is it possible to change this buffer size of 65536 to the needed one of 1817279? Which configuration values do I have to adapt in order to set this value?
I did not find the correct configuration value in the documentation.
I face the attached problem when reading an orc file:
Is it possible to change this buffer size of 65536 to the needed one of 1817279? Which configuration values do I have to adapt in order to set this value?
I did not find the correct configuration value in the documentation.
Share Improve this question edited Feb 3 at 18:51 f_puras 2,5044 gold badges36 silver badges46 bronze badges asked Feb 3 at 16:21 Ruben HartensteinRuben Hartenstein 11 bronze badge1 Answer
Reset to default 0The exceeding buffer-size is related to an issue with the HDFS Erasure Coding (EC) file-encoding. See this issue at Apache ORC:
- Buffer size too small. size · Issue #1939 · apache/orc
It could be traced back to the Hadoop HDFS bug:
- [HDFS-17535] I have confirmed the EC corrupt file, can this corrupt file be restored? - ASF JIRA
So, check your Hadoop HDFS version if it is affected by this bug.