parquet使用笔记

1、错误 Exception in thread "main" org.apache.parquet.column.statistics.StatisticsClassException: Statistics comparator mismatched: SIGNED_INT32_COMPARATOR vs. SIGNED_INT32_COMPARATOR 

完整错误信息如下:

Exception in thread "main" org.apache.parquet.column.statistics.StatisticsClassException: Statistics comparator mismatched: SIGNED_INT32_COMPARATOR vs. SIGNED_INT32_COMPARATOR
at org.apache.parquet.column.statistics.StatisticsClassException.create(StatisticsClassException.java:42)
at org.apache.parquet.column.statistics.Statistics.mergeStatistics(Statistics.java:327)
at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:119)
at org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:147)
at org.apache.parquet.column.impl.ColumnWriterV1.flush(ColumnWriterV1.java:235)
at org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:122)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:169)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:109)
at org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:301)
...

解决办法,升级parquet-avro版本到1.10.1 即可。

原文地址:https://www.cnblogs.com/mylittlecabin/p/11719862.html