In parquet, the estimated_memory_size method of DictEncoder should estimate the actual usage of allocated memory, but instead it uses the data member counting the encoded size of dictionary data, and ignores the hash table memory usage added by the Interner member.
In parquet, the
estimated_memory_sizemethod ofDictEncodershould estimate the actual usage of allocated memory, but instead it uses the data member counting the encoded size of dictionary data, and ignores the hash table memory usage added by theInternermember.