Project: Merge small files on HDFS for Hive table
This is a solution for small file problems on HDFS, but Hive table only.
Here is why I wrote this project: Solving Small Files Problem on CDH4.
This script simply INSERT the requested table/partition to a new table, let data be merged by Hive itself, then INSERT back with compression.
Incoming search terms:
- merge tables in cdh hive
- hive merge smallfiles avgsize
- hdfs merge files
- hive merge files
- hive merge smaller files together
- small file for combining
- HOw to satisfy MERGE IN HIVE
- hive table merge the data small files
- hive table many small files
- hive merge tables
- hive merge small files
- hive getmerge
- hive Failed with exception checkPaths:
- hive creating small files