[Pig-dev] [jira] [Created] (PIG-1972) Cache split information details for data with large number of small part files

Cache split information details for data with large number of small part files
------------------------------------------------------------------------------

Key: PIG-1972
URL: https://issues.apache.org/jira/browse/PIG-1972
Project: Pig
Issue Type: Improvement
Components: impl
Affects Versions: 0.8.0
Environment: Pig 0.8 version with PigMix http://wiki.apache.org/pig/PigMix
Reporter: Rajesh Balamohan


While running scalability benchmarks with Pig 0.8 & PigMix, L14 query listed in http://wiki.apache.org/pig/PigMix showed no scalability characteristics (i.e, for the same problem size response time should decrease as we increase the number of nodes)

Investigating further revealed that L14 query merge-joins small dataset and another large dataset. If the small dataset has many part files with very little amount of data, it causes a huge pressure on NameNode. This is because it is read as a side file in all map slows.

In the environment where I ran the experiment, small dataset was spread across 1900+ part files in HDFS.

Following codepath has the perf issue.
DefaultIndexableLoader--> seekNear() --> initRightLoader() is causing the huge delay. Since
"users_sorted" data is spread across 1900+ small files, FileInputFormat.getSplits() hits the namenode too
frequently.

i.e, (number of machines * number of map slots * 1900+ times). This is the reason why L14 is not scaling up.


Suggestion would be to cache the splitInformation of the small dataset instead of hitting the namenode too frequently.