Fast and Efficient Data Search in Hadoop
Loading...
Files
Date
2018-06-19T10:32:01Z
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Hadoop is the most popular implementation framework of the MapReduce programming
model, and it has a number of performance-critical configuration parameters. However, manually
setting these parameters to their optimal values not only needs in-depth knowledge on Hadoop as well as the job itself, but also requires a large amount of time and efforts. Automatic approaches have therefore been proposed. Their usage, however, is still quite limited due to the intolerably
long searching time. In this proposed system, we introduce MapreducE Self-Adjusting (MESA), a framework that accelerates the searching process for the optimal configuration of a given
Hadoop application. We have devised a novel mechanism by integrating the model trees
algorithm with the genetic algorithm. As such, MESA significantly reduces the searching time by removing unnecessary profiling, modelling, and searching steps, which are mandatory for existing approaches. Our experiments using five benchmarks, each with two input data sets (DS1
and 2xDS1) show that MESA improves the searching efficiency (SE) by factors of 1.37x and
2.18x on average respectively over the state-of-the-art approach.
Description
Keywords
T.POOJITHA REDDY, MANDARA B M, NANDITHA N, 1NH14CS175, 1NH14CS067, 1NH14CS722