Map Reduce based Analysis of Live Website Traffic integrated with improved Performance for Small files using Hadoop
Loading...
Files
Date
2016-07-14T12:03:19Z
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Hadoop, an open source java framework deals with big data. It has mainly two
core components: HDFS (Hadoop distributed file system) which stores large amount of
data in a reliable manner and another is MapReduce which is a programming model
which processes the data in a parallel and distributed manner. Hadoop does not perform
well for small files as a large number of small files pose a heavy burden on the
NameNode of HDFS and an increase in execution time for MapReduce is encountered.
Hadoop is designed to handle huge size files and hence suffers a performance penalty
while dealing with large number of small files.
This research work gives an introduction to HDFS, small file problem and
existing methods to deal with it along with the proposed approach to handle small files.
In proposed approach, merging of small files is done using MapReduce
programming model on Hadoop. This approach improves the performance of Hadoop in
handling small files by ignoring the files whose size is larger than the block size of
Hadoop and also reduces the memory required by NameNode to store them.
We also propose a Traffic analyzer with the combination of Hadoop and Map-
Reduce paradigm. The joint of Hadoop and MapReduce programming tools makes it
possible to provide batch analysis in minimum response time and memory computing
capacity in order to process log in a highly available, efficient and stable way
Description
Keywords
Map Reduce based Analysis of Live Website Traffic integrated with improved Performance for Small files using Hadoop, Sangeetha D, Sannutha S Holla, Sushmitha R, Vaibhav Shekar