Browsing by Author "Sannutha S, Holla"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Item
    Map Reduce based Analysis of Live Website Traffic integrated with improved Performance for Small files using Hadoop
    (2016-07-14T12:03:19Z) Sangeetha, D; Sannutha S, Holla; Sushmitha, R; Vaibhav, Shekar
    Hadoop, an open source java framework deals with big data. It has mainly two core components: HDFS (Hadoop distributed file system) which stores large amount of data in a reliable manner and another is MapReduce which is a programming model which processes the data in a parallel and distributed manner. Hadoop does not perform well for small files as a large number of small files pose a heavy burden on the NameNode of HDFS and an increase in execution time for MapReduce is encountered. Hadoop is designed to handle huge size files and hence suffers a performance penalty while dealing with large number of small files. This research work gives an introduction to HDFS, small file problem and existing methods to deal with it along with the proposed approach to handle small files. In proposed approach, merging of small files is done using MapReduce programming model on Hadoop. This approach improves the performance of Hadoop in handling small files by ignoring the files whose size is larger than the block size of Hadoop and also reduces the memory required by NameNode to store them. We also propose a Traffic analyzer with the combination of Hadoop and Map- Reduce paradigm. The joint of Hadoop and MapReduce programming tools makes it possible to provide batch analysis in minimum response time and memory computing capacity in order to process log in a highly available, efficient and stable way