Clustering of Large Database in Hadoop
Loading...
Files
Date
2017-08-18T09:35:43Z
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
With The growing trend of large volume of data generated everyday,storing and processing the data is a great challenge.The existing solutions like SAS,R,Excel and MapReduce prove to be inefficient.Mapreduce is a part of Apache Hadoop that allows distributive processing of unstructured data, where each distributive node has its own storage. Java serialization is not enough to convert the original message to binary format for a large volume of data,hence,hadoop uses its own serialization technique.Most of the algorithms are iterative in nature.But Mapreduce is a bane as it involves undesirable amount of read and writes to process such iterative algorithms.As an advancement,in this paper,we use the SPARK to implement algorithms like kmeans,linear regression etc on realtime data.Spark is an unified engine that can run Hadoop,Mesos,cloud or standalone.It stores the intermediate result in RAM,thus avoiding read/write from/to disk.Like the Mapreduce,Spark is used for batch processing.In adddition,Spark can be used to handle streaming data,queries and machine learning . It provides various libraries like SaprkSql (mixes the SQL queries with Spark programs) , MLlib (contains algorithms for classification,clustering,regression and so on) and Sparkstreaming (to feed stream data to sprak rograms).In this paper,we prefer to use the cloud services to access the large datasets because cloud is flexible,scalable and ivolves less hardware cost than using our own infrastructure which requires many softwares (hadoop,ubuntu,java etc) to be installed.Once the data is obtained,we then run and evaluate the K-means algorithm on Spark using EMR (Elastic MapReduce) and compare the performance against executing the same algorithm using Mapreduce.
Description
Keywords
Clustering of Large Database in Hadoop, 1NH13IS005, MD Faisal Alam, Pallavi Reddy