TechStack : Hive,HDFS,Python,Hive Thrift Server

The Internet Movie Database (IMDb) is one of the world’s most popular sources for movie, TV and celebrity content with more than 100 million unique visitors per month. IMDb has huge collection of movies database that includes various details of movies along with different ratings and user reviews. This movie reviews affects everyone from audience, film critics to the production company. The idea of project is to analyse the movie and ratings dataset (source: https://www.simplilearn.com/) using big data technologies.

View Code

TechStack : Spark,DataBricks,Python,DataBricks NoteBook

The Project Log-Analysis using Pyspark focuses on learning the basics of Apache Spark, The data is captured from EDGAR Log File Data Set, which provides structured log data. The data set contains around 1 lacs records, with 14 columns {IP,Date,Time,Zone,CIK,Accession,Extension,Code,Size,Index,Noref,NoAgeent,Find,Crawler}. The analysis is mainly done on Size, File type, Status Code and IP. Through this project we have following learning:

Working with pySpark, internal working of spark.sql

Working with DataFrame

Handling different file format linke CSV and JSON

Visualizing data using matplotlib, one of the popular ploting library

View Code

It is a python based project that helps us to get related movie based on the movie you previously watched. The movie are suggested based on the review giveing by other users. We used Pearson Correlation for calculating the relation between two number. If the movies are related it gives a positive result if not its gives negative or else its give zero as a result. Here is just an prototype with a fixed data set. But this could be usefull with tweeks to other set of data. For the purpose of showing the power of pearson correlation we used a fixed data set.

View Code

The project is to improve the voting system in India. People in India does not go for casting the vote.There could be different reason for the same, which lead to seletion of party not by majority but minority of those who gives time for casting vote. To improve the situation and increase the number of voters, This project can do wonders.

I as the leader of team with my two collage mate, Introduce a new concept of using fingerprint of user for voting purpose. Basic idea was to develop an mobile application as well as an Electronic Voting Machine(with fingerprint scanner). The citizen with busy life can cast vote using the mobile application and one with no such faclity can go to the voting booth.

View Code