Responsibilities:
- The project was dividing into three phase analysis of messages using kafka component, streaming using spark jobs and finally resulting into storage of hadoop DB Hbase.
- Kaka : Using Publisher/Consumer/Console-consumer commands for message fetching producing into GSTN portal separate pages.
- Spark: Using streaming jobs of summary for looking GS logs and events per active batches. - Storm: Analyses of topologies through worker logs and dividing the bolts with count of executors on the basis of tasks assigned to it.
- Hive: Running hive queries over schema designed over GSTN portal database and analyzing the data and providing desired result through queries. :: Production based Projects
- Hbase & HDFS: Stirage components for data into Hadoop database using hbase shell commands.
- Oozie: Making hive scripts into automation process using oozie jobs for GSTN records.
- Impala: Working over impala queries to validate the records fetched using hive scripts.