Big Data Engineer Resume Examples

big data engineer

  • Started my career with this startup were I was a part of a team that worked on providing big data related solutions for a fortune 500 retail US giant as well as creating our own automated big data product for Automated data ingestion and data processing.
  • Worked on technologies like: Hadoop, Hive, Java, Maven, Sqoop, Apache Flume, Apache Velocity, Apache Pig, Azkaban, Bash.
  • Worked with cloud based environments like Google Cloud Platform’s DataProc cluster as well as On-Prem clusters like HDP and Cloudera.
  • Working on DR sync with PROD environment. 

senior big data engineer

  • Responsible for the documentation, design, development, and      architecture of Big Data applications.
  • Process customer viewership payload for (Telus International)             Canada Origin Telecommunication (Client).
  • Develop application for ETL (Extraction, Transformation, and Loading) process.
  • Develop applications for processing structured payload at higher speed and real time.
  • Perform the testing with the given scenarios anddummy sample inputs by Business.
  • Maintain data security and privacy by implementing Kerberos Authentication and Authorization in application

big data engineer

  • Responsible for handling almost 20 TB of data on daily basis, from 40 different source like DPI, CDR/VDR related data, Signalling,Browsing, Tealeaf and other applications related data in Hadoop ecosystem.
  • Responsible for enriching, transforming and loading the structured and unstructured data into hive and HBase tables using Pyspark, Scala and shell scripting.
  • POC for the project “City in the Motion” using Docker, Pyspark and Shell Scripting to generate realtime subscriber details using their location data.
  • Foot Fall analysis using subscriber location,Nationality and Age Group for different Malls, places and there frequent visitor based on different client requirements.
  • Worked on Complex sql queries required for Use cases based on the clients requirement.
  • Data Governing and Data Quality check before and after deployments based on use case and for tuning the data from different source files.
  • Perl scripting and shell scripting to automate most of the daily task, related to performance, merging chunk of files on daily basis, auditing files and to automate Spark jobs.

big data engineer

  • Developed and executed plans to monitor standard process adherence.
  • Assemble large, complex data sets that meet functional / non-functional business requirements e.g. Watsons, Robinsons, Waltermart, Mercury Drug, etc. (DATA DOWNLOAD, CLEANSING, TAGGING, BINDING, HARMONIZATION, AND ANALYSIS)
  • Oversees data planning, field work; product scanning store by store, sales and inventory; prepares reports, and communicates findings and recommendations to line and senior management. 
  • Mostly worked on IBM BigSQL, Hive, HBase, MongoDB, Docker, Python, Pyspark, Scala, Spark SQL, Shell/Perl Scripting.

big data engineer

  • Developed module for integrating mongodb for statistics module
  • Developed module for the data acquisition from analyser
  • Worked with S3 module for integrating content to S3 using python
  • Working on the upgrade of GLens Product 

big data engineer

  • Worked with a couple of teams to design the architecture of this Big data application. 
  • Used Java to implement the Levenshtein distance algorithm to map software names that are manually entered by users to system-generated names. 
  • Implemented Spark application using Scala to process and analyze 500+ GBs of data daily. 
  • Implemented Hive queries, joins and UDFs to cleansing on data and storing it on the table. 
  • Scheduled spark application using shell scripting with several dependent data pipelines scripts. 

big data engineer

  • Built Streaming services for applying the processing logic on generated samples using Flink
  • Developed customized Flink JDBC connector
  • Developed Websocket and REST applications for feeding the WebUI
  • Deploy services as containers on Nomad and Consul cluster equipped with the Service Discovery

sr. big data engineer

  • Design, developing high-volume, low-latency, applications for mission-critical systems.
  • Deliver high availability and performance.
  • Development of the Web and backend application for BPO/KPO involving in Sales and Insurance Business model.
  • Contribution in all phases of the Application Development life cycle
  • Writing well designed, testable and efficient code.
  • Prepare and produce releases of software components.
  • Support continuous improvement by investigating alternatives and technologies and presenting these for architectural review.

big data engineer

  • Reporting, Development and Database Environments. 
  • Worked on developing a new system that was used as an interactive framework developed to configure templates, execute setups of all records and basic configuration of the processing engine. Store various resources like images, design, links, etc and merge them as required to produce the template. 
  • Performed UAT testing by making test cases and using XML files to perform testing. 
  • Worked closely with the testing team and later on provided the support for hand over.