Big Data Engineer Resume Examples

big data engineer

  • Responsible for data processing(ETL) and device risk analysis for over 4 million terminal in the Ministry of Public Security, which deals with the virus attack, network flow, business system access data on DataWorks of Alibaba Could Platform during the year Nov.2017 to  Feb. 2019
  • Worked as a big data analyst using Scala on Spark platform analyzing our own company’s data, which includes browser history, software use (developing tools, chatting, gaming) data during the year of  May.2017 to Nov.2017.
  • Worker as a big data model developer using Java programming language on the Hadoop platform to analyze user’s computer data, which also includes business system access during working hour and after work hour data and their key’s data( used to access business system). Finally, We caught more than 10 people stolen data from the business system during the year of Oct.2016 to May.2017
  • Started with building the testing cluster environment of Big data platform in the company, includes Hadoop, Hive, Elsticsearch, and Spark during the first month in the company during the year of Sep.2016 to Oct.2017

big data engineer

  • Implement Java (map-Reduce) based process to cleanse the data before the indexing of the data.
  • Implemented Hive queries as a part of QA automation.
  • Implemented Python based program to enrich the data.
  • Setting up new cluster on Amazons Web Services (AWS) and maintaining for any issues.
  • Taking regular backup of the Indexed data on Amazon’s S3.
  • Transferring data from one cluster to another. 
  • Writing Hbase queries to retrieve the data for other teams.

big data engineer

  • Designed and built data processing pipelines using tools and frameworks in the Hadoop ecosystem
  • Designed and built ETL pipelines to automate ingestion data and facilitate data analysis
  • Built Streaming services including Window Processing using Flink
  • Built Batch services including customized transparent Thrift-Server on data stored on HDFS and Cassandra using Spark
  • Designed the Kafka Topic-partition and Cassandra schema regarding the processing criteria

senior big data engineer

  • Founding member of Consus R&D team to build data infrastructure event streaming for 800 million users per month
  • Led the organization-wide effort to make data lineage, data modeling standardization, and data dictionary.
  • Implementation of data ingestion in Apache Cassandra. 
  • Daily Updates to the Director on the progress of development and tasks assigned.

big data engineer

  • As a BigData Engineer, responsible for the development and support of big data projects including requirement analysis and cross-functional team interactions. 
  • Currently taking care of multiple application build for Barclaycard Business on Cloudera Hadoop ecosystem 
  • Taking handover of new application coming in production from Dev Team.  
  • Support live services (Incidents, Problem, change Management). 
  • Development and deployment of the code of the new project.  
  • Working with the Admin team for patching and CDH upgrade activities. 
  • Status reporting of application in BAU to stakeholders. 

big data engineer

  • Maintenance of and development of the Big Data infrastructure for querying  Turk Telekom’s customer data consists of 40 million+ users and 1 billion transactions per day
  • Development of internal query tools for marketing, visualization and customer behaviour on top of Hadoop ecosystem(Hive, impala)
  • Development of real time spatial visualisation using Hadoop, Spark and Kafka
  • Churn prediction

big data engineer

  • Worked on Apache Nifi for data ingestion, data orchestration,data routing.
  • Designing and developing complex Big data pipelines to get data from various external sources and help in providing insightful data.
  • Performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
  • Experience on AWS services like S3, EC2, DynamoDB and Lambda.
  • Trained freshers by hosting sessions/talks on new technologies

big data engineer

  • Provided status report of team activities against the plan or schedule, inform task accomplishment, issues and status. Coordinated and track the reviews, documentation, test activities
  • Coordinated meetings with the functional managers to discuss project impediments, needed resources or issues/delays in completing the task. 
  • Worked on designing and developing the matching algorithm solution using Spark and Scala.• Leveraged SQOOP, Spark SQL, and Scala to build a robust pipeline for a data repository that is made in HDFS. 
  • Helped and trained to develop the team members. Ensured deliverable is prepared to satisfy the project requirements, schedule. 

big data engineer

  • Developed Map-reduce jobs for Processing the 
  • Write, analyze, review, and rewrite programs, using workflow chart and diagram, and applying knowledge of computer capabilities, subject matter, and symbolic logic.Big Data 
  • Designed And Developed Oozie WorkFlows for Daily Scheduiing
  • Querying and  the Hive Data Warehouse and involved in effective Partitioning  

big data engineer

  • Write, analyze, review, and rewrite programs, using workflow chart and architecture diagram.
  • Part of Change management team which reviews CR’s going in all Hadoop environments. 
  • Worked on Service Improvement activity to reduce recurrent failures, reducing batch run time of application. 
  • Conducted review meeting with Dev team to highlight Data/Logic issues in the application.