Lead Software Engineer (Big Data) Kuala Lumpur, Malaysia
Lead Software Engineer (Big Data) Description
Job #: 74609DESCRIPTION
We are looking for an experienced Lead Software Engineer (Big Data) to be placed in our Office in Bangsar South MY. This As a Lead Engineer will be roll-up the functional and technical aspect of Software Development with a big data technology in Project space to cloud services.
EPAM Systems, Inc. (EPAM) is a global product development, digital platform engineering, and digital and product design agency headquartered in the US. EPAM has been named 5 times in Forbes’ 25 Fastest Growing Public Tech Companies. Kuala Lumpur is the most recent location in which EPAM has acquired a new business. This is your opportunity to join an organization in start-up mode that plans to (at least) double in size every 2-3 years and the team that you are joining will be responsible for that.
Responsibilities
- Lead, design and implement innovative analytical solution using Hadoop, NoSQL and other Big Data related technologies, evaluating new features and architecture in Cloud/ on premise/ Hybrid solutions
- Work with product and engineering teams to understand requirements, evaluate new features and architecture to help drive decisions
- Build collaborative partnerships with architects and key individuals within other functional groups
- Perform detailed analysis of business problems and technical environments and use this in designing quality technical solution
- Actively participate in code review and test solutions to ensure it meets best practice specifications
- Build and foster a high-performance engineering culture, mentor team members and provide team with the tools and motivation
- Write project documentation
Requirements
- Candidate must possess at least Bachelor’s Degree in preferably Computer Science/ Information Technology or equivalent
- Candidates with a minimum of 6 years solid experience in Big Data technologies and Enterprise Software Development
- Engineering experience and practice in Data Management, Data Storage, Data Visualization, Disaster Recovery, Integration, Operation, Security
- Strong experience building data ingestion pipelines (simulating Extract, Transform, Load workload), Data Warehouse or Database architecture
- Strong experience with data modelling; hands-on development experience with modern Big Data components
- Cloud: experience in designing, automation, provisioning, deploying and administering scalable, available and fault tolerant systems
- Good understanding of CI/CD principles and best practices
- Analytical approach to problem-solving with an ability to work at an abstract level and gain consensus; excellent interpersonal, leadership and communication skills
- Motivated, independent, efficient and able to handle several projects: work under pressure with a solid sense for setting priorities
- Ability to work in a fast-paced (start-up like) agile development environment
- Strong experience in high load and IoT Data Platform architectures and infrastructures
- Vast experience with Containers and Resource Management systems: Docker, Kubernetes, Yarn
- Experience in direct customer communications
- Experience in technology / team leading of data-oriented projects
- Solid skills in infrastructure troubleshooting, support and practical experience in performance tuning and optimization, bottleneck problem analysis
- Experienced in different business domains
- English proficiency
Technologies
- Programming Languages: Java/ Scala; Python; SQL; Bash
- Big Data stack: Hadoop, Yarn, HDFS, MapReduce, Hive, Spark, Kafka, Flume, Sqoop, Zookeeper
- NoSQL: Cassandra/ Hbase; MongoDB
- Queues and Stream processing: Kafka Streams; Spark Streaming; Event Hub; IOT Hub; Storage Queues; Service Bus; Stream Analytics
- Data Visualization: Tableau, QlikView
- ETL & Streaming Pipelines: Pentaho; Talend; Apache Oozie, Airflow, NiFi; Streamsets
- Operation: Cluster operation, Cluster planning
- Search: Solr, Elasticsearch/ELK
- InMemory: Ignite, Redis
- Solid Cloud experience with more than 2 leading Cloud providers (AWS/Azure/GCP): Storage; Compute; Networking; Identity and Security; NoSQL; RDBMS and Cubes; Big Data Processing; Queues and Stream Processing; Serverless; Data Analysis and Visualization; ML as a service (SageMaker; TensorFlow)
- Enterprise Design Patterns (ORM, Inversion of Control etc.)
- Development Methods (TDD, BDD, DDD)
- Version Control Systems (Git, SVN)
- Testing: Component/ Integration Testing, Unit testing (JUnit)
- Deep understanding of SQL queries, joins, stored procedures, relational schemas; SQL optimization
- Experience in various messaging systems, such as Kafka, RabbitMQ
- Rest, Thrift, GRPC, SOAP
- Build Systems: Maven, SBT, Ant, Gradle
- Docker, Kubernetes, Yarn
We offer
- Friendly team and enjoyable working environment
- Work-life balance and flexible schedule
- Online training library, mentoring, career development and potential partial grant of certification
- Unlimited access to LinkedIn learning solutions
- Referral bonuses
- Compensation for sick leave and paid time off
- Opportunities for self-realization