Senior Data Engineer

Senior Data Engineer (AWS & Confluent Data/AI Projects)

Work setup: Remote

Schedule :10am-6pm SGT

Required Skills and Qualifications:

• Bachelor's or Master's degree in Computer Science, Software Engineering, or a related quantitative field.
• 3 to 5 years of experience in data engineering, with a significant focus on cloud-based solutions.
• Strong expertise in AWS data services (S3, Glue, EMR, Redshift, Kinesis, Lambda, etc.).
• Extensive hands-on experience with Confluent Platform/Apache Kafka for building real-time data streaming applications.
• Proficiency in programming languages such as Python, PySpark, Scala, or Java.
• Expertise in SQL and experience with various database systems (relational and NoSQL).
• Solid understanding of data warehousing, data lakes, and data modeling concepts (star schema, snowflake schema, etc.).
• Experience with CI/CD pipelines and DevOps practices (Git, Terraform, Jenkins, Azure DevOps, or similar).
• Familiarity with big data processing frameworks like Apache Spark.
• Proven ability to design and implement scalable, fault-tolerant, and secure data architectures.
• Experience working on data projects that support Machine Learning and Artificial Intelligence initiatives.
• Strong problem-solving skills and the ability to troubleshoot complex data issues.
• Excellent communication, interpersonal, and collaboration skills.

Preferred Qualifications (Nice to Have):

• AWS Certifications (e.g., AWS Certified Data Analytics - Specialty, AWS Certified Solutions Architect - Associate/Professional).
• Experience with other streaming technologies (e.g., Flink).
• Knowledge of containerization technologies (Docker, Kubernetes).
• Familiarity with Data Mesh or Data Fabric concepts.
• Experience with data visualization tools (e.g., Tableau, Power BI, QuickSight).
• Understanding of MLOps principles and tools.

Key Responsibilities:

● Architect and Design Data Solutions: Lead the design and architecture of scalable, secure, and efficient data pipelines for both batch and real-time data processing on AWS. This includes data ingestion, transformation, storage, and consumption layers.

● Confluent Kafka Expertise: Design, implement, and optimize highly performant and reliable data streaming solutions using Confluent Platform (Kafka, ksqlDB, Kafka Connect, Schema Registry). Ensure efficient data flow for real-time analytics and AI applications.

● AWS Cloud Native Development: Develop and deploy data solutions leveraging a wide range of AWS services, including but not limited to:

● Data Storage: S3 (Data Lake), RDS, DynamoDB, Redshift, Lake Formation.

● Data Processing: Glue, EMR (Spark), Lambda, Kinesis, MSK (for Kafka integration).

● Orchestration: AWS Step Functions, Airflow (on EC2 or MWAA).

● Analytics & ML: Athena, QuickSight, SageMaker (for MLOps integration).

● Data Pipeline Development: Build, optimize, and maintain complex ETL/ELT pipelines using Python, PySpark, Scala, or Java, ensuring data quality, reliability, and performance.

● AI/ML Integration: Collaborate closely with Data Scientists and Machine Learning Engineers to integrate data pipelines that support the training, deployment, and monitoring of AI/ML models, including potential applications of Generative AI.

● Data Governance & Quality: Implement and enforce data governance policies, data quality checks, and data security best practices (e.g., IAM roles, encryption, VPC networking) across all data assets.

● Performance Optimization: Continuously monitor, tune, and optimize data processing jobs, data storage, and streaming infrastructure for performance, cost-efficiency, and scalability.

● Mentorship and Leadership: Provide technical guidance, mentorship, and support to junior data engineers, fostering a culture of technical excellence and continuous learning.

● Collaboration & Communication: Work effectively with cross-functional teams including Data Scientists, Business Analysts, Product Managers, and other Engineering teams to translate business requirements into robust technical solutions.

● Documentation: Create and maintain comprehensive technical documentation, including data flow diagrams, architectural designs, and operational runbooks.

● Stay Current: Research and evaluate new technologies, tools, and best practices in data engineering, cloud computing, and AI/ML to drive innovation.

Job Type: Full-time

Pay: Php150, Php200,000.00 per month

Benefits:

• Work from home

Experience:

• Data engineering (cloud-based solutions): 3 years (Required)

License/Certification:

• AWS Certification (Required)

Work Location: Remote