Senior Data Engineer

The Senior Data Engineer designs, implements and oversees maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information from structured and unstructured sources in a scalable, repeatable and secure manner. They assist data analysts and data scientists with the extraction of valuable insights from data sets to derive valuable and actionable insights and recommendations that support business requirements. They are involved in rollouts, upgrades, implementation and release of data system changes as required for streamlining of internal practices. They are proficient in database systems, scripting and programming languages required by the organization. They are also familiar with the relevant software platforms on which the solution is deployed on. The Senior Data Engineer possesses natural inclination for understanding business processes and relevant data requirements. They easily build rapport with others and is able to put forth their ideas and recommendations in a persuasive manner, to influence stakeholders and decisions.

Critical Work Functions & Key Tasks

Optimize performance of solutions

• Evaluate existing technologies and technology practices
• Automate processes focusing on repeatability and reliability
• Identify opportunities for improvements and optimization to systems and processes
• Resolve data integrity, performance, and availability issues

Identify business needs

• Translate business' data needs to technical system requirements
• Evaluate suitability of technologies and products in database and data processing for integration and storage
• Assess the suitability of data structures to ensure availability, integrity, quality, scalability and accessibility of data

Build and maintain data pipeline

• Create data storage plans and solutions for information storage and extraction
• Build data flow channels and processing systems to extract, transform, load and integrate data
• Validate data extraction, preparation and processing systems for accuracy of data and outputs
• Lead project rollouts, upgrades, implementation and release of data system changes
• Build a metadata system to ensure documentation and cataloguing of all available data
• Analyze data system performance and develop solutions for improvements
• Ensure the security, privacy and anonymity of users accessing data systems
• Formulate data backup and fail-safe plans as part of business continuity planning

Technical Requirements

• Data Warehousing – Experience with data warehousing solutions such as Snowflake, Google BigQuery, or AWS Redshift.
• Cloud Data Platforms – Proficiency in cloud platforms like AWS (e.g., Redshift, S3, Glue), Google Cloud (BigQuery, Dataflow), or Microsoft Azure (Data Factory, Azure Synapse) for data engineering tasks.
• SQL and NoSQL Databases – Proficiency in both SQL (e.g., PostgreSQL, MySQL) and NoSQL (e.g., MongoDB, Cassandra) for managing and querying data.
• ETL Tools – Expertise in Extract, Transform, Load (ETL) tools such as Apache Nifi, Talend, or Airflow for automating data pipelines.
• Version Control (e.g., Git) – Familiarity with Git or other version control systems for managing code and pipeline versions.
• Python, Scala, or Java – Strong coding skills in languages like Python, Scala, or Java for building and maintaining data pipelines and tools.
• Tableau (preferred) or Power BI – Advanced proficiency for creating interactive data visualizations, dashboards, and reports.

Qualifications

• Bachelor's degree in engineering, Statistics, Computer Science, Data Science, Information Technology, Business Information System or Mathematics
• Can present to a wider group of peers and be convincing in relaying project requirements.
• 5 to 7 years of working experience