Data Pipeline Engineer

Key ResponsibilitiesPipeline Development: Design and maintain end-to-end streaming pipelines using Dataflow and Apache Spark to handle high-velocity data.Specialized Integration: Manage and process niche data streams including LiDAR sensor data, VETA, and Baggage tracking systems.Stream Management: Architect robust messaging patterns using GCP Pub/Sub to...

We are looking for a Data Engineer to join the team of one of our client from the Philippines. This client specializes in developing and implementing systems and applications to increase operational efficiency. They create an environment by providing responsive IT services, solutions, and innovations that positively impact the lives of companies.It's a...

The Data Engineer is IT Team in delivering and supporting Data Analytics Requirements. The resource is expected to collaborate with various stakeholders to deliver needed solutions across the organization. Technical expertise is highly expected from the resource to provide solutions and architecture as needed. The resource will report to the D& IT Business...

Critical Work Functions & Key TasksSource integrations, transforms, DQ rules, testingOptimise solution performance Develop tools to improve data flows between internal and/or external systems andthe data warehouse Test data system configurations to increase efficiency Automate the data collection and analysis processes, data releasing and...

OverviewAs a Software Engineer at Agile Market Intelligence, you will take ownership of our market intelligence technology platform, maintaining and enhancing our cloud-based data pipelines, analytics tools, and reporting systems. You will manage the complete technology stack, from BigQuery data warehouse and Cloud Run infrastructure to React frontends and...

DATA ENGINEERThis is for a full-time position.Job SummaryThe Data Engineer will be responsible for designing and implementing data pipelines using the Microsoft Fabric Data Platform. This involves creating Notebooks to extract, transform, and load data. The role will receive technical guidance from a Senior Data Engineer based in Sydney and day-to-day...

TP ICAP operates at the heart of the world's financial, energy and commodities markets.We are professional intermediaries playing a pivotal role in the world's financial markets, covering FX, Rates, Credit, Equities, Energy & Commodities.Globally, we are a leading provider of market participants, with execution via a range of regulated venues, covering a...

We are seeking a highly skilled and innovative Data Scientist with expertise in Artificial Intelligence (AI), Machine Learning (ML), and Large Language Models (LLMs) to join our team. This role involves designing, developing, and deploying advanced AI solutions—including multimodal LLMs and agentic AI systems—while analyzing large datasets and delivering...

Data Engineer - KafkaLocation: Onsite - BGC, TaguigSchedule: Shifting (must be amenable as required)About the RoleWe are looking for an Expert-Level Data/Integration Engineer with deep experience in Apache Kafka administration. You will be responsible for ensuring the stability, performance, and security of the organization's integration platforms. This role...

We are looking for a QA Data Analyst to ensure the accuracy, reliability, and quality of data across our analytics and reporting platforms. The role focuses on data testing, reconciliation, UAT support, and documentation, working closely with data engineers, analysts, and business stakeholders.This position is ideal for someone with a strong data mindset,...