← Все заказы

Databricks Ingestion Lead

emagine Polska

14 ч назад

Текст заказа

This role is responsible for leading the ingestion workstream that reliably transfers all source data, starting with Oracle, into the bronze layer while providing a functional ingestion framework.
Main Responsibilities
• Own the ingestion inventory: sources, tables, volumes, load patterns and priorities.
• Choose the ingestion pattern per source (Lakeflow Connect, Auto Loader, Azure Data Factory) and document the rationale.
• Design and build a metadata-driven ingestion framework, allowing new sources to be added through configuration rather than new code.
• Implement Change Data Capture (CDC) and incremental loads, accommodating schema changes, deletions, and late-arriving data.
• Design for idempotency and replay through checkpoints, watermarks, and controlled reprocessing.
• Establish monitoring, alerting, load SLAs, and data-quality gates at the bronze layer.
• Plan and execute cut-over operations for each source, including parallel runs, reconciliation, and sign-offs.
• Lead and mentor 2-3 data engineers, and ensure proper handover of the framework and runbooks.
Key Requirements
Extensive experience with Databricks and data ingestion frameworks.
Proficiency in Oracle and related data management.
Strong understanding of CDC and data load strategies.
Experience in metadata-driven architecture design.
Ability to lead and mentor engineering teams.
Nice to Have
Familiarity with Azure Data Factory.
Knowledge of data quality monitoring tools.
Experience with performance optimization of data ingestion processes.
Start: ASAP
Duration: 3-4 months
Location: Copenhagen

Похожие заказы