Текст заказа
Lead Data Engineer
Компания: SNI
Город: Warszawa
Удалённо
SNI is serving as a trusted IT Outsourcing partner in line with the needs of World's most prestigious firms and carried out successful projects worldwide. Responsibilities: Lead data engineering work streams and hands-on technical implementation across multi-functional squads. Design and deploy resilient, scalable batch and streaming pipelines and data processing frameworks. Drive platform migrations from legacy schedulers to modern, cloud-native workflow orchestration tools. Establish data governance policies, lineage tracking, metadata management, and role-based security controls (e.g., Unity Catalog). Implement data virtualization and unified integration patterns across disparate, multi-source enterprise systems. Lead performance tuning, query optimization, cost management, and operational reliability across the data platform. Skills: 6–9 years of professional experience in data engineering and enterprise data platform delivery. Expert-level hands-on experience with Databricks and Apache Spark. Deep knowledge of Data Lakehouse architecture (Delta Lake, medallion architecture, schema enforcement). Advanced proficiency in Python, SQL, and PySpark for data transformation and distributed computing. Strong background in workflow orchestration tools (e.g., Airflow, Databricks Workflows, Dagster) and scheduler migration projects. Practical experience implementing data governance, cataloging, metadata, and data access security frameworks. Proven track record architecting data virtualization and integration solutions across heterogeneous cloud and on-premise sources. PLDE2628