← Все заказы

AWS Data Engineer

Capco

12 ч назад

Текст заказа

Job Title: Data Engineer
About Us
“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount . With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery.
WHY JOIN CAPCO?
You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry.
MAKE AN IMPACT
Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.
#BEYOURSELFATWORK
Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.
CAREER ADVANCEMENT
With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.
DIVERSITY & INCLUSION
We believe that diversity of people and perspective gives us a competitive advantage.
Job Description:
Role: AWS Data Engineer
Location : Bangalore, Pune
Experience-6-12 yrs
Databricks Development and Data Engineering
Design, develop, test, and maintain scalable data pipelines and transformation workflows using Databricks.
Build and optimise data processing solutions using Apache Spark, PySpark, and SQL.
Develop batch and, where required, streaming data pipelines to support enterprise data processing requirements.
Implement robust data ingestion, cleansing, transformation, and enrichment processes across multiple data sources.
Develop and maintain Delta Lake tables, applying appropriate data modelling, partitioning, and optimisation strategies.
Implement reusable, modular, and maintainable data engineering frameworks and components.
Ensure data pipelines meet agreed performance, reliability, scalability, and quality standards.
Data Platform Design and Optimisation
Contribute to the design and implementation of cloud-based data platforms and lakehouse architectures.
Apply Medallion Architecture principles, including Bronze, Silver, and Gold data layers, where appropriate.
Optimise Spark workloads through efficient transformations, query tuning, cluster configuration, and resource management.
Implement appropriate data storage, partitioning, and data lifecycle management strategies.
Monitor and troubleshoot data processing workloads, identifying performance bottlenecks and implementing improvements.
Support the implementation of secure and scalable data platform solutions aligned with enterprise architecture standards.
Data Integration and Orchestration
Design and develop data ingestion and integration pipelines connecting structured, semi-structured, and unstructured data sources.
Implement data orchestration workflows using Databricks Workflows, AWS, or equivalent technologies.
Support integration with enterprise databases, APIs, cloud storage, and downstream analytical platforms.
Develop data validation, reconciliation, and quality control processes to ensure data accuracy and consistency.
Implement appropriate error handling, logging, monitoring, and recovery mechanisms.
Support data migration and modernisation initiatives, including migration from legacy data platforms to cloud-based solutions.
Software Engineering and DevOps
Apply software engineering best practices to data pipeline development, including modular design, code reviews, automated testing, and documentation.
Develop and maintain CI/CD pipelines for deploying Databricks code and configurations across environments.
Use Git, Azure DevOps or equivalent tools for source control, collaboration, and release management.
Implement unit testing, integration testing, and data quality testing for data engineering solutions.
Support automated deployment processes using Databricks Asset Bundles, Terraform, or equivalent tools where appropriate.
Troubleshoot and resolve defects across development, testing, and production environments.
Contribute to continuous improvement of development standards, reusable frameworks, and engineering practices.
Data Governance, Security and Reliability
Implement appropriate data access controls, security configurations, and governance standards within Databricks.
Work with Unity Catalog or equivalent governance capabilities to support data discovery, access management, and lineage.
Apply data quality, metadata management, and data lifecycle principles.
Support compliance with organisational security policies and relevant financial services regulatory requirements.
Implement monitoring and alerting to improve pipeline reliability and operational resilience.
Collaborate with platform, security, and architecture teams to ensure solutions meet enterprise standards.
Consulting, Technical Leadership and Stakeholder Engagement
Engage with client stakeholders to understand business requirements and translate them into scalable technical solutions.
Take ownership of complex technical deliverables or defined data engineering workstreams.
Contribute to solution architecture discussions, technical design decisions, and implementation planning.
Provide technical guidance, code reviews, and mentoring to junior engineers and consultants.
Communicate technical concepts, delivery progress, risks, and dependencies to technical and non-technical stakeholders.
Collaborate with cross-functional teams to ensure alignment between data engineering solutions and business objectives.
Identify opportunities to improve data engineering practices, platform performance, and delivery efficiency.
Support technical estimation, sprint planning, and delivery coordination within Agile teams.
Required Skills and Experience
Essential Technical Skills
Typically 6+ years of experience in data engineering, software engineering, or related technology roles, including substantial hands-on experience with Databricks.
Strong practical experience developing data pipelines and transformation workloads using Databricks.
Advanced proficiency in Python, PySpark, and SQL.
Strong understanding of Apache Spark architecture, distributed data processing, and performance optimisation.
Hands-on experience with Delta Lake, including data ingestion, transformation, optimisation, and data management.
Experience designing and implementing ETL/ELT pipelines for large-scale data processing.
Good understanding of lakehouse architecture, data modelling, and data warehousing concepts.
Strong knowledge and hands-on experience of AWS development in some of the following : Lambda, Glue, Step Functions, IAM roles, Lakeformation, EventBridge, SNS, SQS, EC2,Security Groups, CloudFormation, RDS, DynamoDB, Redshift
Experience with data orchestration tools such as Databricks Workflows or Azure Data Factory.
Practical experience implementing data quality checks, error handling, logging, and monitoring.
Experience using Git and CI/CD pipelines within a data engineering environment.
Understanding of data security, access controls, and governance principles.
Strong troubleshooting and performance-tuning capabilities for Spark and Databricks workloads.
Experience working within Agile software delivery environments.
Consulting and Professional Skills
Strong analytical thinking and structured problem-solving capabilities.
Ability to independently own and deliver complex technical workstrea

Похожие заказы