Data Engineer
AI/ML Computational Science Senior Analyst
|
Full time
|
Experience: 2-5 years
Job No. ATCI-5682610-S2058781
|
Pune
|
Required Skill: Data Engineering,Data Architecture Principles,Databricks Unified Data Analytics Platform,Microsoft Azure Data Services
Project Role : Data Engineer
Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
Must have skills : Data Engineering, Data Architecture Principles, Databricks Unified Data Analytics Platform, Microsoft Azure Data Services
Good to have skills : NA
Minimum 3 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
As a Data Engineer, a typical day involves designing, developing, and maintaining comprehensive data solutions that support the generation, collection, and processing of data. This role requires creating efficient data pipelines and ensuring the integrity and quality of data throughout its lifecycle. The position also involves implementing processes to extract, transform, and load data, facilitating seamless migration and deployment across various systems. Collaboration with different teams to align data strategies and optimize workflows is an integral part of daily activities, ensuring that data infrastructure supports organizational needs effectively and reliably. Utilizes software engineering principles to deploy and maintain fully automated data transformation pipelines that combine a large variety of storage and computation technologies to handle a distribution of data types and volumes in support of data architecture design. A Data Engineer designs data products and data pipelines that are resilient to change, modular, flexible, scalable, reusable, and cost effective.
Roles & Responsibilities:
- Design, develop, and maintain data pipelines and ETL processes using Microsoft Azure services (e.g., Azure Data Factory, Azure Synapse, Azure Databricks, Azure Fabric)
- Utilize Azure data storage accounts for organizing and maintaining data pipeline outputs. (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage)
- Collaborate with data scientists, data analysts, data architects and other stakeholders to understand data requirements and deliver high-quality data solutions
- Optimize data pipelines in the Azure environment for performance, scalability, and reliability
- Ensure data quality and integrity through data validation techniques and frameworks
- Develop and maintain documentation for data processes, configurations, and best practices
- Monitor and troubleshoot data pipeline issues to ensure timely resolution
- Stay current with industry trends and emerging technologies to ensure our data solutions remain cutting-edge
- Manage the CI/CD process for deploying and maintaining data solutions.
- Expected to be an SME, collaborate and manage the team to perform.
- Responsible for team decisions.
- Engage with multiple teams and contribute on key decisions.
- Provide solutions to problems for their immediate team and across multiple teams.
- Lead the design and implementation of scalable data architectures to support business requirements.
- Oversee the continuous improvement of data processing workflows to enhance efficiency and reliability.
- Mentor junior team members to foster skill development and knowledge sharing within the team.
Professional & Technical Skills:
- Must To Have Skills: Proficiency in Data Engineering, Data Architecture Principles, Databricks Unified Data Analytics Platform, Microsoft Azure Data Services.
- Strong experience in building and optimizing data pipelines and ETL frameworks.
- In-depth knowledge of cloud-based data storage and processing solutions.
- Ability to work with large datasets and ensure data quality and consistency.
- Familiarity with scripting and programming languages commonly used in data engineering.
- Experience in monitoring and troubleshooting data workflows to maintain system performance.
- Learning agility
- Technical Leadership
- Consulting and managing business needs
- Strong experience in Python is preferred but experience in other languages such as Scala, Java, C#, etc is accepted
- Experience building spark applications utilizing PySpark.
Additional Information:
- The candidate should have minimum 5 years of experience in Data Engineering.
- This position is based at our Pune office.
- A 15 years full time education is required.
Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
Must have skills : Data Engineering, Data Architecture Principles, Databricks Unified Data Analytics Platform, Microsoft Azure Data Services
Good to have skills : NA
Minimum 3 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
As a Data Engineer, a typical day involves designing, developing, and maintaining comprehensive data solutions that support the generation, collection, and processing of data. This role requires creating efficient data pipelines and ensuring the integrity and quality of data throughout its lifecycle. The position also involves implementing processes to extract, transform, and load data, facilitating seamless migration and deployment across various systems. Collaboration with different teams to align data strategies and optimize workflows is an integral part of daily activities, ensuring that data infrastructure supports organizational needs effectively and reliably. Utilizes software engineering principles to deploy and maintain fully automated data transformation pipelines that combine a large variety of storage and computation technologies to handle a distribution of data types and volumes in support of data architecture design. A Data Engineer designs data products and data pipelines that are resilient to change, modular, flexible, scalable, reusable, and cost effective.
Roles & Responsibilities:
- Design, develop, and maintain data pipelines and ETL processes using Microsoft Azure services (e.g., Azure Data Factory, Azure Synapse, Azure Databricks, Azure Fabric)
- Utilize Azure data storage accounts for organizing and maintaining data pipeline outputs. (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage)
- Collaborate with data scientists, data analysts, data architects and other stakeholders to understand data requirements and deliver high-quality data solutions
- Optimize data pipelines in the Azure environment for performance, scalability, and reliability
- Ensure data quality and integrity through data validation techniques and frameworks
- Develop and maintain documentation for data processes, configurations, and best practices
- Monitor and troubleshoot data pipeline issues to ensure timely resolution
- Stay current with industry trends and emerging technologies to ensure our data solutions remain cutting-edge
- Manage the CI/CD process for deploying and maintaining data solutions.
- Expected to be an SME, collaborate and manage the team to perform.
- Responsible for team decisions.
- Engage with multiple teams and contribute on key decisions.
- Provide solutions to problems for their immediate team and across multiple teams.
- Lead the design and implementation of scalable data architectures to support business requirements.
- Oversee the continuous improvement of data processing workflows to enhance efficiency and reliability.
- Mentor junior team members to foster skill development and knowledge sharing within the team.
Professional & Technical Skills:
- Must To Have Skills: Proficiency in Data Engineering, Data Architecture Principles, Databricks Unified Data Analytics Platform, Microsoft Azure Data Services.
- Strong experience in building and optimizing data pipelines and ETL frameworks.
- In-depth knowledge of cloud-based data storage and processing solutions.
- Ability to work with large datasets and ensure data quality and consistency.
- Familiarity with scripting and programming languages commonly used in data engineering.
- Experience in monitoring and troubleshooting data workflows to maintain system performance.
- Learning agility
- Technical Leadership
- Consulting and managing business needs
- Strong experience in Python is preferred but experience in other languages such as Scala, Java, C#, etc is accepted
- Experience building spark applications utilizing PySpark.
Additional Information:
- The candidate should have minimum 5 years of experience in Data Engineering.
- This position is based at our Pune office.
- A 15 years full time education is required.
15 years full time education
Pune
平等就业机会声明
所有聘用决定均不考虑年龄、种族、信仰、肤色、宗教、性别、国籍、血统、残疾状况、退伍军人身份、性取向、性别认同或表达、基因信息、婚姻状况、公民身份或任何其他受联邦、州或地方法律保护的因素。
求职者在招聘过程中没有义务披露已封存或已删除的定罪或逮捕记录。
埃森哲致力于为我们的男女军人提供退伍军人就业机会。
请阅读埃森哲的招聘和聘用声明，了解更多关于我们在招聘和聘用过程中如何处理您的数据的信息。
We work with one shared purpose: to deliver on the promise of technology and human ingenuity. Every day, more than 775,000 of us help our stakeholders continuously reinvent. Together, we drive positive change and deliver value to our clients, partners, shareholders, communities, and each other.
We believe that delivering value requires innovation, and innovation thrives in an inclusive and diverse environment. We actively foster a workplace free from bias, where everyone feels a sense of belonging and is respected and empowered to do their best work.
At Accenture, we see well-being holistically, supporting our people’s physical, mental, and financial health. We also provide opportunities to keep skills relevant through certifications, learning, and diverse work experiences. We’re proud to be consistently recognized as one of the World’s Best Workplaces™.
Join Accenture to work at the heart of change. Visit us at www.accenture.com.