Azure Data Engineer
Date:
Nov 9, 2024
Location:
Bangalore, KA, IN, 560100
Req ID:
27749
Summary
As an Azure Data Engineer at Gainwell, you will be a part of an innovative healthcare technology company with a mission to serve the healthcare needs of our communities.
Primary skills:
Candidates should possess skills in
- ADF pipeline set up and management
- SQL programming (MS SQL Server – SSMS)
- Issue Resolution & Troubleshooting in Azure environments
- Testing and validations
- Automation activities
- Set up ETL activities in ADF framework
- Identify process efficiencies opportunities
Secondary skills:
- Analyze logs and set up monitoring mechanisms
- Security & Access Management
- Problem solving and root cause analysis
Candidate should have expertise in:
- Azure Data Factory, Azure Data Lake, and Azure SQL Database
- Azure DevOps (for managing CI/CD processes, including pipeline setup, tracking, and performance monitoring)
- Working knowledge of data integration patterns and ETL processes
Key Responsibilities:
Data Pipeline Management:
- Build and execute ADF pipelines for continuous integration/continuous deployment (CI/CD) processes.
- Set up and configure multi-stage pipelines and various environment configurations for development, testing, and production.
- Track pipeline performance and ensure the health of applications, taking corrective actions when needed.
- Design processes for monitoring pipeline activities and system health.
- Integrate data from multiple sources, including SQL databases and third-party APIs.
- Develop data integrations between source and destination systems to support operational and reporting needs.
- Build infrastructure for data extraction, transformation, and loading (ETL) using MS SQL and Azure technologies.
- Identify opportunities for automating tasks within the ADF framework and execute enhancements to improve efficiency.
- Set up Azure Monitor to track performance metrics, logs, and alerts for Azure resources.
SQL Operations:
- Run SQL programs within Azure environments, optimizing for performance and data accuracy.
- Work with data in relational databases and assist with query optimization and troubleshooting.
Issue Resolution & Troubleshooting:
- Perform root cause analysis and debugging to address issues related to application performance.
- Coordinate with the Azure IT team to troubleshoot memory leaks, inefficient memory usage, and other performance bottlenecks in applications.
- Provide rapid response to resolve critical system errors and ensure minimal downtime.
Logs & Monitoring:
- Create and manage rules and alerts for system monitoring.
- Analyze logs from various Azure resources to detect and resolve performance issues.
- Set up comprehensive logging and monitoring solutions for CI/CD processes to ensure system reliability.
Testing & Validation:
- Conduct thorough testing and validation of pipelines and infrastructure before production deployment.
- Collaborate with the IT and development teams to perform functional, performance, and regression testing.
Security & Access Management:
- Coordinate with IT teams to manage access control, granting or revoking access as needed and raising tickets for user management.
- Maintain documentation of current pipelines, refresh processes, and security configurations.
- Coordinate and set up various security levels in reports and applications to ensure data privacy and compliance with security protocols.
- Assist in the review and audit of Azure infrastructure architecture, providing insights for improvements and optimizations.
- Work on creating a unified strategy for Azure cloud transformation, covering governance, automation, security, and operational aspects.
Other Responsibilities:
- Assist in the creation and execution of monthly refresh plans to keep data infrastructure updated.
- Maintain up-to-date documentation for all data pipelines, processes, and operational tasks.
What We’re Looking For:
The ideal candidate should have the following qualifications:
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- Azure certifications such as Azure Data Engineer Associate (or equivalent certifications).
- Working knowledge of Azure DevOps for managing CI/CD processes, including pipeline setup, tracking, and performance monitoring.
- Proficiency in Power BI for reporting and data visualization.
- Adequate experience in Azure Data Factory, Azure Data Lake, and Azure SQL Database.
- Strong working knowledge of data integration patterns and ETL processes for seamless data transfer and transformations.
- Familiarity with Azure Log Analytics to query and analyze logs from various Azure resources, providing insight into performance and system health.
- Good understanding of Azure environments, infrastructure, and resource management.
Soft Skills:
- Good verbal and written communication skills to effectively collaborate with cross-functional teams and business stakeholders.
- Attention to detail and a proactive approach to identifying and addressing potential issues.
- Self-driven with the ability to work independently and manage priorities efficiently.
Other Requirements:
- Ability to work on evening shift to maximize overlap with US teams, ensuring efficient communication and collaboration.
- Eye for details in monitoring, troubleshooting, and pipeline execution.