Azure Data Engineer

Summary

As an Azure Data Engineer at Gainwell, you will be a part of an innovative healthcare technology company with a mission to serve the healthcare needs of our communities.

Primary skills: [ Must have 5 or more years experiene in ADF, SSMS, SQL, PowerBI ]

Candidates should possess skills in

• ADF pipeline set up and management
• SQL programming (MS SQL Server – SSMS)
• Issue Resolution & Troubleshooting in Azure environments
• Testing and validations
• Automation activities
• Set up ETL activities in ADF framework
• Identify process efficiencies opportunities

Secondary skills:

• Analyze logs and set up monitoring mechanisms
• Security & Access Management
• Problem solving and root cause analysis

Candidate should have expertise in:

• Azure Data Factory, Azure Data Lake, and Azure SQL Database
• Azure DevOps (for managing CI/CD processes, including pipeline setup, tracking, and performance monitoring)
• Working knowledge of data integration patterns and ETL processes

Key Responsibilities:

Data Pipeline Management:

• Build and execute ADF pipelines for continuous integration/continuous deployment (CI/CD) processes.
• Set up and configure multi-stage pipelines and various environment configurations for development, testing, and production.
• Track pipeline performance and ensure the health of applications, taking corrective actions when needed.
• Design processes for monitoring pipeline activities and system health.
• Integrate data from multiple sources, including SQL databases and third-party APIs.
• Develop data integrations between source and destination systems to support operational and reporting needs.
• Build infrastructure for data extraction, transformation, and loading (ETL) using MS SQL and Azure technologies.
• Identify opportunities for automating tasks within the ADF framework and execute enhancements to improve efficiency.
• Set up Azure Monitor to track performance metrics, logs, and alerts for Azure resources.

SQL Operations:

• Run SQL programs within Azure environments, optimizing for performance and data accuracy.
• Work with data in relational databases and assist with query optimization and troubleshooting.

Issue Resolution & Troubleshooting:

• Perform root cause analysis and debugging to address issues related to application performance.
• Coordinate with the Azure IT team to troubleshoot memory leaks, inefficient memory usage, and other performance bottlenecks in applications.
• Provide rapid response to resolve critical system errors and ensure minimal downtime.

Logs & Monitoring:

• Create and manage rules and alerts for system monitoring.
• Analyze logs from various Azure resources to detect and resolve performance issues.
• Set up comprehensive logging and monitoring solutions for CI/CD processes to ensure system reliability.

Testing & Validation:

• Conduct thorough testing and validation of pipelines and infrastructure before production deployment.
• Collaborate with the IT and development teams to perform functional, performance, and regression testing.

Security & Access Management:

• Coordinate with IT teams to manage access control, granting or revoking access as needed and raising tickets for user management.
• Maintain documentation of current pipelines, refresh processes, and security configurations.
• Coordinate and set up various security levels in reports and applications to ensure data privacy and compliance with security protocols.
• Assist in the review and audit of Azure infrastructure architecture, providing insights for improvements and optimizations.
• Work on creating a unified strategy for Azure cloud transformation, covering governance, automation, security, and operational aspects.

Other Responsibilities:

• Assist in the creation and execution of monthly refresh plans to keep data infrastructure updated.
• Maintain up-to-date documentation for all data pipelines, processes, and operational tasks.

What We’re Looking For:

The ideal candidate should have the following qualifications:

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Azure certifications such as Azure Data Engineer Associate (or equivalent certifications).
• Working knowledge of Azure DevOps for managing CI/CD processes, including pipeline setup, tracking, and performance monitoring.
• Proficiency in Power BI for reporting and data visualization.
• Adequate experience in Azure Data Factory, Azure Data Lake, and Azure SQL Database.
• Strong working knowledge of data integration patterns and ETL processes for seamless data transfer and transformations.
• Familiarity with Azure Log Analytics to query and analyze logs from various Azure resources, providing insight into performance and system health.
• Good understanding of Azure environments, infrastructure, and resource management.

Soft Skills:

• Good verbal and written communication skills to effectively collaborate with cross-functional teams and business stakeholders.
• Attention to detail and a proactive approach to identifying and addressing potential issues.
• Self-driven with the ability to work independently and manage priorities efficiently.

Other Requirements:

• Ability to work on evening shift to maximize overlap with US teams, ensuring efficient communication and collaboration.
• Eye for details in monitoring, troubleshooting, and pipeline execution.