Data Engineer (SQL, Spark/Pyspark, Databricks)

Summary

Gainwell Technologies seeks experienced Data Engineers specializing in Databricks, Apache Spark, and AWS to join their data analytics team. You will design, develop, and optimize data frameworks for collection, transformation, storage, and reporting on AWS, supporting analytic objectives. This role is strictly involved in the development of the product and does not involve access to Protected Health Information (PHI) & Personally Identifiable Information (PII) or any secured/confidential client data. The work is limited to application development and does not include handling or processing of sensitive health information.

Your role in our mission

• Design and development of highly scalable, robust data pipelines and ETL/ELT solutions using Databricks and Apache Spark.
• Provide expertise in Databricks workspace, cluster, and job management, focusing on performance tuning, cost optimization (DBU reduction), autoscaling, and scheduling.
• Define and enforce technical standards, design patterns, and best practices for data modelling, quality, and schema management across the platform.
• Drive the adoption of DataOps principles by integrating Databricks workloads with enterprise CI/CD tools (e.g., Azure DevOps, GitHub Actions) for automated code and model lifecycle management.
• Act as the technical liaison, engaging with stakeholders to gather requirements, communicating complex designs, and ensuring data solutions meet business needs for quality and consistency.
• Provide technical guidance and mentorship to junior team members, lead project planning and estimation, and create clear architecture and process documentation.
• Focus on maintaining, debugging, and optimizing existing data systems to ensure reliability and low technical debt.
• Ensure data extraction, transformation, loading, normalization, and cleansing.
• Collaborate on the development and documentation of data models, schemas, and source-to-target mappings within the data warehouse.
• Conceptualize and visualize scalable data frameworks.
• Communicate effectively with internal and external stakeholders to support project objectives.

What we're looking for

• Demonstrated expertise in the Databricks platform (AWS/Azure), including deep knowledge of Spark fundamentals, Delta Lake architecture, and Unity Catalog.
• Strong knowledge of cloud platforms (AWS preferred) and advanced proficiency in SQL, data modelling, data warehousing concepts, and SQL optimization.
• Proven experience with CI/CD tools and hands-on implementation of DataOps and MLOps principles.
• Familiarity and experience in implementing data governance and data quality practices.
• Demonstrated ability in project planning, low-level design (LLD), accurate estimation, and managing project delivery at a team level.
• Excellent verbal and written communication skills with proven experience in effective stakeholder management.
• Knowledge of Agile Scrum and SAFe methodologies, with a strong sense of ownership and accountability.
• Proven track record of mentoring and guiding team members to foster technical growth.
• Strong debugging skills and the ability to ask critical questions drive clarity in complex technical situations.                                                                                                     
• Basic Qualifications:

• 6+ years of overall experience with minimum 4 years’ experience with big data technologies on AWS, Azure.
• Bachelor’s degree in computer science or a related field, or equivalent education.
• At least 3 years’ hands-on experience with Databricks or Apache Spark (Python/Scala).
• Proficient in working with various database structures, including transactional and data warehouse environments.
• Databricks and AWS developer/architect certifications are highly desirable.
• Demonstrated ability to lead and mentor engineering teams.

What you should expect in this role

• Excellent communication abilities.
• Capacity to deliver impactful results across multiple concurrent projects.
• Strategic thinker with the ability to balance immediate and long-term objectives.
• Willingness to travel as required.
• Some evening and weekend hours may be required.