Position Overview

In this role, you will utilize your experience as a Lead/Senior Software Engineer – Data Engineering to help design, develop, monitor, troubleshoot and optimize data pipelines on Databricks (AWS) platform.

The hire will be responsible for solutioning and productionizing data storage and retrieval problems to support better performance and scalability. They will be responsible for expanding and optimizing our data and data pipeline architecture, as well as establishing data flow into downstream systems. The Data Engineer will support our software developers, business analysts and data scientists on initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects.

They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products as well as willingness to learn source systems from Business perspective. The right candidate will be excited by the prospect of optimizing or even re-designing our company’s data architecture to support our next generation of products and data initiatives.

Responsibilities

• Design and implement end-to-end data pipelines using Python and Airflow, ensuring efficient scheduling, orchestration, and dependency management for complex workflows.
• Develop high-performance SQL queries for data extraction, transformation, and reporting, with a strong focus on query optimization and scalability across large datasets.
• Automate and modularize ETL processes leveraging Python scripting and reusable Airflow DAG components, adhering to best practices in software engineering and data governance.
• Work with AWS cloud services (S3, Glue, Redshift, Aurora RDS, CloudWatch) to enable scalable, secure, and cloud-native data processing and storage solutions.
• Integrate and transform data from multiple sources such as Alteryx, PostgreSQL, Oracle, and PL/SQL into unified data models to support analytics and reporting needs.

Qualifications

Required Skills:

• Very strong skills in the following technologies:
  • Databricks
  • PySpark
  • Python
  • SQL
  • Apache Airflow - DAGs, Tasks, Scheduled, Operators etc.
  • AWS Services- S3, EC2, Glue, Redshift, PostgreSQL
• Strong written and verbal communication skills with the ability to interact with all levels of the organization. 
• Ability to oversee the work of junior data engineers and mentor them
• Familiarity with agile methodology.

• Good working knowledge of data virtualization, data warehousing concepts, dimensional models and relational databases, and data management.
• Git Based Distributed SW Code Management using GitHub etc.
• SW Code deployment using best CI/CD practices.
• Create and maintain and continuously improve optimal data pipeline architecture.
• Solution performance and scalable data storage and retrieval methods for large datasets for heavy data reliant applications.
• Gain knowledge about large, complex data sets that meet functional / non-functional business requirements.
• Help the application team with performance tuning of queries and help establish best practices
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
• Work with stakeholders including the Product, Business Analysts, Data science and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Keep our data separated and secure across national boundaries through multiple data centers and AWS regions.
• Be attentive to alerts or issues related to the database and take actions to resolve anomalies.

Required Experience & Education: 

• 12 + years of experience
• College degree in any STEM based field.

About Evernorth Health Services