Sr Data Engineer
Amgen
Sr Data Engineer
Job Description
ABOUT AMGEN
What you will do
- Design, develop, and maintain scalable ETL/ELT pipelines to support structured, semi-structured, and unstructured data processing across the Enterprise Data Engineering for Biotech or Pharma functional knowledge of R&D.
- Implement real-time and batch data processing solutions, integrating data from multiple sources into a unified, governed data fabric architecture.
- Optimize big data processing frameworks using Apache Spark, Hadoop, or similar distributed computing technologies to ensure high availability and cost efficiency.
- Work with metadata management and data lineage tracking tools to enable enterprise-wide data discovery and governance.
- Ensure data security, compliance, and role-based access control (RBAC) across data environments.
- Optimize query performance, indexing strategies, partitioning, and caching for large-scale data sets.
- Develop CI/CD pipelines for automated data pipeline deployments, version control, and monitoring.
- Implement data virtualization techniques to provide seamless access to data across multiple storage systems.
- Collaborate with cross-functional teams, including data architects, business analysts, and DevOps teams, to align data engineering strategies with enterprise goals.
- Stay up to date with emerging data technologies and best practices, ensuring continuous improvement of Enterprise Data Fabric architectures.
- Model data for analytics and ML (star/snowflake, Data Vault, semantic layers) and implement robust ELT patterns (dbt or equivalent).
- Build and maintain a lakehouse/warehouse (e.g., Delta Lake/Iceberg/Hudi; Snowflake/Redshift/BigQuery) with partitioning, clustering, and cost/perf optimization.
- Orchestrate workflows with Airflow/Azure Data Factory/Prefect and implement CI/CD for data (Git-based deployments, environments, automated tests).
- Implement data quality and observability (Great Expectations/Deequ, expectations-as-code, lineage/metadata, SLOs and alerting with OpenTelemetry/Prometheus/Datadog).
- Enforce security and governance (RBAC/ABAC, encryption, secrets, tokenization), manage PII/PHI under GDPR/CCPA and secure SDLC for data.
- Partner with analytics, data science, and product to define interfaces, SLAs, and contracts; publish clear docs, runbooks, and diagrams.
- Lead technical discovery, RFCs, and POCs; evaluate vendor tools and guide integrations.
- Mentor engineers; raise the bar on code quality, reviews, and engineering practices.
What we expect of you
- Master’s degree with 6 - 8 years of experience in Computer Science, IT or related field OR Bachelor’s degree with 8 - 12 years of experience in Computer Science, IT or related field
- Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies.
- Proficiency in workflow orchestration, performance tuning on big data processing.
- Strong understanding of AWS services
- Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures.
- Ability to quickly learn, adapt and apply new technologies
- Strong problem-solving and analytical skills
- Excellent communication and teamwork skills
- Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices.
- Excellent analytical and troubleshooting skills
- Strong verbal and written communication skills
- Ability to work effectively with global, virtual teams
- High degree of initiative and self-motivation
- Ability to manage multiple priorities successfully
- Team-oriented, with a focus on achieving team goals
- Strong presentation and public speaking skills
Must-Have Skills
- Databricks
- PySpark
- SparkSQL
- Apache Spark
- AWS
- Python
- SQL
- Scaled Agile methodologies
- Workflow orchestration
- Performance tuning on big data processing
- Data Fabric
- Data Mesh
- Enterprise-wide data architectures
Locations
- Hyderabad, India
Salary
Salary not disclosed
Estimated Salary Rangehigh confidence
80,000 - 120,000 USD / yearly
Source: xAI estimated
* This is an estimated range based on market data and may vary based on experience and qualifications.
Skills Required
- Databricksintermediate
- PySparkintermediate
- SparkSQLintermediate
- Apache Sparkintermediate
- AWSintermediate
- Pythonintermediate
- SQLintermediate
- Scaled Agile methodologiesintermediate
- Workflow orchestrationintermediate
- Performance tuning on big data processingintermediate
- Data Fabricintermediate
- Data Meshintermediate
- Enterprise-wide data architecturesintermediate
Required Qualifications
- Master’s degree with 6 - 8 years of experience in Computer Science, IT or related field OR Bachelor’s degree with 8 - 12 years of experience in Computer Science, IT or related field (experience)
- Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies. (experience)
- Proficiency in workflow orchestration, performance tuning on big data processing. (experience)
- Strong understanding of AWS services (experience)
- Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures. (experience)
- Ability to quickly learn, adapt and apply new technologies (experience)
- Strong problem-solving and analytical skills (experience)
- Excellent communication and teamwork skills (experience)
- Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices. (experience)
- Excellent analytical and troubleshooting skills (experience)
- Strong verbal and written communication skills (experience)
- Ability to work effectively with global, virtual teams (experience)
- High degree of initiative and self-motivation (experience)
- Ability to manage multiple priorities successfully (experience)
- Team-oriented, with a focus on achieving team goals (experience)
- Strong presentation and public speaking skills (experience)
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines to support structured, semi-structured, and unstructured data processing across the Enterprise Data Engineering for Biotech or Pharma functional knowledge of R&D.
- Implement real-time and batch data processing solutions, integrating data from multiple sources into a unified, governed data fabric architecture.
- Optimize big data processing frameworks using Apache Spark, Hadoop, or similar distributed computing technologies to ensure high availability and cost efficiency.
- Work with metadata management and data lineage tracking tools to enable enterprise-wide data discovery and governance.
- Ensure data security, compliance, and role-based access control (RBAC) across data environments.
- Optimize query performance, indexing strategies, partitioning, and caching for large-scale data sets.
- Develop CI/CD pipelines for automated data pipeline deployments, version control, and monitoring.
- Implement data virtualization techniques to provide seamless access to data across multiple storage systems.
- Collaborate with cross-functional teams, including data architects, business analysts, and DevOps teams, to align data engineering strategies with enterprise goals.
- Stay up to date with emerging data technologies and best practices, ensuring continuous improvement of Enterprise Data Fabric architectures.
- Model data for analytics and ML (star/snowflake, Data Vault, semantic layers) and implement robust ELT patterns (dbt or equivalent).
- Build and maintain a lakehouse/warehouse (e.g., Delta Lake/Iceberg/Hudi; Snowflake/Redshift/BigQuery) with partitioning, clustering, and cost/perf optimization.
- Orchestrate workflows with Airflow/Azure Data Factory/Prefect and implement CI/CD for data (Git-based deployments, environments, automated tests).
- Implement data quality and observability (Great Expectations/Deequ, expectations-as-code, lineage/metadata, SLOs and alerting with OpenTelemetry/Prometheus/Datadog).
- Enforce security and governance (RBAC/ABAC, encryption, secrets, tokenization), manage PII/PHI under GDPR/CCPA and secure SDLC for data.
- Partner with analytics, data science, and product to define interfaces, SLAs, and contracts; publish clear docs, runbooks, and diagrams.
- Lead technical discovery, RFCs, and POCs; evaluate vendor tools and guide integrations.
- Mentor engineers; raise the bar on code quality, reviews, and engineering practices.
Tags & Categories
Sr Data Engineer
Amgen
Sr Data Engineer
Job Description
ABOUT AMGEN
What you will do
- Design, develop, and maintain scalable ETL/ELT pipelines to support structured, semi-structured, and unstructured data processing across the Enterprise Data Engineering for Biotech or Pharma functional knowledge of R&D.
- Implement real-time and batch data processing solutions, integrating data from multiple sources into a unified, governed data fabric architecture.
- Optimize big data processing frameworks using Apache Spark, Hadoop, or similar distributed computing technologies to ensure high availability and cost efficiency.
- Work with metadata management and data lineage tracking tools to enable enterprise-wide data discovery and governance.
- Ensure data security, compliance, and role-based access control (RBAC) across data environments.
- Optimize query performance, indexing strategies, partitioning, and caching for large-scale data sets.
- Develop CI/CD pipelines for automated data pipeline deployments, version control, and monitoring.
- Implement data virtualization techniques to provide seamless access to data across multiple storage systems.
- Collaborate with cross-functional teams, including data architects, business analysts, and DevOps teams, to align data engineering strategies with enterprise goals.
- Stay up to date with emerging data technologies and best practices, ensuring continuous improvement of Enterprise Data Fabric architectures.
- Model data for analytics and ML (star/snowflake, Data Vault, semantic layers) and implement robust ELT patterns (dbt or equivalent).
- Build and maintain a lakehouse/warehouse (e.g., Delta Lake/Iceberg/Hudi; Snowflake/Redshift/BigQuery) with partitioning, clustering, and cost/perf optimization.
- Orchestrate workflows with Airflow/Azure Data Factory/Prefect and implement CI/CD for data (Git-based deployments, environments, automated tests).
- Implement data quality and observability (Great Expectations/Deequ, expectations-as-code, lineage/metadata, SLOs and alerting with OpenTelemetry/Prometheus/Datadog).
- Enforce security and governance (RBAC/ABAC, encryption, secrets, tokenization), manage PII/PHI under GDPR/CCPA and secure SDLC for data.
- Partner with analytics, data science, and product to define interfaces, SLAs, and contracts; publish clear docs, runbooks, and diagrams.
- Lead technical discovery, RFCs, and POCs; evaluate vendor tools and guide integrations.
- Mentor engineers; raise the bar on code quality, reviews, and engineering practices.
What we expect of you
- Master’s degree with 6 - 8 years of experience in Computer Science, IT or related field OR Bachelor’s degree with 8 - 12 years of experience in Computer Science, IT or related field
- Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies.
- Proficiency in workflow orchestration, performance tuning on big data processing.
- Strong understanding of AWS services
- Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures.
- Ability to quickly learn, adapt and apply new technologies
- Strong problem-solving and analytical skills
- Excellent communication and teamwork skills
- Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices.
- Excellent analytical and troubleshooting skills
- Strong verbal and written communication skills
- Ability to work effectively with global, virtual teams
- High degree of initiative and self-motivation
- Ability to manage multiple priorities successfully
- Team-oriented, with a focus on achieving team goals
- Strong presentation and public speaking skills
Must-Have Skills
- Databricks
- PySpark
- SparkSQL
- Apache Spark
- AWS
- Python
- SQL
- Scaled Agile methodologies
- Workflow orchestration
- Performance tuning on big data processing
- Data Fabric
- Data Mesh
- Enterprise-wide data architectures
Locations
- Hyderabad, India
Salary
Salary not disclosed
Estimated Salary Rangehigh confidence
80,000 - 120,000 USD / yearly
Source: xAI estimated
* This is an estimated range based on market data and may vary based on experience and qualifications.
Skills Required
- Databricksintermediate
- PySparkintermediate
- SparkSQLintermediate
- Apache Sparkintermediate
- AWSintermediate
- Pythonintermediate
- SQLintermediate
- Scaled Agile methodologiesintermediate
- Workflow orchestrationintermediate
- Performance tuning on big data processingintermediate
- Data Fabricintermediate
- Data Meshintermediate
- Enterprise-wide data architecturesintermediate
Required Qualifications
- Master’s degree with 6 - 8 years of experience in Computer Science, IT or related field OR Bachelor’s degree with 8 - 12 years of experience in Computer Science, IT or related field (experience)
- Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies. (experience)
- Proficiency in workflow orchestration, performance tuning on big data processing. (experience)
- Strong understanding of AWS services (experience)
- Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures. (experience)
- Ability to quickly learn, adapt and apply new technologies (experience)
- Strong problem-solving and analytical skills (experience)
- Excellent communication and teamwork skills (experience)
- Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices. (experience)
- Excellent analytical and troubleshooting skills (experience)
- Strong verbal and written communication skills (experience)
- Ability to work effectively with global, virtual teams (experience)
- High degree of initiative and self-motivation (experience)
- Ability to manage multiple priorities successfully (experience)
- Team-oriented, with a focus on achieving team goals (experience)
- Strong presentation and public speaking skills (experience)
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines to support structured, semi-structured, and unstructured data processing across the Enterprise Data Engineering for Biotech or Pharma functional knowledge of R&D.
- Implement real-time and batch data processing solutions, integrating data from multiple sources into a unified, governed data fabric architecture.
- Optimize big data processing frameworks using Apache Spark, Hadoop, or similar distributed computing technologies to ensure high availability and cost efficiency.
- Work with metadata management and data lineage tracking tools to enable enterprise-wide data discovery and governance.
- Ensure data security, compliance, and role-based access control (RBAC) across data environments.
- Optimize query performance, indexing strategies, partitioning, and caching for large-scale data sets.
- Develop CI/CD pipelines for automated data pipeline deployments, version control, and monitoring.
- Implement data virtualization techniques to provide seamless access to data across multiple storage systems.
- Collaborate with cross-functional teams, including data architects, business analysts, and DevOps teams, to align data engineering strategies with enterprise goals.
- Stay up to date with emerging data technologies and best practices, ensuring continuous improvement of Enterprise Data Fabric architectures.
- Model data for analytics and ML (star/snowflake, Data Vault, semantic layers) and implement robust ELT patterns (dbt or equivalent).
- Build and maintain a lakehouse/warehouse (e.g., Delta Lake/Iceberg/Hudi; Snowflake/Redshift/BigQuery) with partitioning, clustering, and cost/perf optimization.
- Orchestrate workflows with Airflow/Azure Data Factory/Prefect and implement CI/CD for data (Git-based deployments, environments, automated tests).
- Implement data quality and observability (Great Expectations/Deequ, expectations-as-code, lineage/metadata, SLOs and alerting with OpenTelemetry/Prometheus/Datadog).
- Enforce security and governance (RBAC/ABAC, encryption, secrets, tokenization), manage PII/PHI under GDPR/CCPA and secure SDLC for data.
- Partner with analytics, data science, and product to define interfaces, SLAs, and contracts; publish clear docs, runbooks, and diagrams.
- Lead technical discovery, RFCs, and POCs; evaluate vendor tools and guide integrations.
- Mentor engineers; raise the bar on code quality, reviews, and engineering practices.
Tags & Categories