Project Role : Custom Software Engineer
Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.
Must have skills : Microsoft Azure Data Services
Good to have skills : NA
Minimum 3 year(s) of experience is required
Educational Qualification : 15 years full time education
, Roles & Responsibilities: -Develop and Optimize Data Pipelines using PySpark and SQL in Databricks to handle large-scale data transformations and batch/streaming workloads. -Design and Implement ETL Processes by leveraging Databricks notebooks, Delta Lake, and Python-based custom scripts for data ingestion and curation. -Collaborate with Data Analysts and Stakeholders to understand data needs and deliver accurate, performant, and reusable data solutions. -Maintain and Monitor Job Performance, identify bottlenecks, and apply tuning techniques to improve pipeline efficiency on the Databricks platform. Professional & Technical Skills : -Proficient in building scalable, distributed data pipelines using PySpark and Databricks with a deep understanding DataFrame APIs, and Spark optimization techniques. -Experience on Python for scripting, data manipulation, and integrating third-party libraries within data engineering workflows. -Expertise in SQL for data extraction, transformation, complex joins, and performance tuning across relational and distributed data stores. -Experienced in working with Delta Lake, versioned data, schema evolution, and managing CDC (Change Data Capture) scenarios in a cloud-based environment. -Best practices in CI/CD, code versioning (Git), and DevOps for data projects. Additional Information: - The candidate should have minimum 5 years of experience in Microsoft Azure Analytics Services. - This position is based at our Hyderabad office. - A 15 years full time education is required.