Project description
We are seeking a skilled, hands‑on Data Engineer with demonstrated experience in Databricks, Apache Spark, Python, and real‑time data streaming solutions.
Responsibilities
Implement resource composition logic provided by the Product team into scalable and efficient PySpark code.
Develop, optimize, and maintain PySpark applications within the Databricks environment.
Utilize and maintain existing CI/CD pipelines in Databricks for automated build, testing, and deployment.
Write and maintain unit tests to ensure high code quality and reliability.
Perform data validation and quality checks to ensure compliance with defined business rules.
Support and manage production deployments, ensuring system stability and performance.
Troubleshoot and resolve issues across development, testing, and production environments.
Collaborate with Product, QA, DevOps, and healthcare domain teams to ensure accurate implementation.
Maintain technical documentation for code, workflows, and deployment processes.
Pull from and push code to GitHub repositories, following version control and branching best practices.
Skills
Must have
5+ years of experience in data engineering roles.
Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).
4+ years of experience with Databricks data pipelines.
Strong hands‑on experience with PySpark and/or Apache Spark.
Experience working with CI/CD pipelines and DevOps best practices.
Proficiency with CI/CD, GitHub (pull requests, branching, code reviews).
Experience writing unit tests and performing data validation.
Experience supporting production systems and deployments.
Strong analytical and troubleshooting skills.
Nice to have
Knowledge of FHIR (Fast Healthcare Interoperability Resources) standards.
Experience with healthcare data integration projects.
Familiarity with REST APIs and FHIR server integrations.
Experience in Agile/Scrum development environments.
#J-18808-Ljbffr