• the project is focused on accelerating and improving the deployment of data science and machine learning solutions into production

• this is not a classic Data Engineering role focused only on ingestion, ETL pipelines, or data warehouse building

• collaboration at the interface between:

• Data Science

• Data Engineering

• backend development

• cloud platform

• MLOps and production operations

• the task will be to take over Python prototypes and notebooks from Data Scientists and transform them into robust production workflows in Azure Databricks

• the resulting solutions must be scalable, monitored, secure, maintainable, and deployable in a repeatable manner

• collaboration in a small team, typically consisting of two to three Data Scientists and one Production/Data Engineer

• tasks and responsibilities:

• productionalizing ML solutions

• taking over prototypes created in Python or notebooks

• refining experimental code to production quality

• packaging and deploying models into production workflows

• eliminating manual steps and reducing dependence on handoffs between teams

• ensuring reliable operation of production models

• Data and ML pipelines

• designing and implementing robust data and ML pipelines

• automating data processing and model execution

• configuring Databricks jobs and workflows

• resolving dependencies between individual pipeline steps

• implementing validation and control mechanisms

• MLOps and operations

• versioning of code, data, models, and configurations

• deployment automation

• monitoring of production workflows and models

• support for release processes

• resolving operational issues and optimizing stability

• Databricks platform

• co-management of the Azure Databricks environment

• configuring jobs, workflows, and compute resources

• optimizing performance and platform utilization

• working with Apache Spark and Delta Lake

• implementing standards for development and production deployment

• Governance and security

• implementing data security policies

• managing access permissions

• adhering to governance standards