Best Practices in MLOps!

I was a part of an initiative to set up MLOps, so here is a gist of what I thought would be the best in aligning to MLOps. In my previous post, I outlined MLOps and How it differs from DevOps. In my point of view these can be listed as,

1. Naming Conventions

When a particular naming convention is followed throughout the organization, it is pretty easy to identify the variables' functions and meaningful words in them. Also in larger projects, the naming conventions with meaningful names give any onlooker an idea of what it means. It is best to name the inputs, processing tasks, and outputs in a previously agreed format. Using an “_” to get rid of spaces or using a camel case can be followed.

2. Code Quality Checks

Any code written should be in compliance with Code quality checks, it doesn't matter if it is a development environment or a production environment. In code quality check it is measured if it has security breaches, for reliability and maintainability. If the quality gate has passed it is considered a deployable code, while if the quality gate has failed it needs to be analyzed as to why it has failed which means it has to look for code duplications or security aspects and then clear them before the code is deployed. Sonarcube is one main tool that is mostly used in the industry in checking for code quality checks.

3. Tracking Experiments

Data Scientists may come up with different models based on the features which are used in the models. Since it delivers the best model which performs the expectation using the data effectively. It is essential to track the multiple models they come up with to identify which has performed well in the circumstances given. The hyperparameters, the conditions, and the measuring metrics are also logged against these models and how well they perform at the conditions. There are some tools introduced for Model tracking and one I had the chance to look at was MLFlow.

4. Open Communication

It is very important for the whole team to communicate the updates on the project in an open forum. This also paves the way to discuss any queries that come up the way and any blockages. Proper communications and communications is an answer to most questions on any given project. The communication between data engineers, data scientists, platform engineers to ML engineers communicating on the development is a team effort where collaboration of all ideas makes a successful project.

5. Monitoring

In machine learning, monitoring comes in 2 forms. one which is monitoring the resources the other being the models. The compute resources the models are run on can be monitored for health, scaling properties and etc. The models can be monitored for drift tracking, hyperparameters and the matics.

6. Deployments

The continuous integration and continuous deployment can be put in place for MLOps projects as usual. Deployment pipelines in place and different stages of approvals also can be introduced.

7. Resource and Cost Optimization

Lot of AI and ML models run on a high compute power, so therefore cost optimization is a key factor. Using optimized queries, introducing auto termination when the resources are in idle for some time. There are a lot more like having budget alerts for cost updates, can be used in.

8. Feature Store

Feature store is a very recently introduced to ML. It is a central repository more of a feature bank. This stores in the features which is used across the organization. This not only makes the process efficient but also as this has strict regulatory compliance a feature store eliminates data duplication.