Many organizations, including state and local governments, are dipping their toes into machine learning (ML) and artificial intelligence (AI). As we’ve discussed in this blog series, some are already reaping the rewards of AI through increased productivity, cost savings, etc. However, for most embarking on this transformational journey, the results are yet to be seen and for those who are already underway, scaling their results appears as completely uncharted waters. According to a recent study by NewVantage Partners, only 15 percent of organizations surveyed have deployed AI capabilities into widespread production. Most of these leading organizations have significant AI investments, but their path to tangible benefits is challenging, to say the least. AI that is not deployed is nothing more than a costly experiment. These experiments are complex technical accomplishments, but they do not translate into results. In the final installment of this blog series examine how Machine Learning Operations (MLOps) allows governments to easily deploy, monitor, and update models in production, paving the way to AI with measurable results. 

What is MLOps?

Laying an MLOps foundation allows data, development, and production teams to work collaboratively and leverage automation to deploy, monitor, and govern machine learning services and initiatives within an organization.

Depending on an organization’s maturity level, their MLOps infrastructure can be represented by something as simple as a set of vetted and maintained processes such as model lifecycle, model evaluations and production, and model risk.

Four Reasons Why State and Local Governments Need MLOps to Drive AI Results

1. Issues with Deployment

Organizations do not realize the full benefits of AI because models are not often deployed. Or if they are deployed, it is not at the speed or scale to meet the needs of the organization. MLOps simplifies model deployment by streamlining the processes between modeling and production deployments. It should not matter which platform or language the model was built on. An enterprise-grade MLOps system should allow organizations to plug in their models and generate consistent API access for application teams on the other end, regardless of deployment environments and choice of cloud services and providers. MLOps deployment helps you when:

• Multiple teams are used to build models.
• Models are sent to IT but are not making it into production.
• There is a large backlog of models waiting to be deployed.
• A lot of time is spent troubleshooting models during the deployment process.
• A standardized process for elevating models from development to production is missing or flawed.
• There is a complex process for putting models into production that requires updating multiple systems.

2. Issues with Monitoring

Evaluating machine learning model health manually is very time-consuming and distracts resources from model development. MLOps allows both production and AI teams to monitor models in ways specific to machine learning. A robust monitoring infrastructure should be able to proactively monitor data drift, feature importance, and model accuracy issues. Advanced capabilities may include features built to increase trust toward models in production even further. For example, the principle of humility in AI dictates that models should be able to inform not only when predictions are possibly going bad, but also when they’re not confident in the quality of their predictions. MLOps Monitoring helps you when:

• Models are in production, but no monitoring has ever been performed.
• Models are deployed across the organization and in various systems without a consistent way to monitor them.
• Models have been in production for a long time and never refreshed.
• Model performance must be determined with a manual process performed by highly skilled personnel.
• There is no centralized way to view model performance across the entire organization or to offload accountability to operations teams.

3. Issues with Lifecycle Management

Regularly updating models in production and identifying model decay is an extremely intensive process for state governments who, for the most part, lack data science resources and personnel. Additionally, there are concerns that manual code is brittle, and the potential for outages is high. 

MLOps allows for a production model lifecycle management system that automates processes, such as champion/challenger gating, troubleshooting and triage, hot-swap model approvals, and offers a secure workflow to ensure the efficient management of your models’ lifecycle as you scale. MLOps lifecycle management helps you when:

• Models are not being updated in production.
• Data scientists and other related staff do not hear about model decay after initial deployment.
• Data scientists and other related staff are heavily involved in production model updates.
• Only a small percentage of new project demand is met due to high maintenance demands of existing models.

4. Issues with Model Governance

Organizations need time-consuming and costly audit processes in order to ensure compliance as a result of varied deployment processes, modeling languages, and the lack of a centralized view of AI in production across an organization. MLOps offers an enterprise-grade production model governance solution, which can deliver: 

• Model version control 
• Automated documentation 
• Complete and searchable lineage tracking and audit trails for all production models 

This helps minimize legal risks, maintain a transparent production model management pipeline, minimize and even eliminate model bias, and deliver a host of other benefits. MLOps Model governance helps you with:

• Production access control
• Traceable model results
• Model audit trails
• Model upgrade approval workflows

Over the course of this blog series we have discussed the five critical ways that AI can help states solve their hardest problems. MLops enables state and local governments to do just that — to put AI into action.  With AI, states will surge ahead faster than ever before.

About the author

Sara Marshall

Account Executive – Sales, Federal & Public Sector

Sara Marshall is a licensed attorney and strategist with more than a decade of experience in state government, healthcare, and technology. She was voted “Top Up and Comer” in government by State Scoop in 2015, named the “Best Woman in Sales Support” in North America by WISA in 2020, and was recognized as the most valuable player for Regulated Industries at SAP for the last two years prior to joining DataRobot.

Meet Sara Marshall