What is MLOps?Why a working model is only the beginning.

At 06:00, everything appears to have worked

ShelfCast’s morning run is complete. The forecast table is there, every expected store has rows, and the scheduler reports success. The job used model-v17, the version the team approved last week.

A replenishment planner opens the results to prepare tomorrow’s stock plan. For one product at one store, the forecast is 95 units: on time, in the expected format, through a working pipeline.

By the end of tomorrow’s trading day, some stores have empty shelves while other products remain unsold. The job succeeded, but the forecasts now deserve a closer look.

ShelfCast is a fictional retailer created for this series. Its events, schedules, model versions, and numerical examples are synthetic. Following one retailer throughout gives every engineering decision a concrete situation.

For a failed job, you would check logs, retries, credentials, and recent changes. Here the execution path may be healthy, so the questions move to the prediction itself. Were the inputs suitable? Has the relationship the model learned become less useful? Or did the ordering process introduce a problem after the prediction was produced?

The replenishment result alone does not identify the cause. A supplier delay can empty a shelf too. We need evidence that connects the forecast to the data it used, the decision it supported, and the outcome that followed.

We will follow that 95-unit forecast from its inputs to its later outcome: first what the model does, then the system around it, what it must deliver, what we observe, and who responds when something goes wrong. Those decisions become our first ML operating contract.

What is the model actually doing?

Start with an ordinary forecasting rule: use the number sold on the same weekday last week as tomorrow’s estimate. It is easy to explain, cheap to run, and a useful reference when deciding whether something more complicated earns its place.

To train a model, we give a training procedure historical examples and let it fit a relationship between their inputs and the outcome we want to predict. ShelfCast’s inputs might include recent sales, weekday, price, and promotions available by the forecast’s 05:30 input cutoff.

We choose the target, construct the inputs, select the training method, and decide how to judge the result. The saved output of training is the model artifact, which the system uses to calculate future predictions.

Think of training as preparing the forecasting method and inference as using that prepared method. ShelfCast can run inference every morning while keeping model-v17 unchanged. In this example, processing another forecast row does not update its learned parameters. A system that learns during operation would need an explicitly designed update process. [15]

With deterministic inference, a fixed model can return the same prediction for the same input. Exact repeatability still depends on the implementation and execution conditions. [13] The operational difficulty is that the world supplying those inputs and producing the outcomes can change. Learned behavior adds dependencies that the team must understand alongside the application code. [3]

Be precise about what 95 units means

For now, ShelfCast predicts recorded sales. That is an observable quantity. Estimating how much customers would buy with unlimited availability is a different target.

If a store has 80 units and sells all 80, the sales record says 80. It does not tell us whether customers wanted 80, 100, or more. Stockouts can hide demand, so raw sales should not automatically be treated as the full demand outcome. Inventory and availability information matter. [11]

The planner takes the 95-unit forecast as one input to an order. Stock already available, incoming deliveries, shelf life, and supplier lead times may all affect what to buy. We need to keep the prediction and that later decision separate when investigating what happened.

What MLOps takes responsibility for

MLOps means Machine Learning Operations. It connects the development, delivery, observation, and improvement of ML systems. Google’s lifecycle guidance emphasizes automation and checks around data, training, evaluation, and release. [1]

For ShelfCast, that definition becomes six practical questions:

• Purpose: which planning decision should this forecast improve?
• Identity: which data, code, settings, and model produced a particular row?
• Release evidence: why was this version considered suitable to use?
• Current operation: are the required inputs arriving, and are forecasts delivered correctly?
• Current usefulness: when outcomes arrive, how well does the forecast perform for the products and stores that matter?
• Response: who investigates a problem, and what does the planner use in the meantime?

Each question creates work outside the training algorithm. Someone must maintain the sales feed, preserve forecast history, construct reliable evaluation datasets, publish a complete output, and respond when those responsibilities fail.

The research paper Hidden Technical Debt in Machine Learning Systems describes how data dependencies, feedback loops, configuration, and changes in the outside world complicate ML operation. [2]

Where your DevOps knowledge already fits

Version control, repeatable builds, sensible permissions, observable jobs, controlled releases, and recovery procedures all remain useful; MLOps extends the evidence they must handle. A code review may be excellent while the evaluation dataset still contains future information. A deployment may be dependable while a product segment receives poor forecasts.

For ShelfCast, the data owner answers for feed availability, the model owner for the evaluation, the platform owner for execution, and the replenishment owner for what counts as harm and an acceptable fallback. One person may hold several roles; what matters is that each problem reaches someone who can act on it.

The system runs on two clocks

ShelfCast has an execution clock and an evidence clock, and they answer different questions at different times. All times use the retailer’s business timezone, Asia/Kolkata.

On Day 1 at 05:30, we close the information window. Only information available by that cutoff may be used. At 06:00, the job publishes predictions for the Day 2 calendar day. Those predictions support replenishment planning. The Day 2 sales records become usable labels after reconciliation, scheduled here for Day 3 at 06:00.

At publication time, we can check whether the job completed, the required rows exist, the inputs meet their contract, and predictions are valid numbers. We cannot compare those future predictions with finalized Day 2 sales yet. Quality monitoring needs the correct predictions joined to their later outcomes. [9]

The interface should make that wait visible: “Forecast published; outcomes pending.” Showing an accuracy of zero would mislead, and so would a healthy status implying the predictions were already checked against their outcomes.

Once the outcomes arrive, calculate an actual error

The following three rows are a synthetic, fully stocked evaluation cohort. Each row is one product at one store on the target day. Stockouts are excluded so recorded sales are not capped by unavailable stock.

Mean absolute error, or MAE, is 5 units: add the absolute errors, then divide by the three rows: (5 + 6 + 4) / 3 = 5. The model’s predictions were five units away from recorded sales on average in this tiny example. [14]

Weighted absolute percentage error, or WAPE, is 5%: divide the sum of absolute errors by the sum of absolute observed values: 15 / 300 × 100 = 5%. This measures aggregate absolute error relative to observed volume. It is not “95% accuracy,” and it is not the average of three row-level percentage errors. [12]

Neither result is a production acceptance rule. Three invented rows demonstrate arithmetic. A real release decision needs a representative evaluation, a relevant reference method, and requirements connected to the intended use. WAPE can conceal weak low-volume segments; its denominator must be nonzero.

The words “fully stocked” matter: this error describes that cohort only, which does not establish performance during stockouts or across the whole replenishment process. Keep an evaluation’s scope beside its metric.

Every prediction needs a history

When a planner questions the 95-unit forecast for row A, we need a way to retrace it: start with the exact published row, find the model that produced it, and retrieve the input snapshot available to that execution.

In our design, a forecast record carries its identity, store, product, target date, issue time, and released model version. The batch manifest links those rows to an input snapshot, feature-code version, configuration, and execution environment. The model’s release record points back to its training run and evaluation evidence.

That connected history is lineage. Experiment-tracking and model-registry systems can record runs, parameters, metrics, artifacts, and relationships between versions. They still depend on the identifiers and evidence we actually preserve. [6]

For example, sales/latest.csv might be overwritten, so it is a weak reference for reconstructing last Tuesday; ShelfCast should keep or be able to retrieve the exact permitted snapshot. Similarly, a name such as champion can move between versions. The release record should resolve it to the specific artifact used. [16]

This is why a Git commit is only part of a model’s identity. Running the same training code with a different dataset or configuration can produce a different artifact. Conversely, a production incident can occur while the released artifact remains unchanged.

Be clear about what “reproducible” means: identifying an old run, re-running its pipeline, and getting identical numerical output are different claims. An environment file and a random seed are useful records, but the team still needs to examine the data, dependencies, and execution conditions. [13]

A historical test must respect historical knowledge

Suppose we rebuild last month’s training dataset today. An inventory correction now has an event date from last month, but the correction itself arrived yesterday. If a historical forecast uses that corrected value before it was available, its test has borrowed information from the future.

This is data leakage: information enters development or evaluation that the prediction process would not have had at the relevant time. Leakage can make measured performance look better than the system can achieve on genuinely new cases. Learned preprocessing, such as a fitted scaler, must also be fitted on the training portion rather than learning from the test set. [4]

ShelfCast therefore needs both the event’s time and the information’s availability time. A weather forecast issued before 05:30 may be legitimate. The finalized weather observation for Day 2 cannot be an input to Day 1’s historical forecast.

For evaluation, we should simulate successive forecast dates: use earlier, available information; predict the relevant future period; then compare with later outcomes. The split should reflect the intended task. Predicting future days for existing stores differs from claiming the same performance at an entirely new store. [5]

If the historical replay includes training, it may use only labels already usable at that simulated training time. Later outcomes belong in the subsequent scoring step. This applies our information-availability rule to training as well as prediction.

A schema validator can accept a perfectly valid number that came from tomorrow. A chronological split can still contain a feature built using future records. We must check the construction and meaning of the data, as well as its shape.

Build a model, release it, operate it, and learn

ShelfCast does two kinds of work: creating and approving model versions, and producing daily forecasts with the approved version. They run at different frequencies, so they take separate paths and the daily forecast never waits for a new model to be trained.

The path that creates a candidate

A training job reads an identified historical snapshot and checks it against the data contract. Feature preparation builds the inputs, preserving each historical forecast’s information cutoff, and training produces a candidate artifact.

If we propose candidate-v18, that name means only that a candidate exists. We still need an evaluation report. It should compare it with model-v17 and the simple reference forecast on suitable data, and describe the periods tested, important store and product groups, label coverage, and known limitations.

We would also check that the candidate runs with the intended preprocessing and prediction code; a good offline result is useless if the daily job builds different inputs or cannot load the artifact.

The owner reviewing the release brings those findings together. They can approve the version when it meets the agreed policy, reject it, or ask for more work. Some of those decisions may later become automated checks.

The path that produces daily forecasts

The daily job reads the approved artifact and the current permitted input snapshot, predicts for the target date, validates the output, and publishes a versioned forecast dataset for the planner.

This is batch inference: scoring a collection of inputs as a job. Batch jobs may be scheduled or started when needed; online inference returns predictions in response to application requests. ShelfCast’s once-daily planning decision makes a scheduled batch a reasonable starting design. [10]

Publication deserves its own check: a run that wrote half a table before failing must not leave the planner an apparently complete forecast. For this example, we would stage the output, verify the expected rows and identifiers, and then publish that specific completed version. If the job retries, it must not silently create duplicate instructions for the downstream planning process.

Notice that the daily path does not train a model. It uses model-v17 until a separate release decision changes the approved version.

The path back from reality

After the target day, the outcome job reconciles the sales records and joins them to the correct forecast rows, so the review finally has measured error alongside delivery and data evidence.

The feedback arrow leads back to an investigation, not straight to retraining. A broken upstream field or an incorrect store mapping might explain the problem; we may train a new candidate; or we may find the existing model handled an unusual week adequately and leave it in place.

That is why the architecture includes a release gate between a candidate and production. A monitoring signal can start useful work without automatically replacing the model.

CI, CD, and training automation have different jobs

If you maintain CI/CD pipelines, that experience transfers directly. The habit to bring is precision about what each green check establishes, especially when a pipeline now produces a trained model.

ML delivery adds data and model checks. Automated training produces candidates; prediction, training, and release can follow different schedules. [1]

Start with small code and contract checks on every relevant change, and run a full historical evaluation when feature logic, training inputs, or the candidate changes. Before adding an expensive stage, know which release question it answers.

Suppose the team retrains weekly and scores daily. Monday’s training run might produce candidate-v18, while every daily batch continues using model-v17. If the candidate does not meet the agreed requirements, the correct outcome is a completed experiment with no new production release.

“Continuous” does not mean training constantly; the right trigger depends on the application, fresh data, evaluation needs, and cost. ShelfCast can start with an explicit, reviewed training process while automating the daily work that must be dependable.

Watch the system at four different levels

One dashboard can hold several kinds of evidence, but it should keep their meanings clear: platform monitoring, data validation, and model evaluation answer different questions. Monitoring guidance distinguishes input integrity, distribution changes, and measured model performance for this reason. [3]

Three problems that can look similar on a chart

Data quality failure: the promotion feed is missing, but the job replaces every missing value with “no promotion.” The inputs are numerically valid and the run completes, so the investigation belongs at the feed and its missing-value behavior.

Training-serving skew: training calculated “recent sales” from seven complete trading days, while the daily prediction job calculates it from a different window. We use this term for a mismatch between the data preparation or conditions in training and those used for prediction. Shared feature logic can reduce some mismatches; checking the actual inputs remains necessary. [7]

Drift: the distribution or relationship has changed over time. In this article, data drift means a change in the distribution of inputs. Concept drift means a change in the relationship between those inputs and the outcome. Research uses several related naming conventions, so it helps to state ours explicitly. [8]

For example, more promotion days may be an expected input change that the model handles well. A new competitor might change sales patterns that the current features do not explain. A drift alert can help direct attention; it does not by itself establish the business impact or the correct fix.

Choose the response from the evidence

In the promotion-feed example, the response is repairing the missing input; retraining is a different decision. When the inputs are suitable but measured error has worsened, examine the affected periods and groups, compare with the reference method, and use those findings to decide whether a candidate change is justified.

We also need to look beyond the overall average. A forecast that performs reasonably across the chain may be weak for a particular product category or store. Relevant slices help expose these differences, and business metrics help reveal effects that an error score alone does not describe. [7]

Partial labels need the same honesty. If only half of Day 2’s rows have usable sales labels by Day 3, the dashboard shows that gap, and the missing rows stay unknown rather than becoming zero sales.

While the investigation runs, our replenishment owner needs an agreed fallback: perhaps a reviewed reference forecast for supported cases, or explicit planner review. Automatically restoring an older model would not repair a broken source feed. It would also not undo an order already sent to a supplier.

Start with an operating contract you can explain

Before choosing a cloud service, write down what ShelfCast needs to do. For this first design, we want a dependable daily batch and enough preserved history to investigate a forecast.

The table below is our illustrative operating contract. Its schedule and policy are choices for this fictional example. They are not universal MLOps requirements.

There is deliberately no invented rule such as “WAPE below 10% means production ready.” The planning and model owners agree on meaningful requirements, including where an error is expensive and which groups need protection. An aggregate metric informs a decision without being the whole decision.

What needs to exist underneath it?

For this first system, we need versioned code; retrievable input and model artifacts; a train-and-evaluate routine; a durable release record; a scheduled prediction job; validation before publication; outcome reconciliation; monitoring; and an actionable fallback. Existing Git, storage, CI, and scheduling capabilities may cover much of that work.

Dedicated registries and orchestration can support a growing system. Feature stores are optional; add one when the shared-feature requirements justify it. [1]

The familiar platform questions still apply: which identity can publish forecasts, who can approve model versions, and how to protect artifacts, recover failed runs, and limit compute and storage costs.

Consider the deadline itself. If the 05:30 input checks fail, the planner still needs a clear result at 06:00, so the agreement should say when to use the supported fallback and when to ask for manual review. Quietly showing an old forecast as today’s output would hide the problem.

The contract gives everyone one thing to review: the platform engineer checks whether partial output can become visible, the data engineer whether the label deadline is realistic, the planner whether the fallback is usable, and the model owner whether the evaluation has gaps.

Keep the model fixed. Change the evidence.

Return to model-v17 and the forecasts 95, 126, 76. We will keep those predictions fixed for one final illustration. Imagine a second, busier, fully stocked synthetic cohort with recorded sales of 130, 156, 104.

The absolute errors are now 35, 30, 28. Their sum is 93 units, so MAE is 31 units: 93 / 3 = 31. Observed sales total 390 units, making WAPE approximately 23.846%: 93 / 390 × 100.

Neither the model artifact nor these deliberately fixed predictions changed; the measured error changed because the outcomes differed. This is an arithmetic illustration of why version stability cannot establish continuing usefulness; it does not identify a cause, estimate real-world degradation, or prove that retraining will help.

A ten-minute investigation

Return to the timeline in Chapter 4. Select Forecast issued for Day 1 and choose Busier day. Note what you can establish at publication time, then advance to Labels reconciled on Day 3 and calculate the error yourself. Use the incident scenarios above to choose one next piece of evidence: promotion-feed completeness, an affected product group, inventory availability, or a comparison with the reference forecast. Explain why it would help distinguish possible causes.

For a system you actually work with, try the same exercise: identify one prediction, its information cutoff, its released version, its decision, and its later outcome. Any missing link becomes a specific engineering task.

Run the same audit in Python

Download the ShelfCast exercise

Unzip the files and open a terminal there. You need Python 3 (standard library only). The CSV files map Day 1 to 14 September 2026, Day 2 to 15 September, and Day 3 to 16 September. The dates and rows are invented.

# Day 1: the labels are still unknown.
python3 audit_forecasts.py --as-of 2026-09-14T06:00:00+05:30

# Day 3: evaluate the ordinary outcomes.
python3 audit_forecasts.py --as-of 2026-09-16T06:00:00+05:30

# Evaluate the alternate busier day, with the same forecasts.
python3 audit_forecasts.py --as-of 2026-09-16T06:00:00+05:30 --labels busy-day-labels.csv

The first run reports unknown error and zero label coverage; the next two report the calculations above. The alternative labels file is another possible outcome for the same targets, so use it by itself. The script rejects duplicate targets and malformed values, keeps delayed labels unknown, and reports fully stocked cohort exclusions. It audits supplied forecasts; it does not train a model.

Part 2 is planned: we will define ShelfCast’s target and decision more carefully, build a simple reference forecast, and evaluate it using information available at each forecast date. That baseline is what future candidates must beat as we build the pipelines, release process, monitoring, and recovery.