Forecasting

Activity clock onlythe first request wakes it

Activity clock only: asleep from 06:00 to 09:10, waking up from 09:10 to 09:14, awake from 09:14 to 18:29, asleep from 18:29 to 22:00.

With autoResumeup at 08:45

With autoResume: asleep from 06:00 to 08:45, waking up from 08:45 to 08:49, awake from 08:49 to 18:29, asleep from 18:29 to 22:00.

With pauseEarly toopaused at 18:00

With pauseEarly too: asleep from 06:00 to 08:45, waking up from 08:45 to 08:49, awake from 08:49 to 18:00, asleep from 18:00 to 22:00.

06:0008:0010:0012:0014:0016:0018:0020:0022:00

Asleep Waking up Awake09:10 people arrive17:30 people leave

Office hours from 09:10 to 17:30, once learned, with the default idleAfter of an hour. The times are the simulator's (Proving it, What it saves and costs). Waking up is drawn as a few minutes. It takes as long as the workload takes to start.

It always learns

For every managed workload, whatever autoResume says. With ask, it learns the hours it’s paused behind the doorman from whether people woke it. Once confident, it holds off a pause when it expects demand in the hour under way or the next.

With autoResume

It also wakes a paused workload 15 minutes before an hour it expects busy. Turn it on any time. The forecast has been learning all along, and wakes ahead once it acts.

With pauseEarly

Off by default. It also pauses an idle workload before idleAfter runs out.

It does so when it’s sure the hours ahead are quiet, and a request would wake it, never with ask. Otherwise the activity clock decides every pause.

The forecast learns the shape of a day, busy in business hours and quiet at night. It also learns how each weekday departs from that shape. It learns the two apart, from each hour’s CPU utilisation. The simulator, which runs the operator’s own engine and rules, measured every figure on this page.

Phase lifecycle

The forecast earns its way to acting, phase by phase.

1. Observingits first week
2. DailySuggestingscoring each day
3. DailyActiveacts on the day
4. WeeklySuggestingscoring each week
5. FullyActiveacts on both

Phase	Reached when	What it does
Observing	No day of forecasts is scored yet	Collects data, its first week. An hour of the week is scored from the second time it’s seen. No predictions yet
DailySuggesting	Every hour of the day is observed, and a day of forecasts is scored	Scores daily confidence, without acting on it
DailyActive	Daily confidence reaches the threshold	Predictions drive decisions. Weekly patterns are still learning
WeeklySuggesting	Every hour of the week is observed at least once	Earns weekly confidence
FullyActive	Weekly confidence reaches the threshold	Daily and weekly patterns both drive decisions

Case	What it means
The gates	They count hours of the calendar, not data points. A workload observed only from 9 to 5 hasn’t shown its nights, however long it’s watched
A workload paused every night with neither the doorman nor a Prometheus activity query	Its nights are never observed (What an hour records), so it never reaches DailySuggesting. Route it through the doorman for autoResume and the forecast’s veto. That works with request, or with ask, whose paused hours there are learned from whether people woke it. A workload no Service routes requests to, such as a queue worker, needs an activity query that sees its demand. A query of its queue’s length is one
The threshold	The forecast setting’s, and a change applies at the next reconcile. A phase is earned at the threshold and lost 5 points below it. So confidence hovering there doesn’t switch the forecast on and off every hour
status.prediction	A phase per season. dailyPhase and weeklyPhase are each Observing, Suggesting or Active, next to dailyConfidence and weeklyConfidence. DailyActive is dailyPhase: Active with weeklyPhase: Observing
After a regime change	The phase shown, there and in napwake_prediction_phase, is the one the forecast acts in. Left at WeeklySuggesting with nothing scored, it shows DailySuggesting, until a day of its forecasts is scored and it drives decisions again

Forecast setting

How soon the forecast acts is one setting, eager, balanced (the default) or cautious.

Set it on	With
An opted-in workload or its namespace	The annotation napwake.dev/forecast
Every opted-in workload	The Helm value defaults.forecast
A ManagedWorkload you write	spec.prediction.mode
A ManagedWorkload, with a threshold of your own	spec.prediction.confidence, from 50 to 100, which overrides the setting’s. A label-created ManagedWorkload keeps it when its annotations are applied again

apiVersion: apps/v1
kind: Deployment
metadata:
  name: preview-api
  annotations:
    napwake.dev/forecast: eager

Setting	Acts at a confidence of	What it trades
eager	60%	Acts on less regular patterns that balanced waits out. Wakes a workload for nothing more often
balanced	75%	The default. Acts on a weekday pattern after 8 days. Almost never wakes a steadily used workload for nothing
cautious	85%	Acts only on patterns it forecasts well. For workloads costly to keep awake for nothing, such as one on GPUs
A threshold below 60		Gains a morning or so on the most irregular patterns, for two more hours woken for nothing
A threshold above 85		Loses mornings about as fast as it saves hours

A wrong forecast costs a little money, never availability. It only wakes a workload early or keeps it up longer. A paused workload still wakes on a request through the doorman, or with ask on a person’s click. So the choice is between mornings ready and hours awake for nothing. Twenty simulated workloads averaged this over eight weeks of loosely kept office hours. People arrived up to an hour either side of 09:10, and were away one weekday in four.

Setting	Mornings ready, of 30	Hours woken for nothing
eager	10	14
balanced	8	8
cautious	4	5

On steadily kept office hours, every setting acts after 8 days. Every morning after is ready, and none wakes the workload on a weekend. The settings differ only on patterns the forecast can’t get right every day.

How long it takes

Days to DailyActive / FullyActive, learning live from a fresh start, at 40% of the CPU request in use and 1% otherwise. autoResume is on, behind the doorman. The queue worker is woken by a Prometheus activity query instead. The public preview is woken only by people, with wake: ask. “Never” means not within ten weeks.

Workload	eager (60)	balanced (75)	cautious (85)
Busy on weekdays from 09:10 to 17:30, quiet nights and weekends	8 / 14	8 / 14	8 / 14
The same, with people arriving and leaving up to 20 minutes either side and away one weekday in ten (the median of 20)	8 / 14	8 / 14	8 / 14
The same, with people arriving and leaving up to an hour either side and away one weekday in four (the median of 20)	8 / 29	9 / never	10 / never
The same hours every day, weekends included	8 / 14	8 / 14	8 / 14
Always busy	8 / 14	8 / 14	8 / 14
Always idle	8 / 14	8 / 14	8 / 14
A queue worker with no Service, which a query of its queue’s length wakes, busy every day from 07:00 to 19:00, paused every night	8 / 14	8 / 14	8 / 14
A public preview only people wake, with wake: ask. Someone clicks Wake it at 09:05 on weekdays, and it’s quiet nights and weekends	8 / 14	8 / 14	8 / 14

A test in the repo runs these, and fails if they drift from this table.

Pattern	When it acts
Any the forecast can get right	At every setting, once it has scored a day of its second week. The first week is learned, not scored
Office hours	On the Tuesday of the second week. The Monday before is the only morning after the first week that the first request wakes the workload. Every morning after is ready, Mondays included. Weekends stay asleep, since the first weekend taught the forecast they’re quiet
One it can’t get right every day, as in the third row	A higher threshold waits for more evidence
Always busy, or always idle	Acting changes nothing. One is never paused, and the other never woken or held awake by its forecast of no demand
Any, before the forecast acts	Idle detection and wake on request work from the first hour. The first request of the morning wakes the workload, or with ask the first click
Any, with a Prometheus	The forecast learns from its history first, so a few weeks of a steady pattern acts from the first day

When it acts

status.prediction says whether the forecast is acting. If it isn’t, it says what it waits for, and when it’s expected to start.

Field	What it says
acting	Whether the forecast drives decisions. It holds off pauses before hours it expects busy. With autoResume, it wakes the workload 15 minutes ahead of them. With pauseEarly, it brings pauses forward before hours it expects quiet
threshold	The confidence it acts at. That’s the setting’s, or spec.prediction.confidence
reason	One of the reasons below
message	The same in a sentence, with its confidence and threshold once it has them
startsActing	When it’s expected to start acting
history	What it learned from Prometheus when Napwake began managing the workload (Learning from history)

reason	Meaning
Acting	The forecast drives decisions
Observing	It hasn’t scored a day of its forecasts yet, as in its first week, or hasn’t seen every hour of the day
EarningConfidence	Its forecasts of the last day aren’t yet right enough
Relearning	Demand broke from what it had learned, after a regime change
NotLearning	The workload is paused with nothing to see its demand. It has no doorman, and no Prometheus activity query that can be evaluated. Its message says why requests don’t wake it, and how to fix that. With ask, queries that can’t be evaluated leave it NotLearning too, even behind the doorman

A message	Ends with
While the workload is paused	What it learns from then, requests through the doorman, its Prometheus activity queries, or both
While it’s paused behind the doorman and asks before waking	people waking it (a click on its asleep page, a wake request or an activity annotation), with and its Prometheus activity queries if it has them
Behind the doorman, while its queries can’t be evaluated	its Prometheus activity queries failing. Behind the doorman, it learns whatever its queries do

A message while learning, then one while not learning, which says why and how to fix it, with a link:

learning: it has seen 100 of the 168 hours of the week, and scores its forecast of an hour from the second time it sees it; expected to start acting around Tue 24 Mar 00:00 UTC, if the coming days repeat the last week

not learning while paused: no Service routes requests to it; a request can't wake it, so its demand can't be seen, and it learns only the hours it's awake. To learn the hours it's paused too, give it a Service, or, for a workload no request reaches, such as a queue worker, a Prometheus activity query that sees its demand: https://docs.napwake.dev/concepts/wake-on-request/#when-a-workload-isnt-routed

startsActing
How it’s projected	A copy of the forecast runs on through the demand last seen in each hour of the week, until it would act. It has the same phase gates and confidence, and assumes the coming week looks like the last
Unseen hours	An unseen hour of the week takes the same hour on the nearest day it has. An hour of the day it has never seen is never assumed. So a workload paused nightly with nothing watching is never projected to act
How close it is	On office hours, within a day of when the forecast does act, from its second day. Within an hour once it has seen a weekend
When it’s projected	Again every hour, as the forecast learns
When it’s unset	While the forecast is acting. Also when the forecast isn’t expected to act within four weeks, as for a pattern too irregular for its setting. The message then says so

kubectl get managedworkloads -o wide shows the reason in a Forecast column.

Pausing early

With pauseEarly on, a workload idle for 20 minutes is paused without waiting out idleAfter, when all of these hold:

Condition	What it takes
The forecast is acting, and sure the hours ahead are quiet	Every hour until idleAfter would run out, and at least the hour under way and the next, is forecast below cpuThreshold. It’s judged as the veto and autoResume judge, at the same threshold. An unseen hour of the week can’t be called quiet. Nor can the hours after one far quieter than forecast, as on a holiday
A request would wake it	Its wake.mode is Request, the default. A workload that asks before waking never pauses early, since a request doesn’t wake it. The doorman is on and has a Ready pod, and the last pause found nothing wrong with routing (WakeOnRequest). Every Service that selects the pods is one the doorman would route while paused, on every port outside napwake.dev/doorman-ignore-ports. None is headless or has a UDP or SCTP port. None keeps traffic on the caller’s node (internalTrafficPolicy: Local), or has another workload’s Ready pods behind it. Behind GKE container-native load balancing, it has the forwarder. At least one routes TCP, and their ports fit the doorman ports the workload and its namespace may hold
Nothing else holds it	These hold an early pause, as they hold any. An active-until hold, awake dependents, and the hour Napwake waits after a GitOps tool undid its last pause

A wrong forecast then costs a request a wake, a cold start, never its answer. With ask, a wrong guess would cost a person a click, so a workload that asks waits out idleAfter.

Case	What happens
Anything above doesn’t hold, or can’t be read	The workload waits out idleAfter as usual, and the operator logs why
Services without a selector, and ExternalName Services	Not checked, since nothing says which pods they reach
Dry-run	An early pause is a would-be pause, measured like any other
The forecast setting	A separate setting. It says how sure the forecast must be to act at all, and this one whether, once sure, it brings pauses forward. So a cautious forecast can pause early too
The hours of an early pause	Behind the doorman, learned like any paused hour (What an hour records), so the forecast keeps learning

It’s off unless you set napwake.dev/pause-early: "true" on an opted-in workload or its namespace. The Helm value defaults.pauseEarly sets it for every opted-in workload, and spec.idlePolicy.pauseEarly: true on a ManagedWorkload you write.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: preview-api
  annotations:
    napwake.dev/auto-resume: "true"
    napwake.dev/pause-early: "true"

An early pause has a PausedEarly event in place of IdleDetected, naming the hours it expects quiet. Until the workload wakes, status.activity.forecastQuietUntil holds when those hours end. kubectl napwake status says so in NEXT. The event comes first here, then the status line:

idle for 20 minutes, since its CPU use was last above the activity threshold, and the forecast expects the hours from 18:00 to 20:00 UTC to be quiet; pausing before idleAfter (1 hour) runs out

paused early: the forecast expected quiet until Fri 20:00 UTC; wakes on a request or activity

What it saves and costs

The pause comes once the hour people leave in is over, since that hour is learned as busy. On office hours ending at 17:30, it comes at 18:00. The activity clock would have paused at 18:29, an idleAfter after the last busy minute. The longer idleAfter, the more that saves. The cost is a cold start for anyone who comes in an hour forecast quiet, before idleAfter would have run out.

The simulator measures both at balanced, for a workload at 40% of its CPU request in use. It counts the weeks after the forecast acts. For a holiday that’s its week, and for a change of hours the two weeks after.

Workload	Hours awake a week, without → with	Extra cold starts a week
Busy on weekdays from 09:10 to 17:30, quiet nights and weekends	48.7 → 46.3	0
The same, with idleAfter: 2h	53.7 → 46.3	0
The same hours every day, weekends included	68.1 → 64.8	0
Office hours, people arriving and leaving up to 20 minutes either side and away one weekday in ten (the average of 20)	45.2 → 43.3	0
Office hours, people arriving and leaving up to an hour either side and away one weekday in four (the average of 20)	38.2 → 37.8	0
Office hours, the week of a holiday on a Wednesday	39.8 → 37.8	0
Office hours moving to 13:00 to 21:00, the two weeks after	56.0 → 54.7	0
Office hours, someone coming back for ten minutes 15 to 50 minutes after people left, one weekday in three (the average of 20)	50.6 → 48.6	0.6
Office hours, someone working in the evening too, one weekday in three, in three bursts with up to 25 minutes between them (the average of 20)	55.4 → 52.2	0.5
Always busy	168.0 → 168.0	0
Always idle	0.0 → 0.0	0

A test in the repo runs these, and fails if they drift from this table.

	What it means
The default idleAfter of an hour	Half an hour saved a weekday, about 5% of the hours awake. At two hours, an hour and a half a weekday
Learned patterns, holidays and changes of hours included	No cold starts. Nothing is brought forward on a holiday, since the forecast doubts itself from the holiday’s first quiet hour. Nor on the first two days of new hours, while it relearns. And nobody comes back in the hours it pauses through
Loosely kept hours	They save less. There are fewer days people leave to bring a pause forward from, and fewer evenings the forecast is sure of
Cold starts	They come from use it can’t foresee within idleAfter of the last. Someone comes back after the others left, or works in the evening in bursts. Each gap over 20 minutes, in an hour learned as quiet, finds the workload paused
The 20 minutes	At 10, evening work like that row’s costs three times the cold starts, for a quarter more of the hours saved. On office hours, any wait up to half an hour saves the same

Turn it on for workloads used to a pattern. There, an hour after the last use is an hour paid for nothing every day. Leave it off for one people come back to at odd hours, or set active-until while they do.

Learning from history

With a Prometheus URL (prometheus.url, --prometheus-url), a new ManagedWorkload’s forecast starts from the workload’s history. The operator reads up to the last four weeks of its CPU. It learns them in one go, as it would have live. Then it carries on live. A steady pattern acts from the first day. An irregular or short history goes on learning, through the same phase gates and confidence.

History	How it’s read
An hour of history	Measured as a live hour is. That’s the workload’s own containers’ usage against their requests, sidecars injected into its pods left out, at its busiest moment. Above cpuThreshold, it counts as at least twice cpuThreshold
Its pods	Found by the names a Deployment or StatefulSet gives them, so old revisions’ pods are part of its history too
A Prometheus holding more than one cluster’s series, such as Thanos, Mimir or a shared one	Set prometheus.selector (--prometheus-selector), such as cluster="staging". Its matchers are added to the query, so a same-named workload in another cluster isn’t read as this one
The query	One range query per workload, a point an hour, of the busiest five-minute rate of container_cpu_usage_seconds_total in each hour. Prometheus works it out, and answers with at most 672 points
When it fails	The operator asks for one workload at a time, and waits up to 30 seconds. A failing query is tried twice more, two minutes apart, before the forecast learns live only
A reconcile	Never held up. The forecast learns live meanwhile, and takes up what was learned once it’s ready
How often	Once per ManagedWorkload, when it’s new. One that has learned live keeps what it learned. Deleting and creating it again starts it afresh, from history again

What history can’t tell	What it assumes
Hours none of its pods ran in	Quiet, as an hour scaled to zero by something else is live. That includes hours Napwake paused it, if it managed the workload before. One paused behind the doorman or watched by an activity query is quiet anyway. One paused with neither, which live learning skips, can’t be told from an hour nobody used
Its CPU requests	They’re today’s. If they changed over the four weeks, earlier hours are measured against the new ones
Use seen only in requests or activity annotations	None, since that isn’t in Prometheus. So a light-use workload’s history looks quieter than its live hours will
Bursts	Five-minute rates are smoother than the minute-by-minute readings live learning takes. A burst of a minute or two counts for less

status.prediction.history says what came of it, with an event once:

history.reason	Event	Meaning
Learned	ForecastLearnedHistory	Learned hours of history, from and to. The message says how much, such as the forecast learned from 4 weeks of the workload's CPU history in Prometheus
NoHistory	ForecastLearningLive	Prometheus has no CPU for its pods, as for a workload deployed with it. The forecast learns from now
NoCPURequests	ForecastHistoryUnavailable (warning)	Its containers request no CPU, so there’s nothing to measure its use against
QueryFailed	ForecastHistoryUnavailable (warning)	Prometheus didn’t answer three times. The message has its error
NoPrometheusURL	none	The operator has no Prometheus URL, so the forecast learns live from the start

kubectl napwake forecast shows what the forecast would make of your workloads’ history before you opt them in.

What an hour records

The forecast learns CPU utilisation, as a percentage of what the workload’s own containers request, in the pods reporting it. That’s the measure cpuThreshold is set in (Idle detection). The veto and the wake ahead compare the forecast with cpuThreshold directly. So a forecast that keeps an idle workload awake would also wake it once paused.

Each hour is fed in once it’s over, as the busiest moment seen in it. It’s recorded only if it was checked from its start to its end.

While the workload is	The hour records
Awake	The highest utilisation, from CPU read every minute. The first 15 minutes after a wake aren’t counted, whatever woke it, kubectl scale included. After that, an hour the activity clock sees it in use counts as at least twice cpuThreshold. That’s use from CPU above cpuThreshold, an activity annotation or a Prometheus query
Paused behind the doorman (WakeOnRequest=True), with wake.mode: Request	No demand, while a check finds it still paused, since a request would have woken it. A request that wakes it counts as at least twice cpuThreshold, however little it then uses. So do a wake request and an activity annotation. That holds while its Prometheus activity queries can’t be evaluated. The hour is busy if a request woke it or a query found activity, and quiet otherwise
Paused behind the doorman, with wake.mode: Ask	No demand, while a check finds it still paused, since someone who needed it would have woken it. A person’s wake counts as at least twice cpuThreshold. That’s a click on Wake it, a wake request or an activity annotation. So does a query finding activity
With Ask, while the doorman doesn’t route it, or its queries can’t be evaluated	Demand is seen only if someone woke it, so the hour isn’t learned. Nobody can click then, and the doorman sees only people. So an outage of the doorman or of Prometheus can’t teach it quiet hours
With Ask, and no Service to route, such as a queue worker	It learns from its queries while they can be evaluated, as with Request
Paused with a Prometheus activity query	The query runs every minute, as while awake, and wakes the workload when it finds activity. A quiet check finds no demand, and activity counts as at least twice cpuThreshold. A query that can’t be evaluated never wakes it. Without the doorman, that hour isn’t recorded
Paused with neither watching, and wake.mode: Request	Nothing. Demand it can’t see isn’t zero

A workload whose use stays under cpuThreshold, with no other source, is idle to Napwake while awake. So autoResume can’t learn its mornings. Lower cpuThreshold below its use, or add a source that sees it.

Why an hour records what it does

Rule	Why
An hour records its peak	The activity clock counts any minute above cpuThreshold as activity. An hour busy only for its last ten minutes is an hour the workload has to be awake for
The first 15 minutes after a wake don’t count	Starting up isn’t demand. Counted, a wake ahead of a busy hour would teach the forecast that the hour before is busy too. It would wake the workload earlier every week
An hour in use counts as at least twice cpuThreshold	A light-use workload’s use may show only in an annotation or a query. Without the floor, it would learn light use on mornings autoResume had it ready. On mornings people woke it, it would learn a request’s twice cpuThreshold. The forecast would then have it ready only every other morning
A waking request counts as twice cpuThreshold	So the hour people arrive in is learned as busy. A wake request, an activity annotation and a click count the same
With ask, a paused hour behind the doorman nobody woke it in is quiet	Only people wake it then. Someone who needed it would have clicked or asked
Only where people could have woken it	So an outage can’t teach the forecast quiet hours
A query watches a paused workload	That’s how a workload no Service routes requests to, such as a queue worker, learns its nights. It uses a query of its queue’s length
The hour under way is kept in memory	After an operator restart or a leader change, that hour is skipped, unless it was only minutes old. It isn’t recorded from the part seen after

The model

Holt-Winters keeps four components, and moves each a small step towards every hour it sees. A component’s smoothing parameter sets the size of its step.

Component	What it represents	Smoothing parameter
Level	The mean demand	$\alpha = 0.01$
Trend	Whether demand is growing or shrinking	$\beta = 0.001$
Daily components	24 offsets, one per hour of day, the shape of an average day	$\gamma_1 = 0.1$
Weekly components	168 offsets, one per hour of week, how each weekday departs from that shape	$\gamma_2 = 0.5$

Each hour, the model compares the hour’s utilisation $Y(t)$ with what it predicted, for confidence scoring. Then it updates all four by exponential smoothing. It’s additive, since idle workloads spend much of their time at zero demand. A multiplicative model, which divides by the level and the seasonal factors, diverges there.

The forecast $h$ hours ahead adds the level, $h$ hours of trend, and that hour’s daily and weekly offsets. It’s never negative:

$$F(t+h) = \max\Bigl(0,\; L(t) + h \cdot T(t) + D(t+h) + W(t+h)\Bigr)$$

The update equations

$$\begin{aligned} L(t) &= \max\Bigl(0,\; \alpha \cdot \bigl(Y(t) - D(t) - W(t)\bigr) + (1 - \alpha) \cdot \bigl(L(t{-}k) + k \cdot T(t{-}k)\bigr)\Bigr) \\[6pt] T(t) &= \beta \cdot \frac{L(t) - L(t{-}k)}{k} + (1 - \beta) \cdot T(t{-}k) \\[6pt] D(t) &= \gamma_1 \cdot \bigl(Y(t) - L(t) - W(t)\bigr) + (1 - \gamma_1) \cdot D(t - s_1) \\[6pt] W(t) &= \gamma_2 \cdot \bigl(Y(t) - L(t) - D(t)\bigr) + (1 - \gamma_2) \cdot W(t - s_2) \end{aligned}$$

$s_1 = 24$ is the daily season’s length, and $s_2 = 168$ the weekly one’s. $k$ is the hours since the previous observation, 1 unless observations were missed.

After each update, the components are renormalised. The daily components sum to zero, and for each hour of the day, so do the weekly components across the seven days. The shift moves into the component above. That leaves every forecast unchanged, and keeps the level the mean demand, so it’s never negative.

	What the model does
A gap in the observations, from the operator being down or a workload paused with neither the doorman nor an activity query watching	Skips it, since there’s no honest value to fill it with. Slots are wall-clock hours, so skipping one doesn’t shift what was learned
The trend across a gap	Carries it, across a gap of at most a day
The first week	Initialises the model, as the first season does in classical Holt-Winters. An hour of the week seen for the first time sets its weekly component, so its forecast is what was seen. The level, trend and daily components learn only from hours seen again
One week of office hours	Forecasts the next week as the week it saw, weekends quiet
An hour of the week seen for the first time	Doesn’t score it, since it had nothing to forecast it from. The forecast doesn’t act on an hour it hasn’t seen

Confidence scoring

Confidence is one minus the forecast’s error as a share of demand. It’s $1 - \text{WAPE}$, the weighted absolute percentage error. That’s the total absolute error of the hourly forecasts over a window, divided by the total demand in it.

$$C = 1 - \frac{\sum_{i} |F(i) - Y(i)|}{\max\bigl(\sum_{i} Y(i),\; n \cdot f\bigr)}$$

Term	Meaning
A confidence of 75%	The total error over the window is 25% of the demand in it
Zero demand	Defined, unlike an average of per-hour percentage errors. An hour of zero demand forecast as zero isn’t counted as a perfect hour
The floor, $n \cdot f$	$f$ is the workload’s mean hourly demand over about a week, and at least 1% of the CPU requested. So a quiet weekend day is judged against a typical day’s demand. A workload idling at a sliver of its request isn’t judged on its noise
Daily confidence	Scored over the last 24 scored hours
Weekly confidence	Scored over the last 168 scored hours, so a whole week of weekdays and weekend must be forecast well
A window not yet full	No score until it is
An hour of the week seen for the first time	Not scored, or judged an anomaly

Anomaly detection

Traffic patterns change, with a marketing campaign, a new feature or a change of season. Napwake detects these regime changes with z-score anomaly detection. An hour is an anomaly when its error is more than three standard deviations from the recent errors.

1. Each hour, the signed prediction error $e(t) = Y(t) - F(t)$ becomes a two-sided z-score. It uses an exponentially weighted mean $\mu$ and standard deviation $\sigma$ of recent errors, which remember about a week. A sudden surge and a sudden disappearance both count.

   $$z(t) = \frac{|e(t) - \mu|}{\max(\sigma, f)}$$

2. If $z(t) > 3.0$, the hour is an anomaly. The model learns from it only up to 3 standard deviations, so a single spike isn’t learned as a pattern. An error that large in the same direction as the last one in its hour of the week isn’t an anomaly. It’s a weekly pattern, such as a Tuesday night batch job, and it’s learned in full.

3. 3 or more anomalies within 24 observed hours are a regime change.

After	The forecast does this
An anomaly, or any hour in the day after a regime change	It holds the change to that hour of the week until the same hour the next day. If that hour breaks the same way, the change lasted and is made. If it goes as forecast, the day was a one-off, such as a holiday, and the change is dropped. The level and daily components learn as usual. Learned at once, a weekday holiday had the same weekday a week later forecast quieter than it is
An hour far quieter than forecast	It doubts itself. It neither holds off a pause nor wakes the workload. That lasts until an hour goes as forecast, or a regime change has it relearn
An hour far busier than forecast	Nothing, since the workload is in use then anyway
A regime change	It drops at least one level below the phase it held before the anomalies began (below). If they already cost it that much, by lowering its confidence, it stays where they left it. It’s never a second demotion, and never a promotion. It discards the scored errors, the anomaly statistics, and which hours of the week it has observed. Whatever its phase, it acts again only once a full day of its forecasts is scored. A RegimeChange warning event is emitted once

Before a regime change	After
FullyActive	WeeklySuggesting
WeeklySuggesting or DailyActive	DailySuggesting
DailySuggesting	Observing

A change	What happens
A weekday holiday	autoResume wakes the workload. The first quiet office hour is an anomaly, so the forecast doubts the rest, and the activity clock pauses it. That’s an hour and a quarter awake for nothing. Three quiet hours make a regime change. The forecast is off until a day is scored, about 30 hours, so the next morning’s first request wakes the workload. The forecast is back on during that day. That day is busy, so the holiday isn’t learned into the week. The same weekday a week later, and every morning from the second after the holiday, is ready
A Saturday people come in	The same. The Saturday after isn’t woken, and the Monday after is the morning it costs
A lasting change of hours, such as 09:00 to 13:00	A regime change within the day, learned into the week from its second day. For about a week people wake the workload themselves. At balanced, every afternoon from the second Tuesday is ready. Within two weeks it stops waking the workload for nothing. Until then it does a few times, for the old hours and, on weekends, the new ones
A workload nobody uses any more	Woken for its old hours twice, on its first quiet morning and the same weekday a week later. Then it’s left paused

Idle detection and wake on request carry on throughout. A single deviant day can’t be told from the start of a new pattern until the next day shows which it was. So the forecast doesn’t guess. It costs the next morning, and nothing after.

Wall-clock alignment

Seasonal slots are keyed to wall-clock time, not a running counter. So Monday 9am always maps to the same slot, whatever the gaps.

Case	What happens
The timezone	The operator’s, --timezone or the Helm value timezone, UTC by default. Each workload records it in status.prediction.timeZone, and kubectl napwake status shows it
Daylight saving	Business hours keep their slots when the clocks change. Set --timezone to its users’ zone, and a workload in New York or London is up at 08:45 local time every weekday. That holds through the change
Left at UTC	The operator’s clock is an hour off its users’ after the change, so the forecast has to learn their mornings again. In the simulator, four mornings that week find the workload paused. It’s held awake for nothing on others, until about two weeks on
A timezone whose hours start at half or quarter past the hour in UTC, such as India’s	Hours are counted on the local clock, and so are the 15 minutes autoResume wakes ahead
A workload paused for 6 hours without the doorman	It keeps its alignment. The next observation goes into the right hour-of-day slot
An hour paused behind the doorman, asking before waking or not, or watched by an activity query	Recorded as no demand. That’s how the model learns that nights and weekends are quiet

Persistence

The learned state lives in the ManagedWorkload’s status, in status.prediction.state. It’s gzipped JSON, base64-encoded, about 2.5 KB, written with each hourly observation. No external database, persistent volume or ConfigMap is needed, and each workload has its own independent engine.

Case	What happens
What it holds	The model’s components, the scored errors, the anomaly statistics, and the weekly changes held for the next day. Whether the forecast doubts the hour under way. The last hour observed, and the demand last seen in each hour of the week, which startsActing is projected from
Operator restarts and leader changes	The engine survives them. An hour observed before a restart isn’t observed twice, and the hour under way at a restart is skipped. Restarted daily, at a different hour each day, the operator loses those hours and nothing else. The forecast acts within hours of when it would have
A last observed hour more than an hour ahead of the operator’s clock	Not trusted, since a wrong clock recorded it. The next hour is observed as usual, rather than refused until real time catches up
State that can’t be read, from an older version or a hand edit	Discarded with a ForecastReset warning event. The engine starts learning again

Tuning

The forecast setting, or spec.prediction.confidence in its place, is the only thing to tune. The smoothing parameters are fixed, the same for every workload, and can’t be configured.

Parameter	Value	What a higher value would do
$\alpha$	0.01	Level reacts faster to demand changes
$\beta$	0.001	Trend detection is more aggressive
$\gamma_1$	0.1	Daily pattern adapts faster
$\gamma_2$	0.5	Weekly pattern adapts faster

Lower values give smoother, steadier forecasts, and higher ones a more reactive but noisier model. A weekly component updates once a week. So $\gamma_2$ is high enough for a lasting change in a weekday’s hours to be half learned the week after. The first week sets each one outright (The model). The values were chosen by simulating office-hours, always-on, idle and noisy workloads.

Proving it

A forecasting mistake would otherwise show only after weeks in a real cluster. Such mistakes are a wrong slot, a late wake or a false wake at the weekend. The repo’s simulator, internal/forecast/sim, runs the operator’s own forecast engine and decision rules. It runs them through simulated time a minute at a time, as the operator looks at a ManagedWorkload.

It covers the activity clock pausing, the veto, autoResume waking 15 minutes ahead, and the doorman or a query waking the workload. It covers the warm-up after a wake, each hour’s peak fed to the forecast, and restarts through the state saved in status. Months of traffic take milliseconds. A test runs six weeks of office hours through both the reconciler and the simulator. It finds the same wakes and pauses at the same minutes.

The tests run in CI on every change. Besides How long it takes, they check these:

Scenario	What the tests check
Office hours	Once learned, the workload is up at 08:45 every weekday, 25 minutes before people arrive at 09:10. It’s paused an hour after they leave, and left paused at weekends. At every setting, every morning from the day the forecast acts is ready, Mondays included, and it’s never still paused on one
Every day, always busy, always idle, light use seen only through an activity annotation	Each acts as above. The light-use workload is ready every morning
Loosely kept hours	Ready on most mornings once learned. Woken for nothing about twice a month, for an hour or two each time. The settings trade as Forecast setting says
A holiday, a Saturday at work, a permanent change of hours, a workload nobody uses any more, daylight saving in New York and London with and without --timezone, daily operator restarts	As described above
Pauses with neither the doorman nor an activity query	The forecast never acts
Pausing early	On office hours, every weekday’s pause comes at 18:00. Mornings are as ready, and the same hours learned. Nothing is brought forward on a holiday or the first two days of new hours. Nobody finds the workload paused who wouldn’t have. Someone back after 18:00 finds it paused. Without the doorman, nothing is brought forward. The reconciler’s six weeks bring the same pauses forward at the same minutes. What it saves and costs is measured there too
A public preview only people wake, with wake: ask, paused every night and woken by someone at 09:05 on weekdays	It acts after 8 days, as behind the doorman. From then, autoResume has it up at 08:45, before they come. One nobody wakes is learned as quiet, behind the doorman
A queue worker with no Service, woken by a query of its queue’s length, paused every night	It acts after 8 days, as behind the doorman. From then, autoResume has it up before the morning’s messages
Learning from history	Four weeks of office hours have the workload ready from the first morning. The projection made in the first week is within a day of when the forecast starts acting

Run the simulator with go run ./hack/forecastsim. It runs every scenario at every setting, and prints what it measured. These flags change what it runs:

Flag	What it does
-scenario holiday -setting eager -days	Shows one scenario day by day
-pause-early	Brings pauses forward, counting the hours saved and the uses that then found the workload paused
-wake-on-request=false -wake-on-query	Runs a scenario as a queue worker its activity query wakes
-wake-on-ask	Runs a scenario as a workload only people wake, as wake: ask does
-history	Replays a workload’s recorded CPU, as a percentage of its requests. It takes a Prometheus range query’s JSON, or CSV of time,percent rows, with steps at most seven minutes apart

On your own cluster, kubectl napwake forecast replays the forecast over each workload’s Prometheus history. It says whether the forecast would act if opted in today. For the last two weeks, it counts the mornings each workload would have been ready. It also counts how often one would have been woken for nothing.

Named, as kubectl napwake forecast api -n orders, it shows one workload’s forecast. For a workload Napwake manages, that’s its own forecast, from its status and saved state, whatever history Prometheus has. It shows the phase, the hours of the day and week seen, and the hours learned from history and live. It shows whether it’s acting or when it should, why, and the hours of the next day it expects busy. Then comes the replay of its history, day by day, when there’s enough of it.

Next

• Wake on request →A request, or with ask a click, still wakes a paused workload, whatever the forecast says.
• Idle detection →The activity clock decides pauses, unless pauseEarly is on.
• Replay the forecast →See what it would make of your own Prometheus history.