Operations
Troubleshooting
What you see, why, and the command that fixes it.
Opting in and pausing
Section titled “Opting in and pausing”A labelled workload has no ManagedWorkload
Section titled “A labelled workload has no ManagedWorkload”kubectl get managedworkloads -n <namespace> doesn’t list it. The workload’s own events say why:
kubectl describe deployment <name> -n <namespace>| Event | Cause and fix |
|---|---|
LabelIgnored, on the pod template |
On the pods (a chart’s podLabels), where Napwake doesn’t look. Put it on the workload or namespace (Opting in) |
LabelIgnored, not "true" |
Only "true" opts in. Settings such as dry-run are annotations |
LabelIgnored, Napwake’s own |
The operator’s and doorman’s Deployments are never managed |
Protected |
A protected namespace, such as kube-system |
| No event | napwake.dev/ignore: "true", or a ManagedWorkload someone wrote, which wins. Under Kustomize, commonLabels fails on an existing workload (With Kustomize) |
An InvalidSetting warning
Section titled “An InvalidSetting warning”A settings annotation on the workload or its namespace has a value Napwake can’t read. The warning event is on the workload, and names the annotation, where it is, and what Napwake did instead:
napwake.dev/wake="Ask" on the workload isn't "request" or "ask", so the doorman asks before waking it, to be safe| The annotation | What a value that can’t be read does |
|---|---|
| Most settings | It’s skipped. The namespace’s value decides, then the Helm default if there is one, then the built-in one |
napwake.dev/dry-run |
Turns dry-run on |
napwake.dev/wake |
The doorman asks before waking, whatever the namespace says |
Fix the value. Settings annotations lists what each accepts, such as exactly "request" or "ask", in lower case, for wake.
Workload is idle but not being paused
Section titled “Workload is idle but not being paused”-
Dry-run?
Terminal window kubectl get managedworkload my-api -n staging -o jsonpath='{.spec.dryRun}'truemeans it evaluates but doesn’t act.kubectl napwake enable my-api -n stagingends it, however it was set up. On a labelled workload,spec.dryRunfollowsnapwake.dev/dry-run(workload, namespace, thendefaults.dryRun), and an unreadable value, such as"False", means dry-run (by hand). -
Last active?
Terminal window kubectl get managedworkload my-api -n staging -o jsonpath='{.status.activity}'pauseAtis when it pauses. If it keeps moving,lastActivitySourcesays what still counts as activity:Source Cause and fix cpuOver activity.cpuThreshold% of requests, as polling or cron loops can be. Raise it if expected. It takes a decimal, such as"0.5"annotationnapwake.dev/last-activityrefreshed, ornapwake.dev/active-untilahead, on it or its targetrolloutA recent pod template change prometheusAn activity.prometheusquery above zerounobservedA gap in watching, which isn’t activity. An Unobservedevent says how long it wasunobserved, its causeThe operator was down or restarted, or CPU or a query was unreadable for over a couple of checks unobserved, the clockThe gap outlasted the one allowed, so the clock restarted as watching resumed unobserved, repeatingCheck MetricsAvailableandPrometheusAvailable -
CPU readable? Without CPU data a busy workload looks idle, so Napwake doesn’t act.
Terminal window kubectl get managedworkload my-api -n staging \-o jsonpath='{.status.conditions[?(@.type=="MetricsAvailable")]}'FalsereasonCause and fix NoPodMetricsNo metrics from its Ready pods for over 3 minutes. Check metrics-server is installed, with kubectl top pods -n stagingNoCPURequestsNo CPU requests. Set them, or a LimitRange default, then restart its pods, as a default applies only to new ones MetricsUnavailableThe Metrics API failed, error in the message. No metrics.k8s.ioAPI means no metrics-serverZero replicas, and
Unknownjust after a wake or rollout, aren’t errors. -
The forecast?
Terminal window kubectl get managedworkload my-api -n staging \-o jsonpath='{.status.conditions[?(@.type=="IdleVetoed")]}'A confident forecast expecting demand in the hour under way or the next defers the pause. The
IdleVetoed=Truemessage says when the forecast expects demand, and anIdleVetoedevent marks the start. -
Anything else?
kubectl napwake statusgives the reason under NEXT. Or:You see Cause and fix No idlePolicyNothing pauses it but kubectl napwake pausePrometheusAvailable=FalseA query fails, or no --prometheus-urlProtected=TrueProtectedNamespace,NapwakeComponent(Napwake’s own), orPVCsDeletedOnScaleDown, for a StatefulSet whosepersistentVolumeClaimRetentionPolicy.whenScaled: Deletewould lose its PVCs. SetwhenScaled: RetainGitOpsConflict=TrueA GitOps tool undid the last pause. Napwake retries after an hour -
Depended on?
Terminal window kubectl get managedworkload postgres -n staging \-o jsonpath='{.status.conditions[?(@.type=="HeldByDependents")]}'HeldByDependents=Truenames its awake dependents, declared independsOnor learned. It pauses after them.kubectl napwake deps postgres -n stagingshows each link’s source.DependencyCycle=Truemeans two depend on each other, so break the cycle.
kubectl napwake pause didn’t pause it
Section titled “kubectl napwake pause didn’t pause it”kubectl napwake pause my-api -n stagingkubectl get managedworkload my-api -n staging \ -o jsonpath='{.status.conditions[?(@.type=="PauseRequest")]}'pause waits for Paused, or prints why not. PauseRequest keeps that answer until the next request, whatever happens since. It’s True (Paused, AlreadyPaused), as in paused as requested at 2026-10-21 03:32 UTC, or False, starting not paused at, with a reason. Each request is answered once, so fix the reason and run pause again (Pause now).
| Reason | Cause |
|---|---|
ForecastExpectsDemand |
Demand expected within the hour. kubectl napwake pause --yes pauses anyway |
HeldByDependents, DependencyCycle |
Its dependents are awake, or it’s in a dependsOn cycle |
ActiveUntil |
napwake.dev/active-until holds it |
DryRun, Protected, ScaledToZero |
Dry-run, protected, or at zero already |
TargetNotFound, TargetIgnored, DuplicateTarget |
Its target is missing, ignored, or another ManagedWorkload’s |
PredatesStatus |
The request was already on the ManagedWorkload as Napwake took it on, as after a backup restored without status. Set a new value |
Workload paused with pods still running
Section titled “Workload paused with pods still running”PauseIncomplete=True, with a PauseIncomplete warning event instead of Paused, means pods outlived their termination grace period by a minute (one after another for a StatefulSet). Normally it’s Paused once pods stop. Here Napwake stopped waiting, so it’s Paused rather than stuck Pausing, and wakes as usual. It turns False once they’ve gone, or on a wake.
A StatefulSet with the default OrderedReady stops pods one at a time, and not while one isn’t Ready. A Deployment’s can hang terminating on a finalizer or an unreachable node. kubectl describe the pods.
Waking
Section titled “Waking”Requests to a paused workload fail instead of waking it
Section titled “Requests to a paused workload fail instead of waking it”kubectl get managedworkload my-api -n staging \ -o jsonpath='{.status.conditions[?(@.type=="WakeOnRequest")]}'WakeOnRequest |
Cause and fix |
|---|---|
None, or NotPaused |
Not paused, so not routed yet |
Message starts at its last pause: |
Awake now, and the next pause meets the problem named. Fix it now |
NoServices, ServedByOtherPods, NodeLocalTraffic, UnsupportedIPFamily |
No Service it can route. A queue worker can wake on a Prometheus query |
UnsupportedLoadBalancer, or a Service named as not routed |
In GKE NEGs. Use the forwarder |
ForwarderNotReady |
No forwarder pod Ready, normal for seconds after a pause. If it lasts, kubectl describe pods -n staging -l app.kubernetes.io/name=napwake-forwarder for an unpullable image (the operator’s), a quota or LimitRange (doorman.negForwarder.resources), or a NetworkPolicy |
DoormanUnavailable |
No doorman pod Ready. Check kubectl get pods -A -l control-plane=doorman |
NoFreePort, TooManyPorts |
All doorman ports (20000-29999) taken, or over doorman.portsPerWorkload (32 by default) or doorman.portsPerNamespace (256), as the message says. Raise a cap, or list ports in napwake.dev/doorman-ignore-ports |
RoutingFailed |
EndpointSlice writes failed, error in the message, retried with backoff. Check the operator’s RBAC and logs |
DoormanDisabled, TargetNotFound |
doorman.enabled: false, or the workload is gone |
If it’s DoormanRouted and requests still fail (What the caller sees):
| The caller sees | Cause and fix |
|---|---|
| A close after a while, no response | Not Ready by wake.maxWait, or pods unreachable, as a RequestNotServed warning says. Raise maxWait, or see why pods are slow. Counted by napwake_doorman_wakes_total{result=~"timeout|error"} |
| A close at once, nothing sent | A limit, counted by napwake_doorman_wakes_total{result="limited"} |
The app’s own error, such as a 503, just after a wake |
Pods Ready before the app was, as with no readiness probe. A NoReadinessProbe warning, and the WakeOnRequest message, say so. Add a readiness probe that passes once it can serve |
A 503 with no body |
A health check or scraper User-Agent, such as kube-probe/ or Prometheus/, which never wakes it |
A 503 saying it’s asleep and how to wake it |
The workload asks before waking. Only a person’s click on Wake it, or tooling, wakes it. For a caller that isn’t a browser, such as another service or a webhook, set napwake.dev/wake: "request". status.turnedAway counts these requests |
| A 502 from ingress-nginx, or a refused request, as the workload paused | A pod exited the moment it got SIGTERM, before kube-proxy and nginx dropped it. The doorman stands in front of the Service before the pods stop, so it’s the pod’s shutdown. Give the app a preStop sleep of a few seconds (Work in progress at a pause) |
| A 502 from a GKE Ingress or Gateway while paused | In NEGs without the forwarder, so there’s no healthy backend. Just after a pause, GKE takes up to a minute to see health checks pass |
| A 504 from an Ingress at 60 seconds | Raise its upstream timeout, for ingress-nginx nginx.ingress.kubernetes.io/proxy-read-timeout |
| A refusal or timeout at once, every time | A NetworkPolicy blocking the doorman (ports 20000-29999), or it to the pods (Network policies) |
The doorman’s logs name the workload and Service of each held connection (in your install’s namespace):
kubectl logs -n napwake-system -l control-plane=doormanWake it doesn’t wake the workload
Section titled “Wake it doesn’t wake the workload”With ask, a click on Wake it wakes the workload, and the waking-up page follows. So does tooling, with kubectl napwake wake or napwake.dev/wake-requested.
Anything else gets a 503 and wakes nothing, by design. Check the setting first:
kubectl get managedworkload my-api -n staging -o jsonpath='{.spec.wake.mode}'When the asleep page comes back after a click, its notice says why:
| The notice | Cause and fix |
|---|---|
“Nothing woke”, a 403 |
Its token expired because its reloads failed, as after a laptop slept. The page reloads with a fresh one every minute. Or it was shown during an earlier pause, or the browser said the click came from no page (Sec-Fetch-Site: none). Click the button on the page shown now |
| The same, every time | The doorman’s replicas may sign with different keys. Their logs say loading the key the Wake it button is signed with when they can’t read the Secret napwake-doorman-wake-key. Check it exists, and the doorman’s Role still allows reading it. With the Kustomize manifests, or your own, include the Secret and its Role |
“The click came from another address”, a 403 |
The browser says the click came from a page at another address. That’s another site or subdomain, by its Sec-Fetch-Site. Or, over plain HTTP, its Origin isn’t the address the doorman saw the page at. That’s the one at the top of the page. A proxy may rewrite the Host header, as ingress-nginx’s upstream-vhost does. Pass the browser’s Host on, or serve the preview over HTTPS. Clicking again won’t help, so wake it with kubectl napwake wake or napwake.dev/wake-requested meanwhile |
| “Try a current browser” | The click came with neither Sec-Fetch-Site nor an Origin, or with Origin: null, as some browsers from before about 2020 send. So the doorman can’t tell it from a spider posting the page’s token. Open it in a current browser, or wake it with kubectl napwake wake or napwake.dev/wake-requested |
| The same, from a current browser | Something between it and the doorman drops those headers, or a browser extension does. ingress-nginx and most proxies pass them on as they are (How the doorman tells) |
“Try again in a minute”, a 429 with Retry-After: 60 |
A click wakes a workload at most once a minute. Try again shortly |
| “It didn’t wake” | The doorman couldn’t stamp the wake, as when the API server didn’t answer, or a wake rate limit was hit. Its logs say which |
| They see | Cause and fix |
|---|---|
A plain-text 503 in a browser |
To the doorman it isn’t a person’s browser. Its User-Agent names a crawler or a headless browser, or doesn’t start with Mozilla/. Wake it with kubectl napwake wake or napwake.dev/wake-requested |
A 503 saying how to wake it, from an API client, curl or a script |
Only a browser gets the button. Run kubectl napwake wake, or have the tool set napwake.dev/wake-requested |
| No asleep page, and the request fails | It isn’t routed (Requests to a paused workload fail) |
Ask, though nobody set it |
Its namespace or defaults.wake says ask, or its value can’t be read (InvalidSetting) |
status.turnedAway counts what the doorman turned away during the pause, and kubectl napwake status lists the workloads that turned requests away (What it turns away).
Workload stuck in Resuming
Section titled “Workload stuck in Resuming”WaitingForDependencies=True means a waitForReady dependency isn’t Ready. The condition names it and its ready replicas, so check its pods. Otherwise the workload’s own pods aren’t getting Ready.
Napwake waits for every restored replica, but no longer than 2 minutes after the first is Ready, or, while none is, the Deployment’s progressDeadlineSeconds (10 minutes by default, and for a StatefulSet). It’s then Running on its Ready replicas, possibly none, with ResumeIncomplete=True (Resuming → Running). A dependency that never gets Ready is waited for only so long, then skipped with DependencyWontStart. The NapwakeWorkloadStuck alert fires at 15 minutes, and kubectl napwake status lists it under Stuck at 10.
Workload resumed with replicas not Ready
Section titled “Workload resumed with replicas not Ready”ResumeIncomplete=True, with a warning event of that name, means the resume stopped waiting with replicas not Ready. It’s Running on those that are, its KEDA ScaledObject scales it again, and requests go to its Ready pods. With none Ready, the doorman stops holding requests, so they fail until one is.
It’s common when it wasn’t wholly Ready before the pause, with a pod crash-looping or unschedulable. kubectl describe its pods. Once every replica is Ready, later or at the next resume, it turns False with an AllReplicasReady event.
Workload keeps cycling between paused and running
Section titled “Workload keeps cycling between paused and running”The events say what woke it:
| Event | Cause and fix |
|---|---|
WokeByActivity |
An activity annotation changed, or a held request (WokenByRequest names the Service). A monitor requesting on a timer wakes it each time. Point it elsewhere, put its port in the Service’s napwake.dev/doorman-ignore-ports, or have the workload ask before waking (napwake.dev/wake: "ask"). So does a tool refreshing napwake.dev/last-activity on a timer, not on real use |
AutoResume |
The forecast expected demand |
ScaledUp |
Scaled up outside Napwake, by what the event and status.lastScaledUp name. A GitOps tool also sets GitOpsConflict=True. It sets the replicas from Git, undoing every pause until told to leave them (GitOps tools) |
The doorman already ignores connections that close without sending anything, and common probes and scrapers (What wakes a workload).
A PausedWhileInUse warning means its use undid a pause soon after it (Workload paused while in use).
Workload paused while in use
Section titled “Workload paused while in use”A PausedWhileInUse warning event, and kubectl napwake status under Needs attention, say Napwake paused it while it was in use. A request, a click, a wake request or an activity annotation undid the pause soon after it began. Napwake didn’t see the use, usually because the workload uses less CPU in use than its threshold.
kubectl describe managedworkload my-api -n stagingWarning PausedWhileInUse a request came 40 seconds after it paused, so it was in use: it now waits 10 minuteswithout activity before pausing, not 5 minutes, until a pause holds. It used up to 0.6% of its CPU requests whilein use, and up to 0.2% idle: try napwake.dev/cpu-threshold: "0.4"Meanwhile its idle clock waits longer, doubling with each wrong pause up to idleAfter plus an hour. So it’s paused in use less often. Fix the cause, and it pauses on time again:
| You see | Fix |
|---|---|
| A suggested threshold | Set it, on the workload or the ManagedWorkload’s idlePolicy.activity.cpuThreshold. Napwake suggests one only when the workload’s CPU in use sits clearly above its idle CPU |
| No threshold suggested | CPU can’t tell its use from its idle. Count its requests with a Prometheus activity query, such as its ingress’s request rate |
| A caller that keeps a connection open, such as a gRPC client or a connection pool from another workload | Add that workload with dependsOn, or opt it in, so it holds this one awake |
The warning stops once a pause lasts 15 minutes, or idleAfter if shorter, without being undone. See Paused while in use.
You need a paused workload awake now
Section titled “You need a paused workload awake now”kubectl napwake wake my-api -n stagingIt wakes the workload whatever its wake setting, and waits for Running, or says why Napwake won’t wake it. The WakeRequest condition keeps that answer until the next request. False, with a message starting not woken at and the time, gives the reason (Wake now, kubectl plugin).
Dependencies, activity and the forecast
Section titled “Dependencies, activity and the forecast”A dependency isn’t held awake, woken or waited for
Section titled “A dependency isn’t held awake, woken or waited for”kubectl napwake deps my-api -n preview-42That lists its dependencies and their sources. The ManagedWorkload’s conditions say why one is left out:
| You see | Cause and fix |
|---|---|
DependencyNotAllowed=True |
Its namespace’s napwake.dev/dependents-from doesn’t list this one. Annotate it (Across namespaces) |
DependencyNotFound=True |
A dependsOn workload is missing (DependencyNotFound), or outside watchNamespaces or unreadable (DependencyNotVisible). Neither blocks the workload |
A DependencyWakeFailed warning event |
Its wake annotation was refused, such as by a policy engine, error in the message. The workload wakes anyway, and the dependency may stay paused |
A Prometheus query never matches
Section titled “A Prometheus query never matches”PrometheusQueryEmpty=True, reason NoSeries, with a warning event, means an activity query has matched no series since Napwake first ran it. It’s usually a misspelt metric or label. It can’t keep the workload awake or hold off a pause. Test it in Prometheus.
An activity annotation is ignored
Section titled “An activity annotation is ignored”An InvalidActivityAnnotation warning event means a napwake.dev/last-activity, last-request or active-until value isn’t RFC 3339, such as epoch seconds or a bare date. It neither keeps the workload awake nor wakes it. Write it as date -u +%Y-%m-%dT%H:%M:%SZ does (Activity annotations).
Prediction confidence stays at 0
Section titled “Prediction confidence stays at 0”Daily confidence needs a full 24-hour window of observed hours, and weekly 168. An hour isn’t observed while the operator is down, CPU is unreadable, or the workload is paused with neither the doorman nor a Prometheus query to see demand.
With ask, a query that can’t be evaluated leaves it unobserved too, even behind the doorman. Paused that way nightly, it never gets a full day, and status.prediction.message says why requests don’t wake it, and the fix.
status.prediction.reason and message say what the forecast awaits, and startsActing when it should act (When it acts). With a Prometheus URL, a new workload starts from its history, reported in status.prediction.history. Also check metrics-server returns data:
kubectl top pods -n stagingCost data shows $0.00
Section titled “Cost data shows $0.00”| Field | Why |
|---|---|
| All | Priced on what its pods request, so no requests means no cost |
projectedMonthlyCost, projectedMonthlyFreed, projectedMonthlySavings |
pending until 24 hours of the month are tracked, on the first of each month too |
freedThisMonth |
Not paused this month. Dry-run workloads record what they’d free in status.dryRun |
savedThisMonth |
The nodes still fit everything paused, so nothing comes off the bill until nodes go (Freed but not saved). unknown, with why in savingsUnknown, under watchNamespaces (Saved reads unknown) |
Totals restart at zero each calendar month, in UTC (Cost tracking).
Freed but not saved
Section titled “Freed but not saved”FREED THIS MONTH well above SAVED in kubectl napwake status, maybe with Needs attention saying nodes aren’t scaling down. The nodes are still there. Saved counts nothing while a pool’s spare capacity could hold everything paused there (How saved is measured).
-
Is anything scaling the nodes? The first lines of
kubectl napwake statusandkubectl napwake nodesname the node autoscaler, as doesnapwake_node_autoscaler. Withnone found, nothing removes nodes or comes off the bill. Enable Cluster Autoscaler on the node groups, or install Karpenter (Cluster Autoscaler guide). -
What’s keeping the nodes? BLOCKED BY in
kubectl napwake nodes, with how to clear each below its table:BLOCKED BY Cause not autoscaledNo autoscaler found scales it NodePool never consolidates,NodePool consolidates only empty nodesKarpenter consolidateAfter: Never, orconsolidationPolicy: WhenEmptyon a node with more than DaemonSetsscale-down disabled,do-not-disrupt on the nodeNode annotated cluster-autoscaler.kubernetes.io/scale-down-disabled: "true"orkarpenter.sh/do-not-disrupt: "true"PDB allows no disruptionA pod whose PodDisruptionBudget allows none safe-to-evict=false,do-not-disruptPod annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "false"orkarpenter.sh/do-not-disrupt: "true"local storageA pod with emptyDirorhostPath, kept while Cluster Autoscaler’s--skip-nodes-with-local-storageis onbare podA pod no controller would recreate kube-system pod without PDBKept while --skip-nodes-with-system-podsis on -
Busy neighbours? A node goes only once everything on it can, so a cluster of slightly busy nodes keeps them all. Check UTILISATION in
kubectl napwake nodes. Low, with nothing in BLOCKED BY, means the node awaits the autoscaler’s threshold or delay. Middling means it’s in use. A node pool of their own lets pausing non-prod empty whole nodes. -
Spare capacity on purpose? A node group minimum above use, long
scale-down-unneeded-timeorscale-down-delay-after-add, ascale-down-utilization-thresholdunder the nodes’ use, longconsolidateAfter, tight Karpenter disruptionbudgets, or over-provisioning placeholder pods. Saved catches up as those nodes go (Node pools for real savings).
By pool, napwake_capacity_spare_cpu_cores against napwake_capacity_paused_cpu_cores shows which pools still fit what’s paused (Metrics).
Saved reads unknown
Section titled “Saved reads unknown”savedThisMonth, projectedMonthlySavings and SAVED in kubectl napwake status read unknown, with why above the status table and in:
kubectl get managedworkload my-api -n staging -o jsonpath='{.status.cost.savingsUnknown}'Under watchNamespaces the operator can’t see other namespaces’ pods, so it can’t tell what’s left on each node for paused replicas. It can’t measure saved, and exports no napwake_capacity_* or napwake_savings_realised_dollars_per_hour series. Freed is still measured. For saved, install without watchNamespaces, which watches pods cluster-wide, caching only each bound pod’s requests (Data and access).
ManagedWorkloads
Section titled “ManagedWorkloads”Target not found
Section titled “Target not found”TargetAvailable=False, reason TargetNotFound, with a warning event of that name:
kubectl get managedworkload my-api -n staging -o jsonpath='{.status.conditions}'Check the target exists (kubectl get deployment my-api -n staging), target.kind matches (Deployment or StatefulSet), and it shares the ManagedWorkload’s namespace. spec.target can’t change, so create another ManagedWorkload for another workload. Reason TargetIgnored means it’s labelled napwake.dev/ignore.
Two ManagedWorkloads target one workload
Section titled “Two ManagedWorkloads target one workload”The oldest wins. The others report DuplicateTarget=True, naming it, and do nothing:
kubectl get managedworkloads -n staging -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.target.name}{"\n"}{end}'A ManagedByAnotherInstall warning
Section titled “A ManagedByAnotherInstall warning”Two installs watch its namespace, and the other, in the namespace the event names, took this ManagedWorkload first. Give each install its own watchNamespaces (More than one install). If the other is gone, kubectl annotate managedworkload <name> -n <namespace> napwake.dev/operator- hands it over.
A paused workload’s ManagedWorkload lost its status
Section titled “A paused workload’s ManagedWorkload lost its status”It happens to one restored from a backup without status, or deleted without its finalizer and recreated. A PauseRecovered event means Napwake took the pause up again from the workload’s napwake.dev/paused-from (how). It’s Paused again if still at zero, and wakes as usual. Leave that annotation alone. Napwake removes it at hand-back.
A PauseRecordLost warning means there was no record, or the annotation couldn’t be read, as the message says. With no record anywhere, nothing was scaled, and the workload is Running again. If its replicas are at zero anyway, Napwake takes it for scaled to zero outside it and leaves it, so scale it up yourself.
A ManagedWorkload let go of its workload without handing it back
Section titled “A ManagedWorkload let go of its workload without handing it back”A HandBackAbandoned warning event means its namespace is being deleted, and handing the paused workload back failed for 5 minutes, such as when a KEDA webhook that’s down refuses to release its ScaledObject. It lets go rather than block the deletion, and the workload goes with the namespace. The message has the error.
Installing and removing
Section titled “Installing and removing”helm uninstall fails with job napwake-uninstall-guard failed
Section titled “helm uninstall fails with job napwake-uninstall-guard failed”Or, in Helm 4, resource Job/napwake-system/napwake-uninstall-guard not ready. ManagedWorkloads are left, whose workloads would stay paused. Helm 3.18 and later list them after Napwake refuses to uninstall:. Before 3.18, or in any Helm after a helm rollback with one, the guard’s log does:
kubectl logs job/napwake-uninstall-guard -n napwake-systemRemove the napwake.dev/managed label from those workloads and their namespaces, delete the ManagedWorkloads you wrote, and uninstall again.
To keep Napwake, helm rollback napwake -n napwake-system marks the release deployed again, and until then helm upgrade fails with has no deployed releases. One still listed as being deleted is one the operator didn’t finish, usually as it isn’t running. Uninstall with --no-hooks, and restore the workloads by hand. With no CRD (crds.install: false), nothing is guarded. Argo CD 3.3 and later refuses to delete the Application the same way (When helm uninstall refuses).
helm upgrade fails with job napwake-upgrade-guard failed
Section titled “helm upgrade fails with job napwake-upgrade-guard failed”Or, in Helm 4, resource Job/napwake-system/napwake-upgrade-guard not ready. The new watchNamespaces leaves out namespaces where workloads are managed, stranding them paused. Helm 3.18 and later list them after Napwake refuses this upgrade:, or the guard’s log does:
kubectl logs job/napwake-upgrade-guard -n napwake-systemKeep those namespaces, or stop managing their workloads first (Narrowing watchNamespaces). Under Argo CD it fails the sync, in the PreSync hook.
After a refused helm rollback
A rollback to a narrower watchNamespaces is refused the same way, by the pre-rollback hook. Helm still records a revision, pending-rollback in helm history napwake, while the operator runs on unchanged, and helm upgrade fails with another operation (install/upgrade/rollback) is in progress until another revision replaces it. helm rollback napwake -n napwake-system, with no revision, goes back to the one before, the release as it runs, and marks it deployed.
Workloads stay at zero after the operator was removed
Section titled “Workloads stay at zero after the operator was removed”The operator or its CRD went while ManagedWorkloads were left. Nothing scales their workloads up, their Services still point at the doorman’s deleted pods, and deleting the ManagedWorkloads or CRD hangs in Terminating on the napwake.dev/cleanup finalizer. Reinstall and stop managing them, or restore them by hand.
None of these
Section titled “None of these”Debug checklist
Section titled “Debug checklist”Put your install’s namespace for napwake-system.
| Check | Command |
|---|---|
| Operator running | kubectl get pods -n napwake-system |
| CRDs installed | kubectl get crd managedworkloads.napwake.dev |
| Metrics server running | kubectl top nodes |
| RBAC correct | kubectl auth can-i get pods --as=system:serviceaccount:napwake-system:napwake. The ServiceAccount is napwake for a Helm release named napwake, and napwake-controller-manager with install.yaml or the Kustomize manifests |
| Events | kubectl describe managedworkload <name> -n <ns> |
| Logs | kubectl logs -n napwake-system -l control-plane=controller-manager |
| Status | kubectl get managedworkload <name> -n <ns> -o yaml |
Getting help
Section titled “Getting help”- The operator’s version (
kubectl get deploy -n napwake-system -o wideshows the image), the plugin’s (kubectl napwake version), and Kubernetes’ - The ManagedWorkload’s YAML, sanitized
- The relevant operator logs
- The output of
kubectl describe managedworkload