Skip to content

Drain a node

This walkthrough assumes a Norsk Manager with norsk-mgr in front of it, and two local servers registered as workers in the same pool — here both on this machine. Draining is how you take a worker out of service for maintenance without stopping what is running on it: the Manager moves each runtime to another worker in the pool first, and only then lets the node go. Every command below runs against http://127.0.0.1:9333, the default; set NORSK_MGR_URL to point elsewhere.

nodes list shows both, reachable and with cordoned: false — either can take a placement. Keep the two node ids to hand; the commands below take them.

Terminal window
norsk-mgr nodes list
{
"items": [
{
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"state": "running",
"lastActivityAt": "2026-11-21T11:13:50.759Z",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"provider": {
"provider": "cluster"
},
"publicIpAddress": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"proxyUrl": "https://127.0.0.1",
"providerInstanceId": "cluster-TJsfyxPTxihs8AdO",
"spare": false,
"unhealthy": false,
"warmFor": [],
"partiallyWarmFor": [],
"inventory": {
"totalCapacity": 8,
"utilizationCap": 1,
"totalCores": 8,
"gpus": [],
"capabilities": [
{
"name": "norsk.io/intel-qsv",
"count": 1
},
{
"name": "norsk.io/job-socket",
"count": 1
},
{
"name": "norsk.io/secret-references",
"count": 1
},
{
"name": "norsk.io/secret-references-runtime",
"count": 1
}
],
"reservedCapacity": 0,
"reservedCores": 0,
"reservedCapabilities": [],
"reachable": true,
"lastSeen": "2026-11-21T11:13:50.885Z",
"cordoned": false,
"assignedRuntimes": [],
"accessReplica": {
"version": 1795259625379,
"digest": "17166b7ca89b975b2a5f4d559cf41b8561edc083d5ca2e1e9c89422b0050cd3f",
"vouchedAt": "2026-11-21T11:13:50.380Z",
"staleLimitMs": 28800000,
"ageMs": 2398,
"state": "current"
},
"secretsProfile": {
"name": "production",
"matchesManager": true
}
}
},
{
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"state": "running",
"lastActivityAt": "2026-11-21T11:13:52.369Z",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:51.850Z",
"provider": {
"provider": "cluster"
},
"publicIpAddress": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"proxyUrl": "https://127.0.0.1",
"providerInstanceId": "cluster-XNGUdfZgw4rkqh6p",
"spare": false,
"unhealthy": false,
"warmFor": [],
"partiallyWarmFor": [],
"inventory": {
"totalCapacity": 8,
"utilizationCap": 1,
"totalCores": 8,
"gpus": [],
"capabilities": [
{
"name": "norsk.io/intel-qsv",
"count": 1
},
{
"name": "norsk.io/job-socket",
"count": 1
},
{
"name": "norsk.io/secret-references",
"count": 1
},
{
"name": "norsk.io/secret-references-runtime",
"count": 1
}
],
"reservedCapacity": 0,
"reservedCores": 0,
"reservedCapabilities": [],
"reachable": true,
"lastSeen": "2026-11-21T11:13:52.364Z",
"cordoned": false,
"assignedRuntimes": [],
"secretsProfile": {
"name": "production",
"matchesManager": true
}
}
}
],
"nextCursor": null
}

A show format and a production launched from it, as in Launch a show format. One thing matters here: the product template declares how a running instance may be moved (migration.strategy in its manifest). This example product uses stopThenStart — the runtime is stopped on the old worker and started on the new one — which is what a drain runs for each runtime it finds.

Terminal window
norsk-mgr production-template launch evening-news --production-id evening-news-01
{
"launched": [
{
"productionId": "evening-news-01",
"bundleId": "evening-news-01",
"runtimes": [
"main"
],
"launchCredential": "c2663729fedbfee3ba69cec7e105228bde557ec7868f21509120a1f3b55c5c98"
}
],
"failures": []
}

The Manager chose studio-1 for it — runtime get shows the node under nodeId, with the history of where the runtime has been so far (one entry). That is the worker this walkthrough takes out of service.

Terminal window
norsk-mgr runtime get evening-news-01.0.main
{
"job": {
"runtimeRecordId": "evening-news-01.0.main",
"state": "active",
"currentHash": "96993733",
"tags": {
"auto-manager": "true",
"productionId": "evening-news-01",
"runtimeId": "main",
"launchCredentialSha256": "edfd28ced23f3c072bc99123765d1770707458e6009747a6525e1dfdb01e5b47",
"productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6",
"replicaIndex": "0"
},
"startDateTime": "2026-11-21T11:13:53.467Z",
"cloud": "aws",
"description": "",
"writeFiles": [],
"services": [],
"volumes": [],
"managerConfiguration": "",
"norskMediaVersion": "recommended",
"norskServiceParameters": {
"portMappings": [],
"environmentVariables": [],
"cpuSet": [],
"volumeMounts": [],
"bindMounts": [],
"ulimits": []
},
"shape": "",
"availabilityDomain": "",
"subnet": "",
"architecture": "",
"norskCpuList": [],
"autoPlacement": {
"$typeName": "norsk.api.manager.AutoPlacement",
"requirements": {
"$typeName": "norsk.api.manager.JobRequirements",
"requiredCapacity": 1,
"requiredCapabilities": [],
"onRuntimeFailure": 1,
"requiredHostPorts": []
}
},
"productTemplate": {
"productName": "e2e-fixture",
"productVersion": "0.0.1",
"sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a",
"templateName": "evening-news-studio",
"source": {
"kind": "url",
"url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…"
}
},
"productTemplateParameters": {
"INSTANCE_NAME": "evening-news-01"
}
},
"history": [
{
"event": "running",
"timestamp": "2026-11-21T11:13:57.529Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "provisioned",
"timestamp": "2026-11-21T11:13:53.609Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "created",
"timestamp": "2026-11-21T11:13:53.561Z"
}
],
"dropped": false,
"dormant": false,
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"nodeAddress": "127.0.0.1",
"migration": {
"declaredStrategy": "stopThenStart"
}
}

Cordoning a node takes it out of placement: what is running on it keeps running, but nothing new is placed there. It is the first thing a drain does, and it is useful on its own — cordon studio-1 now, and any production launched in the meantime goes to studio-2.

Terminal window
norsk-mgr nodes cordon cluster-TJsfyxPTxihs8AdO
{
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"action": "cordoned"
}

Launch another production from the same format. With studio-1 cordoned, the only place it can go is studio-2.

Terminal window
norsk-mgr production-template launch evening-news --production-id evening-news-02
{
"launched": [
{
"productionId": "evening-news-02",
"bundleId": "evening-news-02",
"runtimes": [
"main"
],
"launchCredential": "156b1374c76aba0ad0e4b21a344bdc056db640bfa9921af11e025adb4e8f80fc"
}
],
"failures": []
}
Terminal window
norsk-mgr runtime get evening-news-02.0.main
{
"job": {
"runtimeRecordId": "evening-news-02.0.main",
"state": "active",
"currentHash": "121555076",
"tags": {
"auto-manager": "true",
"productionId": "evening-news-02",
"runtimeId": "main",
"launchCredentialSha256": "bac01690269e4042f7abb023d76aa4ab4ebca33bfcd69d5f77d6d7737e174e28",
"productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6",
"replicaIndex": "0"
},
"startDateTime": "2026-11-21T11:13:58.722Z",
"cloud": "aws",
"description": "",
"writeFiles": [],
"services": [],
"volumes": [],
"managerConfiguration": "",
"norskMediaVersion": "recommended",
"norskServiceParameters": {
"portMappings": [],
"environmentVariables": [],
"cpuSet": [],
"volumeMounts": [],
"bindMounts": [],
"ulimits": []
},
"shape": "",
"availabilityDomain": "",
"subnet": "",
"architecture": "",
"norskCpuList": [],
"autoPlacement": {
"$typeName": "norsk.api.manager.AutoPlacement",
"requirements": {
"$typeName": "norsk.api.manager.JobRequirements",
"requiredCapacity": 1,
"requiredCapabilities": [],
"onRuntimeFailure": 1,
"requiredHostPorts": []
}
},
"productTemplate": {
"productName": "e2e-fixture",
"productVersion": "0.0.1",
"sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a",
"templateName": "evening-news-studio",
"source": {
"kind": "url",
"url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…"
}
},
"productTemplateParameters": {
"INSTANCE_NAME": "evening-news-02"
}
},
"history": [
{
"event": "running",
"timestamp": "2026-11-21T11:14:02.760Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:51.850Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-XNGUdfZgw4rkqh6p"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "provisioned",
"timestamp": "2026-11-21T11:13:58.815Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:51.850Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-XNGUdfZgw4rkqh6p"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "created",
"timestamp": "2026-11-21T11:13:58.773Z"
}
],
"dropped": false,
"dormant": false,
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"nodeAddress": "127.0.0.1",
"migration": {
"declaredStrategy": "stopThenStart"
}
}

…and the first production is where it was, on studio-1.

nodes drain cordons the node (already done here), migrates every runtime on it to another worker in the pool, and terminates the node once it is empty. A runtime that cannot be moved keeps the node, cordoned, unless you pass --force. --wait polls until the drain ends and exits non-zero if it ended partial; --yes skips the confirmation. Here it is, start to finish:

Terminal window
norsk-mgr nodes drain cluster-TJsfyxPTxihs8AdO --yes --wait
{
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"phase": "done",
"startedAt": "2026-11-21T11:14:03.712Z",
"timeoutMs": 600000,
"force": false,
"runtimes": [
{
"runtimeRecordId": "evening-news-01.0.main",
"status": "migrated",
"migrationId": "17855ee7-1f09-4515-86fc-3f06cdea3b2d"
}
]
}

--wait holds the terminal until the drain ends. From a second terminal, nodes drain-status shows the same view it polls — the drain’s phase and each runtime’s progress — and runtime get shows the runtime mid-move: still bound to the draining node until its new instance is up on the other worker, with the migration recorded in its history.

Terminal window
norsk-mgr nodes drain-status cluster-TJsfyxPTxihs8AdO
{
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"phase": "migrating",
"startedAt": "2026-11-21T11:14:03.712Z",
"timeoutMs": 600000,
"force": false,
"runtimes": [
{
"runtimeRecordId": "evening-news-01.0.main",
"status": "migrating",
"migrationId": "17855ee7-1f09-4515-86fc-3f06cdea3b2d"
}
]
}
Terminal window
norsk-mgr runtime get evening-news-01.0.main
{
"job": {
"runtimeRecordId": "evening-news-01.0.main",
"state": "active",
"currentHash": "96993733",
"tags": {
"auto-manager": "true",
"productionId": "evening-news-01",
"runtimeId": "main",
"launchCredentialSha256": "edfd28ced23f3c072bc99123765d1770707458e6009747a6525e1dfdb01e5b47",
"productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6",
"replicaIndex": "0"
},
"startDateTime": "2026-11-21T11:13:53.467Z",
"cloud": "aws",
"description": "",
"writeFiles": [],
"services": [],
"volumes": [],
"managerConfiguration": "",
"norskMediaVersion": "recommended",
"norskServiceParameters": {
"portMappings": [],
"environmentVariables": [],
"cpuSet": [],
"volumeMounts": [],
"bindMounts": [],
"ulimits": []
},
"shape": "",
"availabilityDomain": "",
"subnet": "",
"architecture": "",
"norskCpuList": [],
"autoPlacement": {
"$typeName": "norsk.api.manager.AutoPlacement",
"requirements": {
"$typeName": "norsk.api.manager.JobRequirements",
"requiredCapacity": 1,
"requiredCapabilities": [],
"onRuntimeFailure": 1,
"requiredHostPorts": []
}
},
"productTemplate": {
"productName": "e2e-fixture",
"productVersion": "0.0.1",
"sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a",
"templateName": "evening-news-studio",
"source": {
"kind": "url",
"url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…"
}
},
"productTemplateParameters": {
"INSTANCE_NAME": "evening-news-01"
}
},
"history": [
{
"event": "stopping",
"timestamp": "2026-11-21T11:14:05.572Z",
"role": "primary",
"reason": "migration",
"additionalReason": "17855ee7-1f09-4515-86fc-3f06cdea3b2d",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"hold": "cordoned",
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "prepared",
"timestamp": "2026-11-21T11:14:05.568Z",
"role": "primary@17855ee7",
"nodeMetadata": {
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:51.850Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-XNGUdfZgw4rkqh6p"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "provisioned",
"timestamp": "2026-11-21T11:14:03.717Z",
"role": "primary@17855ee7",
"nodeMetadata": {
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:51.850Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-XNGUdfZgw4rkqh6p"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "running",
"timestamp": "2026-11-21T11:13:57.529Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "provisioned",
"timestamp": "2026-11-21T11:13:53.609Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "created",
"timestamp": "2026-11-21T11:13:53.561Z"
}
],
"dropped": false,
"dormant": false,
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"nodeAddress": "127.0.0.1",
"migration": {
"declaredStrategy": "stopThenStart"
}
}

The production is live throughout — rollupState is active again, and nothing is broken or awaitingCapacity. The runtime now lives on studio-2: nodeId has changed, and its history carries the move.

Terminal window
norsk-mgr production get evening-news-01
{
"productionId": "evening-news-01",
"pool": "local",
"jobs": [
{
"runtimeId": "main",
"requirements": {
"requiredCapacity": 1,
"requiredCapabilities": [],
"onRuntimeFailure": "avoidSpare"
},
"launch": {
"kind": "productTemplate",
"ref": {
"productName": "e2e-fixture",
"productVersion": "0.0.1",
"sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a",
"templateName": "evening-news-studio",
"source": {
"kind": "url",
"url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…"
}
},
"parameters": {
"INSTANCE_NAME": "evening-news-01"
},
"startDateTime": "2026-11-21T11:13:53.467Z",
"tags": {
"launchCredentialSha256": "edfd28ced23f3c072bc99123765d1770707458e6009747a6525e1dfdb01e5b47",
"productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6"
}
},
"startMode": "auto"
}
],
"state": "live",
"rollupState": "active",
"broken": false,
"resilienceDegraded": false,
"awaitingCapacity": false,
"onLaunch": "avoidSpare",
"identity": {
"productionTemplateId": "evening-news",
"productionTemplateRevision": "1",
"runtimeUrlBase": "/bundles/evening-news-01"
}
}
Terminal window
norsk-mgr runtime get evening-news-01.0.main
{
"job": {
"runtimeRecordId": "evening-news-01.0.main",
"state": "active",
"currentHash": "96993733",
"tags": {
"auto-manager": "true",
"productionId": "evening-news-01",
"runtimeId": "main",
"launchCredentialSha256": "edfd28ced23f3c072bc99123765d1770707458e6009747a6525e1dfdb01e5b47",
"productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6",
"replicaIndex": "0"
},
"startDateTime": "2026-11-21T11:13:53.467Z",
"cloud": "aws",
"description": "",
"writeFiles": [],
"services": [],
"volumes": [],
"managerConfiguration": "",
"norskMediaVersion": "recommended",
"norskServiceParameters": {
"portMappings": [],
"environmentVariables": [],
"cpuSet": [],
"volumeMounts": [],
"bindMounts": [],
"ulimits": []
},
"shape": "",
"availabilityDomain": "",
"subnet": "",
"architecture": "",
"norskCpuList": [],
"autoPlacement": {
"$typeName": "norsk.api.manager.AutoPlacement",
"requirements": {
"$typeName": "norsk.api.manager.JobRequirements",
"requiredCapacity": 1,
"requiredCapabilities": [],
"onRuntimeFailure": 1,
"requiredHostPorts": []
}
},
"productTemplate": {
"productName": "e2e-fixture",
"productVersion": "0.0.1",
"sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a",
"templateName": "evening-news-studio",
"source": {
"kind": "url",
"url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…"
}
},
"productTemplateParameters": {
"INSTANCE_NAME": "evening-news-01"
}
},
"history": [
{
"event": "running",
"timestamp": "2026-11-21T11:14:16.028Z",
"role": "primary@17855ee7",
"nodeMetadata": {
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:51.850Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-XNGUdfZgw4rkqh6p"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "stopped",
"timestamp": "2026-11-21T11:14:15.884Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"hold": "cordoned",
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "stopping",
"timestamp": "2026-11-21T11:14:05.572Z",
"role": "primary",
"reason": "migration",
"additionalReason": "17855ee7-1f09-4515-86fc-3f06cdea3b2d",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"hold": "cordoned",
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "prepared",
"timestamp": "2026-11-21T11:14:05.568Z",
"role": "primary@17855ee7",
"nodeMetadata": {
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:51.850Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-XNGUdfZgw4rkqh6p"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "provisioned",
"timestamp": "2026-11-21T11:14:03.717Z",
"role": "primary@17855ee7",
"nodeMetadata": {
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:51.850Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-XNGUdfZgw4rkqh6p"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "running",
"timestamp": "2026-11-21T11:13:57.529Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "provisioned",
"timestamp": "2026-11-21T11:13:53.609Z",
"role": "primary",
"nodeMetadata": {
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:13:48.099Z",
"providerMetadata": {
"provider": "cluster"
}
},
"runningNodeMetadata": {
"publicDnsName": "",
"privateDnsName": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"publicIpAddress": "127.0.0.1",
"providerMetadata": {
"provider": "cluster",
"instanceId": "cluster-TJsfyxPTxihs8AdO"
},
"proxyUrl": "https://127.0.0.1"
}
},
{
"event": "created",
"timestamp": "2026-11-21T11:13:53.561Z"
}
],
"dropped": false,
"dormant": false,
"nodeId": "cluster-XNGUdfZgw4rkqh6p",
"nodeAddress": "127.0.0.1",
"migration": {
"declaredStrategy": "stopThenStart"
}
}

Once the node is empty the drain terminates it. For a cloud node that destroys the instance; a local server is your machine, so the Manager just lets go of it, closing the worker’s stream. Its registration is untouched — local-servers list still carries it — so it needs no re-registering.

The worker on the drained machine sees its stream to the Manager end and re-registers moments later — usually before you could look — under the same node id, running again, with nothing on it. The cordon is durable and outlives the node: the worker is back, but still off-limits until you say otherwise.

Terminal window
norsk-mgr nodes get cluster-TJsfyxPTxihs8AdO
{
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"state": "running",
"lastActivityAt": "2026-11-21T11:14:18.192Z",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:14:18.144Z",
"provider": {
"provider": "cluster"
},
"publicIpAddress": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"proxyUrl": "https://127.0.0.1",
"providerInstanceId": "cluster-TJsfyxPTxihs8AdO",
"spare": false,
"unhealthy": false,
"warmFor": [
"evening-news-studio"
],
"partiallyWarmFor": [],
"inventory": {
"totalCapacity": 8,
"utilizationCap": 1,
"totalCores": 8,
"gpus": [],
"capabilities": [
{
"name": "norsk.io/intel-qsv",
"count": 1
},
{
"name": "norsk.io/job-socket",
"count": 1
},
{
"name": "norsk.io/secret-references",
"count": 1
},
{
"name": "norsk.io/secret-references-runtime",
"count": 1
}
],
"reservedCapacity": 0,
"reservedCores": 0,
"reservedCapabilities": [],
"reachable": true,
"lastSeen": "2026-11-21T11:14:18.853Z",
"cordoned": true,
"assignedRuntimes": [],
"accessReplica": {
"version": 1795259625379,
"digest": "17166b7ca89b975b2a5f4d559cf41b8561edc083d5ca2e1e9c89422b0050cd3f",
"vouchedAt": "2026-11-21T11:13:50.380Z",
"staleLimitMs": 28800000,
"ageMs": 28517,
"state": "current"
},
"secretsProfile": {
"name": "production",
"matchesManager": true
}
}
}

Maintenance done, nodes uncordon lifts the cordon and studio-1 takes placements again. Nothing moves back on its own — the runtime stays on studio-2 until something else moves it.

Terminal window
norsk-mgr nodes uncordon cluster-TJsfyxPTxihs8AdO
{
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"action": "uncordoned"
}
Terminal window
norsk-mgr nodes get cluster-TJsfyxPTxihs8AdO
{
"nodeId": "cluster-TJsfyxPTxihs8AdO",
"state": "running",
"lastActivityAt": "2026-11-21T11:14:19.067Z",
"tags": {
"pool": "local",
"versionSkew": "none",
"workerProtocolVersion": "2.3",
"workerVersion": "0.1.0-2026-11-21-7114c45"
},
"createdAt": "2026-11-21T11:14:18.144Z",
"provider": {
"provider": "cluster"
},
"publicIpAddress": "127.0.0.1",
"privateIpAddress": "127.0.0.1",
"proxyUrl": "https://127.0.0.1",
"providerInstanceId": "cluster-TJsfyxPTxihs8AdO",
"spare": false,
"unhealthy": false,
"warmFor": [
"evening-news-studio"
],
"partiallyWarmFor": [],
"inventory": {
"totalCapacity": 8,
"utilizationCap": 1,
"totalCores": 8,
"gpus": [],
"capabilities": [
{
"name": "norsk.io/intel-qsv",
"count": 1
},
{
"name": "norsk.io/job-socket",
"count": 1
},
{
"name": "norsk.io/secret-references",
"count": 1
},
{
"name": "norsk.io/secret-references-runtime",
"count": 1
}
],
"reservedCapacity": 0,
"reservedCores": 0,
"reservedCapabilities": [],
"reachable": true,
"lastSeen": "2026-11-21T11:14:18.853Z",
"cordoned": false,
"assignedRuntimes": [],
"accessReplica": {
"version": 1795259625379,
"digest": "17166b7ca89b975b2a5f4d559cf41b8561edc083d5ca2e1e9c89422b0050cd3f",
"vouchedAt": "2026-11-21T11:13:50.380Z",
"staleLimitMs": 28800000,
"ageMs": 28867,
"state": "current"
},
"secretsProfile": {
"name": "production",
"matchesManager": true
}
}
}

Only if you are done with them: stop the two productions, then remove the format and the template.

Terminal window
norsk-mgr production delete evening-news-01 --yes
{
"productionId": "evening-news-01",
"deleted": true
}
Terminal window
norsk-mgr production delete evening-news-02 --yes
{
"productionId": "evening-news-02",
"deleted": true
}
Terminal window
norsk-mgr production-template rm evening-news
{
"status": "ok",
"name": "evening-news"
}
Terminal window
norsk-mgr product-template rm evening-news-studio --yes
{
"status": "ok",
"name": "evening-news-studio"
}

Captured 2026-11-21 against norsk-mgr 0.1.0-2026-11-21-7114c45, Norsk Manager 1.4.2-2026-11-19-3758b1eb — every command and screenshot on this page is from that run.