Drain a node
Before you start
Section titled “Before you start”This walkthrough assumes a Norsk Manager with norsk-mgr in front of it, and two local servers registered as workers in the same pool — here both on this machine. Draining is how you take a worker out of service for maintenance without stopping what is running on it: the Manager moves each runtime to another worker in the pool first, and only then lets the node go. Every command below runs against http://127.0.0.1:9333, the default; set NORSK_MGR_URL to point elsewhere.
Two workers, one pool
Section titled “Two workers, one pool”nodes list shows both, reachable and with cordoned: false — either can take a placement. Keep the two node ids to hand; the commands below take them.
norsk-mgr nodes list{ "items": [ { "nodeId": "cluster-TJsfyxPTxihs8AdO", "state": "running", "lastActivityAt": "2026-11-21T11:13:50.759Z", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "provider": { "provider": "cluster" }, "publicIpAddress": "127.0.0.1", "privateIpAddress": "127.0.0.1", "proxyUrl": "https://127.0.0.1", "providerInstanceId": "cluster-TJsfyxPTxihs8AdO", "spare": false, "unhealthy": false, "warmFor": [], "partiallyWarmFor": [], "inventory": { "totalCapacity": 8, "utilizationCap": 1, "totalCores": 8, "gpus": [], "capabilities": [ { "name": "norsk.io/intel-qsv", "count": 1 }, { "name": "norsk.io/job-socket", "count": 1 }, { "name": "norsk.io/secret-references", "count": 1 }, { "name": "norsk.io/secret-references-runtime", "count": 1 } ], "reservedCapacity": 0, "reservedCores": 0, "reservedCapabilities": [], "reachable": true, "lastSeen": "2026-11-21T11:13:50.885Z", "cordoned": false, "assignedRuntimes": [], "accessReplica": { "version": 1795259625379, "digest": "17166b7ca89b975b2a5f4d559cf41b8561edc083d5ca2e1e9c89422b0050cd3f", "vouchedAt": "2026-11-21T11:13:50.380Z", "staleLimitMs": 28800000, "ageMs": 2398, "state": "current" }, "secretsProfile": { "name": "production", "matchesManager": true } } }, { "nodeId": "cluster-XNGUdfZgw4rkqh6p", "state": "running", "lastActivityAt": "2026-11-21T11:13:52.369Z", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:51.850Z", "provider": { "provider": "cluster" }, "publicIpAddress": "127.0.0.1", "privateIpAddress": "127.0.0.1", "proxyUrl": "https://127.0.0.1", "providerInstanceId": "cluster-XNGUdfZgw4rkqh6p", "spare": false, "unhealthy": false, "warmFor": [], "partiallyWarmFor": [], "inventory": { "totalCapacity": 8, "utilizationCap": 1, "totalCores": 8, "gpus": [], "capabilities": [ { "name": "norsk.io/intel-qsv", "count": 1 }, { "name": "norsk.io/job-socket", "count": 1 }, { "name": "norsk.io/secret-references", "count": 1 }, { "name": "norsk.io/secret-references-runtime", "count": 1 } ], "reservedCapacity": 0, "reservedCores": 0, "reservedCapabilities": [], "reachable": true, "lastSeen": "2026-11-21T11:13:52.364Z", "cordoned": false, "assignedRuntimes": [], "secretsProfile": { "name": "production", "matchesManager": true } } } ], "nextCursor": null}Something to move
Section titled “Something to move”A show format and a production launched from it, as in Launch a show format. One thing matters here: the product template declares how a running instance may be moved (migration.strategy in its manifest). This example product uses stopThenStart — the runtime is stopped on the old worker and started on the new one — which is what a drain runs for each runtime it finds.
Launch it and see where it landed
Section titled “Launch it and see where it landed”norsk-mgr production-template launch evening-news --production-id evening-news-01{ "launched": [ { "productionId": "evening-news-01", "bundleId": "evening-news-01", "runtimes": [ "main" ], "launchCredential": "c2663729fedbfee3ba69cec7e105228bde557ec7868f21509120a1f3b55c5c98" } ], "failures": []}The Manager chose studio-1 for it — runtime get shows the node under nodeId, with the history of where the runtime has been so far (one entry). That is the worker this walkthrough takes out of service.
norsk-mgr runtime get evening-news-01.0.main{ "job": { "runtimeRecordId": "evening-news-01.0.main", "state": "active", "currentHash": "96993733", "tags": { "auto-manager": "true", "productionId": "evening-news-01", "runtimeId": "main", "launchCredentialSha256": "edfd28ced23f3c072bc99123765d1770707458e6009747a6525e1dfdb01e5b47", "productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6", "replicaIndex": "0" }, "startDateTime": "2026-11-21T11:13:53.467Z", "cloud": "aws", "description": "", "writeFiles": [], "services": [], "volumes": [], "managerConfiguration": "", "norskMediaVersion": "recommended", "norskServiceParameters": { "portMappings": [], "environmentVariables": [], "cpuSet": [], "volumeMounts": [], "bindMounts": [], "ulimits": [] }, "shape": "", "availabilityDomain": "", "subnet": "", "architecture": "", "norskCpuList": [], "autoPlacement": { "$typeName": "norsk.api.manager.AutoPlacement", "requirements": { "$typeName": "norsk.api.manager.JobRequirements", "requiredCapacity": 1, "requiredCapabilities": [], "onRuntimeFailure": 1, "requiredHostPorts": [] } }, "productTemplate": { "productName": "e2e-fixture", "productVersion": "0.0.1", "sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a", "templateName": "evening-news-studio", "source": { "kind": "url", "url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…" } }, "productTemplateParameters": { "INSTANCE_NAME": "evening-news-01" } }, "history": [ { "event": "running", "timestamp": "2026-11-21T11:13:57.529Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "provisioned", "timestamp": "2026-11-21T11:13:53.609Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "created", "timestamp": "2026-11-21T11:13:53.561Z" } ], "dropped": false, "dormant": false, "nodeId": "cluster-TJsfyxPTxihs8AdO", "nodeAddress": "127.0.0.1", "migration": { "declaredStrategy": "stopThenStart" }}Cordon it first
Section titled “Cordon it first”Cordoning a node takes it out of placement: what is running on it keeps running, but nothing new is placed there. It is the first thing a drain does, and it is useful on its own — cordon studio-1 now, and any production launched in the meantime goes to studio-2.
norsk-mgr nodes cordon cluster-TJsfyxPTxihs8AdO{ "nodeId": "cluster-TJsfyxPTxihs8AdO", "action": "cordoned"}Prove it: a second production
Section titled “Prove it: a second production”Launch another production from the same format. With studio-1 cordoned, the only place it can go is studio-2.
norsk-mgr production-template launch evening-news --production-id evening-news-02{ "launched": [ { "productionId": "evening-news-02", "bundleId": "evening-news-02", "runtimes": [ "main" ], "launchCredential": "156b1374c76aba0ad0e4b21a344bdc056db640bfa9921af11e025adb4e8f80fc" } ], "failures": []}norsk-mgr runtime get evening-news-02.0.main{ "job": { "runtimeRecordId": "evening-news-02.0.main", "state": "active", "currentHash": "121555076", "tags": { "auto-manager": "true", "productionId": "evening-news-02", "runtimeId": "main", "launchCredentialSha256": "bac01690269e4042f7abb023d76aa4ab4ebca33bfcd69d5f77d6d7737e174e28", "productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6", "replicaIndex": "0" }, "startDateTime": "2026-11-21T11:13:58.722Z", "cloud": "aws", "description": "", "writeFiles": [], "services": [], "volumes": [], "managerConfiguration": "", "norskMediaVersion": "recommended", "norskServiceParameters": { "portMappings": [], "environmentVariables": [], "cpuSet": [], "volumeMounts": [], "bindMounts": [], "ulimits": [] }, "shape": "", "availabilityDomain": "", "subnet": "", "architecture": "", "norskCpuList": [], "autoPlacement": { "$typeName": "norsk.api.manager.AutoPlacement", "requirements": { "$typeName": "norsk.api.manager.JobRequirements", "requiredCapacity": 1, "requiredCapabilities": [], "onRuntimeFailure": 1, "requiredHostPorts": [] } }, "productTemplate": { "productName": "e2e-fixture", "productVersion": "0.0.1", "sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a", "templateName": "evening-news-studio", "source": { "kind": "url", "url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…" } }, "productTemplateParameters": { "INSTANCE_NAME": "evening-news-02" } }, "history": [ { "event": "running", "timestamp": "2026-11-21T11:14:02.760Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-XNGUdfZgw4rkqh6p", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:51.850Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-XNGUdfZgw4rkqh6p" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "provisioned", "timestamp": "2026-11-21T11:13:58.815Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-XNGUdfZgw4rkqh6p", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:51.850Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-XNGUdfZgw4rkqh6p" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "created", "timestamp": "2026-11-21T11:13:58.773Z" } ], "dropped": false, "dormant": false, "nodeId": "cluster-XNGUdfZgw4rkqh6p", "nodeAddress": "127.0.0.1", "migration": { "declaredStrategy": "stopThenStart" }}…and the first production is where it was, on studio-1.
Drain it
Section titled “Drain it”nodes drain cordons the node (already done here), migrates every runtime on it to another worker in the pool, and terminates the node once it is empty. A runtime that cannot be moved keeps the node, cordoned, unless you pass --force. --wait polls until the drain ends and exits non-zero if it ended partial; --yes skips the confirmation. Here it is, start to finish:
norsk-mgr nodes drain cluster-TJsfyxPTxihs8AdO --yes --wait{ "nodeId": "cluster-TJsfyxPTxihs8AdO", "phase": "done", "startedAt": "2026-11-21T11:14:03.712Z", "timeoutMs": 600000, "force": false, "runtimes": [ { "runtimeRecordId": "evening-news-01.0.main", "status": "migrated", "migrationId": "17855ee7-1f09-4515-86fc-3f06cdea3b2d" } ]}While it runs
Section titled “While it runs”--wait holds the terminal until the drain ends. From a second terminal, nodes drain-status shows the same view it polls — the drain’s phase and each runtime’s progress — and runtime get shows the runtime mid-move: still bound to the draining node until its new instance is up on the other worker, with the migration recorded in its history.
norsk-mgr nodes drain-status cluster-TJsfyxPTxihs8AdO{ "nodeId": "cluster-TJsfyxPTxihs8AdO", "phase": "migrating", "startedAt": "2026-11-21T11:14:03.712Z", "timeoutMs": 600000, "force": false, "runtimes": [ { "runtimeRecordId": "evening-news-01.0.main", "status": "migrating", "migrationId": "17855ee7-1f09-4515-86fc-3f06cdea3b2d" } ]}norsk-mgr runtime get evening-news-01.0.main{ "job": { "runtimeRecordId": "evening-news-01.0.main", "state": "active", "currentHash": "96993733", "tags": { "auto-manager": "true", "productionId": "evening-news-01", "runtimeId": "main", "launchCredentialSha256": "edfd28ced23f3c072bc99123765d1770707458e6009747a6525e1dfdb01e5b47", "productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6", "replicaIndex": "0" }, "startDateTime": "2026-11-21T11:13:53.467Z", "cloud": "aws", "description": "", "writeFiles": [], "services": [], "volumes": [], "managerConfiguration": "", "norskMediaVersion": "recommended", "norskServiceParameters": { "portMappings": [], "environmentVariables": [], "cpuSet": [], "volumeMounts": [], "bindMounts": [], "ulimits": [] }, "shape": "", "availabilityDomain": "", "subnet": "", "architecture": "", "norskCpuList": [], "autoPlacement": { "$typeName": "norsk.api.manager.AutoPlacement", "requirements": { "$typeName": "norsk.api.manager.JobRequirements", "requiredCapacity": 1, "requiredCapabilities": [], "onRuntimeFailure": 1, "requiredHostPorts": [] } }, "productTemplate": { "productName": "e2e-fixture", "productVersion": "0.0.1", "sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a", "templateName": "evening-news-studio", "source": { "kind": "url", "url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…" } }, "productTemplateParameters": { "INSTANCE_NAME": "evening-news-01" } }, "history": [ { "event": "stopping", "timestamp": "2026-11-21T11:14:05.572Z", "role": "primary", "reason": "migration", "additionalReason": "17855ee7-1f09-4515-86fc-3f06cdea3b2d", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "hold": "cordoned", "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "prepared", "timestamp": "2026-11-21T11:14:05.568Z", "role": "primary@17855ee7", "nodeMetadata": { "nodeId": "cluster-XNGUdfZgw4rkqh6p", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:51.850Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-XNGUdfZgw4rkqh6p" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "provisioned", "timestamp": "2026-11-21T11:14:03.717Z", "role": "primary@17855ee7", "nodeMetadata": { "nodeId": "cluster-XNGUdfZgw4rkqh6p", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:51.850Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-XNGUdfZgw4rkqh6p" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "running", "timestamp": "2026-11-21T11:13:57.529Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "provisioned", "timestamp": "2026-11-21T11:13:53.609Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "created", "timestamp": "2026-11-21T11:13:53.561Z" } ], "dropped": false, "dormant": false, "nodeId": "cluster-TJsfyxPTxihs8AdO", "nodeAddress": "127.0.0.1", "migration": { "declaredStrategy": "stopThenStart" }}Afterwards
Section titled “Afterwards”The production is live throughout — rollupState is active again, and nothing is broken or awaitingCapacity. The runtime now lives on studio-2: nodeId has changed, and its history carries the move.
norsk-mgr production get evening-news-01{ "productionId": "evening-news-01", "pool": "local", "jobs": [ { "runtimeId": "main", "requirements": { "requiredCapacity": 1, "requiredCapabilities": [], "onRuntimeFailure": "avoidSpare" }, "launch": { "kind": "productTemplate", "ref": { "productName": "e2e-fixture", "productVersion": "0.0.1", "sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a", "templateName": "evening-news-studio", "source": { "kind": "url", "url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…" } }, "parameters": { "INSTANCE_NAME": "evening-news-01" }, "startDateTime": "2026-11-21T11:13:53.467Z", "tags": { "launchCredentialSha256": "edfd28ced23f3c072bc99123765d1770707458e6009747a6525e1dfdb01e5b47", "productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6" } }, "startMode": "auto" } ], "state": "live", "rollupState": "active", "broken": false, "resilienceDegraded": false, "awaitingCapacity": false, "onLaunch": "avoidSpare", "identity": { "productionTemplateId": "evening-news", "productionTemplateRevision": "1", "runtimeUrlBase": "/bundles/evening-news-01" }}norsk-mgr runtime get evening-news-01.0.main{ "job": { "runtimeRecordId": "evening-news-01.0.main", "state": "active", "currentHash": "96993733", "tags": { "auto-manager": "true", "productionId": "evening-news-01", "runtimeId": "main", "launchCredentialSha256": "edfd28ced23f3c072bc99123765d1770707458e6009747a6525e1dfdb01e5b47", "productTemplateRevisionId": "5563d81a-1323-4f2f-9127-820959119ba6", "replicaIndex": "0" }, "startDateTime": "2026-11-21T11:13:53.467Z", "cloud": "aws", "description": "", "writeFiles": [], "services": [], "volumes": [], "managerConfiguration": "", "norskMediaVersion": "recommended", "norskServiceParameters": { "portMappings": [], "environmentVariables": [], "cpuSet": [], "volumeMounts": [], "bindMounts": [], "ulimits": [] }, "shape": "", "availabilityDomain": "", "subnet": "", "architecture": "", "norskCpuList": [], "autoPlacement": { "$typeName": "norsk.api.manager.AutoPlacement", "requirements": { "$typeName": "norsk.api.manager.JobRequirements", "requiredCapacity": 1, "requiredCapabilities": [], "onRuntimeFailure": 1, "requiredHostPorts": [] } }, "productTemplate": { "productName": "e2e-fixture", "productVersion": "0.0.1", "sha256": "f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a", "templateName": "evening-news-studio", "source": { "kind": "url", "url": "http://127.0.0.1:9333/api/v1/product-templates/by-sha/f4d62abfffc6575fefa19d36c8a754f38b1e041b221ebd666992a909bfd71d9a/tar?token=…" } }, "productTemplateParameters": { "INSTANCE_NAME": "evening-news-01" } }, "history": [ { "event": "running", "timestamp": "2026-11-21T11:14:16.028Z", "role": "primary@17855ee7", "nodeMetadata": { "nodeId": "cluster-XNGUdfZgw4rkqh6p", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:51.850Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-XNGUdfZgw4rkqh6p" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "stopped", "timestamp": "2026-11-21T11:14:15.884Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "hold": "cordoned", "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "stopping", "timestamp": "2026-11-21T11:14:05.572Z", "role": "primary", "reason": "migration", "additionalReason": "17855ee7-1f09-4515-86fc-3f06cdea3b2d", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "hold": "cordoned", "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "prepared", "timestamp": "2026-11-21T11:14:05.568Z", "role": "primary@17855ee7", "nodeMetadata": { "nodeId": "cluster-XNGUdfZgw4rkqh6p", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:51.850Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-XNGUdfZgw4rkqh6p" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "provisioned", "timestamp": "2026-11-21T11:14:03.717Z", "role": "primary@17855ee7", "nodeMetadata": { "nodeId": "cluster-XNGUdfZgw4rkqh6p", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:51.850Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-XNGUdfZgw4rkqh6p" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "running", "timestamp": "2026-11-21T11:13:57.529Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "provisioned", "timestamp": "2026-11-21T11:13:53.609Z", "role": "primary", "nodeMetadata": { "nodeId": "cluster-TJsfyxPTxihs8AdO", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:13:48.099Z", "providerMetadata": { "provider": "cluster" } }, "runningNodeMetadata": { "publicDnsName": "", "privateDnsName": "127.0.0.1", "privateIpAddress": "127.0.0.1", "publicIpAddress": "127.0.0.1", "providerMetadata": { "provider": "cluster", "instanceId": "cluster-TJsfyxPTxihs8AdO" }, "proxyUrl": "https://127.0.0.1" } }, { "event": "created", "timestamp": "2026-11-21T11:13:53.561Z" } ], "dropped": false, "dormant": false, "nodeId": "cluster-XNGUdfZgw4rkqh6p", "nodeAddress": "127.0.0.1", "migration": { "declaredStrategy": "stopThenStart" }}What happens to the node
Section titled “What happens to the node”Once the node is empty the drain terminates it. For a cloud node that destroys the instance; a local server is your machine, so the Manager just lets go of it, closing the worker’s stream. Its registration is untouched — local-servers list still carries it — so it needs no re-registering.
The worker rejoins by itself
Section titled “The worker rejoins by itself”The worker on the drained machine sees its stream to the Manager end and re-registers moments later — usually before you could look — under the same node id, running again, with nothing on it. The cordon is durable and outlives the node: the worker is back, but still off-limits until you say otherwise.
norsk-mgr nodes get cluster-TJsfyxPTxihs8AdO{ "nodeId": "cluster-TJsfyxPTxihs8AdO", "state": "running", "lastActivityAt": "2026-11-21T11:14:18.192Z", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:14:18.144Z", "provider": { "provider": "cluster" }, "publicIpAddress": "127.0.0.1", "privateIpAddress": "127.0.0.1", "proxyUrl": "https://127.0.0.1", "providerInstanceId": "cluster-TJsfyxPTxihs8AdO", "spare": false, "unhealthy": false, "warmFor": [ "evening-news-studio" ], "partiallyWarmFor": [], "inventory": { "totalCapacity": 8, "utilizationCap": 1, "totalCores": 8, "gpus": [], "capabilities": [ { "name": "norsk.io/intel-qsv", "count": 1 }, { "name": "norsk.io/job-socket", "count": 1 }, { "name": "norsk.io/secret-references", "count": 1 }, { "name": "norsk.io/secret-references-runtime", "count": 1 } ], "reservedCapacity": 0, "reservedCores": 0, "reservedCapabilities": [], "reachable": true, "lastSeen": "2026-11-21T11:14:18.853Z", "cordoned": true, "assignedRuntimes": [], "accessReplica": { "version": 1795259625379, "digest": "17166b7ca89b975b2a5f4d559cf41b8561edc083d5ca2e1e9c89422b0050cd3f", "vouchedAt": "2026-11-21T11:13:50.380Z", "staleLimitMs": 28800000, "ageMs": 28517, "state": "current" }, "secretsProfile": { "name": "production", "matchesManager": true } }}Put it back into service
Section titled “Put it back into service”Maintenance done, nodes uncordon lifts the cordon and studio-1 takes placements again. Nothing moves back on its own — the runtime stays on studio-2 until something else moves it.
norsk-mgr nodes uncordon cluster-TJsfyxPTxihs8AdO{ "nodeId": "cluster-TJsfyxPTxihs8AdO", "action": "uncordoned"}norsk-mgr nodes get cluster-TJsfyxPTxihs8AdO{ "nodeId": "cluster-TJsfyxPTxihs8AdO", "state": "running", "lastActivityAt": "2026-11-21T11:14:19.067Z", "tags": { "pool": "local", "versionSkew": "none", "workerProtocolVersion": "2.3", "workerVersion": "0.1.0-2026-11-21-7114c45" }, "createdAt": "2026-11-21T11:14:18.144Z", "provider": { "provider": "cluster" }, "publicIpAddress": "127.0.0.1", "privateIpAddress": "127.0.0.1", "proxyUrl": "https://127.0.0.1", "providerInstanceId": "cluster-TJsfyxPTxihs8AdO", "spare": false, "unhealthy": false, "warmFor": [ "evening-news-studio" ], "partiallyWarmFor": [], "inventory": { "totalCapacity": 8, "utilizationCap": 1, "totalCores": 8, "gpus": [], "capabilities": [ { "name": "norsk.io/intel-qsv", "count": 1 }, { "name": "norsk.io/job-socket", "count": 1 }, { "name": "norsk.io/secret-references", "count": 1 }, { "name": "norsk.io/secret-references-runtime", "count": 1 } ], "reservedCapacity": 0, "reservedCores": 0, "reservedCapabilities": [], "reachable": true, "lastSeen": "2026-11-21T11:14:18.853Z", "cordoned": false, "assignedRuntimes": [], "accessReplica": { "version": 1795259625379, "digest": "17166b7ca89b975b2a5f4d559cf41b8561edc083d5ca2e1e9c89422b0050cd3f", "vouchedAt": "2026-11-21T11:13:50.380Z", "staleLimitMs": 28800000, "ageMs": 28867, "state": "current" }, "secretsProfile": { "name": "production", "matchesManager": true } }}Clean up
Section titled “Clean up”Only if you are done with them: stop the two productions, then remove the format and the template.
norsk-mgr production delete evening-news-01 --yes{ "productionId": "evening-news-01", "deleted": true}norsk-mgr production delete evening-news-02 --yes{ "productionId": "evening-news-02", "deleted": true}norsk-mgr production-template rm evening-news{ "status": "ok", "name": "evening-news"}norsk-mgr product-template rm evening-news-studio --yes{ "status": "ok", "name": "evening-news-studio"}Captured 2026-11-21 against norsk-mgr 0.1.0-2026-11-21-7114c45, Norsk Manager 1.4.2-2026-11-19-3758b1eb — every command and screenshot on this page is from that run.