Worker node contract
The precise specification of what a Norsk Manager worker node is: what must be pre-installed on the image, what arrives at boot, what the instance needs from the cloud account, and how the Manager interacts with it over its lifetime. This document is IaC-agnostic — the packer templates and terraform modules in this repo are one implementation of it; any tooling that satisfies it produces a valid worker.
1. Role
Section titled “1. Role”A worker node hosts Norsk media workloads under the Manager’s control. The
inversion to understand first: the worker is the gRPC server; the Manager
dials in (worker.proto, NodeControllerStream). A worker never initiates
a connection to the Manager in cloud mode — it boots, starts its gRPC server,
and waits.
The worker runtime is norsk-ctl in worker mode (a single self-contained
executable). Media workloads run as sibling docker containers that norsk-ctl
launches via the host docker socket. (The legacy WorkerDaemon container is
deprecated; all new deployments use norsk-ctl.)
2. Pre-installed (the image’s job)
Section titled “2. Pre-installed (the image’s job)”| Requirement | Why | Notes |
|---|---|---|
| docker (engine + CLI) | media workloads are containers | worker user must be in the docker group |
| systemd | service supervision | contract is OS-neutral: Ubuntu LTS default; Amazon Linux / Rocky supported |
norsk-ctl binary on PATH | the worker runtime | version pinned per image build |
norsk-ctl-worker.service systemd unit, installed but disabled | boot config enables + starts it | ExecStart=norsk-ctl worker --config-file <path> |
cloud CLI (aws) | registry login; Route53 + Secrets Manager for the wildcard-cert arm | needed even with pre-pulled images (stale-cache fallback) |
| Pre-pulled norsk image layers for the target release channel | the media image is several GB; boot-to-placeable speed is critical | images still run from the registry-tagged references; the AMI is a warm cache. Boot does `docker image inspect |
| GPU drivers + container toolkit (gpu image variant only) | NVENC/NVDEC workloads | not needed on the base variant |
SELinux tools (chcon/semanage) on enforcing bases | cloud-init relabels executed scripts | Amazon Linux / Rocky |
Nothing else. No norsk software runs from the AMI itself — the AMI carries a binary, a unit file, and a warm image cache.
3. Delivered at boot (cloud-init, generated by the Manager)
Section titled “3. Delivered at boot (cloud-init, generated by the Manager)”The Manager generates a per-node #cloud-config at provision time and passes
it as instance user-data: for norsk-ctl workers gzipped, then base64 text,
which is what EC2 stores and counts against its 16 KB limit (cloud-init
decodes and unpacks it). With side-loaded launch data (§3.1) the same
items arrive through a per-worker Secrets Manager entry instead, and the user
data is only a loader. It delivers:
-
The worker config file — a JSON
DaemonConfig, written to disk (base64-embedded in the cloud-init). Contains:nodeId, node tags,workerServicePort(deployment default 8110),healthPingPeriod, an Erlang-stylecookie(random 32 bytes), node metadata. This is how the node learns its identity — the Manager also stampsnodeIdinto instance tags and uses it as therunInstancesclient token (idempotency). -
Registry login — when the deployment’s registry requires auth (e.g. ECR:
aws ecr get-login-password | docker login). Skipped for anonymous registries (Docker Hub default). Credentials are written underDOCKER_CONFIG=/etc/norsk/docker, which the worker unit’s environment file also sets, so thedocker/docker composecalls norsk-ctl shells out to inherit them. The cloud-init itself logs in once; norsk-ctl is not registry-aware and does not re-authenticate, and an ECR token lasts 12 hours. The refresh is the image’s: the worker AMI carriesnorsk-ecr-login.timer(packer/files/norsk-ecr-login), which every 6 h re-logs in to every ECR registry found in the host’s docker configs —/etc/norsk/dockerincluded — and re-tags the mirrored proxy images under their upstream names, so a node that evicts an image (--image-store-max) can pull it again after the boot token has expired. An image built without the timer keeps only its boot-time login. -
Optional support login — only when the deployment sets
ec2ProviderConfig.supportUser(terraformworker_support_user): a cloud-initusers:entry namedsupportUser.namewithsupportUser.sshAuthorizedKeyandsudo: ALL=(ALL) NOPASSWD:ALL. Absent (the default) = nousers:block; nothing in the cloud-init lets anyone log in (§6). (TheworkerConfigurationS3 sync into<wd>/container-configis WorkerDaemon-only — norsk-ctl reads nothing from it, so the NorskCtl cloud-init no longer runs it.) -
Optional extra files — caller-supplied
writeFilesfrom the job’s node requirements (inline or URL-fetched). -
Optional storage sidecar key — only when the deployment sets
ec2ProviderConfig.workerStorage(terraformworker_storage, withworker_storage_key_secret_arnandworker_storage_bucket_region), and only in the NorskCtl flavour. The media engines reads3://file inputs through norsk-ctl’s storage sidecar, which authenticates with its own read-only access key, never the instance role: containers cannot reach IMDS (§4). The cloud-init writes:/usr/local/bin/norsk-worker-storage-key-fetch(0755, run as root): reads the key’s secret from Secrets Manager as the worker role — a JSON object{"accessKeyId": ..., "secretAccessKey": ...}, optionally with"sessionToken"— and writes it as norsk-ctl’s three-line key file (access key id, secret, optional session token). It swaps the file in by rename only when the content changed, and leaves the old key in place when the secret is unreadable or lacks either field;norsk-worker-storage-key.service(oneshot, runs the script) andnorsk-worker-storage-key.timer(OnActiveSec=1min,OnUnitActiveSec=5min,RandomizedDelaySec=30s), enabled by the cloud-init;- the key itself: directory
/var/lib/norsk-ctl/storage-key/(0700, owned by the worker user) holding onlykey(0600, owned by the worker user; a.key.tmpexists only while it is being replaced). norsk-ctl refuses any other layout. The directory is bind-mounted read-only into the storage sidecar and no other container.
The script runs once before the worker starts (a failure is logged and the timer retries), then the timer keeps the key current: a new secret value reaches the sidecar within about 6 minutes (up to five for the timer, then up to a minute for the sidecar to use the new key), with no restart. The worker role needs
secretsmanager:GetSecretValueon that one secret (modules/iamworker_storage). The worker config file (item 1) then carries astoragefield — norsk-ctl’sstorage:block,{"provider": "AWS", "region": <bucket region>, "credentials": {"kind": "key-file", "path": "/var/lib/norsk-ctl/storage-key/key"}}— whichnorsk-ctl workeruses to start the sidecar (it wins over aconfig.yamlstorage:block on the node; until the key lands it retries every minute). One bucket region per deployment: the sidecar has one S3 remote. AbsentworkerStorage= no script, no timer, nostoragefield, and no sidecar, as before. Only workers launched after the setting is applied get it. -
Service start — enable + start
norsk-ctl-worker.service.
To read a worker’s user data:
aws ec2 describe-instance-attribute --instance-id <id> --attribute userData \ --query UserData.Value --output text | base64 -d | base64 -d | gunzipOn the instance (IMDSv2 is required):
T=$(curl -sX PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')curl -sH "X-aws-ec2-metadata-token: $T" http://169.254.169.254/latest/user-data | base64 -d | gunzipDrop the final gunzip for WorkerDaemon workers and for norsk-ctl workers
launched before the Manager began compressing user data (after the base64
decodes, compressed data starts with the gzip bytes 1f 8b).
3.1 Side-loaded launch data (workerLaunchSecret)
Section titled “3.1 Side-loaded launch data (workerLaunchSecret)”When the Manager’s provider config sets workerLaunchSecret (terraform:
manager-host worker_launch_secret_name_prefix, from the iam module’s
worker_launch_secret_name_prefix output, with iam worker_launch_secret,
on by default in this kit), a norsk-ctl worker’s start-up data does not go in
its user data. Everything above (the config file with its enrollment cookie,
worker.env, every script and unit, the extra files, and the commands, in
the same order) goes in a Secrets Manager entry of its own instead, and the
user data is a short loader. WorkerDaemon workers are never side-loaded.
- The entry is named
<prefix><sha256 of the node id, 64 lowercase hex>(the kit’s prefix is<name>-worker-launch/), in the worker’s region, encrypted with the account’s defaultaws/secretsmanagerkey. Its value is JSON:{"version": 1, "files": [{"path", "permissions", "content"}], "runcmd": [...]}, each file’s content base64 of its exact bytes,permissionsits octal mode (absent or null: 0644). It is taggedid3as.Norsk.LaunchSecret=workerandid3as.Norsk.Manager=<fingerprint>at creation, andid3as.Norsk.InstanceArn=<the instance's ARN>once the instance exists. The instance carriesid3as.Norsk.LaunchSecret=<entry name>. - Who can read it: the worker role may read an entry only when its
id3as.Norsk.InstanceArntag is the requesting instance (secretsmanager:ResourceTag/id3as.Norsk.InstanceArn=${ec2:SourceInstanceARN}), so a worker can read its own entry and no other. The limit is the same as for user data today: any root process on the worker, including a product container running as uid 0 with host networking (the caveat inpacker/scripts/25-imds-guard.sh), can use the instance role, and so can read that worker’s own entry. - The loader (
/usr/local/sbin/norsk-worker-launch, 0700, run once fromruncmd) fetches the entry with the instance role, retrying every 5 s for up to 15 minutes (it is refused until the Manager has labelled the entry, and IAM is eventually consistent); then writes each file (0600 first, then its mode, renamed into place; directories with cloud-init’s umask, 022) and runs the commands in order withsh, as cloud-init’swrite_filesandruncmdwould have. It needsbash,aws(CLI v2),jqandbase64, which the kit’s worker images have. If the entry never becomes readable, or holds something it cannot use, it writes onenorsk-worker-launch:line to/var/log/cloud-init-output.log, exits 1 having run nothing, and the node expires unregistered like any node that never registers. The payload never goes on a command line or into a log. - Lifecycle: the Manager creates the entry before
RunInstances(no entry, no launch: that create fails and is retried), deletes it if the launch fails, and deletes it (with no recovery window) once the instance is terminated. An hourly sweep deletes this Manager’s entries that no instance names any more (after a 30-minute grace) and labels any live one the launch could not. A stopped worker keeps its entry until it is terminated. - The user data then holds the hostname, the optional support login (it must be created by cloud-init itself) and the loader: no cookie, no config, a few kilobytes whatever the deployment turns on, so EC2’s 16 KB limit no longer constrains worker options.
To read a side-loaded worker’s launch data (with the Manager role’s rights):
aws secretsmanager get-secret-value --secret-id <prefix><sha256 of node id> \ --query SecretString --output text | jqThe user-data commands above still apply: to the loader of a side-loaded worker, and to every worker launched before or without the setting.
norsk-ctl worker (cloud-init mode, its default) then: reads the config file
— the Manager writes it to
<workerWorkingDirectory>/worker.<nodeId>.config (in this kit
/var/lib/norsk-ctl/worker-mode/worker.<nodeId>.config) and points the unit
at it through /etc/norsk/worker.env; the binary’s own --config-file
default of /mnt/worker.config is never used on a cloud node — starts its
gRPC server on workerServicePort, and waits for the Manager to dial in. On connect it sends
WorkerHello (including any persisted running jobs, for reconciliation).
No license is baked into the image or delivered by IaC, and the hello
carries none: each job brings its product’s license, which the worker
writes beside the instance and deletes when the job stops.
/etc/norsk/worker.env also carries the settings newer than some worker
AMIs, as variables rather than flags: NORSK_WORKER_LOG_FORWARD
(host:port of the node’s log agent, written by the same cloud-init when the
deployment runs one — every media engine the worker launches forwards its
log records there) and NORSK_WORKER_ORPHAN_TERMINATE_AFTER (seconds
without word from the Manager before the worker terminates its own instance;
the worker role needs ec2:TerminateInstances on itself, which
modules/iam grants). norsk-ctl worker refuses flags it does not know, so
a flag would stop an older worker from starting at all; an older worker
simply ignores the variables and runs without the feature.
Which of the two cloud-init flavours the Manager generates is selected by
ec2ProviderConfig.workerRuntime: NorskCtl (what this kit renders —
enables the image’s pre-installed norsk-ctl-worker.service) or absent (the
legacy WorkerDaemon container path, deprecated).
Worker DNS records are never cleaned up. Under worker_cert_mode = wildcard-route53 each worker’s cloud-init UPSERTs w-<ip-with-dashes>.<zone>
into the delegated workers zone; the Manager’s terminate path has no Route53
step and this kit has no sweeper, so the zone holds one A record per node
ever launched. A recycled private IP re-resolves an old name — harmless,
because the name encodes the IP — and the kit sets force_destroy on the
zone so terraform destroy can remove it with the records in it. Cleanup on
terminate is a Manager-side change.
4. Instance requirements (the cloud account’s job)
Section titled “4. Instance requirements (the cloud account’s job)”What the Manager’s runInstances call assumes exists — i.e. the terraform
surface:
| Requirement | Detail |
|---|---|
| AMI | per region and per architecture (x86_64 / aarch64), referenced from the Manager’s imageVersionTable by release channel (latest / recommended / previousRecommended / lts) |
| Subnet(s) | one or more per region; the Manager round-robins across them. The primary interface gets a public address unless the region’s worker config says associatePublicIp: false (§4.1) |
| Worker security group | must admit the Manager host on workerServicePort (8110). Media ingest/egress ports per workflow are deployment-specific — size the SG for the products you run. Outbound, the worker reaches the manager host on 9333 (NORSK_MGR_PUBLIC_URL) for its product-template tar — GET /api/v1/product-templates/by-sha/<sha256>/tar?token=<t>, the per-launch token frozen into the job’s ProductTemplateRef.url; the worker fetches the URL as given and verifies the bytes against sha256 — and for workdir assets (/api/v1/workdir-assets/<sha256>). A job frozen before tokens existed (a token-less or by-name URL) is refused there: relaunch the production |
| Worker IAM role + instance profile | registry pull (if ECR), SSM core; with side-loaded launch data (§3.1), secretsmanager:GetSecretValue on <name>-worker-launch/* only where the entry is tagged with the requesting instance; S3 read only on the object ARNs in worker_asset_s3_arns (the examples: the deployment bucket’s assets/*), for production working-directory assets named by s3:// URL, which the worker fetches with its instance role. The manager host’s role needs iam:PassRole on this role |
| IMDSv2 | the Manager requests httpTokens: required and instance-metadata-tags. Hop limit 1 for the norsk-ctl worker, which runs on the host and never calls IMDS, so a product container on a bridge network cannot reach IMDS (and the enrollment cookie, which is in the user data, or in the worker’s launch entry when side-loaded, §3.1); hop limit 2 only for the legacy WorkerDaemon, which reads IMDS from inside its container. A host-networked product can still reach IMDS; a per-uid firewall rule for that is planned. |
| Root volume | the Manager requests 100 GB. With worker storage on (§3 item 5), the storage sidecar caches up to 20 GB of read objects under the worker’s state directory on the root volume (/home/norsk/.norsk-ctl/storage/ in this kit), so size the pool’s root volume to allow for it |
| Tags | the Manager stamps id3as.Norsk.NodeType=worker + Name=Norsk Worker + id3as.Norsk.Manager=<fingerprint> (the launching Manager’s identity; it discovers only instances carrying its own — a second Manager in the account leaves them alone, and a worker nobody tagged is logged and left alone until an operator tags it) + job tags. Do not reuse this tag key on non-worker instances — discovery polls by it |
Subnet / security-group / role names reach the Manager via the
WORKER_CONFIG_FILE JSON (see the manager host contract § 4) — they are
deployment config, not baked into images.
4.1 Workers without a public address
Section titled “4.1 Workers without a public address”The region’s worker config may say associatePublicIp: false (absent means
true). The Manager then launches workers with no public address and dials
them on their private one, so the subnets must be private ones with their
own way out: a NAT gateway, or whatever the VPC routes to (a transit
gateway, a firewall). In the kit this is worker_network = private-nat or
existing in examples/clean-account.
What such a worker can still do: everything outbound — push outputs to a destination, publish to S3 (the kit adds an S3 gateway endpoint to the private route tables, so that traffic costs nothing at the NAT), pull images, ship logs, call the AWS APIs, fetch its product template and workdir assets from the manager host (which it reaches by private address in every mode). What it cannot do, because nothing outside the VPC can reach it:
- take media in from the internet — no SRT, RTMP or WHIP ingest from a public source. Sources have to arrive over a VPN, Direct Connect, a transit gateway or from inside the VPC;
- serve a product UI to a browser outside the VPC. Operators reach a
private worker’s previews over the same private routing, and the
deployment’s
advertisemust beprivate(the worker’s private IP in the URLs) ornone(no URL — the UI says previews are unavailable);publicis refused, since there is no address to advertise.
The worker security group’s direct-access rules (the UI proxy port, MoQ)
should admit the ranges those operators arrive from — the VPC’s CIDR by
default in the kit — rather than public operator addresses, which cannot
route to a private subnet. Everything else in this contract is unchanged:
the Manager still dials workerServicePort (over the VPC), the worker still
fetches from port 9333 on the manager host’s private address, the wildcard
certificate arm still works (the worker registers its private address, and
needs egress to the Route53 and Secrets Manager APIs to do so). A load
balancer in front of private workers for public ingress is not part of this
contract.
5. Lifecycle (what the Manager does to the node)
Section titled “5. Lifecycle (what the Manager does to the node)”- Discovery: per-region poll of
describeInstancesfiltered onid3as.Norsk.NodeType=worker; the Manager recovers the node’s config if it restarts and re-learns the fleet: from the worker’s launch entry when the instance carriesid3as.Norsk.LaunchSecret(§3.1), else from its user data. - Health: the Manager health-pings over the gRPC stream. Missed pings beyond the configured count → the node is stopped. A bare TCP drop is not death — only ping timeout is.
- Reaping: nodes stopped longer than a threshold are terminated; elastic nodes empty longer than the linger window are terminated. The fleet is fully Manager-owned — external tooling must never “clean up” running workers (that’s a failure the Manager will respond to by re-placing workloads).
- Cordon holds a node: a cordoned node is not reaped. Nothing new is placed on it, and neither the Manager’s reapers nor its hot-spare housekeeping terminate it, so you can ssh in (§3.3) and look around a node that has gone bad. If a held node stops answering health pings, its workloads are still re-placed elsewhere, but the instance is left running until you uncordon it. Only an explicit terminate, evict or drain takes a cordoned node down. Uncordon it when you are done: that returns it to the fleet, and the Manager then reaps it as usual.
- Spot: the Manager launches spot instances when the placement tier says
so (
marketType: spot); on-demand otherwise. The node’s one market duty is to watch the instance metadata service for a spot interruption or rebalance notice (worker-mode/interruption-watch.ts) and report it to the Manager, which drains the node.
6. Out of scope for the node
Section titled “6. Out of scope for the node”- No inbound access needed from anywhere except the Manager (+ media ports).
- No cluster-mode registration (that path —
--manager-urlwith a join token from--token-fileor--token— is for operator-owned bring-your-own machines, not cloud workers). - No credentials on disk beyond the instance profile — unless the deployment
opts into a
supportUser(§3.3), which is exactly one ssh key with sudo, chosen by the operator.