Skip to content

Worker node contract

The precise specification of what a Norsk Manager worker node is: what must be pre-installed on the image, what arrives at boot, what the instance needs from the cloud account, and how the Manager interacts with it over its lifetime. This document is IaC-agnostic — the packer templates and terraform modules in this repo are one implementation of it; any tooling that satisfies it produces a valid worker.

A worker node hosts Norsk media workloads under the Manager’s control. The inversion to understand first: the worker is the gRPC server; the Manager dials in (worker.proto, NodeControllerStream). A worker never initiates a connection to the Manager in cloud mode — it boots, starts its gRPC server, and waits.

The worker runtime is norsk-ctl in worker mode (a single self-contained executable). Media workloads run as sibling docker containers that norsk-ctl launches via the host docker socket. (The legacy WorkerDaemon container is deprecated; all new deployments use norsk-ctl.)

RequirementWhyNotes
docker (engine + CLI)media workloads are containersworker user must be in the docker group
systemdservice supervisioncontract is OS-neutral: Ubuntu LTS default; Amazon Linux / Rocky supported
norsk-ctl binary on PATHthe worker runtimeversion pinned per image build
norsk-ctl-worker.service systemd unit, installed but disabledboot config enables + starts itExecStart=norsk-ctl worker --config-file <path>
cloud CLI (aws)registry login; Route53 + Secrets Manager for the wildcard-cert armneeded even with pre-pulled images (stale-cache fallback)
Pre-pulled norsk image layers for the target release channelthe media image is several GB; boot-to-placeable speed is criticalimages still run from the registry-tagged references; the AMI is a warm cache. Boot does `docker image inspect
GPU drivers + container toolkit (gpu image variant only)NVENC/NVDEC workloadsnot needed on the base variant
SELinux tools (chcon/semanage) on enforcing basescloud-init relabels executed scriptsAmazon Linux / Rocky

Nothing else. No norsk software runs from the AMI itself — the AMI carries a binary, a unit file, and a warm image cache.

3. Delivered at boot (cloud-init, generated by the Manager)

Section titled “3. Delivered at boot (cloud-init, generated by the Manager)”

The Manager generates a per-node #cloud-config at provision time and passes it as instance user-data: for norsk-ctl workers gzipped, then base64 text, which is what EC2 stores and counts against its 16 KB limit (cloud-init decodes and unpacks it). With side-loaded launch data (§3.1) the same items arrive through a per-worker Secrets Manager entry instead, and the user data is only a loader. It delivers:

  1. The worker config file — a JSON DaemonConfig, written to disk (base64-embedded in the cloud-init). Contains: nodeId, node tags, workerServicePort (deployment default 8110), healthPingPeriod, an Erlang-style cookie (random 32 bytes), node metadata. This is how the node learns its identity — the Manager also stamps nodeId into instance tags and uses it as the runInstances client token (idempotency).

  2. Registry login — when the deployment’s registry requires auth (e.g. ECR: aws ecr get-login-password | docker login). Skipped for anonymous registries (Docker Hub default). Credentials are written under DOCKER_CONFIG=/etc/norsk/docker, which the worker unit’s environment file also sets, so the docker/docker compose calls norsk-ctl shells out to inherit them. The cloud-init itself logs in once; norsk-ctl is not registry-aware and does not re-authenticate, and an ECR token lasts 12 hours. The refresh is the image’s: the worker AMI carries norsk-ecr-login.timer (packer/files/norsk-ecr-login), which every 6 h re-logs in to every ECR registry found in the host’s docker configs — /etc/norsk/docker included — and re-tags the mirrored proxy images under their upstream names, so a node that evicts an image (--image-store-max) can pull it again after the boot token has expired. An image built without the timer keeps only its boot-time login.

  3. Optional support login — only when the deployment sets ec2ProviderConfig.supportUser (terraform worker_support_user): a cloud-init users: entry named supportUser.name with supportUser.sshAuthorizedKey and sudo: ALL=(ALL) NOPASSWD:ALL. Absent (the default) = no users: block; nothing in the cloud-init lets anyone log in (§6). (The workerConfiguration S3 sync into <wd>/container-config is WorkerDaemon-only — norsk-ctl reads nothing from it, so the NorskCtl cloud-init no longer runs it.)

  4. Optional extra files — caller-supplied writeFiles from the job’s node requirements (inline or URL-fetched).

  5. Optional storage sidecar key — only when the deployment sets ec2ProviderConfig.workerStorage (terraform worker_storage, with worker_storage_key_secret_arn and worker_storage_bucket_region), and only in the NorskCtl flavour. The media engines read s3:// file inputs through norsk-ctl’s storage sidecar, which authenticates with its own read-only access key, never the instance role: containers cannot reach IMDS (§4). The cloud-init writes:

    • /usr/local/bin/norsk-worker-storage-key-fetch (0755, run as root): reads the key’s secret from Secrets Manager as the worker role — a JSON object {"accessKeyId": ..., "secretAccessKey": ...}, optionally with "sessionToken" — and writes it as norsk-ctl’s three-line key file (access key id, secret, optional session token). It swaps the file in by rename only when the content changed, and leaves the old key in place when the secret is unreadable or lacks either field;
    • norsk-worker-storage-key.service (oneshot, runs the script) and norsk-worker-storage-key.timer (OnActiveSec=1min, OnUnitActiveSec=5min, RandomizedDelaySec=30s), enabled by the cloud-init;
    • the key itself: directory /var/lib/norsk-ctl/storage-key/ (0700, owned by the worker user) holding only key (0600, owned by the worker user; a .key.tmp exists only while it is being replaced). norsk-ctl refuses any other layout. The directory is bind-mounted read-only into the storage sidecar and no other container.

    The script runs once before the worker starts (a failure is logged and the timer retries), then the timer keeps the key current: a new secret value reaches the sidecar within about 6 minutes (up to five for the timer, then up to a minute for the sidecar to use the new key), with no restart. The worker role needs secretsmanager:GetSecretValue on that one secret (modules/iam worker_storage). The worker config file (item 1) then carries a storage field — norsk-ctl’s storage: block, {"provider": "AWS", "region": <bucket region>, "credentials": {"kind": "key-file", "path": "/var/lib/norsk-ctl/storage-key/key"}} — which norsk-ctl worker uses to start the sidecar (it wins over a config.yaml storage: block on the node; until the key lands it retries every minute). One bucket region per deployment: the sidecar has one S3 remote. Absent workerStorage = no script, no timer, no storage field, and no sidecar, as before. Only workers launched after the setting is applied get it.

  6. Service start — enable + start norsk-ctl-worker.service.

To read a worker’s user data:

Terminal window
aws ec2 describe-instance-attribute --instance-id <id> --attribute userData \
--query UserData.Value --output text | base64 -d | base64 -d | gunzip

On the instance (IMDSv2 is required):

Terminal window
T=$(curl -sX PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
curl -sH "X-aws-ec2-metadata-token: $T" http://169.254.169.254/latest/user-data | base64 -d | gunzip

Drop the final gunzip for WorkerDaemon workers and for norsk-ctl workers launched before the Manager began compressing user data (after the base64 decodes, compressed data starts with the gzip bytes 1f 8b).

3.1 Side-loaded launch data (workerLaunchSecret)

Section titled “3.1 Side-loaded launch data (workerLaunchSecret)”

When the Manager’s provider config sets workerLaunchSecret (terraform: manager-host worker_launch_secret_name_prefix, from the iam module’s worker_launch_secret_name_prefix output, with iam worker_launch_secret, on by default in this kit), a norsk-ctl worker’s start-up data does not go in its user data. Everything above (the config file with its enrollment cookie, worker.env, every script and unit, the extra files, and the commands, in the same order) goes in a Secrets Manager entry of its own instead, and the user data is a short loader. WorkerDaemon workers are never side-loaded.

  • The entry is named <prefix><sha256 of the node id, 64 lowercase hex> (the kit’s prefix is <name>-worker-launch/), in the worker’s region, encrypted with the account’s default aws/secretsmanager key. Its value is JSON: {"version": 1, "files": [{"path", "permissions", "content"}], "runcmd": [...]}, each file’s content base64 of its exact bytes, permissions its octal mode (absent or null: 0644). It is tagged id3as.Norsk.LaunchSecret=worker and id3as.Norsk.Manager=<fingerprint> at creation, and id3as.Norsk.InstanceArn=<the instance's ARN> once the instance exists. The instance carries id3as.Norsk.LaunchSecret=<entry name>.
  • Who can read it: the worker role may read an entry only when its id3as.Norsk.InstanceArn tag is the requesting instance (secretsmanager:ResourceTag/id3as.Norsk.InstanceArn = ${ec2:SourceInstanceARN}), so a worker can read its own entry and no other. The limit is the same as for user data today: any root process on the worker, including a product container running as uid 0 with host networking (the caveat in packer/scripts/25-imds-guard.sh), can use the instance role, and so can read that worker’s own entry.
  • The loader (/usr/local/sbin/norsk-worker-launch, 0700, run once from runcmd) fetches the entry with the instance role, retrying every 5 s for up to 15 minutes (it is refused until the Manager has labelled the entry, and IAM is eventually consistent); then writes each file (0600 first, then its mode, renamed into place; directories with cloud-init’s umask, 022) and runs the commands in order with sh, as cloud-init’s write_files and runcmd would have. It needs bash, aws (CLI v2), jq and base64, which the kit’s worker images have. If the entry never becomes readable, or holds something it cannot use, it writes one norsk-worker-launch: line to /var/log/cloud-init-output.log, exits 1 having run nothing, and the node expires unregistered like any node that never registers. The payload never goes on a command line or into a log.
  • Lifecycle: the Manager creates the entry before RunInstances (no entry, no launch: that create fails and is retried), deletes it if the launch fails, and deletes it (with no recovery window) once the instance is terminated. An hourly sweep deletes this Manager’s entries that no instance names any more (after a 30-minute grace) and labels any live one the launch could not. A stopped worker keeps its entry until it is terminated.
  • The user data then holds the hostname, the optional support login (it must be created by cloud-init itself) and the loader: no cookie, no config, a few kilobytes whatever the deployment turns on, so EC2’s 16 KB limit no longer constrains worker options.

To read a side-loaded worker’s launch data (with the Manager role’s rights):

Terminal window
aws secretsmanager get-secret-value --secret-id <prefix><sha256 of node id> \
--query SecretString --output text | jq

The user-data commands above still apply: to the loader of a side-loaded worker, and to every worker launched before or without the setting.

norsk-ctl worker (cloud-init mode, its default) then: reads the config file — the Manager writes it to <workerWorkingDirectory>/worker.<nodeId>.config (in this kit /var/lib/norsk-ctl/worker-mode/worker.<nodeId>.config) and points the unit at it through /etc/norsk/worker.env; the binary’s own --config-file default of /mnt/worker.config is never used on a cloud node — starts its gRPC server on workerServicePort, and waits for the Manager to dial in. On connect it sends WorkerHello (including any persisted running jobs, for reconciliation). No license is baked into the image or delivered by IaC, and the hello carries none: each job brings its product’s license, which the worker writes beside the instance and deletes when the job stops.

/etc/norsk/worker.env also carries the settings newer than some worker AMIs, as variables rather than flags: NORSK_WORKER_LOG_FORWARD (host:port of the node’s log agent, written by the same cloud-init when the deployment runs one — every media engine the worker launches forwards its log records there) and NORSK_WORKER_ORPHAN_TERMINATE_AFTER (seconds without word from the Manager before the worker terminates its own instance; the worker role needs ec2:TerminateInstances on itself, which modules/iam grants). norsk-ctl worker refuses flags it does not know, so a flag would stop an older worker from starting at all; an older worker simply ignores the variables and runs without the feature.

Which of the two cloud-init flavours the Manager generates is selected by ec2ProviderConfig.workerRuntime: NorskCtl (what this kit renders — enables the image’s pre-installed norsk-ctl-worker.service) or absent (the legacy WorkerDaemon container path, deprecated).

Worker DNS records are never cleaned up. Under worker_cert_mode = wildcard-route53 each worker’s cloud-init UPSERTs w-<ip-with-dashes>.<zone> into the delegated workers zone; the Manager’s terminate path has no Route53 step and this kit has no sweeper, so the zone holds one A record per node ever launched. A recycled private IP re-resolves an old name — harmless, because the name encodes the IP — and the kit sets force_destroy on the zone so terraform destroy can remove it with the records in it. Cleanup on terminate is a Manager-side change.

4. Instance requirements (the cloud account’s job)

Section titled “4. Instance requirements (the cloud account’s job)”

What the Manager’s runInstances call assumes exists — i.e. the terraform surface:

RequirementDetail
AMIper region and per architecture (x86_64 / aarch64), referenced from the Manager’s imageVersionTable by release channel (latest / recommended / previousRecommended / lts)
Subnet(s)one or more per region; the Manager round-robins across them. The primary interface gets a public address unless the region’s worker config says associatePublicIp: false (§4.1)
Worker security groupmust admit the Manager host on workerServicePort (8110). Media ingest/egress ports per workflow are deployment-specific — size the SG for the products you run. Outbound, the worker reaches the manager host on 9333 (NORSK_MGR_PUBLIC_URL) for its product-template tar — GET /api/v1/product-templates/by-sha/<sha256>/tar?token=<t>, the per-launch token frozen into the job’s ProductTemplateRef.url; the worker fetches the URL as given and verifies the bytes against sha256 — and for workdir assets (/api/v1/workdir-assets/<sha256>). A job frozen before tokens existed (a token-less or by-name URL) is refused there: relaunch the production
Worker IAM role + instance profileregistry pull (if ECR), SSM core; with side-loaded launch data (§3.1), secretsmanager:GetSecretValue on <name>-worker-launch/* only where the entry is tagged with the requesting instance; S3 read only on the object ARNs in worker_asset_s3_arns (the examples: the deployment bucket’s assets/*), for production working-directory assets named by s3:// URL, which the worker fetches with its instance role. The manager host’s role needs iam:PassRole on this role
IMDSv2the Manager requests httpTokens: required and instance-metadata-tags. Hop limit 1 for the norsk-ctl worker, which runs on the host and never calls IMDS, so a product container on a bridge network cannot reach IMDS (and the enrollment cookie, which is in the user data, or in the worker’s launch entry when side-loaded, §3.1); hop limit 2 only for the legacy WorkerDaemon, which reads IMDS from inside its container. A host-networked product can still reach IMDS; a per-uid firewall rule for that is planned.
Root volumethe Manager requests 100 GB. With worker storage on (§3 item 5), the storage sidecar caches up to 20 GB of read objects under the worker’s state directory on the root volume (/home/norsk/.norsk-ctl/storage/ in this kit), so size the pool’s root volume to allow for it
Tagsthe Manager stamps id3as.Norsk.NodeType=worker + Name=Norsk Worker + id3as.Norsk.Manager=<fingerprint> (the launching Manager’s identity; it discovers only instances carrying its own — a second Manager in the account leaves them alone, and a worker nobody tagged is logged and left alone until an operator tags it) + job tags. Do not reuse this tag key on non-worker instances — discovery polls by it

Subnet / security-group / role names reach the Manager via the WORKER_CONFIG_FILE JSON (see the manager host contract § 4) — they are deployment config, not baked into images.

The region’s worker config may say associatePublicIp: false (absent means true). The Manager then launches workers with no public address and dials them on their private one, so the subnets must be private ones with their own way out: a NAT gateway, or whatever the VPC routes to (a transit gateway, a firewall). In the kit this is worker_network = private-nat or existing in examples/clean-account.

What such a worker can still do: everything outbound — push outputs to a destination, publish to S3 (the kit adds an S3 gateway endpoint to the private route tables, so that traffic costs nothing at the NAT), pull images, ship logs, call the AWS APIs, fetch its product template and workdir assets from the manager host (which it reaches by private address in every mode). What it cannot do, because nothing outside the VPC can reach it:

  • take media in from the internet — no SRT, RTMP or WHIP ingest from a public source. Sources have to arrive over a VPN, Direct Connect, a transit gateway or from inside the VPC;
  • serve a product UI to a browser outside the VPC. Operators reach a private worker’s previews over the same private routing, and the deployment’s advertise must be private (the worker’s private IP in the URLs) or none (no URL — the UI says previews are unavailable); public is refused, since there is no address to advertise.

The worker security group’s direct-access rules (the UI proxy port, MoQ) should admit the ranges those operators arrive from — the VPC’s CIDR by default in the kit — rather than public operator addresses, which cannot route to a private subnet. Everything else in this contract is unchanged: the Manager still dials workerServicePort (over the VPC), the worker still fetches from port 9333 on the manager host’s private address, the wildcard certificate arm still works (the worker registers its private address, and needs egress to the Route53 and Secrets Manager APIs to do so). A load balancer in front of private workers for public ingress is not part of this contract.

5. Lifecycle (what the Manager does to the node)

Section titled “5. Lifecycle (what the Manager does to the node)”
  • Discovery: per-region poll of describeInstances filtered on id3as.Norsk.NodeType=worker; the Manager recovers the node’s config if it restarts and re-learns the fleet: from the worker’s launch entry when the instance carries id3as.Norsk.LaunchSecret (§3.1), else from its user data.
  • Health: the Manager health-pings over the gRPC stream. Missed pings beyond the configured count → the node is stopped. A bare TCP drop is not death — only ping timeout is.
  • Reaping: nodes stopped longer than a threshold are terminated; elastic nodes empty longer than the linger window are terminated. The fleet is fully Manager-owned — external tooling must never “clean up” running workers (that’s a failure the Manager will respond to by re-placing workloads).
  • Cordon holds a node: a cordoned node is not reaped. Nothing new is placed on it, and neither the Manager’s reapers nor its hot-spare housekeeping terminate it, so you can ssh in (§3.3) and look around a node that has gone bad. If a held node stops answering health pings, its workloads are still re-placed elsewhere, but the instance is left running until you uncordon it. Only an explicit terminate, evict or drain takes a cordoned node down. Uncordon it when you are done: that returns it to the fleet, and the Manager then reaps it as usual.
  • Spot: the Manager launches spot instances when the placement tier says so (marketType: spot); on-demand otherwise. The node’s one market duty is to watch the instance metadata service for a spot interruption or rebalance notice (worker-mode/interruption-watch.ts) and report it to the Manager, which drains the node.
  • No inbound access needed from anywhere except the Manager (+ media ports).
  • No cluster-mode registration (that path — --manager-url with a join token from --token-file or --token — is for operator-owned bring-your-own machines, not cloud workers).
  • No credentials on disk beyond the instance profile — unless the deployment opts into a supportUser (§3.3), which is exactly one ssh key with sudo, chosen by the operator.