Skip to content

Install, upgrade, back up

The operator runbook for a deployment built from this kit (examples/clean-account). It says what to run, what each step changes, and — because this matters more than anything else in it — which operations replace the manager host and what that costs you (§ 6). Read that section before any day-2 change.

Commands are written as tofu …; terraform … runs the same configuration (required_version >= 1.9 for the example root), but pick one binary per checkout and stay with it — the kit’s README (“Tooling”) says why.

  • One manager host (EC2). It runs norsk-mgr — the operator API, UI and CLI, as a systemd service — which in turn runs the Norsk Manager container (the orchestration daemon: job store, worker provisioning, worker supervision) and the front door (oauth2-proxy + nginx, the only thing on the public side).
  • A worker fleet the Manager owns. Workers are EC2 instances the Manager launches and terminates itself as productions need them. Terraform never creates or destroys a worker; it only builds what those launches assume (subnets, security groups, the worker instance role, the AMI to use).
  • Substrate: a VPC with public subnets — and, for workers that must not carry a public IP, private subnets behind NAT, or your own VPC (§ 3.2 “Worker networking”) — two security groups, two instance roles, a handful of Secrets Manager secrets (worker keypair, license, admin password, and — with TLS — the certificates’ state), an S3 bucket, and optionally a Route53 record + Let’s Encrypt certificate.

Everything the deployment remembers lives under /var/lib/norsk-mgr on the manager host — on its own EBS data volume (manager_data_volume_id), which outlives the host: a replacement re-attaches it and finds its state. norsk-mgr backup ships an archive of it to the deployment bucket daily (§ 8). Keep § 6 and § 8 in mind from the start.

NeedDetail
An AWS account and credentials with rights to create VPCs, security groups, IAM roles/instance profiles, EC2 instances and AMIs, Secrets Manager secrets, S3 buckets, and (for TLS) Route53 records in your zoneTerraform and Packer both run under these credentials. Run them from an operator workstation, not from inside the deployment.
OpenTofu ≥ 1.9 (or Terraform ≥ 1.9)examples/clean-account (its variable validations read other variables, which needs 1.9); the modules on their own need 1.6.
Packer ≥ 1.10Builds the two AMIs. packer init fetches the amazon plugin.
The AWS CLI, plus the Session Manager pluginHost access is SSM only — no SSH keys are installed. aws ssm start-session --target <instance id> is how you get a shell.
Your Norsk Manager license — a JSON fileIt must be a V2 license envelope; norsk-mgr refuses to start the Manager on anything else. It is stored in Secrets Manager and fetched by the host at boot.
The Manager imagenorskvideo/norsk-manager:<tag> on Docker Hub, or the reference you were given. Pin a tag; do not deploy latest.
A norsk-mgr versionThe version (or release-channel name) to bake into the manager-host AMI, or a direct binary URL if that is what you were given.
Optional: a Route53 hosted zoneIf you want the operator UI on a hostname with a Let’s Encrypt certificate (dns_zone). Without it the Manager is plain HTTP on port 9081, and a browser cannot use the operator UI there: the session cookie is a __Host- cookie, which browsers store only over HTTPS, so sign-in loops back to the sign-in page; and products’ WebSockets (Studio’s live connection among them) fail the worker proxy’s Origin check even through a tunnel. Without a zone, drive the Manager with norsk-mgr on the host (§ 4).

From packer/ (variables and OS/arch options: packer/README.md):

Terminal window
cd packer
packer init worker.pkr.hcl
packer init manager-host.pkr.hcl
# Worker: docker, the norsk-ctl worker binary, a warm cache of the media
# image and of the two proxy images (nginx, oauth2-proxy).
packer build \
-var region=<your region> \
-var norsk_media_tag=<media image tag> \
-var norsk_ctl_version=<norsk-ctl version or channel, e.g. latest> \
worker.pkr.hcl
# Manager host: docker, norsk-mgr, an enabled norsk-mgr.service, a warm
# cache of the Manager image and of the two proxy images.
packer build \
-var region=<your region> \
-var norsk_manager_tag=<manager image tag — the value you will give manager_tag> \
-var norsk_mgr_version=<norsk-mgr version or channel, e.g. latest> \
manager-host.pkr.hcl # add -var norsk_mgr_binary_url=… if you were given a URL

Each build prints an AMI id and appends it to manifest-worker.json / manifest-manager-host.json. Build in the region you will deploy to (or add -var 'ami_regions=[…]' to copy). Both binaries resolve from their S3 release channels (latest is the blessed release, rc the hourly candidate, nightly the nightly build), the Manager image from Docker Hub; for a deployment that must not reach Docker Hub / quay.io at all, see § 5.5 (ECR mirror) and bake with registry_auth=ecr.

Terminal window
cd ../examples/clean-account
cp terraform.example.tfvars terraform.tfvars # gitignored; keep it out of version control

The variables you must set:

VariableWhat
regionThe deployment region. Everything — host, workers, secrets — is single-region.
deployment_namePrefixes every resource; lowercase letters, digits, hyphens.
admin_cidrsThe networks allowed to reach the operator UI/API. This is the only public surface: keep it to your office/VPN egress.
manager_ami, worker_ami_x86_64The AMI ids from 3.1 (worker_ami_arm64 too if you built one).
license_jsonYour license, e.g. license_json = file("license.json").

Worth deciding now, because changing them later replaces the host (§ 6):

VariableWhat
dns_zone, tls_email (and hostname)Set dns_zone to your Route53 zone name and the UI becomes https://<hostname>.<dns_zone>/ with a Let’s Encrypt certificate issued at boot (DNS-01, via a Route53 grant scoped to that zone). hostname defaults to deployment_name. Adding, removing or changing it on a live deployment replaces the manager host, and the operator URL changes with it; workers already running keep the old URL (they send sign-in to it), so drain each one afterwards (norsk-mgr nodes drain <node> --yes) and the Manager moves its work to a fresh worker that has the new URL.
admin_user, admin_passwordThe first operator login. Leave admin_password empty and one is generated. Either way it lives in Secrets Manager and is applied at boot; it is never written into the instance’s user-data.
worker_support_userOptional break-glass ssh login on every worker ({ name, ssh_authorized_key }). Absent, workers carry no credentials beyond their instance role; reach them by SSM.
media_ingressInbound media ports on the worker security group (an SRT listener, say). Nothing is open by default. Only reachable from the internet with public workers — see below.
worker_networkWhere the workers live: public (the default), private-nat or existing. The decision is the next block.
worker_grpc_transportHow the Manager talks to its workers. mtls, the default: the deployment gets its own CA, the Manager and the workers each hold a certificate from it, and each refuses a peer that holds none. plaintext: no TLS on that channel, which the security groups alone protect. A deployment installed before mtls became the default (2026-10-12), that never set this, ran plaintext: set worker_grpc_transport = "plaintext" before your next apply to keep it, or plan for that apply to replace the host and turn over the workers.

Worker networking. The default puts workers in public subnets with a public IP each: media can reach them from the internet, operators’ browsers dial their previews directly, and there is nothing to pay for beyond the instances. If your policy says no instance gets a public address, pick one of the private modes — but read the limit first, because it matters more than the terraform:

A private worker cannot be reached from outside the VPC. It can send media out — pushed outputs, S3, log shipping, image pulls — but it cannot take media in from the internet (no SRT, RTMP or WHIP ingest from a public source) and cannot serve a product UI to a browser outside the VPC. The mode suits sources that arrive over a VPN, Direct Connect, a transit gateway or from inside the VPC, and operators who reach the workers the same way. A load balancer in front of private workers is not part of the kit.

ModeBuildsChoose it when
publicthe kit’s VPC, public subnetsSources or viewers on the internet talk to the workers directly; you want the simplest, cheapest layout. Our own deployments run this.
private-natthe kit’s VPC, plus a private subnet per AZ and a NAT gateway; nat_per_az = true for one gateway per AZYou are starting from a clean account and workers must have no public address. Costs about USD 35 a month per NAT gateway plus about USD 0.045 per GB through it — media pushed out through the NAT is the expensive part, and the obvious alternative (public workers) is free. S3 traffic bypasses the NAT through a gateway endpoint.
existingnothing — you pass vpc_id, manager_subnet_id (a public subnet: the manager host keeps a public address in every mode) and worker_subnet_idsYou already have a VPC with its own egress (a NAT, a transit gateway, a firewall) and would rather plug into it than be handed a second one. Your subnets must reach the AWS APIs, the container registry and your log and media destinations. worker_public_ip = true if the worker subnets are public ones.

In either private mode set worker_advertise to private (operators reach the workers over your routing, and the preview URLs carry private IPs) or none; public is refused. The worker security group’s direct-access rules then admit worker_access_cidrs — the VPC’s CIDR unless you set it to the ranges your operators arrive from. media_ingress and the example ingest security groups still work, for sources that can route to the VPC. Worker TLS (worker_cert_mode = wildcard-route53) works too: the per-worker DNS record carries the private address, which publishes your VPC’s internal addressing in a public zone — acceptable to most, forbidden by some. private-nat lists the NAT addresses as the nat_public_ips output: they are what your workers’ outbound traffic arrives from, for allowlisting at the far end. The mode is a bootstrap setting (§ 6): moving a live deployment between modes replaces the manager host, and the workers it launches afterwards land in the new subnets.

Tunables — changed later with an apply and a reload, no replacement (§ 6):

VariableWhat
manager_tag (manager_image, manager_image_pull)The Manager image. Required — there is no default, because a stale one silently hands an unpinned apply an old Manager: give the nightly label (<version>-<YYYY-MM-DD>-<sha8>) the shipped norsk-mgr was built against (its @norskvideo/norsk-manager-sdk pin), never a moving latest.
pool_instance_types, pool_root_volume_gibThe default placement pool seeded on FIRST boot: which instance types workers may be, and their root volume. After that the pool is the API’s (norsk-mgr pool …), and these only matter to a fresh store.
migration_launching_deadline_msHow long a migration waits for the new runtime.
worker_ami_x86_64 / worker_ami_arm64The worker AMIs (§ 5.3).
data_volume_size_gbThe data volume; grows in place.
backup_scheduleWhen the daily backup runs (§ 8).

The full list with descriptions is variables.tf; the example tfvars shows the optional groups (media egress via CloudFront, ECR instead of Docker Hub).

Terminal window
tofu init
tofu plan -out=plan.out # read it: for a first install everything is "create"
tofu apply plan.out

What happens on the host after apply: cloud-init mounts the data volume at /var/lib/norsk-mgr (formatting it the first time); writes norsk-mgr’s config.yaml, the Manager’s startup config and the per-region worker networking file there; points the daemon at the launcher config (an SSM parameter terraform wrote from the same values — § 6) and at the backup target; fetches the license (and, with TLS, restores or issues the certificate); restarts norsk-mgr; waits for its API; creates the admin login from the password secret; starts the Manager container; enables the daily backup timer. Allow a few minutes after apply returns. Progress is in /var/log/cloud-init-output.log on the host.

State is local to the directory (terraform.tfstate) and contains the license, the admin password and the workers’ keypair private key. Protect it like a credential, and configure a remote backend if that is your policy (versions.tf).

Terminal window
tofu output operator_url # https://<fqdn>/ or http://<ip>:9081/
tofu output admin_user
tofu output -raw admin_password # sensitive: not shown without -raw

Open the URL, sign in with those. If nothing answers, the usual reason is that your current IP is not in admin_cidrs (§ 10).

This needs dns_zone. On the plain-HTTP URL a browser cannot sign in: the session cookie is a __Host- cookie, which browsers store only over HTTPS, so the sign-in page comes straight back. An SSM port-forward to localhost:9081 gets past sign-in (browsers treat localhost as secure), but products’ WebSockets then fail the worker proxy’s Origin check, so Studio loads without its live connection. Without a zone, use norsk-mgr on the manager host (aws ssm start-session --target <instance>, then sudo -u norsk norsk-mgr …).

Then, in the UI or with the CLI (norsk-mgr, on the host or with an API token from anywhere — norsk-mgr token create; API tokens walks through minting and revoking one):

  • Pools — the default pool is seeded from pool_instance_types on first boot; edit or add pools under Pools (norsk-mgr pool …). The seed applies once; after that the pool store on the host is authoritative.
  • Products — register the product images you are licensed for (norsk-mgr product add --license-file <file> registers everything the license entitles).
  • Launch a production. The Manager launches the first worker for it; you will see an EC2 instance tagged id3as.Norsk.NodeType=worker appear. Never terminate those instances from the console — the fleet is Manager-owned; use norsk-mgr nodes drain/terminate or the Nodes page.

4.1 Signing in through your OpenID Provider

Section titled “4.1 Signing in through your OpenID Provider”

Operators sign in with the local accounts norsk-mgr keeps (admin_user and any you add). You can put your own OpenID Provider beside them — Entra ID, Okta, Keycloak, Google, anything that speaks OpenID Connect. The sign-in page then offers both; the local accounts stay, and are the way in when the provider is down. It needs TLS (dns_zone).

At the provider, register a confidential client for norsk-mgr:

  • redirect URI https://<hostname>.<dns_zone>/oauth2/callback;
  • to have signing out of norsk-mgr sign out at the provider too, the post-logout redirect URI https://<hostname>.<dns_zone>/, and sign_out_at_provider = true in identity_provider below. It is off by default: a provider refuses a post-logout redirect URI it has not been told about. The provider’s end-session endpoint must accept POST (Keycloak’s and Entra ID’s do; Okta’s is unconfirmed). Off, or where the provider publishes no end-session endpoint, sign-out ends only norsk-mgr’s session. Only a norsk-mgr release that signs out at the provider knows the setting; an older one ignores it, so sign-out there stays norsk-mgr’s alone;
  • the scopes openid, profile and email, plus whatever your provider needs to release group membership. email is not optional: the front door signs nobody in unless the provider gives an email claim for them and marks it verified (email_verified), and the user sees only a bare Permission Denied. A directory whose users have no address, or only unverified ones, has to be made to send verified addresses first;
  • group membership in the ID token, under a claim you name below (often groups). Where the provider can, have it send stable group ids rather than names: a role is given to exactly what the claim carries, and a name can be renamed, or reused for another group, underneath it. Entra ID sends object ids; Keycloak’s group-membership mapper and Okta send names, so there keep the name of a group that holds a role unique, and never rename it.

Then in tfvars:

identity_provider = {
id = "corp-sso" # norsk-mgr's own name for it; choose once
issuer_url = "https://login.example.com/realms/example"
client_id = "norsk-mgr"
scopes = ["openid", "profile", "email"]
groups_claim = "groups"
# sign_out_at_provider = true # sign out at the provider too (above)
}
identity_provider_client_secret = "<the client's secret>"

issuer_url is the issuer exactly as the provider’s tokens state it (its /.well-known/openid-configuration says). id is recorded against every user who signs in through the provider, so a later change makes them different users; pick a short, permanent name: at most 64 lowercase letters, digits, ., - and _, starting with a letter or digit. An id with a slash, backslash, quote, whitespace or an invisible (zero-width, bidi) character, and the reserved local, manager, api-token, ctl, indeterminate and issuer-refused, are refused; any other is accepted with a warning. The secret goes to Secrets Manager and is fetched onto the host at boot.

What a provider user can do. Signing in proves who they are and which groups they are in; it gives them nothing by itself. A user whose groups hold no role is refused, and the audit log (/var/lib/norsk-mgr/logs/access-audit.jsonl) says why: no-mapped-group when their groups map to nothing, no-groups-asserted when the provider sent no groups at all — that one is almost always the provider’s configuration (the claim name or a missing scope), not norsk-mgr’s. Groups are given roles in norsk-mgr itself, not in tfvars — on the Access page (Provider groups), or on the host:

Terminal window
norsk-mgr group roles news-ops viewer@fleet operator@production[class=news]
norsk-mgr group list
norsk-mgr group roles news-ops --clear # removes the group

A member holds everything their groups hold, from their next request; no restart and no new sign-in. fleet is the whole fleet, and the narrower scopes are the ones an account’s scoped roles use. Name the group exactly as the provider sends it.

Taking a role away from a group bites at once too. Taking a user out of a group at the provider does not: norsk-mgr learns their groups when they sign in, and a sign-in is not refreshed from the provider, so they keep what that group gave them until the sign-in ends (24 hours by default). End their sessions as well (below) to take it away now.

Workers differ in one direction. A session on a worker, made when someone follows a link to it, carries only those of their groups that held a role at that moment. A role given to a group afterwards reaches someone who already has such a session only when it ends (with their norsk-mgr sign-in, within 24 hours) and they follow a link again; at norsk-mgr itself, and through its /bundles/ links, it applies within a minute. Taking a role away from a group applies everywhere within seconds, because workers decide from what each group holds now.

A group may hold admin. Then whoever manages that group’s membership at the provider decides who administers norsk-mgr, and an admin can create local accounts, which outlive any later change to the provider or its groups. Guard a group that holds admin as you guard the admin password. An admin through a group never counts towards the last-admin rule either: norsk-mgr will not let its last local account that can manage users lose that, because local accounts are the way back in when the provider is down, misconfigured or replaced. Keep one.

Entra ID: users in many groups. For a user in more than about 200 groups, Entra ID leaves the groups claim out of the token and sends a pointer to Microsoft Graph instead (the groups overage). norsk-mgr does not follow it, so that user is refused with no-groups-asserted. Either limit the claim to the groups that matter (on the app registration, Token configuration, groups claim, Groups assigned to the application, and assign those groups to the enterprise application), or define app roles on the registration, assign them to those groups, set groups_claim = "roles", and give roles in norsk-mgr to the app roles’ values: the roles claim has no overage.

Getting a groups claim, provider by provider. Most providers send no group membership until they are told to, and a user then signs in to nothing (no-groups-asserted). Where to set it:

  • Entra ID: the app registration’s Token configuration, Add groups claim, security groups, ID token, Group ID. The claim is groups and carries object ids. See above for users in many groups.
  • Okta: the OIDC app’s Sign On tab, OpenID Connect ID Token: a groups claim named groups, with a filter that matches the groups you will give roles to (Matches regex .* sends every group). On a custom authorization server, add a groups claim of type Groups, included in the ID token, instead.
  • Keycloak: a Group Membership mapper on the client (or a client scope it uses): token claim name groups, Add to ID token on, and Full group path off, unless you mean to give roles to paths such as /eng/ops.
  • Google: its sign-in puts no group membership in the ID token, so on its own it signs everyone in to nothing. Put a broker in front of it that adds groups (Keycloak or Dex, say) and configure that broker here instead.

Whatever the provider, groups_claim must name the claim it actually sends, and a value given a role in norsk-mgr must match what the claim carries, character for character.

Who has signed in. norsk-mgr remembers every provider user who has made a request in the last 90 days (fleet.providerUserRetentionDays in its config): the Access page’s Provider users, or norsk-mgr provider-user list, shows each with their groups and what those give them. They are kept in /var/lib/norsk-mgr/provider-users.json (mode 0600): each user’s provider id and subject, display name, email, groups (in the order sent; past 100, only those that hold a role here and the earliest of the rest, to 100 unless more than that hold a role), and when they were first and last seen. A user unseen for the retention period is dropped from the file at its next write, but the file is in every norsk-mgr backup archive (§ 8), so those names and emails live as long as your backups do: set the backup bucket’s retention with that in mind.

Ending a session now. End sessions on the Access page, or norsk-mgr user revoke-sessions <name> for a local account and norsk-mgr provider-user revoke-sessions <issuer> <sub> for a provider user, ends every session they have: at norsk-mgr on their next request, and on every worker within seconds. They may sign in again at once; to keep someone out, delete the local account, or take them out of their groups at the provider. A provider user of the configured provider can be ended even if norsk-mgr does not list them. On a worker older than this release, ending a user’s sessions locks them out of that worker entirely until the sign-in length has passed, even if they sign in again.

Taking someone out of a group at the provider does not end their sessions; end them as well. Ending a provider user’s sessions also revokes every API token they own, and every token bound below one; the reply’s revokedTokens says how many. If the tokens could not be revoked, the sessions have still ended: the reply carries tokenRevocationFailed, the CLI exits non-zero, and asking again revokes them. Ending a local account’s sessions does not touch its API tokens, which hold only what the account holds now: an admin lists them with norsk-mgr token list and revokes each with norsk-mgr token revoke <id>.

API tokens. A provider user may mint personal API tokens like anyone, holding no more than their groups give them. Because norsk-mgr cannot ask the provider about someone who is not signed in, such a token lasts 30 days at most from the sign-in it was minted in (fleet.providerPatMaxDays sets it lower), holds its role only while the groups it was minted with still give it, and drops any group the user is no longer in each time they sign in. Taking a group out of a role here bites at once; taking a user out of a group at the provider bites at their next sign-in, or when the token expires. A token minted with another token is bound to it: it expires no later and is revoked with it, so only signing in again starts a fresh 30 days.

Changing it later. Setting the client secret for the first time, or removing it, replaces the manager host (§ 6). The provider’s details are tunables: change them in tfvars, apply, then on the host

Terminal window
norsk-mgr reload-launcher-config

The reload recreates the front door on the new details before it answers, with no restart. norsk-mgr’s own decisions take the new details as the file is written, a moment before the front door is recreated: on a change that keeps the id, the old directory can still sign someone in during that moment, and they hold its groups’ roles until the recreate. Taking those roles away first (below) closes it. The reply’s ingress says what the front door now serves: "outcome": "applied" is done; anything else comes with a message, and the command exits non-zero. A provider whose clientSecretFile cannot be read is refused, and nothing is written. norsk-mgr automanager status shows the same ingress at any time: pendingRestart: true means what the front door serves is not yet what is configured — a recreate still to come, a discovery of the provider still pending or due a retry, or a configuration it cannot render, which renderError then names.

What a change does to people already signed in:

  • id or issuer_url changed, the provider removed, or one added: every sign-in at norsk-mgr ends, local accounts’ included. The front door’s session key is replaced, because a session from the old configuration can no longer be attributed to a directory. Everyone signs in again; choose a quiet moment. If the new issuer cannot be discovered at the reload, the front door falls back to local accounts only, and changes the key again when a later discovery succeeds, so everyone signs in twice.
  • client_id, scopes or groups_claim changed: the front door is recreated (a few seconds without it) and sessions survive.
  • A new id makes every provider user a different user, and every group role given under the old id stops granting: roles are recorded against the provider’s id, not its directory. Workers stop accepting the old id’s users within seconds, and their API tokens hold nothing.

Retiring or replacing a provider. Before you remove the provider, change its id, or point it at another directory under the same id:

  1. Take the old groups’ roles away while the old provider is still the configured one: norsk-mgr group list, then norsk-mgr group roles <group> --clear for each. The API edits only the configured provider’s groups, so this cannot be done afterwards. A role left behind is keyed by the provider’s id and what the group claim carried, so it grants again to whatever directory is next configured under that id, to whoever that directory puts in a group of that name.
  2. Revoke the provider users’ API tokens, one by one: norsk-mgr token list (a provider user’s token names its owner’s issuer), then norsk-mgr token revoke <id>. Ending a user’s sessions (step 3) revokes their tokens too, but once the provider has changed only users norsk-mgr still lists can be ended, so revoke them here. Such a token holds nothing while its provider’s id is not the configured one, but a provider configured again under that id — even with a different issuer_url — revives it, and under the same id it goes on holding what its groups give. Give a new provider a new id.

Then change it and reload, and afterwards:

  1. End the provider users’ sessions: norsk-mgr provider-user list, then norsk-mgr provider-user revoke-sessions <issuer> <sub> for each (or End sessions on the Access page). The new session key ends their sign-ins at norsk-mgr, but not their sessions on workers, which under the same id go on until they expire; this ends those within seconds.
  2. Remove norsk-mgr’s client at the old provider.

The old provider’s users stay in Provider users, marked as not the current provider and holding nothing, until they have gone unseen for the retention period (90 days by default, as little as 7). That list is not a list of tokens: a user it has forgotten may still own a live token, which norsk-mgr token list still shows.

Four things get upgraded independently. None of them is zero-downtime for the control plane, and there is no rollback other than re-applying the previous version, so: upgrade with no productions running where you can. Running workloads are designed to survive a Manager restart (the workers keep running their jobs and the Manager re-adopts them when it reconnects), but that is resilience, not a maintenance procedure.

The Manager’s store across an upgrade. The Manager daemon keeps productions, runtimes, the audit log and node state in a store whose records carry a schema version. The first start of a newer Manager on an older store takes a backup of it before changing anything — into <jobStoreBackupDir>-pre-upgrade, beside the rolling backups, where nothing purges it — and then migrates it; the Manager’s log says so. A Manager never starts on a store written by a newer Manager: it refuses, naming both schema versions, rather than half-read records it does not understand. So rolling the Manager image back past an upgrade that migrated the store means rolling the store back too: stop the Manager, point its jobStoreBackupDir at the -pre-upgrade directory, and move the live job-store directory’s contents aside, leaving one placeholder file in it — the daemon treats a directory that will not open as damaged and restores from its backups on start (an EMPTY directory would instead start a fresh store).

norsk-mgr follows the same rules for the files it keeps. config.yaml carries a configVersion (a file without one — what the deploy kit writes — is version 0). A newer norsk-mgr reads an older file as it is and upgrades it the first time it writes settings, keeping the file it started from as config.yaml.v<n> beside it. An older norsk-mgr refuses to start on a config.yaml stamped newer than it understands, so rolling the binary back past such an upgrade means putting that config.yaml.v<n> back as config.yaml. norsk-mgr’s operational state — cordons, migration history, production templates and API tokens — lives in the Manager’s store (once the Manager offers its key/value store) alongside the Manager’s own, so it rolls back with the Manager.

How a release reaches you. Three artifacts, three variables:

ArtifactBuilt byPublished asYour variable
Manager imagenorsk CI (nightly)norskvideo/norsk-manager:<version>-<YYYY-MM-DD>-<sha8> on Docker Hub (plus a moving :nightly)manager_tag (tfvars; § 5.1)
norsk-mgr binarynorsk-ctl CI, nightly with the ctl binariess3://norsk.video/norsk-mgr/<version>/norsk-mgr-<version>-linux-<x64|arm64>; channel pointers latest, rcnorsk_mgr_version (packer; § 5.2)
norsk-ctl binarynorsk-ctl CI, nightlys3://norsk.video/norsk-ctl/<version>/…; the same channel pointersnorsk_ctl_version (packer; § 5.3)

The two binaries are cut from one norsk-ctl commit, so their versions share the <YYYY-MM-DD>-<sha7> stamp (norsk-mgr --version, norsk-ctl --version); the Manager image tag norsk-mgr was built against is its @norskvideo/norsk-manager-sdk pin — a release note names it, and the root’s manager_tag default carries the one this kit shipped with. Picking a Manager tag: docker manifest inspect norskvideo/norsk-manager:<tag> (or the Docker Hub tag list) shows what exists; use the dated label, never latest/nightly, so a manager start cannot silently move you. Then: image → tfvars + apply + reload (§ 5.1); binaries → bake + tfvars + apply (§ 5.2, § 5.3). With an ECR mirror (§ 5.5) the image step is preceded by a mirror-to-ecr.sh run.

5.1 The Manager image (the orchestration daemon)

Section titled “5.1 The Manager image (the orchestration daemon)”

The Manager runs as a container that norsk-mgr starts from managerImage:managerTag in config.yaml, pulling per managerImagePull (missing — pull only when the tag is not present locally — unless you set otherwise). Once started it is supervised: a container found exited is started again, and more than managerRestartLimit restarts within managerRestartWindowSeconds (5 in 600 s unless you set otherwise) stop it and make norsk-mgr exit for systemd to decide — the unit restarts it, and after three such exits in an hour leaves it failed for you (systemctl reset-failed norsk-mgr && systemctl start norsk-mgr). norsk-mgr manager stop removes the container, which is what ends the supervision; manager start begins it again.

From the registry (the normal path). Set manager_tag in terraform.tfvars and apply — the plan updates the launcher-config parameter and nothing else (no host replacement: § 6). Then, on the host:

Terminal window
norsk-mgr reload-launcher-config # writes config.yaml from the parameter; reports changedKeys
norsk-mgr manager stop
norsk-mgr manager start # pulls the new tag, starts it
norsk-mgr manager status

(Editing managerTag: in /var/lib/norsk-mgr/config.yaml by hand still works for a quick trial, but the next reload or daemon boot puts the parameter’s value back — tfvars is the record.)

From an image tarball (no registry access, or a build you were sent). Upload the docker saved image to the deployment bucket (tofu output worker_config_bucket), for example under images/, set manager_image_s3_uri to its s3:// URI in tfvars, and on the host:

Terminal window
/usr/local/bin/norsk-load-manager-image # fetch + docker load + tag as managerImage:managerTag
norsk-mgr manager stop
norsk-mgr manager start

With manager_image_s3_uri set the kit writes managerImagePull: never, so a start never replaces the loaded image with the registry’s copy of the same tag. The loader reads managerImage:managerTag from config.yaml at run time, so a tag change is the registry flow above plus a re-run of the loader; the S3 URI itself is baked into the script by cloud-init (a bootstrap value), so the first time you adopt this channel on an existing host either pass through a host replacement (§ 6) or run the aws s3 cp + docker load + docker tag by hand once.

The tarball is the record. Its identity — the S3 object’s ETag, read at plan time, or manager_image_s3_hash when you pass one — is in user-data, so it is part of the bootstrap fingerprint: after a new tarball is uploaded to the same key, the next tofu apply replaces the host (§ 6), and the new host loads it at boot. Before this, a new tarball never reached an existing host through terraform at all. Loading in place first (above) is still the quick route; the apply that follows replaces the host regardless. A root that uploads the tarball in the same apply must pass manager_image_s3_hash = filemd5("<file>") (the ETag is unknown until apply); the host then also verifies the download against it.

Check it took. docker inspect --format '{{.Id}}' norskvideo/norsk-manager:<tag> must change across the load; the Dashboard shows the daemon’s build once it reconnects (§ 10, version banner).

5.2 norsk-mgr itself (the operator API/UI)

Section titled “5.2 norsk-mgr itself (the operator API/UI)”

The supported path is a new manager-host AMI: rebuild with packer (norsk_mgr_version — a channel such as latest, or the exact version a release note names; the bake resolves it against the S3 store and verifies the checksum), set manager_ami in tfvars, apply. A new AMI always replaces the host, so do this together with anything else that replaces it and follow § 6 and § 8 first.

If you were given a standalone norsk-mgr binary to run ahead of an AMI, it can be swapped in place — the binary is self-contained (the UI is embedded), and the Manager container is untouched:

Terminal window
cp /usr/local/bin/norsk-mgr /usr/local/bin/norsk-mgr.prev
install -o root -g root -m 0755 <new binary> /usr/local/bin/norsk-mgr
systemctl restart norsk-mgr
norsk-mgr --version

On a Manager cluster do this for every host, standbys first and the active member last, which is what scripts/roll-norsk-mgr.sh does over SSM (--version resolves a version or channel against the S3 store as the bake does and verifies the checksum; --url takes a binary from anywhere the hosts can fetch). The Manager containers are not touched, so there is no failover; each host is back as a standby or the active member before the next is rolled:

Terminal window
scripts/roll-norsk-mgr.sh --region eu-west-1 --hosts i-aaa,i-bbb,i-ccc --version latest

A swapped binary is lost on host replacement (the AMI’s copy comes back; the data volume carries no binaries), so treat it as a bridge to the next AMI, not a release mechanism. After any systemctl restart norsk-mgr, check the front door (§ 10, “front door down after a restart”).

Bake with packer (worker.pkr.hcl, new norsk_ctl_version / media tag), set worker_ami_x86_64 (and _arm64) in tfvars and apply. The AMI table is a tunable: the plan updates the launcher-config parameter — every release channel for both architectures points at the new AMI (the x86_64 id stands in for aarch_64 when you have no arm AMI) — and the host is not replaced. Then on the host:

Terminal window
norsk-mgr reload-launcher-config # reports managerOverridesChanged: true, restartRequired: true
norsk-mgr manager stop && norsk-mgr manager start
norsk-mgr manager overrides get # the table the Manager now runs with

The Manager reads the overrides file at container start, hence the restart. (The old ritual — norsk-mgr manager overrides set overrides.json with a hand-written table — still works for a one-off test of an AMI, but the next reload or daemon boot writes the parameter’s table over it; tfvars is the record.)

Only newly launched workers get the new AMI. Existing nodes keep the old one until recycled: cordon the node (norsk-mgr nodes cordon <id>), move its productions (norsk-mgr runtime migrate <runtimeRecordId> --to <nodeId>, for products that support migration) or wait for them to end, then norsk-mgr nodes terminate <id> --yes. The Nodes page does the same. Drain a node shows the whole move, cordon to uncordon, with the CLI’s output at each step.

A just-created AMI is pending for a minute or so; a worker launch in that window fails — wait for available before switching.

Workers run the registry reference in norsk_media_image at whatever tag the registry serves (the AMI only pre-pulls it as a warm cache). Product images are whatever you registered with norsk-mgr product add. Neither involves this kit; a new worker AMI (§ 5.3) is only needed to make first boots fast again.

5.5 Running from an ECR mirror (no Docker Hub / quay.io egress)

Section titled “5.5 Running from an ECR mirror (no Docker Hub / quay.io egress)”

Set ecr_mirror = true. The apply creates ECR repositories for the whole image set (norsk, norsk-manager, norsk-moq-relay, nginx, oauth2-proxy), grants both instance roles pull, and points the deployment at them: manager_image and norsk_media_image become <registry>/<prefix>norsk-manager and …/norsk, container_registry becomes the registry host (the login target; the repository prefix, if any, rides proxy_images_registry), and the manager host (cloud-init) and every worker (the Manager’s cloud-init, plus the timer the AMI carries) docker login at boot and every 6 h — an ECR token lasts 12 h, and a worker’s --image-store-max can evict an image a stale token could not re-pull. The proxy images keep their upstream names (the software asks for nginx:<tag> literally): each host pulls the mirror’s copy and tags it so. The MoQ relay each worker runs for itself is handled the same way, at the norsk release the worker’s norsk-ctl is pinned to: mirror-to-ecr.sh mirrors it at --manager-tag unless --relay-tag names another (norsk-ctl node-images on a worker prints the one it wants). Nothing in the kit can push; you fill the mirror:

Terminal window
tofu apply # the repositories first
tofu output ecr_mirror_command # → the exact invocation for this manager_tag
bash packer/scripts/mirror-to-ecr.sh --registry <ecr_registry_url>[/<prefix>] \
--manager-tag <manager_tag> --media-tag <media tag> # your credentials

then bake both AMIs with registry_auth=ecr and container_registry=<the ecr_packer_registry output> — the registry plus your repository prefix, the string every image reference starts with (packer/README.md). Every later manager_tag or media-tag change is a mirror-to-ecr.sh run first, then the usual § 5.1 / § 5.3. Turning ecr_mirror on or off on a live deployment changes the registry settings, which are bootstrap values: the host is replaced (§ 6).

6. Configuration changes and host replacement

Section titled “6. Configuration changes and host replacement”

modules/manager-host sorts every input into one of two sets, and tofu plan tells you which one you touched.

Tunables — manager_image / manager_tag / manager_image_pull, the pool seed (pool_instance_types, pool_root_volume_*), migration_launching_deadline_ms, both worker AMI ids, and the identity_provider details (§ 4.1). These are published as the launcher config: a YAML document holding config.yaml’s content plus a managerOverrides block with the worker AMI table, written by terraform to an SSM parameter (tofu output launcher_config_uri; launcher_config_uri in tfvars to bring your own s3:// or file:// document instead). norsk-mgr fetches it at every boot and on demand:

Terminal window
tofu apply # the plan updates aws_ssm_parameter.launcher_config only
# on the host:
norsk-mgr reload-launcher-config # fetch, validate the whole document, write config.yaml

The reply says what took effect: changedKeys; restartRequiredKeys (a key the daemon reads only at boot — externalUrl is one — needs systemctl restart norsk-mgr); managerOverridesChanged (the AMI table — needs norsk-mgr manager stop && norsk-mgr manager start); seedOnlyKeys (placementPools: a seed for a fresh store, ignored on a store that has one); ingress, when the front door was recreated for the identity provider (§ 4.1). A document the validator rejects is 400 and changes nothing.

Two consequences to know:

  • The parameter is the record. With NORSK_MGR_LAUNCHER_CONFIG set — every host this kit builds — a settings change made through the UI, the CLI (norsk-mgr settings set) or a hand edit of config.yaml lasts until the next reload or daemon boot, when the parameter’s content is written back. Change tunables in tfvars.
  • A boot that cannot reach the parameter carries on on the config.yaml cloud-init wrote (the same content as the parameter at the time of the last replacement), with a launcher-config-fetch-failed audit entry and a warning in journalctl -u norsk-mgr. Fix the cause (the instance role, the SSM endpoint) and norsk-mgr reload-launcher-config.

Bootstrap — everything else that lands in user-data, plus the manager AMI: manager_ami, the Manager’s sys.config except the AMI table (worker_service_port, worker_support_user, worker_proxy_*, worker_advertise, worker_image_store_max, manager_initialisation_settle_ms, worker_orphan_terminate_after, container_registry (+ region), norsk_media_image, the worker cert secrets), the worker networking file (subnets, security group, role, worker_network and worker_public_ip), dns_zone / hostname / tls_email, admin_user, the secret ARNs (so setting log_output_config, ice_servers or identity_provider_client_secret for the first time, or clearing one, replaces the host; changing its content does not — § 12), manager_image_s3_uri, the launcher-config and backup URIs, the data volume id, and any edit to the module’s templates. Changing one of these replaces the manager host. The plan shows it as terraform_data.bootstrap updating and module.manager_host.aws_instance.this must be replaced — read the plan before every apply. (Mechanically: the instance ignores user-data changes and is replaced when the fingerprint of user-data-with-tunables-pinned changes, so a tunable never replaces it and a bootstrap change always does; tofu output -raw bootstrap_fingerprint in the module is that value.)

Does not touch the host at all: admin_cidrs, ssh_cidrs, media_ingress, moq_admin_cidrs, extra_worker_security_groups (security-group rules), license_json and admin_password (new secret versions; the host does not re-read them — § 7), tags, data_volume_size_gb (grows the volume in place), backup_schedule (rewrites the timer on the next replacement; edit /etc/systemd/system/norsk-mgr-backup.timer by hand until then), the media-egress and ECR groups.

What survives a replacement

  • Everything under /var/lib/norsk-mgr — it is the data volume (aws_ebs_volume.manager_data, prevent_destroy): norsk-mgr’s store (pools, products and licenses, product templates and their tars, production templates, machine types, cordons, migrations, API tokens, extra logins), the Manager’s metadata store with all job and production history, its rotated exports, the overrides file, a loaded image’s config.yaml entry. The old host is stopped before the volume is detached, so the filesystem is clean; the new host mounts it from bootcmd, before norsk-mgr can start, and refuses to start without it.

  • The public address: the Elastic IP moves to the new instance, so the DNS record and operator_url do not change.

  • The private address, with the Elastic IP (allocate_eip, the default): the host sits on its own network interface, which the new instance takes over. That matters because a production’s jobs carry norsk-mgr’s address for workers (http://<private IP>:9333) in their product-template URLs, and the Manager keeps those jobs and sends them unchanged whenever it restarts or re-places one. Without the Elastic IP the host gets a new private address, and a production that is running across the replacement can no longer fetch its template on a new worker: delete productions first, or keep the default.

    Adopting this on a host built before it (a kit older than 2026-10-12): the first apply replaces the host onto the new interface, so the private address changes that once. Delete running productions before that apply and launch them again after. Later replacements keep the address.

  • With TLS: the Let’s Encrypt certificate state. It is pushed to a Secrets Manager secret on every issue/renew and restored before certbot runs on the new host, so a replacement re-issues nothing and burns no rate limit.

  • Everything terraform owns: secrets, roles, security groups, the bucket.

  • The workers that are running — as EC2 instances. See the last bullet.

What does NOT survive — what lives on the root volume:

  • A swapped norsk-mgr binary (§ 5.2) and a docker loaded image tarball (§ 5.1: re-run /usr/local/bin/norsk-load-manager-image).
  • Running productions, as far as the control plane is concerned during the replacement: the new Manager starts from the same job store and re-adopts the workers it knew, but placement, migration and recovery are down for the minutes in between. Prefer a quiet window.

So a replacement is: § 8 backup (belt and braces) → tofu apply → wait for cloud-init → verify (§ 10). Batch the changes that force one.

7. Rotating the admin password, the license and the provider’s client secret

Section titled “7. Rotating the admin password, the license and the provider’s client secret”

All three are Secrets Manager secrets that the host reads once, at boot. A new secret version does not reach a running host by itself; the kit has no automatic rotation path today.

Admin password. Change it on the host, where it takes effect at once:

Terminal window
echo -n '<new password>' | norsk-mgr user set admin --password-stdin

Then put the same value in admin_password in tfvars and apply (a new secret version; the plan shows no host replacement) so that the next replacement bootstraps the login with it. Additional operators are the same command with another name, and norsk-mgr user role <name> <role> gives each one a role (§ 9).

License. Update license_json in tfvars and apply — that stores the new version, no host replacement. Then, on the host, fetch it into place and restart the Manager (which stages the file on every start):

Terminal window
aws secretsmanager get-secret-value --region <region> \
--secret-id <deployment_name>-manager-license --query SecretString --output text \
> /var/lib/norsk-mgr/license-source.json
chown norsk:norsk /var/lib/norsk-mgr/license-source.json
norsk-mgr manager stop && norsk-mgr manager start

manager start validates the file before it replaces the previously staged copy, so a bad file leaves the working license in place and reports why. The alternative — a host replacement, which fetches the secret fresh — costs what § 6 says it costs.

OpenID Provider client secret (§ 4.1). Most providers let a client hold two secrets at once, so add the new one at the provider first and the old one keeps working until you are done. Put the new one in identity_provider_client_secret and apply — a new secret version, no host replacement. Then, on the host, fetch it into place and reload:

Terminal window
f=/var/lib/norsk-mgr/identity-provider-client-secret
(umask 077; aws secretsmanager get-secret-value --region <region> \
--secret-id <deployment_name>-identity-provider-client-secret \
--query SecretString --output text > $f.new)
chown norsk:norsk $f.new && chmod 0600 $f.new && mv $f.new $f
norsk-mgr reload-launcher-config

The file is replaced whole, by a rename, because a front door that finds it empty or unreadable will not start (§ 10). The reload sees that the front door was set up with another secret and recreates it; the reply says "outcome": "applied", and sessions survive. Sign in through the provider once, then delete the old secret there. Without the reload the front door takes the new file within 15 minutes anyway.

A host created by an earlier kit has the file at mode 0640, and keeps it, because cloud-init runs only once; norsk-mgr warns about it at every start. Rotating as above sets 0600, or set it by hand: chmod 0600 /var/lib/norsk-mgr/identity-provider-client-secret.

The role-store replica’s signing key. norsk-mgr replica rotate-key replaces the key norsk-mgr signs the replica (every worker’s copy of who may do what) and identity assertions with; the Manager endorses the new key to every worker and the replica goes out again under it. Nothing is re-enrolled, and the change is audited.

The fleet discovery tier secret, with discovery_replica_hosts. The two replica hosts run no norsk-mgr, so they do not follow a rotation on their own: each reads the tier secret, and during a rotation the second one, from Secrets Manager at every start (<name>-discovery-tier, <name>-discovery-tier-alt; the second holds none outside a rotation). scripts/rotate-discovery-tier.sh takes each step at the Manager and carries it to them: on the manager host it writes the secrets in force into the two secrets (the role’s discovery-tier-rotation grant, so no secret reaches your machine or SSM’s command history), then restarts each replica host’s service in turn and waits for it to run again.

Terminal window
out="$(tofu output -json discovery_rotation)"
scripts/rotate-discovery-tier.sh stage --from-output "$out" # wait the settle time it prints
scripts/rotate-discovery-tier.sh flip --from-output "$out" # wait again: every node gets a new token
scripts/rotate-discovery-tier.sh finish --from-output "$out"

Each run first brings the replica hosts up to what is in force, so a step taken by hand with norsk-mgr manager discovery rotate-secret is caught up too, and sync does only that. A later tofu apply leaves both secrets as the rotation wrote them. Without replica hosts (discovery_rotation is null) every replica is one norsk-mgr runs, and norsk-mgr manager discovery rotate-secret alone rotates.

Three layers, innermost first:

  1. The Manager’s own store exports. After every change the daemon writes a whole-store export, manager/data/jobStoreBackup/<id>.khepri (one consistent file; the newest three are kept, then hourly for a day and daily for a week) and runs norsk-mgr’s hook, manager_backup.sh, which copies it to manager/data/jobStoreSnapshots/<utc>-<id>.khepri (keeping seven). Same volume — this is what lets the daemon rebuild a store it lost, and what norsk-mgr backup ships.

  2. The data volume (§ 6): survives the host. It does not survive the AZ, an accidental tofu destroy with prevent_destroy removed, or a filesystem you broke by hand.

  3. norsk-mgr backup — an archive of the whole store to a URI, off the host. The kit runs it daily (norsk-mgr-backup.timer, 00:00 UTC + a jitter; backup_schedule in tfvars) to s3://<deployment bucket>/norsk-mgr/backups/ (tofu output backup_target_uri; backup_target_uri in tfvars for another prefix — with its own IAM grant — or a file:// path). By hand, any time:

    Terminal window
    norsk-mgr backup # to the configured target
    norsk-mgr backup --to file:///tmp/b # somewhere else

    The archive is norsk-mgr-backup-<utc>.tar.gz: store/ (everything under /var/lib/norsk-mgr except logs, the extracted UI assets and the tls/ copy of the certificate), job-store-backup/ (one export: the newest hook snapshot, else the daemon’s newest), job-store-live/ (the live store directory, only when the Manager is stopped — a running store is never copied), and a manifest.json with versions and every file’s size and sha256. journalctl -u norsk-mgr-backup has the timer’s runs.

Restore — onto this host after a disaster, or onto a new deployment’s host to move a Manager. Stop the Manager first; the daemon may stay up but is restarted afterwards:

Terminal window
norsk-mgr manager stop
norsk-mgr restore --from s3://<bucket>/norsk-mgr/backups/norsk-mgr-backup-<utc>.tar.gz --yes
systemctl restart norsk-mgr
norsk-mgr manager start

restore refuses while the Manager container is running (--force to override — do not), keeps whatever it replaces under /var/lib/norsk-mgr/.pre-restore-<utc>/, and puts the store back one of two ways: the archived live directory when the backup was taken with the Manager stopped, otherwise the export alone in the daemon’s backup directory with the live directory left empty, which makes the Manager import it on that manager start (it logs Metadata store is empty but exports exist; restoring the newest). Run it as root (sudo) so file ownership is restored; the Manager’s own tree is made writable for its container user either way. A cluster member (§ 13) restores through the seed: every member stopped and emptied, the restore on the seed, the seed started first.

Restoring config.yaml from a backup is transient when a launcher config is configured (§ 6): the next reload or boot writes the parameter’s content back. Restoring manager/data from a different deployment brings back that deployment’s view of which workers exist; expect the daemon to reconcile against what EC2 actually reports and to raise alerts for nodes that are gone.

Before this kit’s data volume (a host built from a commit older than aws_ebs_volume.manager_data): § 11 is the one-time adoption.

  • One manager host by default. One norsk-mgr, one Manager container, one data volume in one AZ: the control plane is a single point of failure. Workers keep running their jobs while it is down, but nothing places, migrates or recovers until it is back. Three hosts as a cluster (§ 13) is how to remove it; the kit’s root composes one host today, and the module takes the cluster block for a root that composes three.
  • One region. Host, workers, secrets and the worker networking config are all in region.
  • Private workers take no inbound from outside the VPC (§ 3.2 “Worker networking”): no internet ingest, no product UI for an outside browser. There is no load balancer or ingress proxy in the kit for that case.
  • Roles, but no tenancy. Logins and API tokens hold a role (viewer, operator, builder, admin), optionally scoped to part of the fleet, and enforcement is permissive until you set accessEnforcement: enforced. A deployment is still one team’s.
  • Port 9333 is open to the worker fleet, but the API is gated. Workers fetch product-template tars (per-launch token) and workdir assets (by sha) from norsk-mgr on port 9333, which is open to the whole worker security group; every other /api/* route needs a bearer token, the mgr-nginx proxy secret, or the proxy secret presented on loopback by the on-host CLI, so a worker — or a product container it runs — gets 401 elsewhere.
  • Backups are daily and the restore is manual (§ 8); no automatic license/password rotation (§ 7).
  • The Manager’s sys.config is still delivered by host replacement (§ 6): the worker proxy/advertise/support-user/image-store/registry knobs live there and are not part of the launcher config yet. Only the AMI table rides the overrides channel.
  • With ECR as the registry, the login is refreshed every 6 h by a timer in both AMIs (norsk-ecr-login.timer; a token lasts 12 h). A worker built from an AMI older than that timer still has only its boot-time login, and cannot re-pull an image it evicts after 12 hours — rebake (§ 5.3) or recycle such workers within the day.
  • With worker TLS (worker_cert_mode = wildcard-route53), each worker registers a DNS record at boot and nothing removes it on termination; the delegated zone grows by one record per worker ever launched. Harmless (names encode the IP) but untidy.

Where things are on the host (aws ssm start-session --target <instance id>; find the id in the EC2 console by the <deployment_name>-manager Name tag — it changes on every replacement):

WhatWhere
norsk-mgr (operator API, proxy lifecycle, Manager lifecycle)journalctl -u norsk-mgr -f; systemctl status norsk-mgr
The Manager containerdocker ps (norsk-manager); docker logs norsk-manager; its log files under /var/lib/norsk-mgr/manager/logs/
norsk-mgr’s structured log/var/lib/norsk-mgr/logs/norsk-mgr/debug.json — one JSON object per line, each API request’s lines sharing its requestId (the X-Request-Id response header). The Manager’s own debug.json (under /var/lib/norsk-mgr/manager/logs/) logs each call norsk-mgr makes while serving a request under the same requestId, with the gRPC method as grpcMethod
The front doordocker ps (the proxy and nginx containers)
Boot / cloud-init/var/log/cloud-init-output.log; cloud-init status
Configuration/var/lib/norsk-mgr/config.yaml, manager.sys.config, worker-config.json; the launcher config the daemon fetches: aws ssm get-parameter --name /norsk-mgr/<deployment>/launcher-config --query Parameter.Value --output text
The data volumefindmnt /var/lib/norsk-mgr (must show the nvme…/xvdf device, not the root disk); /usr/local/bin/norsk-data-volume re-runs the mount
Backupssystemctl list-timers norsk-mgr-backup.timer; journalctl -u norsk-mgr-backup; aws s3 ls <backup_target_uri>/
A workeraws ssm start-session --target <worker instance id> → journalctl -u norsk-ctl-worker -f, docker ps

Health and metrics.

  • GET /api/health — on the host without any credential (curl http://127.0.0.1:9333/api/health), or through the front door with a login or an API token (curl -H "Authorization: Bearer <token>" https://<fqdn>/api/health; norsk-mgr token create probe --role viewer mints one). ok = API up, connected to the Manager, every store readable. degraded (still HTTP 200) = up but the Manager channel is down or a store file is unreadable — the body says which (daemon, store, unreadableStores). down (503) = boot failed or still starting, with reason.
  • GET /api/manager/client (or norsk-mgr manager client) — is the Manager daemon answering. norsk-mgr manager status — is the container running. norsk-mgr automanager status — placement engine connection and counters.
  • GET /metrics — Prometheus text format, every series norsk_mgr_*. Served on the host’s loopback (a scraper on the host) and through the front door to a logged-in session; anyone else gets 403 unless the service runs with NORSK_MGR_METRICS_OPEN=1. norsk_mgr_daemon_connected, norsk_mgr_store_unreadable, norsk_mgr_nodes{state} and norsk_mgr_alerts_active{severity} are the ones to alert on. The Manager daemon’s own metrics follow them in the same scrape (§ 12.5).
  • Version banner. The Dashboard shows a banner when the Manager daemon’s wire-protocol version differs from norsk-mgr’s. Minor skew: they interoperate; fields one side added are ignored by the other — upgrade the lagging side (§ 5.1 or § 5.2) when convenient. Major skew: norsk-mgr has refused to connect; health reports degraded with daemon: disconnected until one side is upgraded so the majors match, then systemctl restart norsk-mgr. Unknown: the daemon predates the handshake and is assumed compatible.

norsk-mgr will not start; systemctl status says a dependency failed (var-lib-norsk\x2dmgr.mount). The data volume is not mounted, and the unit is meant to refuse rather than build a second store on the root disk. lsblk — is the volume attached (tofu output manager_data_volume_id, aws ec2 describe-volumes --volume-ids <id>)? If attached, run /usr/local/bin/norsk-data-volume and read what it says (it waits up to five minutes for the device, formats only a blank one, and adds the fstab line by UUID). If a replacement ran while the old host still held the volume, the attachment is in the plan’s output; tofu apply again.

launcher-config-fetch-failed in the audit log / “launcher config NOT applied” in the journal. The daemon booted on the local config.yaml because the parameter could not be read: the instance role lacks ssm:GetParameter on it (the plan shows the LauncherConfigRead statement), the region is wrong (AWS_REGION in the service drop-in), or launcher_config_uri points somewhere that does not exist. Fix, then norsk-mgr reload-launcher-config — no restart needed for the fetch itself.

The UI does not answer at all. First suspect: your current IP is not in admin_cidrs (operator addresses drift). Check the manager security group in the console, or fix admin_cidrs and apply — that is a security-group change only, no host replacement. Then systemctl status norsk-mgr and docker ps on the host.

Front door down: port 443/9081 refuses connections. Two causes, and the journal tells them apart. On the host the CLI still works either way: it reaches norsk-mgr on loopback, not through the front door.

  • The identity provider’s client secret. The front door will not start with a provider whose clientSecretFile it cannot read, or finds empty — a fetch that failed half way, a file left owned by root — because a sign-in page that cannot complete a provider sign-in would hide the fault. journalctl -u norsk-mgr says identityProvider.clientSecretFile … could not be read, or is empty, and norsk-mgr automanager status shows it as ingress.renderError. A reload refuses such a provider, and a running front door keeps its previous configuration, so this shows after a restart or a host replacement. Put the secret back (§ 7: owner norsk, mode 0600: only norsk-mgr reads it), or take identity_provider out of tfvars and apply — the provider’s details are a tunable, so that does not replace the host — then norsk-mgr reload-launcher-config. The reload starts the front door; its reply says "outcome": "applied" once it is up, and curl -sk https://127.0.0.1/ answers. While the front door is down, for whatever reason, every reload tries to start it, and exits non-zero if it cannot.
  • A restart race. After systemctl restart norsk-mgr the new process can find the outgoing proxy containers still stopping and skip starting its own. The journal has no refusal; docker ps shows the proxy containers Exited. A second systemctl restart norsk-mgr starts them cleanly.

Certificate not issued (TLS deployments). cloud-init’s certbot step logs its failure and carries on, so the proxy comes up without a certificate. On the host: certbot certificates, then re-run certbot certonly --dns-route53 -d <fqdn> -n --agree-tos -m <tls_email> --deploy-hook /usr/local/bin/norsk-mgr-tls-deploy. The usual causes are the Route53 zone id not matching dns_zone, or the instance role missing the zone grant (both are terraform’s — re-check the plan).

The Manager container will not start. norsk-mgr manager start prints the reason: an invalid license file (must be a V2 envelope), an image that cannot be pulled (managerImagePull: never with no local image; a registry that needs a login the host does not have), or — after a hand restore that chowned manager/data — a permission error on the store’s log file (norsk-mgr restore leaves that tree writable for the container; a tar xzf by hand must not chown it).

Workers never appear / launches fail. norsk-mgr alert list and the placement log (norsk-mgr placement recent) name the reason: no AMI in the image table for the instance’s architecture, an AMI still pending, an instance type not offered in the subnet’s AZ, a service quota, a missing iam:PassRole. The worker’s own boot is in its journalctl -u norsk-ctl-worker over SSM; a worker that boots but never connects usually cannot reach the manager on its private IP (a security-group edit).

Something is being re-placed or stopped that should not be. Do not change config.yaml through the API (norsk-mgr settings set, the hot-spares and policy routes) while productions are running: those paths reload the placement engine, which reconnects and re-hydrates from the Manager. Make such changes with no productions running — through tfvars and norsk-mgr reload-launcher-config (§ 6), which applies the same way.

11. Adopting the data volume on a host built before it existed

Section titled “11. Adopting the data volume on a host built before it existed”

A deployment applied from a commit before aws_ebs_volume.manager_data has its state on the root volume. Applying the current kit creates the volume and — because the volume id and the daemon’s launcher-config / backup environment are bootstrap values — replaces the host, whose new cloud-init would format the empty volume and start with an empty store. Move the state across first, once:

  1. On the current host, with no productions running: norsk-mgr manager stop && norsk-mgr backup --to file:///tmp/pre-volume and copy the archive off the host (aws s3 cp to the deployment bucket; the role can read it back from there).
  2. tofu plan: expect the volume, the SSM parameter, the IAM statements and the host replacement. tofu apply. The new host mounts a blank volume, boots empty (the launcher config gives it the same config.yaml as before, but no pools, products, templates or history).
  3. On the new host, after cloud-init status --wait:
    Terminal window
    norsk-mgr manager stop
    norsk-mgr restore --from s3://<bucket>/…/norsk-mgr-backup-<utc>.tar.gz --yes
    systemctl restart norsk-mgr
    norsk-mgr manager start
    restore puts the store back onto the mounted volume; the Manager rebuilds its job store from the archived backup on that start (§ 8).
  4. Verify (§ 10): the Products and Productions pages show what the old host had; norsk-mgr audit list shows the restore.

From then on, replacements keep the volume and step 3 is never needed again. (An EBS snapshot of the old root volume, taken before step 2, is the belt to this braces.)

12. Running productions: stop, pause, TURN, logs, metrics, configuration as files

Section titled “12. Running productions: stop, pause, TURN, logs, metrics, configuration as files”

A stop takes a runtime down and keeps it down: nothing restarts it until you activate it again, and it gives up its node (the capacity is free for other work, and an idle elastic node can be reaped).

Terminal window
norsk-mgr runtime stop <runtime-id> # docker's default 10 s to finish
norsk-mgr runtime stop <runtime-id> --grace 60 # a minute to finish cleanly
norsk-mgr runtime stop <runtime-id> --now # kill the containers at once
norsk-mgr runtime activate <runtime-id> # place and start it again

A pause stops it too, but keeps its place: the runtime stays bound to its node, the capacity it reserved stays reserved, and the node is held so it is not reaped or scaled in. resume starts it on that same node — or places it afresh if the node has gone meanwhile.

Terminal window
norsk-mgr runtime pause <runtime-id> [--grace <s> | --now]
norsk-mgr runtime resume <runtime-id>

Use a pause for a short break you want back quickly and in the same place; a paused runtime still costs its share of the node. Use a stop for anything longer. A production’s page in the UI has the same actions on each of its runtimes (Stop, Stop now, Pause, Resume, Activate), and a paused runtime is badged as such.

--grace is 0–600 seconds: the time the product’s containers get between the stop signal and being killed, for a clean end (closing a recording, finishing a segment). A worker older than the Manager’s protocol 1.1 does not know it and uses docker’s default.

This kit runs no TURN server. If browsers or media servers sit behind NAT, run your own (coturn, or a hosted service) and give the list in tfvars:

ice_servers = [
{
url = "turn:198.51.100.15:3478" # what the media server dials
reported_url = "turn:turn.example.com:3478" # what the browser is told (omit if the same)
username = "norsk"
credential = "<password>"
},
{ url = "stun:stun.example.com:3478" },
]

The list is stored in Secrets Manager (TURN entries carry credentials), fetched to /var/lib/norsk-mgr/ice-servers.json at boot, and handed at launch to every product whose template declares GLOBAL_ICE_SERVERS (Studio does). No field may contain whitespace, #, $, quotes or backslashes. The credential is readable inside the product container, as any launch parameter is.

Setting it for the first time replaces the manager host (§ 6). To change it afterwards, tofu apply (a new secret version) and fetch it on the host, as root; no restart is needed, because norsk-mgr reads the file at every launch:

Terminal window
aws secretsmanager get-secret-value --region <region> --secret-id <deployment_name>-ice-servers \
--query SecretString --output text > /var/lib/norsk-mgr/ice-servers.json

Running productions keep the list they started with.

Everything writes structured logs on its own host: norsk-mgr to /var/lib/norsk-mgr/logs/norsk-mgr/debug.json, the Manager under /var/lib/norsk-mgr/manager/logs/, each media engine on its worker. Each line is one JSON object: time (epoch microseconds), level, text, and optional data.

To ship them elsewhere, give a fluent-bit output config: its [OUTPUT] sections, saying where logs go (Loki, OpenSearch, CloudWatch Logs, S3, an HTTP endpoint — anything fluent-bit writes to):

log_output_config = <<-EOT
[OUTPUT]
Name loki
Match *
Host loki.example.com
Port 443
tls On
http_user norsk
http_passwd <password>
labels job=norsk
EOT

It is stored in Secrets Manager and fetched to the manager host at boot. Then:

  • the Manager container runs fluent-bit over its own logs and norsk-mgr’s, so the host runs one log agent. Match on the tags manager-erlang, manager-stderr and manager-mgr;
  • on workers, each node’s log agent ships the node’s logs with the same file (below).

What ships, by tag — match on these in Match:

TagWhat
manager-erlang, manager-stderr, manager-mgrthe Manager and norsk-mgr, on the manager host
media-erlang, media-rust, media-stderreach media engine
worker-journalthe worker service’s own journal
product.file.<json|text>.<instance>.<service>a product’s declared log files, e.g. Studio’s JSON debug.log
product.stdout.<json|text>.<service>a product service’s declared stdout

Every record carries a source field (norsk-mgr, norsk-manager-erlang, norsk-media-erlang, …). Every engine and product record also carries which runtime it came from:

  • NORSK_RUNTIME_ID — <production>.<replica>.<job>, e.g. studio4.0.main: filter on its prefix for a production, on the whole value for one runtime;
  • NORSK_LAUNCH_ID and NORSK_NODE_ID — this launch of the runtime (new every launch, so a moved runtime’s records are told apart), and the node (- when not known). Records from before carried NORSK_INSTANCE_ID instead of NORSK_LAUNCH_ID.

Which product services ship, and how, is declared by each product template (its manifest’s logs: block); Norsk’s own products declare theirs.

Things to know:

  • Credentials. Either use an output that carries its own credentials — a user and password or a token in the file (Loki, OpenSearch with basic auth, an HTTP endpoint, Splunk, Datadog…), which is why the file is a secret — or an AWS output that uses the instance role.

  • CloudWatch Logs. Set cloudwatch_log_group (and optionally cloudwatch_log_retention_days, default 30 — CloudWatch’s own default is to keep logs forever). The kit creates the group with that retention and grants the manager host’s role and the workers’ permission to write to that group and nothing else. Name the same group in the output with auto_create_group Off:

    cloudwatch_log_group = "/norsk/logs"
    log_output_config = <<-EOT
    [OUTPUT]
    Name cloudwatch_logs
    Match *
    region eu-west-1
    log_group_name /norsk/logs
    log_stream_prefix norsk-
    auto_create_group Off
    EOT

    Any other AWS output (S3, SigV4 OpenSearch) needs a policy for it attached to the <deployment_name>-manager and <deployment_name>-worker roles.

    To see one production’s logs, use Logs Insights on the group, not the streams (streams are per tag, and every production shares them):

    fields @timestamp, @logStream, NORSK_RUNTIME_ID, @message
    | filter NORSK_RUNTIME_ID like /^studio4\./
    | sort @timestamp desc

    stats count() by NORSK_RUNTIME_ID shows what is logging.

  • Datadog. Its API key rides in the file, so there is nothing to grant and no cloudwatch_log_group. Set Host to your Datadog site’s log intake (http-intake.logs.datadoghq.com for US1, http-intake.logs.datadoghq.eu for EU, and so on):

    log_output_config = <<-EOT
    [OUTPUT]
    Name datadog
    Match *
    Host http-intake.logs.datadoghq.eu
    TLS on
    apikey <datadog API key>
    dd_service norsk
    dd_source norsk
    dd_tags env:production
    EOT

    Use an API key for this deployment alone, so it can be revoked on its own. NORSK_RUNTIME_ID and the other fields arrive as attributes: make it a facet to filter by production or runtime. Several [OUTPUT] sections with different Match patterns send different tags to different places — e.g. Match product.* to one index and Match media-* to another.

  • Workers: the log agent. The Manager runs a log-agent node service on every worker node: one host-networked fluent-bit that alone fetches the output config from its secret, as the worker role, and that every engine forwards to. So the config is never in worker user data or in an engine container, AWS outputs authenticate from workers, and the worker service’s journal and declared product logs ship too. A node whose agent fails is not given work, and raises an alert after three failures. (Until 2026-10-13 a deployment could instead hand the config to every engine; that path is gone, and the log_agent variable with it.)

  • Changing it. Setting it for the first time replaces the manager host (§ 6). After that, tofu apply writes a new secret version; fetch it on the host as root and restart the Manager container, which reads it at start:

    Terminal window
    aws secretsmanager get-secret-value --region <region> --secret-id <deployment_name>-log-output-config \
    --query SecretString --output text > /var/lib/norsk-mgr/log-output.conf
    norsk-mgr manager stop && norsk-mgr manager start

    Each worker’s agent fetches the secret when it starts: new workers get the new version, and running ones when their agent restarts.

12.4 Workers whose Manager never comes back

Section titled “12.4 Workers whose Manager never comes back”

A worker that hears nothing from its Manager for worker_orphan_terminate_after seconds (default 86400, one day) terminates its own instance. That covers a Manager host lost without a restore, or a deployment torn down around workers it missed: without it, those instances keep running and billing with nothing left to stop them. The worker role may terminate its own instance and no other.

The timeout must outlast any Manager outage you expect, upgrades included. If the Manager is down for longer than this, every worker in the fleet terminates itself. Set it to 0 to switch it off. A worker built from an AMI older than this feature ignores the setting and never terminates itself.

The Manager daemon serves Prometheus metrics on host port 6788, bound to loopback: the Erlang VM’s, and the mailbox depth and memory of each of its long-lived processes (norsk_manager_process_message_queue_len{process}, norsk_manager_process_memory_bytes{process}). A mailbox that keeps growing is the process that cannot keep up — the node controller or a cloud provider, typically. The port is manager_metrics_port in modules/manager-host (0 turns it off). norsk-mgr’s own /metrics appends them after its own series, so the host stays one scrape target (§ 10). norsk_mgr_manager_metrics_up says whether that part of the scrape worked.

Worth an alert beyond those in § 10: a mailbox that stays high; norsk_mgr_placement_drift_corrected_total, which counts the times placement’s view of which node runs what disagreed with the Manager’s and was corrected (a steady rise means something keeps moving jobs behind placement’s back); and norsk_mgr_manager_metrics_up == 0, which means the daemon’s metrics have gone missing.

12.6 Configuration as files: export, review, apply

Section titled “12.6 Configuration as files: export, review, apply”

norsk-mgr export writes the operator configuration — clouds, machine types, pools, hot spares, product templates (with their tars), production templates and node services — to a directory of files you can keep in version control. The log agent node service that norsk-mgr creates from the deployment’s own configuration is left out, because each deployment creates its own, and --prune never removes it. norsk-mgr apply makes a Manager match such a directory:

Terminal window
norsk-mgr export ./mgr-config # everything
norsk-mgr export ./mgr-config --only pools,hot-spares
norsk-mgr apply ./mgr-config --dry-run # print the plan, change nothing
norsk-mgr apply ./mgr-config # create and update
norsk-mgr apply ./mgr-config --prune # ...and delete what the directory does not name

apply touches only the kinds the directory has files for, and without --prune it never deletes anything. Applying the same directory twice changes nothing the second time. It is also how to copy a setup from one deployment to another: export from staging, apply to production. Set NORSK_MGR_URL and NORSK_MGR_TOKEN (an API token whose role may make those changes) to run it from anywhere but the host.

Production templates also keep their own history: norsk-mgr production-template history <name> lists the last 20 revisions, show <name> --revision N prints one, and restore <name> N makes it current again (as a new revision, so a restore can itself be undone). The template’s page in the UI has the same history.

Node services keep the same history: norsk-mgr node-service history <name>, node-service revision <name> N and node-service restore <name> N, and the History button on the service’s card. A node runs the revision it started with, so the history is also how you find out what an older node is running. A restore, like any save, reaches only nodes that start afterwards.

13. Three manager hosts: a Manager cluster

Section titled “13. Three manager hosts: a Manager cluster”

One manager host is one point of failure. Three hosts, each running its own norsk-mgr and Manager container, form a cluster: the Managers hold one replicated store (a Raft quorum, so any two of the three carry on), exactly one of them is primary and makes every decision, and the norsk-mgr beside it is the active one. The other two norsk-mgrs are standby: an operator who reaches any of the three sees the same deployment, because a standby authenticates the operator itself and forwards the request to the active one. Everything the deployment remembers — productions, pools, templates, accounts and roles, API tokens, the proxy’s login accounts and the session cookie secret — is in the replicated store; product-template tars and uploaded assets are copied to every member as soon as their record appears.

Kill the primary (the host, its Manager, or both) and within about ten seconds a survivor is primary at the next epoch and its norsk-mgr is active. Running productions are adopted, not restarted: the new primary re-dials every worker, finds each runtime where it was, and writes no history. A worker that restarts while the member it was given is dead registers with a survivor from the member list every worker carries. The dead member, restarted, rejoins as a standby. While no member is primary (the election) a standby’s UI shows “the Manager cluster is taking over” and retries by itself; nothing needs doing.

The active norsk-mgr dying while its Manager stays primary is the single-host story: the kit’s norsk-mgr.service is Restart=on-failure, so systemd starts it again and it reconnects to the still-primary Manager as active, hydrating as any restart does. The Manager container is norsk-mgr’s to keep up: its supervisor starts a container it finds exited (after a crash, a host stop or a reboot) within a ceiling, and starts none while the Manager is held (§ 13.7). The standbys can tell this case from a network fault between them and the active one: every norsk-mgr announces itself to its own Manager every few seconds, and the primary’s row in the cluster carries when its norsk-mgr last did. Once three intervals pass without one, norsk-mgr manager cluster on any member shows the primary’s controlPlaneStale: true (with controlPlaneAgeMs), norsk-mgr manager client on a standby shows activeControlPlane.down: true, a request through a standby is refused with 503 STANDBY_ACTIVE_DOWN (“its service manager should be restarting it”) rather than the generic STANDBY_ACTIVE_UNREACHABLE, and the standby’s UI banner says the active norsk-mgr is down. All of it clears by itself when the restarted process announces again; the Manager does not fail over for this, and the epoch does not move.

Each host is a manager host as in § 3, with its own data volume, and the same norsk-mgr version. Then on every host, the same command with its own --cluster-self:

norsk-mgr init --cluster 10.0.1.10,10.0.1.11,10.0.1.12 --cluster-self 10.0.1.11 \
--cluster-secret <secret> --cluster-cookie <cookie> ... (the usual init flags)
  • --cluster lists every member, the seed first, as the others reach it: private IPs, or DNS names that resolve inside the VPC. The Manager’s node name is manager@<host>, so the entry must not change for the life of the host.
  • The secret (what a standby’s forward carries) and the cookie (the Managers’ Erlang cookie) are the same on every member. Leave both off on the first host and init mints and prints them; pass them to the others.
  • --cluster-witnesses <host> names a member that votes and replicates but never becomes primary: an on-prem pair’s small third machine.

init writes the cluster block of config.yaml; from it norsk-mgr starts the Manager container as that node with that cookie, publishes the distribution ports, and writes the membership into the Manager’s overrides file, so the Manager’s sys.config is the single-host one unchanged. The hosts must reach one another on tcp 4369 and 9101–9110 (Erlang distribution), 6787 and 6789 (the Manager) and 9333 (norsk-mgr); the kit’s modules/security opens exactly those, member to member, with manager_cluster = true, and modules/manager-host takes the same block as its cluster variable. The kit’s root (examples/clean-account) composes the three with manager_cluster = {} (its README, “Manager cluster”): each member in its own AZ with a fixed private address, its own data volume, launcher config and hostname (<hostname>-a.<dns_zone> and so on), the secret and cookie minted once and shared. Those hosts run init --cluster themselves at boot; the command above is for hosts you build by hand.

Start the seed first with an empty Manager data directory, then the others, each also empty: each joins the seed on its first tick and replicates the store. norsk-mgr manager cluster on any member shows the members, the primary and the epoch. The order matters only once: after that any member may restart in any order.

  • One to three. Keep the existing Manager as the seed: run init --cluster on it with itself first and restart norsk-mgr; its store carries on. Bring the other two up with the same list and EMPTY data directories.
  • Retiring or replacing a member. norsk-mgr manager cluster remove <node> retires it: the store’s membership drops it, every worker is told, and if it was the primary it hands leadership over first. Then stop its norsk-mgr and take it out of every member’s list (re-run init --cluster or edit config.yaml, and restart at leisure: the list is only used to find a cluster to join). A retired member that starts again with its old data joins as a new member, so wipe its Manager data directory before reusing the host. A replacement starts with a list naming itself and the current members; the others need not know it yet.
  • A member lost its disk. Give it an empty data directory and start it with the same list: it joins the cluster the others still have. The seed included: a seed with an empty store joins an existing cluster, and only restores a backup export when no other member has a cluster.
  • A member restarted while the others are down comes up with its data and waits (its log says “cluster has no leader yet”), answering nothing until a majority is back. It never restores, never wipes, never leads alone.
  • A member with data that is not the cluster’s (a restored copy, a Manager that ran alone) refuses to join and says so: wipe it to join, or make it the seed and give the others empty directories.
  • Backup and restore. norsk-mgr backup on any member ships that member’s copy; the Manager’s own exports (jobStoreBackupDir) are whole-store snapshots. To restore: stop every member, empty every Manager data directory, put the chosen export alone in the seed’s backup directory, start the seed (it imports the newest export beside an empty store), then the others with empty directories. A restore starts a new store, so the epoch restarts with it.
  • Members far apart. The lease and Raft timers assume one site; members in different regions want clusterTimers in the Manager’s overrides (cluster.timers in manager-overrides.json: leaseIntervalMs, leaseTimeoutMs, leaseFailures, raftTickMs), every one larger, and the workers’ orphan timeout must exceed the demotion window, leaseFailures × (leaseIntervalMs + leaseTimeoutMs).
  • Never downgrade the Manager image below the version a store was written with.

config.yaml’s cluster.distributionTls (certFile, keyFile, caFile; init --cluster-dist-cert/-key/-ca) runs the Erlang distribution between the members over TLS: each presents its certificate and verifies the peer’s against the CA, both ways. Without it the distribution is plaintext, protected by the security group alone.

The kit does this by default (manager_cluster.distribution_tls, true): each member gets a certificate for its fixed address, as IP and DNS subject alternative names, under the deployment’s own CA — the worker mTLS CA, minted for this alone when worker mTLS is off — and the pair and the CA are secrets the member’s host fetches at boot and hourly into /var/lib/norsk-mgr/cluster-dist-tls, which norsk-mgr mounts into the Manager container. A renewal (any apply in a certificate’s last year re-issues it) lands in that directory and is read for the next distribution connection; nothing restarts. Outside the kit, issue the three files yourself: the certificate’s subject alternative names must cover the host part of the member’s node name (the address or name in --cluster), it needs both server and client usage, and norsk-mgr mounts the directory the three files share, so rotate them in place rather than pointing config.yaml at new paths.

Every member keeps its own cluster event log beside its store: role changes and the reason (promoted at which epoch, demoted because the lease failed or another member leads), lease failures, members joining, rejoining after a wipe or retiring, the store’s leader changing, a takeover settling, replayed launches, a stale norsk-mgr. norsk-mgr manager cluster events merges every member’s log newest first (--member, --kind to filter; --scope local for one member’s own), the Cluster page shows the same timeline grouped by epoch, each member’s log rides the structured log under cluster.events, and the primary mirrors them into the audit log. A member that does not answer is listed beside the events rather than hidden: “c has nothing since 14:02” is itself part of the story. When something looks wrong, read this first.

13.6 Lost quorum: taking members out, and the Big Red Button

Section titled “13.6 Lost quorum: taking members out, and the Big Red Button”

Losing two members at once, or any split that leaves no majority, stops the cluster: the store cannot commit, the primary steps down, and norsk-mgr on the survivor answers 503 until a majority is back. Workers keep running what they were running; nothing new is launched or placed. If the lost members are not coming back soon, the survivor can be made a one-member cluster that keeps the state and leads alone, and the others rejoin — wiping their stale copies — as they return. The order, so that no two members can ever form a quorum of their own on the old state:

  1. On each member you can still reach but intend to take out: norsk-mgr manager cluster leave --yes. Its Manager marks its copy for discarding and stops; started again it runs as a follower only, joining nothing that has no leader, so two members that left never form a quorum between themselves.
  2. Stop those instances in the cloud console, if you can.
  3. On the survivor: norsk-mgr manager cluster force-standalone --yes (the Cluster page’s red button, behind a typed confirmation). Before it fires, it asks every peer it can still reach to leave, and its answer lists them as lockedOut and the rest as unreachable. The survivor’s Manager restarts alone at the next generation and the production is served again within the usual failover budget. Refused while a primary is reachable: then the cluster is not lost, retire members instead.
  4. Bring the others back one at a time. A member whose copy is at a lower generation than a reachable peer’s wipes it and rejoins; when the last is back the cluster is whole again (manager cluster status shows generation, and standalone with the members still missing until then; the UI says so, and the timeline records standalone.forced, member.rejoined-after-wipe and standalone.cluster-restored).

The one case nothing can prevent is two members cut off from you but able to see each other: they keep a quorum between themselves and run on. The worker epoch fence keeps them off the workers once the survivor leads at a higher epoch, and they step down and rejoin the moment they reach it.

13.7 Looking at a host before it rejoins: the hold

Section titled “13.7 Looking at a host before it rejoins: the hold”

To inspect a host (gather logs, read its store, check its disk) without its Manager coming back into the cluster the moment it boots, hold it. Two ways to set the same hold:

  • an instance tag, from the cloud console or CLI with the instance stopped: id3as.Norsk.ManagerHold = <reason> on the manager host’s instance (norsk-mgr reads it from the instance metadata service at boot, every minute and before every start; the kit’s manager hosts expose their tags on IMDS for this);
  • on the host: norsk-mgr manager hold --reason "<why>", which also stops a running Manager (the container is kept).

Held, norsk-mgr manager status shows hold (source, reason, since, who), manager start answers 409, the Infrastructure page says held, and the audit log carries the hold and its release. norsk-mgr itself stays up, so manager status, manager cluster status, manager cluster events, the backup verbs and the UI all work on the held host. norsk-mgr manager release lets it back in (remove the tag first if that is how it was held; release says so while the tag still holds): the supervisor starts the Manager within seconds, and it joins — or wipes and rejoins — as above.

Since this change norsk-mgr owns the Manager container’s life entirely (there is no Docker restart policy on it): after a host stop or reboot it is norsk-mgr that starts the Manager, unless held. Hosts built before it carry a container with Docker’s old unless-stopped policy until the next manager start (or supervisor restart) recreates it; on such a host a tag hold stops the Manager a few seconds after boot rather than before it.

13.8 Adding a member, and making a cluster from one Manager

Section titled “13.8 Adding a member, and making a cluster from one Manager”

A Manager can be added to a running cluster — or to a single Manager, which is how a one-host deployment becomes a cluster — without touching any existing member’s config.yaml. On any member (the Cluster page’s “Add a member” form does the same):

norsk-mgr manager cluster add-member 203.0.113.10 # the new host's fixed private address

The cluster records that it expects manager@203.0.113.10 (manager cluster status lists it under configured until it joins; the timeline says member.added), every member’s norsk-mgr takes it into its configuration (a cluster-members.json beside config.yaml, merged over the kit’s cluster block on every read, so a launcher-config reload does not lose it), and the answer prints what to run on the new host: the init --cluster with the existing members first and the new one as --cluster-self, then manager start. It joins as a standby. Retiring it (manager cluster remove) forgets it everywhere. With distribution TLS on, the new host needs a certificate for its address under the deployment CA first (the kit’s scripts/mint-member-cert.sh).

With the deployment kit, the Manager can launch the host too: norsk-mgr manager cluster launch-member 203.0.113.10 --cert <pem> --key <pem> (or the same form with “launch the host too”). The kit publishes a member-launch bundle — what a manager host of the deployment is made of — and config.yaml’s managerLaunch.bundleUri names it; norsk-mgr writes the member’s certificate and key to Secrets Manager (the key never rides user-data and is never logged or audited), writes its launcher-config parameter, creates its data volume, launches the instance with the fixed address, and records the member as above. The host boots, installs and joins on its own. Such a host is not Terraform-managed (it carries id3as.Norsk.ManagerMember=<label>): retiring the member does not terminate it, and it has no DNS name or sign-in of its own — operators sign in through a Terraform-managed member’s address and it forwards. Without the bundle the verb answers 409 and names the manual path.

If the launched host boots but does not join, cloud-init on it has carried on past a failed step: cloud-init status and the tail of /var/log/cloud-init-output.log name it (the first live launch failed its certificate fetch, which the kit’s grant now covers). Fix the cause, re-run the step (systemctl start norsk-mgr-cluster-dist-tls.service for the certificates) and norsk-mgr manager start on the host: it joins from there; nothing has to be relaunched.

13.9 Lease-protected jobs, and maintenance on a single Manager

Section titled “13.9 Lease-protected jobs, and maintenance on a single Manager”

Some products hold something that must never be in use by two encoders at once: an Akamai entry point, a multicast address, a SIP number. Such a product declares it in its manifest (lease: { protected: true, graceMs? }), and the Manager — which does not need to know what the thing is — applies one rule to its instances:

  • Lost uncleanly, it waits. When the worker running one drops off (its node stops answering, the Manager restarts or fails over without hearing from it), the instance is not started anywhere else until its grace has passed since the Manager last knew the worker alive (the product’s graceMs, else the deployment’s nodeControllerConfig.leaseGraceMs, 60 s by default). The production shows Waiting on a job lease with the time it may start again, and the AutoManager tries again then; nothing counts against the restart ceiling.
  • The worker stops it first. A worker that has had no lease acknowledgement from a live Manager for its lease deadline (norsk-ctl worker --lease-deadline, 30 s by default) stops its lease-protected jobs and leaves every other job running. The deadline is shorter than the grace, so by the time the Manager may start the instance elsewhere, the old one has stopped.
  • A known exit is free at once. A stop, a crash the worker reports, or a worker that comes back without the instance: the Manager knows it is not running and starts it again at once.
  • Never two at once. While a worker runs it, starting it on another node is refused — including an overlap migration; give such a product the default stopThenStart.

norsk-mgr manager leases status (and the Cluster page) lists every lease held: the job, its holder, and when it may start again if the holder is gone.

Taking a single Manager down. With one Manager, its maintenance would otherwise stop every lease-protected job at the deadline. Pause leases first:

norsk-mgr manager leases pause --reason "patching the Manager host"
# … stop the Manager, do the work, start it again …
norsk-mgr manager leases resume

While paused, workers keep their lease-protected jobs without refreshing, and no lease-protected instance whose worker was lost is started again — past its grace or not — until you resume; new launches are not refused. The pause is stored with the deployment’s state, so it holds across the Manager’s restart, and each worker keeps it on disk across its own. Resuming restarts every grace from the resume. Both are admin actions and are audited. In a cluster there is no need: the other members keep the leases alive through a member’s maintenance.

13.10 Fleet discovery: replicas in every deployment

Section titled “13.10 Fleet discovery: replicas in every deployment”

Fleet discovery (norsk-discovery) is how a workflow on one worker finds a name another worker publishes — an SRT output by name, a Norsk link to another engine. Each worker runs a node discovery daemon, and each node declares itself to the deployment’s replicas, which hold the fleet’s whole directory and agree, by majority, when a node has died. A replica keeps that directory in a state file, so a replica that restarts restores it — including the nodes that died while it was down, which it then buries. While a replica is down the nodes keep resolving what they already know; what waits is anything published or lost in the meantime.

There are four ways to run the replicas, one code path:

DeploymentReplicas
A Manager clusterone beside each member’s norsk-mgr
One Manager, cloudthe manager host and two small replica hosts, each in its own availability zone (the kit’s discovery_replica_hosts)
One Manager, on-premthe manager host and two machines you name
One Manager, smallthe manager host alone: survives its restart, not its loss

On by default. norsk-mgr runs its replica unless told not to: NORSK_MGR_DISCOVERY_ENABLE=0, or discovery: { enabled: false } in config.yaml (the kit’s discovery = false writes that). With no discovery section, or discovery: {}, it is a replica set of one. Open tcp 7001 from the workers to the replica hosts (the kit’s modules/security discovery, also on by default). Changing it through the kit rides the launcher config: a norsk-mgr restart, not a new host.

A Manager cluster. With manager_cluster and discovery, the kit lists the members as the replica set and each member as its own self:

discovery:
replicas:
- { id: 10-0-1-10, address: 10.0.1.10:7001, gossip: 10.0.1.10:7002 }
- { id: 10-0-2-11, address: 10.0.2.11:7001, gossip: 10.0.2.11:7002 }
- { id: 10-0-3-12, address: 10.0.3.12:7001, gossip: 10.0.3.12:7002 }
self: 10-0-1-10

The set must be odd: a cluster grown to four uses its first three, and the fourth member runs no replica. The members reconcile on 7001 and gossip on 7002, between themselves only.

One tier secret. Every replica holds the same tier secret, the key each node’s identity token is minted from. The Manager keeps the first one published and hands it to every norsk-mgr, active or standby, which runs its replica with it, so a failover to another member, or a rebuilt host, keeps every node’s token: no node daemon is recreated. norsk-mgr manager discovery status says whether this host’s replica holds the secret in force (secretsAgreed), which replicas the Manager hands the nodes, and whether this host’s replica is running.

A single Manager in the cloud. The kit’s discovery_replica_hosts (with discovery = true) adds the two replica hosts and writes the set into the manager host’s config.yaml. Every replica’s config names every replica, so the manager host’s private address is fixed with it on: pass its current address as discovery_replica_hosts.manager_private_ip, or the manager host is replaced once. The replica hosts read the deployment’s tier secret from Secrets Manager; on a deployment that has already run discovery, pass the one in force as discovery_tier_secret (norsk-mgr manager discovery replica-run replica-1 shows it), or they refuse every node.

Replicas on hosts without norsk-mgr (on-prem). Name every replica in the manager host’s config.yaml, the manager host’s own as self:

discovery:
replicas:
- { id: mgr, address: 192.0.2.10:7001, gossip: 192.0.2.10:7002 }
- { id: r1, address: 192.0.2.11:7001, gossip: 192.0.2.11:7002 }
- { id: r2, address: 192.0.2.12:7001, gossip: 192.0.2.12:7002 }
self: mgr

Then, for each of the others, on the manager host:

norsk-mgr manager discovery replica-run r1

prints the docker run that starts replica r1 — the same image and replica-set settings norsk-mgr runs its own with, persistent under /var/lib/norsk-discovery, with the tier secret in force. Run it on that machine. It carries the tier secret, so it is an admin action and is audited. The machine must not be one that scales away: a replica set is fixed, and changing it means restarting every replica with the new list.

What a replica’s loss costs. With three, nothing: the other two carry the directory and still agree on deaths, and the lost one refills from them when it comes back. With one, a restart costs nothing either (the state file), but while it is down no new name is seen across workers and no dead node is buried, and losing the host for good loses the fleet’s directory until the nodes declare themselves to its replacement.

Rotating the tier secret. Three steps, each taken with one command and each a state every replica can run in, so no node’s token is refused at any point:

norsk-mgr manager discovery rotate-secret stage # every replica also accepts a new secret
norsk-mgr manager discovery rotate-secret flip # it mints node tokens; every node is sent its new one
norsk-mgr manager discovery rotate-secret finish # the old one is no longer accepted

stage makes the new secret (or takes --secret). Each norsk-mgr, active or standby, restarts its replica with each step within the settleSeconds the command prints (about 80 s), so wait that long before the next; manager discovery status shows the rotationPhase and, on each host, secretsAgreed. A step out of order is refused. Each replica restarts once per step and restores its state file, so a step costs a moment per replica, not the directory.

Replicas no norsk-mgr runs do not follow on their own: status lists them as externalReplicas. After each step, re-run each with replica-run <id> (the command it prints removes the old container first), then give the next step --external-rerun; flip and finish are refused until you do.

The kit’s cloud replica hosts (discovery_replica_hosts) read both secrets from Secrets Manager at every start, so they are re-run by writing those and restarting the hosts: the kit’s scripts/rotate-discovery-tier.sh <step> does that, and takes the step at the Manager, in one command (its runbook, section 7).