Install, upgrade, back up
The operator runbook for a deployment built from this kit
(examples/clean-account). It says what to run, what each step changes, and
— because this matters more than anything else in it — which operations
replace the manager host and what that costs you (§ 6). Read that section
before any day-2 change.
Commands are written as tofu …; terraform … runs the same configuration
(required_version >= 1.9 for the example root), but pick one binary per checkout and stay with
it — the kit’s README (“Tooling”) says why.
1. What you are installing
Section titled “1. What you are installing”- One manager host (EC2). It runs
norsk-mgr— the operator API, UI and CLI, as a systemd service — which in turn runs the Norsk Manager container (the orchestration daemon: job store, worker provisioning, worker supervision) and the front door (oauth2-proxy + nginx, the only thing on the public side). - A worker fleet the Manager owns. Workers are EC2 instances the Manager launches and terminates itself as productions need them. Terraform never creates or destroys a worker; it only builds what those launches assume (subnets, security groups, the worker instance role, the AMI to use).
- Substrate: a VPC with public subnets — and, for workers that must not carry a public IP, private subnets behind NAT, or your own VPC (§ 3.2 “Worker networking”) — two security groups, two instance roles, a handful of Secrets Manager secrets (worker keypair, license, admin password, and — with TLS — the certificates’ state), an S3 bucket, and optionally a Route53 record + Let’s Encrypt certificate.
Everything the deployment remembers lives under /var/lib/norsk-mgr on the
manager host — on its own EBS data volume (manager_data_volume_id),
which outlives the host: a replacement re-attaches it and finds its state.
norsk-mgr backup ships an archive of it to the deployment bucket daily
(§ 8). Keep § 6 and § 8 in mind from the start.
2. Prerequisites
Section titled “2. Prerequisites”| Need | Detail |
|---|---|
| An AWS account and credentials with rights to create VPCs, security groups, IAM roles/instance profiles, EC2 instances and AMIs, Secrets Manager secrets, S3 buckets, and (for TLS) Route53 records in your zone | Terraform and Packer both run under these credentials. Run them from an operator workstation, not from inside the deployment. |
| OpenTofu ≥ 1.9 (or Terraform ≥ 1.9) | examples/clean-account (its variable validations read other variables, which needs 1.9); the modules on their own need 1.6. |
| Packer ≥ 1.10 | Builds the two AMIs. packer init fetches the amazon plugin. |
| The AWS CLI, plus the Session Manager plugin | Host access is SSM only — no SSH keys are installed. aws ssm start-session --target <instance id> is how you get a shell. |
| Your Norsk Manager license — a JSON file | It must be a V2 license envelope; norsk-mgr refuses to start the Manager on anything else. It is stored in Secrets Manager and fetched by the host at boot. |
| The Manager image | norskvideo/norsk-manager:<tag> on Docker Hub, or the reference you were given. Pin a tag; do not deploy latest. |
| A norsk-mgr version | The version (or release-channel name) to bake into the manager-host AMI, or a direct binary URL if that is what you were given. |
| Optional: a Route53 hosted zone | If you want the operator UI on a hostname with a Let’s Encrypt certificate (dns_zone). Without it the Manager is plain HTTP on port 9081, and a browser cannot use the operator UI there: the session cookie is a __Host- cookie, which browsers store only over HTTPS, so sign-in loops back to the sign-in page; and products’ WebSockets (Studio’s live connection among them) fail the worker proxy’s Origin check even through a tunnel. Without a zone, drive the Manager with norsk-mgr on the host (§ 4). |
3. First install
Section titled “3. First install”3.1 Build the two AMIs
Section titled “3.1 Build the two AMIs”From packer/ (variables and OS/arch options: packer/README.md):
cd packerpacker init worker.pkr.hclpacker init manager-host.pkr.hcl
# Worker: docker, the norsk-ctl worker binary, a warm cache of the media# image and of the two proxy images (nginx, oauth2-proxy).packer build \ -var region=<your region> \ -var norsk_media_tag=<media image tag> \ -var norsk_ctl_version=<norsk-ctl version or channel, e.g. latest> \ worker.pkr.hcl
# Manager host: docker, norsk-mgr, an enabled norsk-mgr.service, a warm# cache of the Manager image and of the two proxy images.packer build \ -var region=<your region> \ -var norsk_manager_tag=<manager image tag — the value you will give manager_tag> \ -var norsk_mgr_version=<norsk-mgr version or channel, e.g. latest> \ manager-host.pkr.hcl # add -var norsk_mgr_binary_url=… if you were given a URLEach build prints an AMI id and appends it to manifest-worker.json /
manifest-manager-host.json. Build in the region you will deploy to (or
add -var 'ami_regions=[…]' to copy). Both binaries resolve from their
S3 release channels (latest is the blessed release, rc the hourly candidate, nightly the nightly build),
the Manager image from Docker Hub; for a deployment that must not reach
Docker Hub / quay.io at all, see § 5.5 (ECR mirror) and bake with
registry_auth=ecr.
3.2 Fill in the variables
Section titled “3.2 Fill in the variables”cd ../examples/clean-accountcp terraform.example.tfvars terraform.tfvars # gitignored; keep it out of version controlThe variables you must set:
| Variable | What |
|---|---|
region | The deployment region. Everything — host, workers, secrets — is single-region. |
deployment_name | Prefixes every resource; lowercase letters, digits, hyphens. |
admin_cidrs | The networks allowed to reach the operator UI/API. This is the only public surface: keep it to your office/VPN egress. |
manager_ami, worker_ami_x86_64 | The AMI ids from 3.1 (worker_ami_arm64 too if you built one). |
license_json | Your license, e.g. license_json = file("license.json"). |
Worth deciding now, because changing them later replaces the host (§ 6):
| Variable | What |
|---|---|
dns_zone, tls_email (and hostname) | Set dns_zone to your Route53 zone name and the UI becomes https://<hostname>.<dns_zone>/ with a Let’s Encrypt certificate issued at boot (DNS-01, via a Route53 grant scoped to that zone). hostname defaults to deployment_name. Adding, removing or changing it on a live deployment replaces the manager host, and the operator URL changes with it; workers already running keep the old URL (they send sign-in to it), so drain each one afterwards (norsk-mgr nodes drain <node> --yes) and the Manager moves its work to a fresh worker that has the new URL. |
admin_user, admin_password | The first operator login. Leave admin_password empty and one is generated. Either way it lives in Secrets Manager and is applied at boot; it is never written into the instance’s user-data. |
worker_support_user | Optional break-glass ssh login on every worker ({ name, ssh_authorized_key }). Absent, workers carry no credentials beyond their instance role; reach them by SSM. |
media_ingress | Inbound media ports on the worker security group (an SRT listener, say). Nothing is open by default. Only reachable from the internet with public workers — see below. |
worker_network | Where the workers live: public (the default), private-nat or existing. The decision is the next block. |
worker_grpc_transport | How the Manager talks to its workers. mtls, the default: the deployment gets its own CA, the Manager and the workers each hold a certificate from it, and each refuses a peer that holds none. plaintext: no TLS on that channel, which the security groups alone protect. A deployment installed before mtls became the default (2026-10-12), that never set this, ran plaintext: set worker_grpc_transport = "plaintext" before your next apply to keep it, or plan for that apply to replace the host and turn over the workers. |
Worker networking. The default puts workers in public subnets with a public IP each: media can reach them from the internet, operators’ browsers dial their previews directly, and there is nothing to pay for beyond the instances. If your policy says no instance gets a public address, pick one of the private modes — but read the limit first, because it matters more than the terraform:
A private worker cannot be reached from outside the VPC. It can send media out — pushed outputs, S3, log shipping, image pulls — but it cannot take media in from the internet (no SRT, RTMP or WHIP ingest from a public source) and cannot serve a product UI to a browser outside the VPC. The mode suits sources that arrive over a VPN, Direct Connect, a transit gateway or from inside the VPC, and operators who reach the workers the same way. A load balancer in front of private workers is not part of the kit.
| Mode | Builds | Choose it when |
|---|---|---|
public | the kit’s VPC, public subnets | Sources or viewers on the internet talk to the workers directly; you want the simplest, cheapest layout. Our own deployments run this. |
private-nat | the kit’s VPC, plus a private subnet per AZ and a NAT gateway; nat_per_az = true for one gateway per AZ | You are starting from a clean account and workers must have no public address. Costs about USD 35 a month per NAT gateway plus about USD 0.045 per GB through it — media pushed out through the NAT is the expensive part, and the obvious alternative (public workers) is free. S3 traffic bypasses the NAT through a gateway endpoint. |
existing | nothing — you pass vpc_id, manager_subnet_id (a public subnet: the manager host keeps a public address in every mode) and worker_subnet_ids | You already have a VPC with its own egress (a NAT, a transit gateway, a firewall) and would rather plug into it than be handed a second one. Your subnets must reach the AWS APIs, the container registry and your log and media destinations. worker_public_ip = true if the worker subnets are public ones. |
In either private mode set worker_advertise to private (operators
reach the workers over your routing, and the preview URLs carry private
IPs) or none; public is refused. The worker security group’s
direct-access rules then admit worker_access_cidrs — the VPC’s CIDR
unless you set it to the ranges your operators arrive from. media_ingress
and the example ingest security groups still work, for sources that can
route to the VPC. Worker TLS (worker_cert_mode = wildcard-route53) works
too: the per-worker DNS record carries the private address, which publishes
your VPC’s internal addressing in a public zone — acceptable to most,
forbidden by some. private-nat lists the NAT addresses as the
nat_public_ips output: they are what your workers’ outbound traffic
arrives from, for allowlisting at the far end. The mode is a bootstrap
setting (§ 6): moving a live deployment between modes replaces the manager
host, and the workers it launches afterwards land in the new subnets.
Tunables — changed later with an apply and a reload, no replacement (§ 6):
| Variable | What |
|---|---|
manager_tag (manager_image, manager_image_pull) | The Manager image. Required — there is no default, because a stale one silently hands an unpinned apply an old Manager: give the nightly label (<version>-<YYYY-MM-DD>-<sha8>) the shipped norsk-mgr was built against (its @norskvideo/norsk-manager-sdk pin), never a moving latest. |
pool_instance_types, pool_root_volume_gib | The default placement pool seeded on FIRST boot: which instance types workers may be, and their root volume. After that the pool is the API’s (norsk-mgr pool …), and these only matter to a fresh store. |
migration_launching_deadline_ms | How long a migration waits for the new runtime. |
worker_ami_x86_64 / worker_ami_arm64 | The worker AMIs (§ 5.3). |
data_volume_size_gb | The data volume; grows in place. |
backup_schedule | When the daily backup runs (§ 8). |
The full list with descriptions is variables.tf; the example tfvars shows
the optional groups (media egress via CloudFront, ECR instead of Docker Hub).
3.3 Apply
Section titled “3.3 Apply”tofu inittofu plan -out=plan.out # read it: for a first install everything is "create"tofu apply plan.outWhat happens on the host after apply: cloud-init mounts the data volume at
/var/lib/norsk-mgr (formatting it the first time); writes norsk-mgr’s
config.yaml, the Manager’s startup config and the per-region worker
networking file there; points the daemon at the launcher config (an SSM
parameter terraform wrote from the same values — § 6) and at the backup
target; fetches the license (and, with TLS, restores or issues the
certificate); restarts norsk-mgr; waits for its API; creates the admin
login from the password secret; starts the Manager container; enables the
daily backup timer. Allow a few minutes after apply returns. Progress is in
/var/log/cloud-init-output.log on the host.
State is local to the directory (terraform.tfstate) and contains the
license, the admin password and the workers’ keypair private key. Protect it
like a credential, and configure a remote backend if that is your policy
(versions.tf).
4. First login and first production
Section titled “4. First login and first production”tofu output operator_url # https://<fqdn>/ or http://<ip>:9081/tofu output admin_usertofu output -raw admin_password # sensitive: not shown without -rawOpen the URL, sign in with those. If nothing answers, the usual reason is
that your current IP is not in admin_cidrs (§ 10).
This needs dns_zone. On the plain-HTTP URL a browser cannot sign in:
the session cookie is a __Host- cookie, which browsers store only over
HTTPS, so the sign-in page comes straight back. An SSM port-forward to
localhost:9081 gets past sign-in (browsers treat localhost as secure),
but products’ WebSockets then fail the worker proxy’s Origin check, so
Studio loads without its live connection. Without a zone, use norsk-mgr on
the manager host (aws ssm start-session --target <instance>, then sudo -u norsk norsk-mgr …).
Then, in the UI or with the CLI (norsk-mgr, on the host or with an API
token from anywhere — norsk-mgr token create; API tokens walks through minting and revoking one):
- Pools — the
defaultpool is seeded frompool_instance_typeson first boot; edit or add pools under Pools (norsk-mgr pool …). The seed applies once; after that the pool store on the host is authoritative. - Products — register the product images you are licensed for
(
norsk-mgr product add --license-file <file>registers everything the license entitles). - Launch a production. The Manager launches the first worker for it;
you will see an EC2 instance tagged
id3as.Norsk.NodeType=workerappear. Never terminate those instances from the console — the fleet is Manager-owned; usenorsk-mgr nodes drain/terminateor the Nodes page.
4.1 Signing in through your OpenID Provider
Section titled “4.1 Signing in through your OpenID Provider”Operators sign in with the local accounts norsk-mgr keeps (admin_user and
any you add). You can put your own OpenID Provider beside them — Entra ID,
Okta, Keycloak, Google, anything that speaks OpenID Connect. The sign-in page
then offers both; the local accounts stay, and are the way in when the
provider is down. It needs TLS (dns_zone).
At the provider, register a confidential client for norsk-mgr:
- redirect URI
https://<hostname>.<dns_zone>/oauth2/callback; - to have signing out of norsk-mgr sign out at the provider too, the post-logout
redirect URI
https://<hostname>.<dns_zone>/, andsign_out_at_provider = trueinidentity_providerbelow. It is off by default: a provider refuses a post-logout redirect URI it has not been told about. The provider’s end-session endpoint must accept POST (Keycloak’s and Entra ID’s do; Okta’s is unconfirmed). Off, or where the provider publishes no end-session endpoint, sign-out ends only norsk-mgr’s session. Only a norsk-mgr release that signs out at the provider knows the setting; an older one ignores it, so sign-out there stays norsk-mgr’s alone; - the scopes
openid,profileandemail, plus whatever your provider needs to release group membership.emailis not optional: the front door signs nobody in unless the provider gives anemailclaim for them and marks it verified (email_verified), and the user sees only a bare Permission Denied. A directory whose users have no address, or only unverified ones, has to be made to send verified addresses first; - group membership in the ID token, under a claim you name below (often
groups). Where the provider can, have it send stable group ids rather than names: a role is given to exactly what the claim carries, and a name can be renamed, or reused for another group, underneath it. Entra ID sends object ids; Keycloak’s group-membership mapper and Okta send names, so there keep the name of a group that holds a role unique, and never rename it.
Then in tfvars:
identity_provider = { id = "corp-sso" # norsk-mgr's own name for it; choose once issuer_url = "https://login.example.com/realms/example" client_id = "norsk-mgr" scopes = ["openid", "profile", "email"] groups_claim = "groups" # sign_out_at_provider = true # sign out at the provider too (above)}identity_provider_client_secret = "<the client's secret>"issuer_url is the issuer exactly as the provider’s tokens state it (its
/.well-known/openid-configuration says). id is recorded against every
user who signs in through the provider, so a later change makes them
different users; pick a short, permanent name: at most 64 lowercase letters, digits,
., - and _, starting with a letter or digit. An id with a slash, backslash,
quote, whitespace or an invisible (zero-width, bidi) character, and the reserved
local, manager, api-token, ctl, indeterminate and issuer-refused, are
refused; any other is accepted with a warning. The secret goes to Secrets
Manager and is fetched onto the host at boot.
What a provider user can do. Signing in proves who they are and which
groups they are in; it gives them nothing by itself. A user whose groups hold
no role is refused, and the audit log
(/var/lib/norsk-mgr/logs/access-audit.jsonl) says why: no-mapped-group
when their groups map to nothing, no-groups-asserted when the provider sent
no groups at all — that one is almost always the provider’s configuration
(the claim name or a missing scope), not norsk-mgr’s. Groups are given roles
in norsk-mgr itself, not in tfvars — on the Access page (Provider
groups), or on the host:
norsk-mgr group roles news-ops viewer@fleet operator@production[class=news]norsk-mgr group listnorsk-mgr group roles news-ops --clear # removes the groupA member holds everything their groups hold, from their next request; no
restart and no new sign-in. fleet is the whole fleet, and the narrower
scopes are the ones an account’s scoped roles use. Name the group exactly as
the provider sends it.
Taking a role away from a group bites at once too. Taking a user out of a group at the provider does not: norsk-mgr learns their groups when they sign in, and a sign-in is not refreshed from the provider, so they keep what that group gave them until the sign-in ends (24 hours by default). End their sessions as well (below) to take it away now.
Workers differ in one direction. A session on a worker, made when someone
follows a link to it, carries only those of their groups that held a role at
that moment. A role given to a group afterwards reaches someone who already
has such a session only when it ends (with their norsk-mgr sign-in, within 24
hours) and they follow a link again; at norsk-mgr itself, and through its
/bundles/ links, it applies within a minute. Taking a role away from a group
applies everywhere within seconds, because workers decide from what each
group holds now.
A group may hold admin. Then whoever manages that group’s membership at
the provider decides who administers norsk-mgr, and an admin can create local
accounts, which outlive any later change to the provider or its groups. Guard
a group that holds admin as you guard the admin password. An admin through
a group never counts towards the last-admin rule either: norsk-mgr will not
let its last local account that can manage users lose that, because local
accounts are the way back in when the provider is down, misconfigured or
replaced. Keep one.
Entra ID: users in many groups. For a user in more than about 200
groups, Entra ID leaves the groups claim out of the token and sends a
pointer to Microsoft Graph instead (the groups overage). norsk-mgr does not
follow it, so that user is refused with no-groups-asserted. Either limit
the claim to the groups that matter (on the app registration, Token
configuration, groups claim, Groups assigned to the application, and
assign those groups to the enterprise application), or define app roles on
the registration, assign them to those groups, set groups_claim = "roles",
and give roles in norsk-mgr to the app roles’ values: the roles claim has
no overage.
Getting a groups claim, provider by provider. Most providers send no
group membership until they are told to, and a user then signs in to
nothing (no-groups-asserted). Where to set it:
- Entra ID: the app registration’s Token configuration, Add groups
claim, security groups, ID token, Group ID. The claim is
groupsand carries object ids. See above for users in many groups. - Okta: the OIDC app’s Sign On tab, OpenID Connect ID Token: a groups
claim named
groups, with a filter that matches the groups you will give roles to (Matches regex.*sends every group). On a custom authorization server, add agroupsclaim of type Groups, included in the ID token, instead. - Keycloak: a Group Membership mapper on the client (or a client scope
it uses): token claim name
groups, Add to ID token on, and Full group path off, unless you mean to give roles to paths such as/eng/ops. - Google: its sign-in puts no group membership in the ID token, so on its own it signs everyone in to nothing. Put a broker in front of it that adds groups (Keycloak or Dex, say) and configure that broker here instead.
Whatever the provider, groups_claim must name the claim it actually sends,
and a value given a role in norsk-mgr must match what the claim carries,
character for character.
Who has signed in. norsk-mgr remembers every provider user who has made
a request in the last 90 days (fleet.providerUserRetentionDays in its
config): the Access page’s Provider users, or norsk-mgr provider-user list, shows each with their groups and what those give them. They are
kept in /var/lib/norsk-mgr/provider-users.json (mode 0600): each user’s
provider id and subject, display name, email, groups (in the order sent; past 100,
only those that hold a role here and the earliest of the rest, to 100 unless
more than that hold a role), and when they were first and last
seen. A user unseen for the retention period is dropped from the file at its
next write, but the file is in every norsk-mgr backup archive (§ 8), so
those names and emails live as long as your backups do: set the backup
bucket’s retention with that in mind.
Ending a session now. End sessions on the Access page, or
norsk-mgr user revoke-sessions <name> for a local account and
norsk-mgr provider-user revoke-sessions <issuer> <sub> for a provider user,
ends every session they have: at norsk-mgr on their next request, and on
every worker within seconds. They may sign in again at once; to keep someone
out, delete the local account, or take them out of their groups at the
provider. A provider user of the configured provider can be ended even if
norsk-mgr does not list them. On a worker older than this release, ending
a user’s sessions locks them out of that worker entirely until the sign-in
length has passed, even if they sign in again.
Taking someone out of a group at the provider does not end their sessions;
end them as well. Ending a provider user’s sessions also revokes every API
token they own, and every token bound below one; the reply’s
revokedTokens says how many. If the tokens could not be revoked, the
sessions have still ended: the reply carries tokenRevocationFailed, the
CLI exits non-zero, and asking again revokes them.
Ending a local account’s sessions does not
touch its API tokens, which hold only what the account holds now: an admin
lists them with norsk-mgr token list and revokes each with
norsk-mgr token revoke <id>.
API tokens. A provider user may mint personal API tokens like anyone,
holding no more than their groups give them. Because norsk-mgr cannot ask the
provider about someone who is not signed in, such a token lasts 30 days at
most from the sign-in it was minted in (fleet.providerPatMaxDays sets it
lower), holds its role only while
the groups it was minted with still give it, and drops any group the user is
no longer in each time they sign in. Taking a group out of a role here bites
at once; taking a user out of a group at the provider bites at their next
sign-in, or when the token expires.
A token minted with another token is bound to it: it expires no later and
is revoked with it, so only signing in again starts a fresh 30 days.
Changing it later. Setting the client secret for the first time, or removing it, replaces the manager host (§ 6). The provider’s details are tunables: change them in tfvars, apply, then on the host
norsk-mgr reload-launcher-configThe reload recreates the front door on the new details before it answers,
with no restart. norsk-mgr’s own decisions take the new details as the file
is written, a moment before the front door is recreated: on a change that
keeps the id, the old directory can still sign someone in during that
moment, and they hold its groups’ roles until the recreate. Taking those
roles away first (below) closes it. The reply’s ingress says what the
front door now serves:
"outcome": "applied" is done; anything else comes with a message, and the
command exits non-zero. A provider whose clientSecretFile cannot be read is
refused, and nothing is written. norsk-mgr automanager status shows the
same ingress at any time: pendingRestart: true means what the front door
serves is not yet what is configured — a recreate still to come, a discovery
of the provider still pending or due a retry, or a configuration it cannot
render, which renderError then names.
What a change does to people already signed in:
idorissuer_urlchanged, the provider removed, or one added: every sign-in at norsk-mgr ends, local accounts’ included. The front door’s session key is replaced, because a session from the old configuration can no longer be attributed to a directory. Everyone signs in again; choose a quiet moment. If the new issuer cannot be discovered at the reload, the front door falls back to local accounts only, and changes the key again when a later discovery succeeds, so everyone signs in twice.client_id,scopesorgroups_claimchanged: the front door is recreated (a few seconds without it) and sessions survive.- A new
idmakes every provider user a different user, and every group role given under the oldidstops granting: roles are recorded against the provider’sid, not its directory. Workers stop accepting the oldid’s users within seconds, and their API tokens hold nothing.
Retiring or replacing a provider. Before you remove the provider, change
its id, or point it at another directory under the same id:
- Take the old groups’ roles away while the old provider is still the
configured one:
norsk-mgr group list, thennorsk-mgr group roles <group> --clearfor each. The API edits only the configured provider’s groups, so this cannot be done afterwards. A role left behind is keyed by the provider’sidand what the group claim carried, so it grants again to whatever directory is next configured under thatid, to whoever that directory puts in a group of that name. - Revoke the provider users’ API tokens, one by one:
norsk-mgr token list(a provider user’s token names its owner’sissuer), thennorsk-mgr token revoke <id>. Ending a user’s sessions (step 3) revokes their tokens too, but once the provider has changed only users norsk-mgr still lists can be ended, so revoke them here. Such a token holds nothing while its provider’sidis not the configured one, but a provider configured again under thatid— even with a differentissuer_url— revives it, and under the sameidit goes on holding what its groups give. Give a new provider a newid.
Then change it and reload, and afterwards:
- End the provider users’ sessions:
norsk-mgr provider-user list, thennorsk-mgr provider-user revoke-sessions <issuer> <sub>for each (or End sessions on the Access page). The new session key ends their sign-ins at norsk-mgr, but not their sessions on workers, which under the sameidgo on until they expire; this ends those within seconds. - Remove norsk-mgr’s client at the old provider.
The old provider’s users stay in Provider users, marked as not the current
provider and holding nothing, until they have gone unseen for the retention
period (90 days by default, as little as 7). That list is not a list of
tokens: a user it has forgotten may still own a live token, which
norsk-mgr token list still shows.
5. Day 2: upgrading
Section titled “5. Day 2: upgrading”Four things get upgraded independently. None of them is zero-downtime for the control plane, and there is no rollback other than re-applying the previous version, so: upgrade with no productions running where you can. Running workloads are designed to survive a Manager restart (the workers keep running their jobs and the Manager re-adopts them when it reconnects), but that is resilience, not a maintenance procedure.
The Manager’s store across an upgrade. The Manager daemon keeps
productions, runtimes, the audit log and node state in a store whose records
carry a schema version. The first start of a newer Manager on an older store
takes a backup of it before changing anything — into
<jobStoreBackupDir>-pre-upgrade, beside the rolling backups, where nothing
purges it — and then migrates it; the Manager’s log says so. A Manager never
starts on a store written by a newer Manager: it refuses, naming both schema
versions, rather than half-read records it does not understand. So rolling
the Manager image back past an upgrade that migrated the store means rolling
the store back too: stop the Manager, point its jobStoreBackupDir at the
-pre-upgrade directory, and move the live job-store directory’s contents
aside, leaving one placeholder file in it — the daemon treats a directory
that will not open as damaged and restores from its backups on start (an
EMPTY directory would instead start a fresh store).
norsk-mgr follows the same rules for the files it keeps. config.yaml
carries a configVersion (a file without one — what the deploy kit
writes — is version 0). A newer norsk-mgr reads an older file as it is and
upgrades it the first time it writes settings, keeping the file it started
from as config.yaml.v<n> beside it. An older norsk-mgr refuses to start
on a config.yaml stamped newer than it understands, so rolling the binary
back past such an upgrade means putting that config.yaml.v<n> back as
config.yaml. norsk-mgr’s operational state — cordons, migration history,
production templates and API tokens — lives in the Manager’s store
(once the Manager offers its key/value store) alongside the Manager’s own,
so it rolls back with the Manager.
How a release reaches you. Three artifacts, three variables:
| Artifact | Built by | Published as | Your variable |
|---|---|---|---|
| Manager image | norsk CI (nightly) | norskvideo/norsk-manager:<version>-<YYYY-MM-DD>-<sha8> on Docker Hub (plus a moving :nightly) | manager_tag (tfvars; § 5.1) |
norsk-mgr binary | norsk-ctl CI, nightly with the ctl binaries | s3://norsk.video/norsk-mgr/<version>/norsk-mgr-<version>-linux-<x64|arm64>; channel pointers latest, rc | norsk_mgr_version (packer; § 5.2) |
norsk-ctl binary | norsk-ctl CI, nightly | s3://norsk.video/norsk-ctl/<version>/…; the same channel pointers | norsk_ctl_version (packer; § 5.3) |
The two binaries are cut from one norsk-ctl commit, so their versions
share the <YYYY-MM-DD>-<sha7> stamp (norsk-mgr --version,
norsk-ctl --version); the Manager image tag norsk-mgr was built against
is its @norskvideo/norsk-manager-sdk pin — a release note names it, and
the root’s manager_tag default carries the one this kit shipped with.
Picking a Manager tag: docker manifest inspect norskvideo/norsk-manager:<tag> (or the Docker Hub tag list) shows what
exists; use the dated label, never latest/nightly, so a manager start cannot silently move you. Then: image → tfvars + apply + reload
(§ 5.1); binaries → bake + tfvars + apply (§ 5.2, § 5.3). With an ECR
mirror (§ 5.5) the image step is preceded by a mirror-to-ecr.sh run.
5.1 The Manager image (the orchestration daemon)
Section titled “5.1 The Manager image (the orchestration daemon)”The Manager runs as a container that norsk-mgr starts from
managerImage:managerTag in config.yaml, pulling per managerImagePull
(missing — pull only when the tag is not present locally — unless you set
otherwise). Once started it is supervised: a container found exited is
started again, and more than managerRestartLimit restarts within
managerRestartWindowSeconds (5 in 600 s unless you set otherwise) stop it
and make norsk-mgr exit for systemd to decide — the unit restarts it, and
after three such exits in an hour leaves it failed for you
(systemctl reset-failed norsk-mgr && systemctl start norsk-mgr).
norsk-mgr manager stop removes the container, which is what ends the
supervision; manager start begins it again.
From the registry (the normal path). Set manager_tag in
terraform.tfvars and apply — the plan updates the launcher-config
parameter and nothing else (no host replacement: § 6). Then, on the host:
norsk-mgr reload-launcher-config # writes config.yaml from the parameter; reports changedKeysnorsk-mgr manager stopnorsk-mgr manager start # pulls the new tag, starts itnorsk-mgr manager status(Editing managerTag: in /var/lib/norsk-mgr/config.yaml by hand still
works for a quick trial, but the next reload or daemon boot puts the
parameter’s value back — tfvars is the record.)
From an image tarball (no registry access, or a build you were sent).
Upload the docker saved image to the deployment bucket
(tofu output worker_config_bucket), for example under images/, set
manager_image_s3_uri to its s3:// URI in tfvars, and on the host:
/usr/local/bin/norsk-load-manager-image # fetch + docker load + tag as managerImage:managerTagnorsk-mgr manager stopnorsk-mgr manager startWith manager_image_s3_uri set the kit writes managerImagePull: never,
so a start never replaces the loaded image with the registry’s copy of the
same tag. The loader reads managerImage:managerTag from config.yaml at
run time, so a tag change is the registry flow above plus a re-run of the
loader; the S3 URI itself is baked into the script by cloud-init (a
bootstrap value), so the first time you adopt this channel on an existing
host either pass through a host replacement (§ 6) or run the aws s3 cp +
docker load + docker tag by hand once.
The tarball is the record. Its identity — the S3 object’s ETag, read at
plan time, or manager_image_s3_hash when you pass one — is in user-data,
so it is part of the bootstrap fingerprint: after a new tarball is uploaded
to the same key, the next tofu apply replaces the host (§ 6), and the new
host loads it at boot. Before this, a new tarball never reached an existing
host through terraform at all. Loading in place first (above) is still the
quick route; the apply that follows replaces the host regardless. A root
that uploads the tarball in the same apply must pass
manager_image_s3_hash = filemd5("<file>") (the ETag is unknown until
apply); the host then also verifies the download against it.
Check it took. docker inspect --format '{{.Id}}' norskvideo/norsk-manager:<tag> must change across the load; the Dashboard
shows the daemon’s build once it reconnects (§ 10, version banner).
5.2 norsk-mgr itself (the operator API/UI)
Section titled “5.2 norsk-mgr itself (the operator API/UI)”The supported path is a new manager-host AMI: rebuild with packer
(norsk_mgr_version — a channel such as latest, or the exact version a
release note names; the bake resolves it against the S3 store and verifies
the checksum), set manager_ami in tfvars, apply. A new AMI always
replaces the host, so do this together with anything else that replaces it
and follow § 6 and § 8 first.
If you were given a standalone norsk-mgr binary to run ahead of an AMI,
it can be swapped in place — the binary is self-contained (the UI is
embedded), and the Manager container is untouched:
cp /usr/local/bin/norsk-mgr /usr/local/bin/norsk-mgr.previnstall -o root -g root -m 0755 <new binary> /usr/local/bin/norsk-mgrsystemctl restart norsk-mgrnorsk-mgr --versionOn a Manager cluster do this for every host, standbys first and the active
member last, which is what scripts/roll-norsk-mgr.sh does over SSM
(--version resolves a version or channel against the S3 store as the
bake does and verifies the checksum; --url takes a binary from anywhere
the hosts can fetch). The Manager containers are not touched, so there is
no failover; each host is back as a standby or the active member before
the next is rolled:
scripts/roll-norsk-mgr.sh --region eu-west-1 --hosts i-aaa,i-bbb,i-ccc --version latestA swapped binary is lost on host replacement (the AMI’s copy comes back;
the data volume carries no binaries), so treat it as a bridge to the next
AMI, not a release mechanism. After any
systemctl restart norsk-mgr, check the front door (§ 10, “front door
down after a restart”).
5.3 Workers (a new worker AMI)
Section titled “5.3 Workers (a new worker AMI)”Bake with packer (worker.pkr.hcl, new norsk_ctl_version / media tag),
set worker_ami_x86_64 (and _arm64) in tfvars and apply. The AMI table
is a tunable: the plan updates the launcher-config parameter — every
release channel for both architectures points at the new AMI (the x86_64 id
stands in for aarch_64 when you have no arm AMI) — and the host is not
replaced. Then on the host:
norsk-mgr reload-launcher-config # reports managerOverridesChanged: true, restartRequired: truenorsk-mgr manager stop && norsk-mgr manager startnorsk-mgr manager overrides get # the table the Manager now runs withThe Manager reads the overrides file at container start, hence the restart.
(The old ritual — norsk-mgr manager overrides set overrides.json with a
hand-written table — still works for a one-off test of an AMI, but the
next reload or daemon boot writes the parameter’s table over it; tfvars is
the record.)
Only newly launched workers get the new AMI. Existing nodes keep the
old one until recycled: cordon the node (norsk-mgr nodes cordon <id>),
move its productions (norsk-mgr runtime migrate <runtimeRecordId> --to <nodeId>,
for products that support migration) or wait for them to end, then
norsk-mgr nodes terminate <id> --yes. The Nodes page does the same.
Drain a node shows the whole move, cordon to uncordon, with the CLI’s output at each step.
A just-created AMI is pending for a minute or so; a worker launch in that
window fails — wait for available before switching.
5.4 The media image the workers run
Section titled “5.4 The media image the workers run”Workers run the registry reference in norsk_media_image at whatever tag
the registry serves (the AMI only pre-pulls it as a warm cache). Product
images are whatever you registered with norsk-mgr product add. Neither
involves this kit; a new worker AMI (§ 5.3) is only needed to make first
boots fast again.
5.5 Running from an ECR mirror (no Docker Hub / quay.io egress)
Section titled “5.5 Running from an ECR mirror (no Docker Hub / quay.io egress)”Set ecr_mirror = true. The apply creates ECR repositories for the whole
image set (norsk, norsk-manager, norsk-moq-relay, nginx,
oauth2-proxy), grants both
instance roles pull, and points the deployment at them: manager_image and
norsk_media_image become <registry>/<prefix>norsk-manager and
…/norsk, container_registry becomes the registry host (the login
target; the repository prefix, if any, rides proxy_images_registry), and
the manager host
(cloud-init) and every worker (the Manager’s cloud-init, plus the timer the
AMI carries) docker login at boot and every 6 h — an ECR token lasts 12 h,
and a worker’s --image-store-max can evict an image a stale token could
not re-pull. The proxy images keep their upstream names (the software asks
for nginx:<tag> literally): each host pulls the mirror’s copy and tags it
so. The MoQ relay each worker runs for itself is handled the same way, at
the norsk release the worker’s norsk-ctl is pinned to: mirror-to-ecr.sh
mirrors it at --manager-tag unless --relay-tag names another
(norsk-ctl node-images on a worker prints the one it wants). Nothing in the kit can push; you fill the mirror:
tofu apply # the repositories firsttofu output ecr_mirror_command # → the exact invocation for this manager_tagbash packer/scripts/mirror-to-ecr.sh --registry <ecr_registry_url>[/<prefix>] \ --manager-tag <manager_tag> --media-tag <media tag> # your credentialsthen bake both AMIs with registry_auth=ecr and container_registry=<the ecr_packer_registry output> — the registry plus your repository prefix,
the string every image reference starts with (packer/README.md). Every
later manager_tag or
media-tag change is a mirror-to-ecr.sh run first, then the usual § 5.1 /
§ 5.3. Turning ecr_mirror on or off on a live deployment changes the
registry settings, which are bootstrap values: the host is replaced (§ 6).
6. Configuration changes and host replacement
Section titled “6. Configuration changes and host replacement”modules/manager-host sorts every input into one of two sets, and tofu plan tells you which one you touched.
Tunables — manager_image / manager_tag / manager_image_pull, the
pool seed (pool_instance_types, pool_root_volume_*),
migration_launching_deadline_ms, both worker AMI ids, and the
identity_provider details (§ 4.1). These are published
as the launcher config: a YAML document holding config.yaml’s content
plus a managerOverrides block with the worker AMI table, written by
terraform to an SSM parameter (tofu output launcher_config_uri;
launcher_config_uri in tfvars to bring your own s3:// or file://
document instead). norsk-mgr fetches it at every boot and on demand:
tofu apply # the plan updates aws_ssm_parameter.launcher_config only# on the host:norsk-mgr reload-launcher-config # fetch, validate the whole document, write config.yamlThe reply says what took effect: changedKeys; restartRequiredKeys (a
key the daemon reads only at boot — externalUrl is one — needs systemctl restart norsk-mgr); managerOverridesChanged (the AMI table — needs
norsk-mgr manager stop && norsk-mgr manager start); seedOnlyKeys
(placementPools: a seed for a fresh store, ignored on a store that has
one); ingress, when the front door was recreated for the identity
provider (§ 4.1). A document the validator rejects is 400 and changes
nothing.
Two consequences to know:
- The parameter is the record. With
NORSK_MGR_LAUNCHER_CONFIGset — every host this kit builds — a settings change made through the UI, the CLI (norsk-mgr settings set) or a hand edit ofconfig.yamllasts until the next reload or daemon boot, when the parameter’s content is written back. Change tunables in tfvars. - A boot that cannot reach the parameter carries on on the
config.yamlcloud-init wrote (the same content as the parameter at the time of the last replacement), with alauncher-config-fetch-failedaudit entry and a warning injournalctl -u norsk-mgr. Fix the cause (the instance role, the SSM endpoint) andnorsk-mgr reload-launcher-config.
Bootstrap — everything else that lands in user-data, plus the manager
AMI: manager_ami, the Manager’s sys.config except the AMI table
(worker_service_port, worker_support_user, worker_proxy_*,
worker_advertise, worker_image_store_max,
manager_initialisation_settle_ms, worker_orphan_terminate_after,
container_registry (+ region),
norsk_media_image, the worker cert secrets), the worker networking file
(subnets, security group, role, worker_network and worker_public_ip),
dns_zone / hostname / tls_email,
admin_user, the secret ARNs (so setting log_output_config, ice_servers or
identity_provider_client_secret for the
first time, or clearing one, replaces the host; changing its content does
not — § 12), manager_image_s3_uri, the launcher-config
and backup URIs, the data volume id, and any edit to the module’s
templates. Changing one of these replaces the manager host. The plan
shows it as terraform_data.bootstrap updating and
module.manager_host.aws_instance.this must be replaced — read the plan
before every apply. (Mechanically: the instance ignores user-data changes
and is replaced when the fingerprint of user-data-with-tunables-pinned
changes, so a tunable never replaces it and a bootstrap change always
does; tofu output -raw bootstrap_fingerprint in the module is that
value.)
Does not touch the host at all: admin_cidrs, ssh_cidrs,
media_ingress, moq_admin_cidrs, extra_worker_security_groups
(security-group rules), license_json and admin_password (new secret
versions; the host does not re-read them — § 7), tags,
data_volume_size_gb (grows the volume in place), backup_schedule
(rewrites the timer on the next replacement; edit
/etc/systemd/system/norsk-mgr-backup.timer by hand until then), the
media-egress and ECR groups.
What survives a replacement
-
Everything under
/var/lib/norsk-mgr— it is the data volume (aws_ebs_volume.manager_data,prevent_destroy):norsk-mgr’s store (pools, products and licenses, product templates and their tars, production templates, machine types, cordons, migrations, API tokens, extra logins), the Manager’s metadata store with all job and production history, its rotated exports, the overrides file, a loaded image’sconfig.yamlentry. The old host is stopped before the volume is detached, so the filesystem is clean; the new host mounts it frombootcmd, beforenorsk-mgrcan start, and refuses to start without it. -
The public address: the Elastic IP moves to the new instance, so the DNS record and
operator_urldo not change. -
The private address, with the Elastic IP (
allocate_eip, the default): the host sits on its own network interface, which the new instance takes over. That matters because a production’s jobs carry norsk-mgr’s address for workers (http://<private IP>:9333) in their product-template URLs, and the Manager keeps those jobs and sends them unchanged whenever it restarts or re-places one. Without the Elastic IP the host gets a new private address, and a production that is running across the replacement can no longer fetch its template on a new worker: delete productions first, or keep the default.Adopting this on a host built before it (a kit older than 2026-10-12): the first apply replaces the host onto the new interface, so the private address changes that once. Delete running productions before that apply and launch them again after. Later replacements keep the address.
-
With TLS: the Let’s Encrypt certificate state. It is pushed to a Secrets Manager secret on every issue/renew and restored before certbot runs on the new host, so a replacement re-issues nothing and burns no rate limit.
-
Everything terraform owns: secrets, roles, security groups, the bucket.
-
The workers that are running — as EC2 instances. See the last bullet.
What does NOT survive — what lives on the root volume:
- A swapped
norsk-mgrbinary (§ 5.2) and adocker loaded image tarball (§ 5.1: re-run/usr/local/bin/norsk-load-manager-image). - Running productions, as far as the control plane is concerned during the replacement: the new Manager starts from the same job store and re-adopts the workers it knew, but placement, migration and recovery are down for the minutes in between. Prefer a quiet window.
So a replacement is: § 8 backup (belt and braces) → tofu apply → wait for
cloud-init → verify (§ 10). Batch the changes that force one.
7. Rotating the admin password, the license and the provider’s client secret
Section titled “7. Rotating the admin password, the license and the provider’s client secret”All three are Secrets Manager secrets that the host reads once, at boot. A new secret version does not reach a running host by itself; the kit has no automatic rotation path today.
Admin password. Change it on the host, where it takes effect at once:
echo -n '<new password>' | norsk-mgr user set admin --password-stdinThen put the same value in admin_password in tfvars and apply (a new
secret version; the plan shows no host replacement) so that the next
replacement bootstraps the login with it. Additional operators are the
same command with another name, and norsk-mgr user role <name> <role>
gives each one a role (§ 9).
License. Update license_json in tfvars and apply — that stores the
new version, no host replacement. Then, on the host, fetch it into place
and restart the Manager (which stages the file on every start):
aws secretsmanager get-secret-value --region <region> \ --secret-id <deployment_name>-manager-license --query SecretString --output text \ > /var/lib/norsk-mgr/license-source.jsonchown norsk:norsk /var/lib/norsk-mgr/license-source.jsonnorsk-mgr manager stop && norsk-mgr manager startmanager start validates the file before it replaces the previously staged
copy, so a bad file leaves the working license in place and reports why.
The alternative — a host replacement, which fetches the secret fresh — costs
what § 6 says it costs.
OpenID Provider client secret (§ 4.1). Most providers let a client hold
two secrets at once, so add the new one at the provider first and the old
one keeps working until you are done. Put the new one in
identity_provider_client_secret and apply — a new secret version, no host
replacement. Then, on the host, fetch it into place and reload:
f=/var/lib/norsk-mgr/identity-provider-client-secret(umask 077; aws secretsmanager get-secret-value --region <region> \ --secret-id <deployment_name>-identity-provider-client-secret \ --query SecretString --output text > $f.new)chown norsk:norsk $f.new && chmod 0600 $f.new && mv $f.new $fnorsk-mgr reload-launcher-configThe file is replaced whole, by a rename, because a front door that finds it
empty or unreadable will not start (§ 10). The reload sees that the front
door was set up with another secret and recreates it; the reply says
"outcome": "applied", and sessions survive. Sign in through the provider
once, then delete the old secret there. Without the reload the front door
takes the new file within 15 minutes anyway.
A host created by an earlier kit has the file at mode 0640, and keeps it,
because cloud-init runs only once; norsk-mgr warns about it at every
start. Rotating as above sets 0600, or set it by hand:
chmod 0600 /var/lib/norsk-mgr/identity-provider-client-secret.
The role-store replica’s signing key. norsk-mgr replica rotate-key
replaces the key norsk-mgr signs the replica (every worker’s copy of who
may do what) and identity assertions with; the Manager endorses the new key
to every worker and the replica goes out again under it. Nothing is
re-enrolled, and the change is audited.
The fleet discovery tier secret, with discovery_replica_hosts. The two
replica hosts run no norsk-mgr, so they do not follow a rotation on their
own: each reads the tier secret, and during a rotation the second one, from
Secrets Manager at every start (<name>-discovery-tier,
<name>-discovery-tier-alt; the second holds none outside a rotation).
scripts/rotate-discovery-tier.sh takes each step at the Manager and carries
it to them: on the manager host it writes the secrets in force into the two
secrets (the role’s discovery-tier-rotation grant, so no secret reaches
your machine or SSM’s command history), then restarts each replica host’s
service in turn and waits for it to run again.
out="$(tofu output -json discovery_rotation)"scripts/rotate-discovery-tier.sh stage --from-output "$out" # wait the settle time it printsscripts/rotate-discovery-tier.sh flip --from-output "$out" # wait again: every node gets a new tokenscripts/rotate-discovery-tier.sh finish --from-output "$out"Each run first brings the replica hosts up to what is in force, so a step
taken by hand with norsk-mgr manager discovery rotate-secret is caught up
too, and sync does only that. A later tofu apply leaves both secrets as
the rotation wrote them. Without replica hosts (discovery_rotation is
null) every replica is one norsk-mgr runs, and norsk-mgr manager discovery rotate-secret alone rotates.
8. Backup and restore
Section titled “8. Backup and restore”Three layers, innermost first:
-
The Manager’s own store exports. After every change the daemon writes a whole-store export,
manager/data/jobStoreBackup/<id>.khepri(one consistent file; the newest three are kept, then hourly for a day and daily for a week) and runsnorsk-mgr’s hook,manager_backup.sh, which copies it tomanager/data/jobStoreSnapshots/<utc>-<id>.khepri(keeping seven). Same volume — this is what lets the daemon rebuild a store it lost, and whatnorsk-mgr backupships. -
The data volume (§ 6): survives the host. It does not survive the AZ, an accidental
tofu destroywithprevent_destroyremoved, or a filesystem you broke by hand. -
norsk-mgr backup— an archive of the whole store to a URI, off the host. The kit runs it daily (norsk-mgr-backup.timer, 00:00 UTC + a jitter;backup_schedulein tfvars) tos3://<deployment bucket>/norsk-mgr/backups/(tofu output backup_target_uri;backup_target_uriin tfvars for another prefix — with its own IAM grant — or afile://path). By hand, any time:Terminal window norsk-mgr backup # to the configured targetnorsk-mgr backup --to file:///tmp/b # somewhere elseThe archive is
norsk-mgr-backup-<utc>.tar.gz:store/(everything under/var/lib/norsk-mgrexcept logs, the extracted UI assets and thetls/copy of the certificate),job-store-backup/(one export: the newest hook snapshot, else the daemon’s newest),job-store-live/(the live store directory, only when the Manager is stopped — a running store is never copied), and amanifest.jsonwith versions and every file’s size and sha256.journalctl -u norsk-mgr-backuphas the timer’s runs.
Restore — onto this host after a disaster, or onto a new deployment’s host to move a Manager. Stop the Manager first; the daemon may stay up but is restarted afterwards:
norsk-mgr manager stopnorsk-mgr restore --from s3://<bucket>/norsk-mgr/backups/norsk-mgr-backup-<utc>.tar.gz --yessystemctl restart norsk-mgrnorsk-mgr manager startrestore refuses while the Manager container is running (--force to
override — do not), keeps whatever it replaces under
/var/lib/norsk-mgr/.pre-restore-<utc>/, and puts the store back one of
two ways: the archived live directory when the backup was taken with the
Manager stopped, otherwise the export alone in the daemon’s backup
directory with the live directory left empty, which makes the Manager
import it on that manager start (it logs Metadata store is empty but exports exist; restoring the newest). Run it as root (sudo) so file
ownership is restored; the Manager’s own tree is made writable for its
container user either way. A cluster member (§ 13) restores through the
seed: every member stopped and emptied, the restore on the seed, the seed
started first.
Restoring config.yaml from a backup is transient when a launcher config
is configured (§ 6): the next reload or boot writes the parameter’s content
back. Restoring manager/data from a different deployment brings back
that deployment’s view of which workers exist; expect the daemon to
reconcile against what EC2 actually reports and to raise alerts for nodes
that are gone.
Before this kit’s data volume (a host built from a commit older than
aws_ebs_volume.manager_data): § 11 is the one-time adoption.
9. Limitations
Section titled “9. Limitations”- One manager host by default. One
norsk-mgr, one Manager container, one data volume in one AZ: the control plane is a single point of failure. Workers keep running their jobs while it is down, but nothing places, migrates or recovers until it is back. Three hosts as a cluster (§ 13) is how to remove it; the kit’s root composes one host today, and the module takes the cluster block for a root that composes three. - One region. Host, workers, secrets and the worker networking config
are all in
region. - Private workers take no inbound from outside the VPC (§ 3.2 “Worker networking”): no internet ingest, no product UI for an outside browser. There is no load balancer or ingress proxy in the kit for that case.
- Roles, but no tenancy. Logins and API tokens hold a role (
viewer,operator,builder,admin), optionally scoped to part of the fleet, and enforcement is permissive until you setaccessEnforcement: enforced. A deployment is still one team’s. - Port 9333 is open to the worker fleet, but the API is gated. Workers
fetch product-template tars (per-launch token) and workdir assets (by sha)
from
norsk-mgron port 9333, which is open to the whole worker security group; every other/api/*route needs a bearer token, the mgr-nginx proxy secret, or the proxy secret presented on loopback by the on-host CLI, so a worker — or a product container it runs — gets 401 elsewhere. - Backups are daily and the restore is manual (§ 8); no automatic license/password rotation (§ 7).
- The Manager’s
sys.configis still delivered by host replacement (§ 6): the worker proxy/advertise/support-user/image-store/registry knobs live there and are not part of the launcher config yet. Only the AMI table rides the overrides channel. - With ECR as the registry, the login is refreshed every 6 h by a timer in
both AMIs (
norsk-ecr-login.timer; a token lasts 12 h). A worker built from an AMI older than that timer still has only its boot-time login, and cannot re-pull an image it evicts after 12 hours — rebake (§ 5.3) or recycle such workers within the day. - With worker TLS (
worker_cert_mode = wildcard-route53), each worker registers a DNS record at boot and nothing removes it on termination; the delegated zone grows by one record per worker ever launched. Harmless (names encode the IP) but untidy.
10. Troubleshooting
Section titled “10. Troubleshooting”Where things are on the host (aws ssm start-session --target <instance id>; find the id in the EC2 console by the <deployment_name>-manager Name
tag — it changes on every replacement):
| What | Where |
|---|---|
norsk-mgr (operator API, proxy lifecycle, Manager lifecycle) | journalctl -u norsk-mgr -f; systemctl status norsk-mgr |
| The Manager container | docker ps (norsk-manager); docker logs norsk-manager; its log files under /var/lib/norsk-mgr/manager/logs/ |
norsk-mgr’s structured log | /var/lib/norsk-mgr/logs/norsk-mgr/debug.json — one JSON object per line, each API request’s lines sharing its requestId (the X-Request-Id response header). The Manager’s own debug.json (under /var/lib/norsk-mgr/manager/logs/) logs each call norsk-mgr makes while serving a request under the same requestId, with the gRPC method as grpcMethod |
| The front door | docker ps (the proxy and nginx containers) |
| Boot / cloud-init | /var/log/cloud-init-output.log; cloud-init status |
| Configuration | /var/lib/norsk-mgr/config.yaml, manager.sys.config, worker-config.json; the launcher config the daemon fetches: aws ssm get-parameter --name /norsk-mgr/<deployment>/launcher-config --query Parameter.Value --output text |
| The data volume | findmnt /var/lib/norsk-mgr (must show the nvme…/xvdf device, not the root disk); /usr/local/bin/norsk-data-volume re-runs the mount |
| Backups | systemctl list-timers norsk-mgr-backup.timer; journalctl -u norsk-mgr-backup; aws s3 ls <backup_target_uri>/ |
| A worker | aws ssm start-session --target <worker instance id> → journalctl -u norsk-ctl-worker -f, docker ps |
Health and metrics.
GET /api/health— on the host without any credential (curl http://127.0.0.1:9333/api/health), or through the front door with a login or an API token (curl -H "Authorization: Bearer <token>" https://<fqdn>/api/health;norsk-mgr token create probe --role viewermints one).ok= API up, connected to the Manager, every store readable.degraded(still HTTP 200) = up but the Manager channel is down or a store file is unreadable — the body says which (daemon,store,unreadableStores).down(503) = boot failed or still starting, withreason.GET /api/manager/client(ornorsk-mgr manager client) — is the Manager daemon answering.norsk-mgr manager status— is the container running.norsk-mgr automanager status— placement engine connection and counters.GET /metrics— Prometheus text format, every seriesnorsk_mgr_*. Served on the host’s loopback (a scraper on the host) and through the front door to a logged-in session; anyone else gets 403 unless the service runs withNORSK_MGR_METRICS_OPEN=1.norsk_mgr_daemon_connected,norsk_mgr_store_unreadable,norsk_mgr_nodes{state}andnorsk_mgr_alerts_active{severity}are the ones to alert on. The Manager daemon’s own metrics follow them in the same scrape (§ 12.5).- Version banner. The Dashboard shows a banner when the Manager daemon’s
wire-protocol version differs from
norsk-mgr’s. Minor skew: they interoperate; fields one side added are ignored by the other — upgrade the lagging side (§ 5.1 or § 5.2) when convenient. Major skew:norsk-mgrhas refused to connect; health reportsdegradedwithdaemon: disconnecteduntil one side is upgraded so the majors match, thensystemctl restart norsk-mgr. Unknown: the daemon predates the handshake and is assumed compatible.
norsk-mgr will not start; systemctl status says a dependency failed
(var-lib-norsk\x2dmgr.mount). The data volume is not mounted, and the
unit is meant to refuse rather than build a second store on the root disk.
lsblk — is the volume attached (tofu output manager_data_volume_id,
aws ec2 describe-volumes --volume-ids <id>)? If attached, run
/usr/local/bin/norsk-data-volume and read what it says (it waits up to
five minutes for the device, formats only a blank one, and adds the fstab
line by UUID). If a replacement ran while the old host still held the
volume, the attachment is in the plan’s output; tofu apply again.
launcher-config-fetch-failed in the audit log / “launcher config NOT
applied” in the journal. The daemon booted on the local config.yaml
because the parameter could not be read: the instance role lacks
ssm:GetParameter on it (the plan shows the LauncherConfigRead
statement), the region is wrong (AWS_REGION in the service drop-in), or
launcher_config_uri points somewhere that does not exist. Fix, then
norsk-mgr reload-launcher-config — no restart needed for the fetch itself.
The UI does not answer at all. First suspect: your current IP is not in
admin_cidrs (operator addresses drift). Check the manager security group
in the console, or fix admin_cidrs and apply — that is a security-group
change only, no host replacement. Then systemctl status norsk-mgr and
docker ps on the host.
Front door down: port 443/9081 refuses connections. Two causes, and the
journal tells them apart. On the host the CLI still works either way: it
reaches norsk-mgr on loopback, not through the front door.
- The identity provider’s client secret. The front door will not start
with a provider whose
clientSecretFileit cannot read, or finds empty — a fetch that failed half way, a file left owned by root — because a sign-in page that cannot complete a provider sign-in would hide the fault.journalctl -u norsk-mgrsaysidentityProvider.clientSecretFile … could not be read, or is empty, andnorsk-mgr automanager statusshows it asingress.renderError. A reload refuses such a provider, and a running front door keeps its previous configuration, so this shows after a restart or a host replacement. Put the secret back (§ 7: ownernorsk, mode0600: onlynorsk-mgrreads it), or takeidentity_providerout of tfvars and apply — the provider’s details are a tunable, so that does not replace the host — thennorsk-mgr reload-launcher-config. The reload starts the front door; its reply says"outcome": "applied"once it is up, andcurl -sk https://127.0.0.1/answers. While the front door is down, for whatever reason, every reload tries to start it, and exits non-zero if it cannot. - A restart race. After
systemctl restart norsk-mgrthe new process can find the outgoing proxy containers still stopping and skip starting its own. The journal has no refusal;docker psshows the proxy containersExited. A secondsystemctl restart norsk-mgrstarts them cleanly.
Certificate not issued (TLS deployments). cloud-init’s certbot step
logs its failure and carries on, so the proxy comes up without a
certificate. On the host: certbot certificates, then re-run
certbot certonly --dns-route53 -d <fqdn> -n --agree-tos -m <tls_email> --deploy-hook /usr/local/bin/norsk-mgr-tls-deploy. The usual causes are
the Route53 zone id not matching dns_zone, or the instance role missing
the zone grant (both are terraform’s — re-check the plan).
The Manager container will not start. norsk-mgr manager start prints
the reason: an invalid license file (must be a V2 envelope), an image that
cannot be pulled (managerImagePull: never with no local image; a registry
that needs a login the host does not have), or — after a hand restore that
chowned manager/data — a permission error on the store’s log file
(norsk-mgr restore leaves that tree writable for the container; a tar xzf by hand must not chown it).
Workers never appear / launches fail. norsk-mgr alert list and the
placement log (norsk-mgr placement recent) name the reason: no AMI in the
image table for the instance’s architecture, an AMI still pending, an
instance type not offered in the subnet’s AZ, a service quota, a missing
iam:PassRole. The worker’s own boot is in its journalctl -u norsk-ctl-worker over SSM; a worker that boots but never connects usually
cannot reach the manager on its private IP (a security-group edit).
Something is being re-placed or stopped that should not be. Do not
change config.yaml through the API (norsk-mgr settings set, the
hot-spares and policy routes) while productions are running: those paths
reload the placement engine, which reconnects and re-hydrates from the
Manager. Make such changes with no productions running — through tfvars
and norsk-mgr reload-launcher-config (§ 6), which applies the same way.
11. Adopting the data volume on a host built before it existed
Section titled “11. Adopting the data volume on a host built before it existed”A deployment applied from a commit before aws_ebs_volume.manager_data
has its state on the root volume. Applying the current kit creates the
volume and — because the volume id and the daemon’s launcher-config /
backup environment are bootstrap values — replaces the host, whose
new cloud-init would format the empty volume and start with an empty
store. Move the state across first, once:
- On the current host, with no productions running:
norsk-mgr manager stop && norsk-mgr backup --to file:///tmp/pre-volumeand copy the archive off the host (aws s3 cpto the deployment bucket; the role can read it back from there). tofu plan: expect the volume, the SSM parameter, the IAM statements and the host replacement.tofu apply. The new host mounts a blank volume, boots empty (the launcher config gives it the sameconfig.yamlas before, but no pools, products, templates or history).- On the new host, after
cloud-init status --wait:Terminal window norsk-mgr manager stopnorsk-mgr restore --from s3://<bucket>/…/norsk-mgr-backup-<utc>.tar.gz --yessystemctl restart norsk-mgrnorsk-mgr manager startrestoreputs the store back onto the mounted volume; the Manager rebuilds its job store from the archived backup on that start (§ 8). - Verify (§ 10): the Products and Productions pages show what the old host
had;
norsk-mgr audit listshows the restore.
From then on, replacements keep the volume and step 3 is never needed again. (An EBS snapshot of the old root volume, taken before step 2, is the belt to this braces.)
12. Running productions: stop, pause, TURN, logs, metrics, configuration as files
Section titled “12. Running productions: stop, pause, TURN, logs, metrics, configuration as files”12.1 Stopping and pausing a runtime
Section titled “12.1 Stopping and pausing a runtime”A stop takes a runtime down and keeps it down: nothing restarts it until you activate it again, and it gives up its node (the capacity is free for other work, and an idle elastic node can be reaped).
norsk-mgr runtime stop <runtime-id> # docker's default 10 s to finishnorsk-mgr runtime stop <runtime-id> --grace 60 # a minute to finish cleanlynorsk-mgr runtime stop <runtime-id> --now # kill the containers at oncenorsk-mgr runtime activate <runtime-id> # place and start it againA pause stops it too, but keeps its place: the runtime stays bound to
its node, the capacity it reserved stays reserved, and the node is held so
it is not reaped or scaled in. resume starts it on that same node — or
places it afresh if the node has gone meanwhile.
norsk-mgr runtime pause <runtime-id> [--grace <s> | --now]norsk-mgr runtime resume <runtime-id>Use a pause for a short break you want back quickly and in the same place; a paused runtime still costs its share of the node. Use a stop for anything longer. A production’s page in the UI has the same actions on each of its runtimes (Stop, Stop now, Pause, Resume, Activate), and a paused runtime is badged as such.
--grace is 0–600 seconds: the time the product’s containers get between
the stop signal and being killed, for a clean end (closing a recording,
finishing a segment). A worker older than the Manager’s protocol 1.1 does
not know it and uses docker’s default.
12.2 TURN and STUN for WebRTC
Section titled “12.2 TURN and STUN for WebRTC”This kit runs no TURN server. If browsers or media servers sit behind NAT, run your own (coturn, or a hosted service) and give the list in tfvars:
ice_servers = [ { url = "turn:198.51.100.15:3478" # what the media server dials reported_url = "turn:turn.example.com:3478" # what the browser is told (omit if the same) username = "norsk" credential = "<password>" }, { url = "stun:stun.example.com:3478" },]The list is stored in Secrets Manager (TURN entries carry credentials),
fetched to /var/lib/norsk-mgr/ice-servers.json at boot, and handed at
launch to every product whose template declares GLOBAL_ICE_SERVERS
(Studio does). No field may contain whitespace, #, $, quotes or
backslashes. The credential is readable inside the product container, as
any launch parameter is.
Setting it for the first time replaces the manager host (§ 6). To change
it afterwards, tofu apply (a new secret version) and fetch it on the host,
as root; no restart is needed, because norsk-mgr reads the file at every
launch:
aws secretsmanager get-secret-value --region <region> --secret-id <deployment_name>-ice-servers \ --query SecretString --output text > /var/lib/norsk-mgr/ice-servers.jsonRunning productions keep the list they started with.
12.3 Logs, and shipping them
Section titled “12.3 Logs, and shipping them”Everything writes structured logs on its own host: norsk-mgr to
/var/lib/norsk-mgr/logs/norsk-mgr/debug.json, the Manager under
/var/lib/norsk-mgr/manager/logs/, each media engine on its worker. Each
line is one JSON object: time (epoch microseconds), level, text, and
optional data.
To ship them elsewhere, give a fluent-bit
output config: its [OUTPUT] sections, saying where logs go (Loki,
OpenSearch, CloudWatch Logs, S3, an HTTP endpoint — anything fluent-bit
writes to):
log_output_config = <<-EOT [OUTPUT] Name loki Match * Host loki.example.com Port 443 tls On http_user norsk http_passwd <password> labels job=norskEOTIt is stored in Secrets Manager and fetched to the manager host at boot. Then:
- the Manager container runs fluent-bit over its own logs and
norsk-mgr’s, so the host runs one log agent. Match on the tagsmanager-erlang,manager-stderrandmanager-mgr; - on workers, each node’s log agent ships the node’s logs with the same file (below).
What ships, by tag — match on these in Match:
| Tag | What |
|---|---|
manager-erlang, manager-stderr, manager-mgr | the Manager and norsk-mgr, on the manager host |
media-erlang, media-rust, media-stderr | each media engine |
worker-journal | the worker service’s own journal |
product.file.<json|text>.<instance>.<service> | a product’s declared log files, e.g. Studio’s JSON debug.log |
product.stdout.<json|text>.<service> | a product service’s declared stdout |
Every record carries a source field (norsk-mgr,
norsk-manager-erlang, norsk-media-erlang, …). Every engine and product
record also carries which runtime it came from:
NORSK_RUNTIME_ID—<production>.<replica>.<job>, e.g.studio4.0.main: filter on its prefix for a production, on the whole value for one runtime;NORSK_LAUNCH_IDandNORSK_NODE_ID— this launch of the runtime (new every launch, so a moved runtime’s records are told apart), and the node (-when not known). Records from before carriedNORSK_INSTANCE_IDinstead ofNORSK_LAUNCH_ID.
Which product services ship, and how, is declared by each product template
(its manifest’s logs: block); Norsk’s own products declare theirs.
Things to know:
-
Credentials. Either use an output that carries its own credentials — a user and password or a token in the file (Loki, OpenSearch with basic auth, an HTTP endpoint, Splunk, Datadog…), which is why the file is a secret — or an AWS output that uses the instance role.
-
CloudWatch Logs. Set
cloudwatch_log_group(and optionallycloudwatch_log_retention_days, default 30 — CloudWatch’s own default is to keep logs forever). The kit creates the group with that retention and grants the manager host’s role and the workers’ permission to write to that group and nothing else. Name the same group in the output withauto_create_group Off:cloudwatch_log_group = "/norsk/logs"log_output_config = <<-EOT[OUTPUT]Name cloudwatch_logsMatch *region eu-west-1log_group_name /norsk/logslog_stream_prefix norsk-auto_create_group OffEOTAny other AWS output (S3, SigV4 OpenSearch) needs a policy for it attached to the
<deployment_name>-managerand<deployment_name>-workerroles.To see one production’s logs, use Logs Insights on the group, not the streams (streams are per tag, and every production shares them):
fields @timestamp, @logStream, NORSK_RUNTIME_ID, @message| filter NORSK_RUNTIME_ID like /^studio4\./| sort @timestamp descstats count() by NORSK_RUNTIME_IDshows what is logging. -
Datadog. Its API key rides in the file, so there is nothing to grant and no
cloudwatch_log_group. SetHostto your Datadog site’s log intake (http-intake.logs.datadoghq.comfor US1,http-intake.logs.datadoghq.eufor EU, and so on):log_output_config = <<-EOT[OUTPUT]Name datadogMatch *Host http-intake.logs.datadoghq.euTLS onapikey <datadog API key>dd_service norskdd_source norskdd_tags env:productionEOTUse an API key for this deployment alone, so it can be revoked on its own.
NORSK_RUNTIME_IDand the other fields arrive as attributes: make it a facet to filter by production or runtime. Several[OUTPUT]sections with differentMatchpatterns send different tags to different places — e.g.Match product.*to one index andMatch media-*to another. -
Workers: the log agent. The Manager runs a
log-agentnode service on every worker node: one host-networked fluent-bit that alone fetches the output config from its secret, as the worker role, and that every engine forwards to. So the config is never in worker user data or in an engine container, AWS outputs authenticate from workers, and the worker service’s journal and declared product logs ship too. A node whose agent fails is not given work, and raises an alert after three failures. (Until 2026-10-13 a deployment could instead hand the config to every engine; that path is gone, and thelog_agentvariable with it.) -
Changing it. Setting it for the first time replaces the manager host (§ 6). After that,
tofu applywrites a new secret version; fetch it on the host as root and restart the Manager container, which reads it at start:Terminal window aws secretsmanager get-secret-value --region <region> --secret-id <deployment_name>-log-output-config \--query SecretString --output text > /var/lib/norsk-mgr/log-output.confnorsk-mgr manager stop && norsk-mgr manager startEach worker’s agent fetches the secret when it starts: new workers get the new version, and running ones when their agent restarts.
12.4 Workers whose Manager never comes back
Section titled “12.4 Workers whose Manager never comes back”A worker that hears nothing from its Manager for
worker_orphan_terminate_after seconds (default 86400, one day)
terminates its own instance. That covers a Manager host lost without a
restore, or a deployment torn down around workers it missed: without it,
those instances keep running and billing with nothing left to stop them.
The worker role may terminate its own instance and no other.
The timeout must outlast any Manager outage you expect, upgrades
included. If the Manager is down for longer than this, every worker in the
fleet terminates itself. Set it to 0 to switch it off. A worker built
from an AMI older than this feature ignores the setting and never
terminates itself.
12.5 The Manager daemon’s metrics
Section titled “12.5 The Manager daemon’s metrics”The Manager daemon serves Prometheus metrics on host port 6788, bound to
loopback: the Erlang VM’s, and the mailbox depth and memory of each of its
long-lived processes (norsk_manager_process_message_queue_len{process},
norsk_manager_process_memory_bytes{process}). A mailbox that keeps
growing is the process that cannot keep up — the node controller or a
cloud provider, typically. The port is
manager_metrics_port in modules/manager-host (0 turns it off).
norsk-mgr’s own /metrics appends them after its own series, so the host
stays one scrape target (§ 10). norsk_mgr_manager_metrics_up says whether
that part of the scrape worked.
Worth an alert beyond those in § 10: a mailbox that stays high;
norsk_mgr_placement_drift_corrected_total, which counts the times
placement’s view of which node runs what disagreed with the Manager’s and
was corrected (a steady rise means something keeps moving jobs behind
placement’s back); and norsk_mgr_manager_metrics_up == 0, which means the
daemon’s metrics have gone missing.
12.6 Configuration as files: export, review, apply
Section titled “12.6 Configuration as files: export, review, apply”norsk-mgr export writes the operator configuration — clouds, machine
types, pools, hot spares, product templates (with their tars), production
templates and node services — to a directory of files you can keep in
version control. The log agent node service that norsk-mgr creates from
the deployment’s own configuration is left out, because each deployment
creates its own, and --prune never removes it. norsk-mgr apply makes a Manager match such a directory:
norsk-mgr export ./mgr-config # everythingnorsk-mgr export ./mgr-config --only pools,hot-sparesnorsk-mgr apply ./mgr-config --dry-run # print the plan, change nothingnorsk-mgr apply ./mgr-config # create and updatenorsk-mgr apply ./mgr-config --prune # ...and delete what the directory does not nameapply touches only the kinds the directory has files for, and without
--prune it never deletes anything. Applying the same directory twice
changes nothing the second time. It is also how to copy a setup from one
deployment to another: export from staging, apply to production. Set
NORSK_MGR_URL and NORSK_MGR_TOKEN (an API token whose role may make those changes) to
run it from anywhere but the host.
Production templates also keep their own history: norsk-mgr production-template history <name> lists the last 20 revisions, show <name> --revision N prints one, and restore <name> N makes it current
again (as a new revision, so a restore can itself be undone). The template’s
page in the UI has the same history.
Node services keep the same history: norsk-mgr node-service history <name>, node-service revision <name> N and node-service restore <name> N, and the History button on the service’s card. A node runs the revision it
started with, so the history is also how you find out what an older node is
running. A restore, like any save, reaches only nodes that start
afterwards.
13. Three manager hosts: a Manager cluster
Section titled “13. Three manager hosts: a Manager cluster”One manager host is one point of failure. Three hosts, each running its own
norsk-mgr and Manager container, form a cluster: the Managers hold one
replicated store (a Raft quorum, so any two of the three carry on), exactly
one of them is primary and makes every decision, and the norsk-mgr
beside it is the active one. The other two norsk-mgrs are standby:
an operator who reaches any of the three sees the same deployment, because
a standby authenticates the operator itself and forwards the request to the
active one. Everything the deployment remembers — productions, pools,
templates, accounts and roles, API tokens, the proxy’s login accounts and
the session cookie secret — is in the replicated store; product-template
tars and uploaded assets are copied to every member as soon as their record
appears.
13.1 What a failure looks like
Section titled “13.1 What a failure looks like”Kill the primary (the host, its Manager, or both) and within about ten
seconds a survivor is primary at the next epoch and its norsk-mgr is
active. Running productions are adopted, not restarted: the new primary
re-dials every worker, finds each runtime where it was, and writes no
history. A worker that restarts while the member it was given is dead
registers with a survivor from the member list every worker carries. The
dead member, restarted, rejoins as a standby. While no member is primary
(the election) a standby’s UI shows “the Manager cluster is taking over”
and retries by itself; nothing needs doing.
The active norsk-mgr dying while its Manager stays primary is the
single-host story: the kit’s norsk-mgr.service is Restart=on-failure,
so systemd starts it again and it reconnects to the still-primary Manager
as active, hydrating as any restart does. The Manager container is
norsk-mgr’s to keep up: its supervisor starts a container it finds exited
(after a crash, a host stop or a reboot) within a ceiling, and starts none
while the Manager is held (§ 13.7). The standbys can tell this
case from a network fault between them and the active one: every
norsk-mgr announces itself to its own Manager every few seconds, and the
primary’s row in the cluster carries when its norsk-mgr last did. Once
three intervals pass without one, norsk-mgr manager cluster on any member
shows the primary’s controlPlaneStale: true (with controlPlaneAgeMs),
norsk-mgr manager client on a standby shows
activeControlPlane.down: true, a request through a standby is refused
with 503 STANDBY_ACTIVE_DOWN (“its service manager should be restarting
it”) rather than the generic STANDBY_ACTIVE_UNREACHABLE, and the
standby’s UI banner says the active norsk-mgr is down. All of it clears
by itself when the restarted process announces again; the Manager does not
fail over for this, and the epoch does not move.
13.2 Installing three hosts
Section titled “13.2 Installing three hosts”Each host is a manager host as in § 3, with its own data volume, and the
same norsk-mgr version. Then on every host, the same command with its
own --cluster-self:
norsk-mgr init --cluster 10.0.1.10,10.0.1.11,10.0.1.12 --cluster-self 10.0.1.11 \ --cluster-secret <secret> --cluster-cookie <cookie> ... (the usual init flags)--clusterlists every member, the seed first, as the others reach it: private IPs, or DNS names that resolve inside the VPC. The Manager’s node name ismanager@<host>, so the entry must not change for the life of the host.- The secret (what a standby’s forward carries) and the cookie (the
Managers’ Erlang cookie) are the same on every member. Leave both off on
the first host and
initmints and prints them; pass them to the others. --cluster-witnesses <host>names a member that votes and replicates but never becomes primary: an on-prem pair’s small third machine.
init writes the cluster block of config.yaml; from it norsk-mgr
starts the Manager container as that node with that cookie, publishes the
distribution ports, and writes the membership into the Manager’s overrides
file, so the Manager’s sys.config is the single-host one unchanged. The
hosts must reach one another on tcp 4369 and 9101–9110 (Erlang
distribution), 6787 and 6789 (the Manager) and 9333
(norsk-mgr); the kit’s modules/security opens exactly those, member to
member, with manager_cluster = true, and modules/manager-host takes the
same block as its cluster variable. The kit’s root (examples/clean-account)
composes the three with manager_cluster = {} (its README, “Manager
cluster”): each member in its own AZ with a fixed private address, its own
data volume, launcher config and hostname (<hostname>-a.<dns_zone> and so
on), the secret and cookie minted once and shared. Those hosts run init --cluster themselves at boot; the command above is for hosts you build by
hand.
Start the seed first with an empty Manager data directory, then the
others, each also empty: each joins the seed on its first tick and
replicates the store. norsk-mgr manager cluster on any member shows the
members, the primary and the epoch. The order matters only once: after
that any member may restart in any order.
13.3 Operating a cluster
Section titled “13.3 Operating a cluster”- One to three. Keep the existing Manager as the seed: run
init --clusteron it with itself first and restartnorsk-mgr; its store carries on. Bring the other two up with the same list and EMPTY data directories. - Retiring or replacing a member.
norsk-mgr manager cluster remove <node>retires it: the store’s membership drops it, every worker is told, and if it was the primary it hands leadership over first. Then stop itsnorsk-mgrand take it out of every member’s list (re-runinit --clusteror editconfig.yaml, and restart at leisure: the list is only used to find a cluster to join). A retired member that starts again with its old data joins as a new member, so wipe its Manager data directory before reusing the host. A replacement starts with a list naming itself and the current members; the others need not know it yet. - A member lost its disk. Give it an empty data directory and start it with the same list: it joins the cluster the others still have. The seed included: a seed with an empty store joins an existing cluster, and only restores a backup export when no other member has a cluster.
- A member restarted while the others are down comes up with its data and waits (its log says “cluster has no leader yet”), answering nothing until a majority is back. It never restores, never wipes, never leads alone.
- A member with data that is not the cluster’s (a restored copy, a Manager that ran alone) refuses to join and says so: wipe it to join, or make it the seed and give the others empty directories.
- Backup and restore.
norsk-mgr backupon any member ships that member’s copy; the Manager’s own exports (jobStoreBackupDir) are whole-store snapshots. To restore: stop every member, empty every Manager data directory, put the chosen export alone in the seed’s backup directory, start the seed (it imports the newest export beside an empty store), then the others with empty directories. A restore starts a new store, so the epoch restarts with it. - Members far apart. The lease and Raft timers assume one site; members
in different regions want
clusterTimersin the Manager’s overrides (cluster.timersinmanager-overrides.json:leaseIntervalMs,leaseTimeoutMs,leaseFailures,raftTickMs), every one larger, and the workers’ orphan timeout must exceed the demotion window,leaseFailures × (leaseIntervalMs + leaseTimeoutMs). - Never downgrade the Manager image below the version a store was written with.
13.4 TLS between the Managers
Section titled “13.4 TLS between the Managers”config.yaml’s cluster.distributionTls (certFile, keyFile, caFile;
init --cluster-dist-cert/-key/-ca) runs the Erlang distribution between
the members over TLS: each presents its certificate and verifies the
peer’s against the CA, both ways. Without it the distribution is plaintext,
protected by the security group alone.
The kit does this by default (manager_cluster.distribution_tls, true):
each member gets a certificate for its fixed address, as IP and DNS
subject alternative names, under the deployment’s own CA — the worker mTLS
CA, minted for this alone when worker mTLS is off — and the pair and the
CA are secrets the member’s host fetches at boot and hourly into
/var/lib/norsk-mgr/cluster-dist-tls, which norsk-mgr mounts into the
Manager container. A renewal (any apply in a certificate’s last year
re-issues it) lands in that directory and is read for the next distribution
connection; nothing restarts. Outside the kit, issue the three files
yourself: the certificate’s subject alternative names must cover the host
part of the member’s node name (the address or name in --cluster), it
needs both server and client usage, and norsk-mgr mounts the directory
the three files share, so rotate them in place rather than pointing
config.yaml at new paths.
13.5 Seeing what the cluster did, and why
Section titled “13.5 Seeing what the cluster did, and why”Every member keeps its own cluster event log beside its store: role
changes and the reason (promoted at which epoch, demoted because the lease
failed or another member leads), lease failures, members joining, rejoining
after a wipe or retiring, the store’s leader changing, a takeover settling,
replayed launches, a stale norsk-mgr. norsk-mgr manager cluster events
merges every member’s log newest first (--member, --kind to filter;
--scope local for one member’s own), the Cluster page shows the same
timeline grouped by epoch, each member’s log rides the structured log under
cluster.events, and the primary mirrors them into the audit log. A member
that does not answer is listed beside the events rather than hidden: “c has
nothing since 14:02” is itself part of the story. When something looks
wrong, read this first.
13.6 Lost quorum: taking members out, and the Big Red Button
Section titled “13.6 Lost quorum: taking members out, and the Big Red Button”Losing two members at once, or any split that leaves no majority, stops the
cluster: the store cannot commit, the primary steps down, and norsk-mgr
on the survivor answers 503 until a majority is back. Workers keep running
what they were running; nothing new is launched or placed. If the lost
members are not coming back soon, the survivor can be made a one-member
cluster that keeps the state and leads alone, and the others rejoin —
wiping their stale copies — as they return. The order, so that no two
members can ever form a quorum of their own on the old state:
- On each member you can still reach but intend to take out:
norsk-mgr manager cluster leave --yes. Its Manager marks its copy for discarding and stops; started again it runs as a follower only, joining nothing that has no leader, so two members that left never form a quorum between themselves. - Stop those instances in the cloud console, if you can.
- On the survivor:
norsk-mgr manager cluster force-standalone --yes(the Cluster page’s red button, behind a typed confirmation). Before it fires, it asks every peer it can still reach to leave, and its answer lists them aslockedOutand the rest asunreachable. The survivor’s Manager restarts alone at the next generation and the production is served again within the usual failover budget. Refused while a primary is reachable: then the cluster is not lost, retire members instead. - Bring the others back one at a time. A member whose copy is at a
lower generation than a reachable peer’s wipes it and rejoins; when the
last is back the cluster is whole again (
manager cluster statusshowsgeneration, andstandalonewith the members still missing until then; the UI says so, and the timeline recordsstandalone.forced,member.rejoined-after-wipeandstandalone.cluster-restored).
The one case nothing can prevent is two members cut off from you but able to see each other: they keep a quorum between themselves and run on. The worker epoch fence keeps them off the workers once the survivor leads at a higher epoch, and they step down and rejoin the moment they reach it.
13.7 Looking at a host before it rejoins: the hold
Section titled “13.7 Looking at a host before it rejoins: the hold”To inspect a host (gather logs, read its store, check its disk) without its Manager coming back into the cluster the moment it boots, hold it. Two ways to set the same hold:
- an instance tag, from the cloud console or CLI with the instance stopped:
id3as.Norsk.ManagerHold = <reason>on the manager host’s instance (norsk-mgrreads it from the instance metadata service at boot, every minute and before every start; the kit’s manager hosts expose their tags on IMDS for this); - on the host:
norsk-mgr manager hold --reason "<why>", which also stops a running Manager (the container is kept).
Held, norsk-mgr manager status shows hold (source, reason, since, who),
manager start answers 409, the Infrastructure page says held, and the
audit log carries the hold and its release. norsk-mgr itself stays up, so
manager status, manager cluster status, manager cluster events, the
backup verbs and the UI all work on the held host. norsk-mgr manager release lets it back in (remove the tag first if that is how it was held;
release says so while the tag still holds): the supervisor starts the
Manager within seconds, and it joins — or wipes and rejoins — as above.
Since this change norsk-mgr owns the Manager container’s life entirely
(there is no Docker restart policy on it): after a host stop or reboot it
is norsk-mgr that starts the Manager, unless held. Hosts built before it
carry a container with Docker’s old unless-stopped policy until the next
manager start (or supervisor restart) recreates it; on such a host a tag
hold stops the Manager a few seconds after boot rather than before it.
13.8 Adding a member, and making a cluster from one Manager
Section titled “13.8 Adding a member, and making a cluster from one Manager”A Manager can be added to a running cluster — or to a single Manager, which
is how a one-host deployment becomes a cluster — without touching any
existing member’s config.yaml. On any member (the Cluster page’s “Add a
member” form does the same):
norsk-mgr manager cluster add-member 203.0.113.10 # the new host's fixed private addressThe cluster records that it expects manager@203.0.113.10 (manager cluster status lists it under configured until it joins; the timeline
says member.added), every member’s norsk-mgr takes it into its
configuration (a cluster-members.json beside config.yaml, merged over
the kit’s cluster block on every read, so a launcher-config reload does not
lose it), and the answer prints what to run on the new host: the
init --cluster with the existing members first and the new one as
--cluster-self, then manager start. It joins as a standby. Retiring it
(manager cluster remove) forgets it everywhere. With distribution TLS on,
the new host needs a certificate for its address under the deployment CA
first (the kit’s scripts/mint-member-cert.sh).
With the deployment kit, the Manager can launch the host too:
norsk-mgr manager cluster launch-member 203.0.113.10 --cert <pem> --key <pem> (or the same form with “launch the host too”). The kit publishes a
member-launch bundle — what a manager host of the deployment is made of —
and config.yaml’s managerLaunch.bundleUri names it; norsk-mgr writes
the member’s certificate and key to Secrets Manager (the key never rides
user-data and is never logged or audited), writes its launcher-config
parameter, creates its data volume, launches the instance with the fixed
address, and records the member as above. The host boots, installs and
joins on its own. Such a host is not Terraform-managed (it carries
id3as.Norsk.ManagerMember=<label>): retiring the member does not
terminate it, and it has no DNS name or sign-in of its own — operators
sign in through a Terraform-managed member’s address and it forwards.
Without the bundle the verb answers 409 and names the manual path.
If the launched host boots but does not join, cloud-init on it has carried
on past a failed step: cloud-init status and the tail of
/var/log/cloud-init-output.log name it (the first live launch failed its
certificate fetch, which the kit’s grant now covers). Fix the cause, re-run
the step (systemctl start norsk-mgr-cluster-dist-tls.service for the
certificates) and norsk-mgr manager start on the host: it joins from
there; nothing has to be relaunched.
13.9 Lease-protected jobs, and maintenance on a single Manager
Section titled “13.9 Lease-protected jobs, and maintenance on a single Manager”Some products hold something that must never be in use by two encoders at
once: an Akamai entry point, a multicast address, a SIP number. Such a
product declares it in its manifest (lease: { protected: true, graceMs? }),
and the Manager — which does not need to know what the thing is — applies
one rule to its instances:
- Lost uncleanly, it waits. When the worker running one drops off (its
node stops answering, the Manager restarts or fails over without hearing
from it), the instance is not started anywhere else until its grace has
passed since the Manager last knew the worker alive (the product’s
graceMs, else the deployment’snodeControllerConfig.leaseGraceMs, 60 s by default). The production shows Waiting on a job lease with the time it may start again, and the AutoManager tries again then; nothing counts against the restart ceiling. - The worker stops it first. A worker that has had no lease
acknowledgement from a live Manager for its lease deadline (
norsk-ctl worker --lease-deadline, 30 s by default) stops its lease-protected jobs and leaves every other job running. The deadline is shorter than the grace, so by the time the Manager may start the instance elsewhere, the old one has stopped. - A known exit is free at once. A stop, a crash the worker reports, or a worker that comes back without the instance: the Manager knows it is not running and starts it again at once.
- Never two at once. While a worker runs it, starting it on another
node is refused — including an
overlapmigration; give such a product the defaultstopThenStart.
norsk-mgr manager leases status (and the Cluster page) lists every lease
held: the job, its holder, and when it may start again if the holder is gone.
Taking a single Manager down. With one Manager, its maintenance would otherwise stop every lease-protected job at the deadline. Pause leases first:
norsk-mgr manager leases pause --reason "patching the Manager host"# … stop the Manager, do the work, start it again …norsk-mgr manager leases resumeWhile paused, workers keep their lease-protected jobs without refreshing, and no lease-protected instance whose worker was lost is started again — past its grace or not — until you resume; new launches are not refused. The pause is stored with the deployment’s state, so it holds across the Manager’s restart, and each worker keeps it on disk across its own. Resuming restarts every grace from the resume. Both are admin actions and are audited. In a cluster there is no need: the other members keep the leases alive through a member’s maintenance.
13.10 Fleet discovery: replicas in every deployment
Section titled “13.10 Fleet discovery: replicas in every deployment”Fleet discovery (norsk-discovery) is how a workflow on one worker finds a name another worker publishes — an SRT output by name, a Norsk link to another engine. Each worker runs a node discovery daemon, and each node declares itself to the deployment’s replicas, which hold the fleet’s whole directory and agree, by majority, when a node has died. A replica keeps that directory in a state file, so a replica that restarts restores it — including the nodes that died while it was down, which it then buries. While a replica is down the nodes keep resolving what they already know; what waits is anything published or lost in the meantime.
There are four ways to run the replicas, one code path:
| Deployment | Replicas |
|---|---|
| A Manager cluster | one beside each member’s norsk-mgr |
| One Manager, cloud | the manager host and two small replica hosts, each in its own availability zone (the kit’s discovery_replica_hosts) |
| One Manager, on-prem | the manager host and two machines you name |
| One Manager, small | the manager host alone: survives its restart, not its loss |
On by default. norsk-mgr runs its replica unless told not to:
NORSK_MGR_DISCOVERY_ENABLE=0, or discovery: { enabled: false } in
config.yaml (the kit’s discovery = false writes that). With no discovery
section, or discovery: {}, it is a replica set of one. Open tcp 7001 from
the workers to the replica hosts (the kit’s modules/security discovery,
also on by default). Changing it through the kit rides the launcher config:
a norsk-mgr restart, not a new host.
A Manager cluster. With manager_cluster and discovery, the kit lists
the members as the replica set and each member as its own self:
discovery: replicas: - { id: 10-0-1-10, address: 10.0.1.10:7001, gossip: 10.0.1.10:7002 } - { id: 10-0-2-11, address: 10.0.2.11:7001, gossip: 10.0.2.11:7002 } - { id: 10-0-3-12, address: 10.0.3.12:7001, gossip: 10.0.3.12:7002 } self: 10-0-1-10The set must be odd: a cluster grown to four uses its first three, and the fourth member runs no replica. The members reconcile on 7001 and gossip on 7002, between themselves only.
One tier secret. Every replica holds the same tier secret, the key each
node’s identity token is minted from. The Manager keeps the first one
published and hands it to every norsk-mgr, active or standby, which runs its
replica with it, so a failover to another member, or a rebuilt host, keeps
every node’s token: no node daemon is recreated. norsk-mgr manager discovery status says whether this host’s replica holds the secret in force
(secretsAgreed), which replicas the Manager hands the nodes, and whether
this host’s replica is running.
A single Manager in the cloud. The kit’s discovery_replica_hosts
(with discovery = true) adds the two replica hosts and writes the set into
the manager host’s config.yaml. Every replica’s config names every replica,
so the manager host’s private address is fixed with it on: pass its current
address as discovery_replica_hosts.manager_private_ip, or the manager host
is replaced once. The replica hosts read the deployment’s tier secret from
Secrets Manager; on a deployment that has already run discovery, pass the
one in force as discovery_tier_secret (norsk-mgr manager discovery replica-run replica-1 shows it), or they refuse every node.
Replicas on hosts without norsk-mgr (on-prem). Name every replica in the
manager host’s config.yaml, the manager host’s own as self:
discovery: replicas: - { id: mgr, address: 192.0.2.10:7001, gossip: 192.0.2.10:7002 } - { id: r1, address: 192.0.2.11:7001, gossip: 192.0.2.11:7002 } - { id: r2, address: 192.0.2.12:7001, gossip: 192.0.2.12:7002 } self: mgrThen, for each of the others, on the manager host:
norsk-mgr manager discovery replica-run r1prints the docker run that starts replica r1 — the same image and
replica-set settings norsk-mgr runs its own with, persistent under
/var/lib/norsk-discovery, with the tier secret in force. Run it on that
machine. It carries the tier secret, so it is an admin action and is
audited. The machine must not be one that scales away: a replica set is
fixed, and changing it means restarting every replica with the new list.
What a replica’s loss costs. With three, nothing: the other two carry the directory and still agree on deaths, and the lost one refills from them when it comes back. With one, a restart costs nothing either (the state file), but while it is down no new name is seen across workers and no dead node is buried, and losing the host for good loses the fleet’s directory until the nodes declare themselves to its replacement.
Rotating the tier secret. Three steps, each taken with one command and each a state every replica can run in, so no node’s token is refused at any point:
norsk-mgr manager discovery rotate-secret stage # every replica also accepts a new secretnorsk-mgr manager discovery rotate-secret flip # it mints node tokens; every node is sent its new onenorsk-mgr manager discovery rotate-secret finish # the old one is no longer acceptedstage makes the new secret (or takes --secret). Each norsk-mgr, active
or standby, restarts its replica with each step within the settleSeconds
the command prints (about 80 s), so wait that long before the next;
manager discovery status shows the rotationPhase and, on each host,
secretsAgreed. A step out of order is refused. Each replica restarts once
per step and restores its state file, so a step costs a moment per replica,
not the directory.
Replicas no norsk-mgr runs do not follow on their own: status lists them
as externalReplicas. After each step, re-run each with replica-run <id>
(the command it prints removes the old container first), then give the next
step --external-rerun; flip and finish are refused until you do.
The kit’s cloud replica hosts (discovery_replica_hosts) read both secrets
from Secrets Manager at every start, so they are re-run by writing those and
restarting the hosts: the kit’s scripts/rotate-discovery-tier.sh <step>
does that, and takes the step at the Manager, in one command (its runbook,
section 7).