Manager host contract
The precise specification of the manager host: the single instance per deployment that runs the norsk-mgr operator executable, the Norsk Manager container, and the operator front door. IaC-agnostic, like the worker contract (worker node).
1. Role
Section titled “1. Role”One manager host per deployment. It runs three things:
norsk-mgr— a single self-contained executable (systemd service). The operator API/UI/CLI backend, and the process that owns everything else: it starts/stops the Manager container, hosts the AutoManager placement engine against it, and fronts per-job runtime UIs through its proxy.- The Manager container (
norskvideo/norsk-manager) — the orchestration daemon: durable job/bundle store, worker provisioning (EC2 API calls originate here), worker supervision, event stream. Driven entirely by norsk-mgr over local docker. - The front door — norsk-mgr’s bundled oauth2-proxy + nginx. This is the only public surface.
2. Pre-installed (the image’s job)
Section titled “2. Pre-installed (the image’s job)”| Requirement | Why |
|---|---|
| docker | the Manager container + proxy containers |
| systemd | supervises norsk-mgr (norsk-mgr.service, modelled on norsk-ctl’s unit) |
norsk-mgr binary on PATH | the operator runtime; version pinned per image build |
cloud CLI (aws) | secret/license staging at boot, registry login if needed |
Pre-pulled norskvideo/norsk-manager image | fast first boot |
3. Delivered at boot (cloud-init, from terraform / your IaC)
Section titled “3. Delivered at boot (cloud-init, from terraform / your IaC)”Unlike workers (whose boot config the Manager generates per-node), the manager host’s boot config comes from the IaC layer:
~/.norsk-mgr/config.yaml— operator config: license source, manager image/tag, placement pools seed, clouds config, bands/placement/hot-spare settings.- The Manager license — staged to
~/.norsk-mgr/manager/license.json(norsk-mgr validates it is a V2 envelope before replacing any previously staged copy, and bind-mounts it read-only into the container). Source it from Secrets Manager or S3 in cloud deployments. - Provider wiring — the two config channels of §4.
- Service start — enable + start
norsk-mgr.service. norsk-mgr then pulls/starts the Manager container and the proxy. - Registry login, when the host must pull from an authenticated registry.
Distinct from the worker case: a worker’s login is generated by the
Manager into its cloud-init, but nothing generates one here. norsk-mgr
shells out to
docker, so it uses whatever credential state the host already has, and the product images it pulls (product add --image ...) come from wherever the operator points it. If those live in a private registry, the IaC must log in as the user norsk-mgr runs as — a login performed as root does not give the service user credentials. Themodules/manager-hostcloud-init does this whencontainer_registry_regionis set:aws ecr get-login-password | docker loginat boot as root, as thenorskuser and intoDOCKER_CONFIG=/etc/norsk/docker, repeated every 6 h bynorsk-ecr-login.timer(an ECR token lasts 12 hours), which also keeps the proxy images (nginx,oauth2-proxy— referenced by their upstream names) tagged from the mirror’s copies. The AMI ships the same refresher (packer/files/norsk-ecr-login). - Bootstrap operator login — once
/api/readyanswers,norsk-mgr user set <user> --password <pw>creates the first oauth2-proxy login (idempotent; re-runs update the password). The password must not ride in user-data. User-data is readable by every container on the host over IMDS (the hop limit is 2 so the Manager container can reach its role credentials), by anyone withec2:DescribeInstanceAttribute, and it sits in/var/lib/cloud/instance/user-data.txt. Stage it as a Secrets Manager secret (thesecretsmodule’s<name>-admin-password), render only the ARN into user-data, grant the instance roleGetSecretValueon it, and fetch it at boot.modules/manager-hostdoes this, piping the secret intonorsk-mgr user set --password-stdinso it is never in a process’s argv.
4. The two provider config channels
Section titled “4. The two provider config channels”The Manager daemon learns its AWS surface through two channels; any IaC must produce both:
- Cloud/provider config (
ec2ProviderConfig): regions, the per-region × per-arch × per-channelimageVersionTableof worker AMI ids, the container registry (+ region, when ECR), the optional workersupportUserssh login, secret regions. This flows through norsk-mgr —/api/cloudsplus the manager-overrides file (~/.norsk-mgr/manager/data/manager-overrides.json, bind-mounted to the container’s/var/data/manager-overrides.json). The packer AMI pipeline emits animageVersionTablefragment as a build artifact precisely so this channel can be fed without hand-copying AMI ids. - Per-region worker networking (
WORKER_CONFIG_FILE): a JSON file mapping region →{ workerSubnets: [...], workerSecurityGroup, workerServerRole, associatePublicIp }— the last optional,truewhen absent;falselaunches workers with no public address (worker-node contract §4.1). Terraform renders it from its outputs; norsk-mgr mounts it into the Manager container and sets the env var. (Historically this data traveled via manager-instance tags — that mechanism is superseded by the file.)
5. Instance requirements
Section titled “5. Instance requirements”| Requirement | Detail |
|---|---|
| Instance role | ec2:RunInstances/Describe*/Terminate*/Stop*/CreateTags (scope where practical), iam:PassRole on the worker role, iam:CreateServiceLinkedRole for spot.amazonaws.com only (a spot tier’s first launch in an account that has never run spot creates AWSServiceRoleForEC2Spot; without it every spot launch fails ServiceLinkedRoleCreationNotPermitted), registry pull (if ECR), S3 read (the deployment bucket — manager-image channel), Secrets Manager read (keypair + license + admin-password secrets), with side-loaded worker launch data (worker-node contract §3.1) create, tag, read, describe and delete on <name>-worker-launch/* plus secretsmanager:ListSecrets (names and tags only, for the hourly sweep of orphaned entries), SSM core |
| Security group | public (operator CIDRs): the ONE front-door port in use — 443 with TLS, 9081 (plain HTTP) without; 9333 from the worker security group only (template tars + workdir assets — see ports below). Nothing else inbound |
| Networking | a public IP / EIP (operators reach the UI; the host reaches cloud APIs + workers; Let’s Encrypt is issued to it) — in a public subnet even when the workers are in private ones. Egress to worker subnets on workerServicePort (8110); the workers reach it back on 9333 by its private address |
| Persistent state | ~/.norsk-mgr/ (operator config, pools.json, staged license, product-template tars, tokens) and ~/.norsk-mgr/manager/{data,logs} (bind-mounted /var/data, /var/log — the Manager’s RocksDB job store, its backup engine and the hook’s snapshots live here). This is the deployment’s durable state: the kit puts it on a dedicated EBS volume the root owns (data_volume_id), mounted before norsk-mgr starts and re-attached to a replacement host; norsk-mgr backup archives it off-host (NORSK_MGR_BACKUP_TARGET, daily timer) |
| Config source | NORSK_MGR_LAUNCHER_CONFIG in the service environment: a file:// / s3:// / ssm:/// URI holding config.yaml’s content + a managerOverrides block. Fetched at every daemon boot and by norsk-mgr reload-launcher-config; unreadable → boot continues on the local file, audited. The kit writes it to an SSM parameter from tfvars; it is the record — API/hand edits of config.yaml last until the next reload |
| Instance role, additionally | ssm:GetParameter on that parameter; s3:PutObject/GetObject under the backup prefix + ListBucket on its bucket |
| Keypair secret | a Secrets Manager secret (worker instance key + worker auth-token signing), generated per-deployment by the secrets module. Name it: instance_key_secret_arn makes cloud-init fetch that ARN to /var/lib/norsk-mgr/instance-key-source.json, norsk-mgr mount it at /home/id3as/instance-key.json (managerInstanceKeyFile) and the sys.config render secretRegions => [ ] + secretPath. Left empty, the Manager instead LISTS every secret in each secretRegions region and takes one tagged id3as:defaultInstanceKey=true — account-wide, so two deployments in one account steal each other’s key (2026-09-18). Both examples name it, and the secrets module tags the key only with instance_key_discovery_tag = true |
6. Ports
Section titled “6. Ports”| Port | What | Exposure |
|---|---|---|
| 443 | oauth2-proxy TLS (operator UI/API) — tls_mode = certbot-route53; cloud-init sets PROXY_PORT=443 | public (operator CIDRs) |
| 9081 | oauth2-proxy plain HTTP (operator UI/API) — tls_mode = none. norsk-mgr’s DEFAULT_PROXY_HTTP_PORT; with no TLS listener this IS the front door, not a redirect. CLI/smoke tests only: a browser cannot sign in over it (__Host- session cookie) | public (operator CIDRs), only without TLS |
| 81 | HTTP → HTTPS redirect, bound by norsk-mgr only under TLS | not opened (nothing needs it) |
| 9444 | norsk-mgr’s default TLS port — never bound by this kit (PROXY_PORT overrides it). A host that leaves PROXY_PORT unset serves TLS here instead of 443 | not opened |
| 9333 | norsk-mgr backend | worker security group only (plus localhost and mgr-nginx). Workers fetch GET /api/v1/product-templates/by-sha/<sha256>/tar?token=<t> (the per-launch token frozen into the job; the route refuses any other non-loopback caller) and GET /api/v1/workdir-assets/<sha256> (content-addressed). Advertised to workers as NORSK_MGR_PUBLIC_URL=http://<private-ip>:9333 |
| 6789 | Manager gRPC | localhost only — norsk-mgr/AutoManager connect locally; workers never dial in (the Manager dials out to them). The kit’s config.yaml sets managerGrpcBind: 127.0.0.1, so the container publishes it on loopback only (Docker-published ports sit ahead of host firewalls). A host with local-server workers binds its private address instead: they register over this port. |
| 6788 | Manager web/status | localhost only |
Every control connection is Manager→worker; the one worker→manager path is the :9333 fetch above.
What :9333 does and does not protect. The backend gates every /api/*
request itself (norsk-ctl e11c0d45, backend/src/lib/api-auth.ts): a
bearer API token holding a role, the proxy secret mgr-nginx stamps after the
oauth2 session, or the proxy secret presented on loopback by the CLI on the
host; anything else is 401. Once a proxy secret exists, a bare loopback
request is nobody. The tar route admits a worker only
with the token minted for that launch (?token=, 256 random bits, persisted
in product-templates/tar-tokens.json, revoked when the template is deleted
and pruned after 24 h once no production carries it); the by-name
/:name/tar route and a token-less by-sha fetch are refused from anywhere
but localhost / mgr-nginx. Those two GET routes are the only /api/* bypasses: every other route
answers a worker (or a product container on one) with 401 unless it holds a
token. The security-group rule is still the network boundary, so do not widen
manager_backend_from_workers beyond the worker SG. NORSK_MGR_BIND
(new) can pin the backend to 127.0.0.1 on a single-host install, but not
here: mgr-nginx reaches it via host.docker.internal and workers over the
VPC, so the deployment leaves it unset.
7. Failure / recovery posture
Section titled “7. Failure / recovery posture”- The Manager’s durable state (RocksDB under
manager/data) survives container restarts via the bind mounts; norsk-mgr restarts the container on boot. Instance replacement keeps the data volume, so the store and the job history come back on the new host; a lost volume =norsk-mgr restorefrom the newest archive (or accept re-seeding config and losing job history — the fleet itself is re-discovered by tag polling, and running workloads are reconciled viaWorkerHello). - The Manager can be held. norsk-mgr owns the Manager container’s life
(no Docker restart policy): it starts a container it finds exited — after
a crash, a host stop or a reboot — up to a ceiling, and it starts nothing
while the Manager is held. A hold is a file beside config.yaml
(
norsk-mgr manager hold --reason …/manager releaseon the host) or the instance tagid3as.Norsk.ManagerHold(set in the console while the host is stopped; read from the metadata service at boot and before every start; needsinstance_metadata_tagson EC2). Held, a running Manager is stopped and kept,manager startanswers 409, andmanager statussays held, by what and why. norsk-mgr itself keeps running, so the host’s status, logs and store can be read. This is how a host is looked at before it is allowed back into the cluster. - A further member can be launched by a running Manager. With a cluster
the root publishes a member-launch bundle (modules/manager-host output
member_launch: the cloud-init and launcher config with per-member values as@@PLACEHOLDERS@@, plus the AMI, profile, security group, subnets and names) to the deployment bucket, and the manager role may pass itself, write member certificate secrets, write launcher-config parameters and make data volumes. The operator supplies the member’s distribution certificate (minted from the CA in state byscripts/mint-member-cert.sh); norsk-mgr does the rest (runbook § 13.8). - A replacement is triggered by the host-bootstrap set only (the module’s
terraform_data.bootstrapfingerprint); the tunables in config.yaml and the worker AMI table change through the launcher config without one. - norsk-mgr’s AutoManager reconnects with retry/backoff and rebuilds all in-memory state from the Manager’s event-stream replay; nothing above the daemon is durable.
- Two gRPC health names on :6789, and which one a probe wants. The
Manager’s
grpc.health.v1service answers two names: the empty service name""is liveness —SERVINGas soon as the process is up and its stores are open — andnorsk.api.manager.Manageris readiness —NOT_SERVINGthrough the initialisation settle after every start (up to 60 s while it re-hydrates and re-discovers the fleet), thenSERVING. The manager image’s ownHEALTHCHECK(grpc_health_probe -addr :6789, no-service) is the liveness one, so Docker never reports the container unhealthy for settling. Anything that RESTARTS the Manager when a probe fails — a watchdog, an orchestrator liveness probe, an autoheal sidecar — must use"": keyed on the readiness name it restarts the daemon during every settle and it never comes up. Anything that WAITS for the Manager before using it (norsk-mgr’s connect, an operator script) wantsnorsk.api.manager.Manager. This kit ships no probe of its own beyond the image’s:norsk-mgr.serviceisRestart=on-failureon the process, and norsk-mgr reports the containerrunningfromdocker inspect(State.Status, neither name) and/api/manager/clientis its own gRPC channel state, not a health probe.