Operator guide for Valet Dedicated 0.10.1

Observe

This page says what the installation watches, where its logs go, and how long it keeps them. Alarms, access logs, and telemetry export are off until you set their inputs. Log retention is on.

Alarms

Set alerts_topic_arn to an SNS topic in the installation's region. The module then creates CloudWatch alarms and notifies the topic when each goes to ALARM and back to OK. Alarm names start with the installation name. Tune the thresholds with the alarms input; see the reference.

<name>-alb-target-5xx

Fires when
Targets return more than 10 5xx responses in 5 minutes

<name>-alb-elb-5xx

Fires when
The load balancer returns more than 10 5xx responses of its own in 5 minutes

<name>-api-unhealthy-hosts, <name>-dashboard-unhealthy-hosts

Fires when
A target group has had an unhealthy target for 5 minutes

<name>-<service>-running

Fires when
The service reports no running task for 5 minutes. One alarm each for api, worker, inference, dashboard, and build

<name>-db-cpu

Fires when
The database's CPU averages above 80% for 15 minutes

<name>-db-storage

Fires when
Free storage falls below 10% of the allocated storage

<name>-db-connections

Fires when
Connections stay above 80% of 225 for 15 minutes

<name>-appdb-cpu, -appdb-storage, -appdb-connections

Fires when
The same three, for the app database. Created only when app_database.enabled is true

<name>-efs-burst-credits

Fires when
The code-storage file system's burst credit balance falls below 1 TB

The service alarms catch a service with no running task. They cannot catch a service that runs fewer tasks than desired but more than none. Container Insights would report that count, and the module turns it off because it costs about $1.40 a day per cluster. Until the first valet-dedicated upgrade starts the services, the service alarms are in ALARM.

The module creates no IAM for the alarms. CloudWatch publishes to the topic with its own service principal. A topic with the default policy accepts it. If you encrypt the topic with a customer managed KMS key, the key policy must let CloudWatch publish:

{
  "Sid": "AllowCloudWatchAlarms",
  "Effect": "Allow",
  "Principal": { "Service": "cloudwatch.amazonaws.com" },
  "Action": ["kms:Decrypt", "kms:GenerateDataKey*"],
  "Resource": "*"
}

Wire the topic to your channel

Create the topic yourself and pass its ARN. Then subscribe a channel to it.

  • Email. Subscribe an address to the topic with protocol email and confirm the message AWS sends.
  • PagerDuty. Create an Amazon CloudWatch integration on a PagerDuty service. Subscribe the integration URL it gives you to the topic with protocol https.
  • Slack. In AWS Chatbot, configure a Slack channel and add the topic to it. Or subscribe a Lambda function that posts to a Slack webhook.

Send metrics and traces to your collector

Set telemetry to export Valet's own metrics and traces over OTLP/HTTP. The api, worker, inference, dashboard, and build services export. Agent runtimes do not. Without telemetry, nothing exports.

  1. Create a Secrets Manager secret in the installation's region. Its plain-text value is the token your collector expects.
  2. Set the input. endpoint is an https URL with a host and an optional port, and no path. Valet adds /v1/metrics and /v1/traces.
telemetry = {
  endpoint     = "https://otel.example.com:4318"
  token_secret = "arn:aws:secretsmanager:us-east-2:123456789012:secret:otlp-token-AbCdEf"
}

The module passes the endpoint to each task as VALET_OTLP_ENDPOINT and the secret as VALET_OTLP_TOKEN. It never puts the token in plain environment or state. It grants each task's execution role secretsmanager:GetSecretValue on that one secret. You can pass the secret's name instead of its ARN; the module then grants the name with Secrets Manager's six-character suffix.

Every export carries one header, auth_header, set to auth_prefix followed by the token. The defaults send Authorization: Bearer <token>. An empty auth_prefix also sends Bearer , so the header always carries a prefix.

The tasks read the secret when they start. After you change the secret or the input, run valet-dedicated upgrade to start new tasks.

The tasks need a route to the endpoint. The workload subnets' NAT gateway provides one to the public internet. For a collector on your own network, make sure the platform security group can reach it on the port you set.

Datadog

Datadog's direct OTLP intake wants the API key bare in the dd-api-key header. Valet always sends a prefix, so point Valet at a collector that holds the key. Run the OpenTelemetry Collector (contrib) or the Datadog Agent with its OTLP receiver on your network. Use the Collector's bearertokenauth extension on the receiver, and the datadog exporter with your API key. Then set endpoint to the Collector's https address and token_secret to the bearer token you gave the extension.

extensions:
  bearertokenauth:
    token: ${env:VALET_INGEST_TOKEN}
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: bearertokenauth
exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: datadoghq.com
service:
  extensions: [bearertokenauth]
  pipelines:
    metrics: { receivers: [otlp], exporters: [datadog] }
    traces: { receivers: [otlp], exporters: [datadog] }

Terminate TLS in front of the Collector, because Valet always exports over https.

A generic OTLP collector

Any collector that accepts OTLP/HTTP over https works. Match its authentication to Valet's one header. For a bearer token, keep the defaults. For a collector that reads another header, set auth_header, such as X-Api-Key, and auth_prefix. Valet adds a service.name resource attribute to every metric and span, such as valet-api.

Logs

Each service and task family writes to the log group /ecs/<name>/<family> in CloudWatch. The families are api, worker, inference, dashboard, build, migrate, runtime, and builder. ECS Exec sessions write to /ecs/<name>/platform/exec and /ecs/<name>/workloads/exec.

The module keeps each group's events for log_retention_days, 365 by default. Raise it to keep more, or set 0 to keep them forever. Lowering it deletes older events when you apply.

The database's PostgreSQL log goes to the group /aws/rds/instance/<name>-db/postgresql. It records each connection and disconnection with its role. The module keeps its events for audit_retention_days. See what the database records.

Audit events

Valet records each security decision, such as a sign-in or a permission check, in its database and in a CloudWatch log group named /valet/<name>/audit. Each event is one JSON line with a version field and no secret values. The api writes the lines to a stream named api/<hostname>, one stream per running task, within seconds of the decision. The audit log page lists every field and how to read or forward the group.

The group includes allowed and refused operator and superuser writes, including settings, platform roles, caps, source edits, and runtime operations. Their permission names start with operator., such as operator.set_setting and operator.suspend_user. Settings events carry detail.name and optional detail.reason, never the value. The top-level reason describes the admission decision: empty on allow, permission_not_held on denial, or authorization_read_failed on error. Operator text stays in detail.reason for every outcome.

Other write events carry target identifiers and reviewed non-secret changes: platform role, org role enforcement, billing exemption, and runtime image. Each request-derived detail string is capped at 256 bytes on a UTF-8 boundary. The fixed operator-token actor identifies installation-token calls. GitHub catalog and deployment calls use actor_kind=ci; detail carries verified repository, workflow, and ref claims, never a token.

These events record admission; an allowed request can still fail in its handler. Missing or unverified credentials produce no row and increment audit.events{outcome=unauthenticated_skipped}. Read-only procedures, including settings and audit-log reads, produce no event. Source clone and release download URL minting record admission without the URL or token. Suspension produces one operator.suspend_user admission event with the target email in detail. The former post-success user.suspend event is retired. Impersonation remains a session event.

The module keeps the group's events for audit_retention_days, 400 by default, and no other log group uses that input. Raise it to match your compliance window, or set 0 to keep them forever. Lowering it deletes older events when you apply. Only the api task can write to the group. You can read it, or forward it, with the permissions you already hold on your account. The database copy is the source of truth; the group is a best-effort second copy, and a crash can lose the last few seconds.

Load balancer access logs

Set alb_access_logs_bucket to the name of an S3 bucket you own. The load balancer writes its access logs there under the prefix <name>. The module does not create the bucket or its policy. Create both first, because AWS checks the policy when it turns logging on.

The bucket must use SSE-S3 encryption, not a KMS key, and must be in the installation's region. Its policy must let the Elastic Load Balancing log delivery service write under the prefix. In regions opened in or after August 2022, name the service principal. Replace BUCKET, NAME, and ACCOUNT:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "logdelivery.elasticloadbalancing.amazonaws.com" },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::BUCKET/NAME/AWSLogs/ACCOUNT/*"
    }
  ]
}

In older regions, such as us-east-1, us-east-2, and us-west-2, name the regional load balancer account as the principal instead: "AWS": "arn:aws:iam::ELB_ACCOUNT:root". The AWS documentation lists ELB_ACCOUNT for each region.