Administer

Custom checkers

What a custom checker is#

A release policy runs a checker on every release of an artifact, app or agent, and refuses the releases that fail. Valet ships two checkers, catalog:osv and catalog:secrets. A custom checker is one your organization writes. It is an HTTPS endpoint that implements the Checker API described below. You add it by name, URL and headers, and a policy names it as org:<name>, the way a policy names a Valet checker.

Valet doesn't care what runs behind the URL. It can be an app in your organization, an adapter for a vendor you already pay, or a service on your own infrastructure.

Write a checker when a rule is yours alone:

• A license, package age or exact-version rule for an app's packages. Valet's checkers don't check these.
• A scan by a vendor your security team already uses.
• A check against an internal list, such as approved domains or required headers in a site's files.

A checker can skip the services it has no opinion on and report no findings for them. Policies apply to every release in the organization.

What leaves Valet#

Adding a checker agrees to send release files to its URL. For every release that a policy naming the checker evaluates, Valet sends the checker links to the release's source, its build, or both, as the checker asks. Once the checker fetches the files, they are outside Valet's control, and the checker can keep them.

• Source is the files the release was published from.
• Build is what Valet produced from them: for an app, the build and its launch layers, and for an agent, the bundle Valet deploys. An artifact's or skill's build is a list of file hashes, which Valet never sends. Their source is sent.
• Valet never sends env vars, other services' files, or other policies' findings.

When you add a checker, the endpoint says which files it asks for, and Valet saves them as the checker's approved inputs. The dashboard, the CLI and set_checker all say so before they save. An endpoint can't widen the agreement on its own. If a later answer from the endpoint asks for a file set you haven't approved, Valet marks the checker unavailable until a member adds it again. An answer that asks for fewer files narrows what Valet sends.

Valet calls the endpoint only over HTTPS, only at a public address, and follows no redirect.

The Checker API#

This section is the reference. Checker API version 1 is the only version.

Overview#

Valet calls four operations on the URL you add. Each is a POST to <url>/v1/<operation> with a JSON body and the headers you configured. Each answer must be HTTP 200 with a JSON body.

Every request body carries "api_version": 1 and "org": {"id": "…", "name": "acme"}. Valet sends Content-Type: application/json and Accept: application/json too.

A checker reports its result one of three ways: in the answer to start, in the answer to a later poll, or by calling back. The checker fetches the release's files through short-lived links, which Valet sends in the start body.

Each side ignores fields it doesn't know. Within version 1, Valet may add optional request fields and starts to read optional response fields only when they are present, so a checker should ignore fields it doesn't know.

Versioning#

The version is in the path (/v1/) and in describe's api_versions. A checker must list 1 there, or Valet refuses it. A change that an existing checker could misread would be a new version at a new path, and Valet would speak both for a period.

Describe#

describe says what the checker is. The body holds only api_version and org.

{
  "api_version": 1,
  "org": { "id": "0192f1c4-7b3a-7d44-9e0a-2c5d8f1b3a60", "name": "acme" }
}

The answer:

{
  "api_versions": [1],
  "display_name": "Snyk Open Source",
  "description": "Tests the release's packages with Snyk.",
  "inputs": ["build"],
  "fields": [{ "name": "snyk_org", "description": "Snyk organization id.", "required": true }]
}

A field has a name, a description and required. A name uses lowercase letters, digits, _ and -, starts with a letter, is unique, and is at most 64 characters, so --set name=value can name it. A description is at most 1,000 characters. A field is never secret: the checker holds its own credentials. A display name or description with a control character is refused.

The answer may be 64 KiB. Valet describes the endpoint when you add a checker, when you verify it, and when a listing finds its description more than an hour old. It describes a checker at most once every 10 seconds. A failed describe records the failure as the checker's last_error and keeps the last good description. A checker that fails describe when you add it isn't saved.

Validate#

validate asks whether the checker accepts a policy's settings.

{
  "api_version": 1,
  "org": { "id": "0192f1c4-7b3a-7d44-9e0a-2c5d8f1b3a60", "name": "acme" },
  "settings": { "snyk_org": "acme-prod" }
}

The answer is {"ok": true}, or a refusal:

{ "ok": false, "reason": "no Snyk organization acme-prod" }

A refusal fails valet policies add with policy.invalid_settings and the reason, cut at 500 characters. A reason must not quote a secret. ok is required.

Start#

start sends the release.

{
  "api_version": 1,
  "idempotency_key": "0192f1c4-7b3e-7c1a-9d2e-5a1f3c4b6d7e",
  "evaluation_id": "0192f1c4-7b3d-7a10-8e21-1b2c3d4e5f60",
  "org": { "id": "0192f1c4-7b3a-7d44-9e0a-2c5d8f1b3a60", "name": "acme" },
  "policy": "snyk",
  "service": { "id": "0192f1c4-7a01-7b52-8c33-4d6e7f809a1b", "name": "shop", "kind": "app" },
  "release": {
    "id": "0192f1c4-7b2c-7e65-a4d7-3b8c9d0e1f2a",
    "commit": "4be1c0d9f3a2…",
    "base_commit": "9a1fe2d7c5b4…"
  },
  "settings": { "snyk_org": "acme-prod" },
  "archives": [
    {
      "name": "build",
      "format": "tar+zstd",
      "url": "https://api.valet.dev/checker/v1/archives/eyJhbGciOi…",
      "expires_at": "2026-10-20T16:04:05Z"
    }
  ],
  "callback": {
    "url": "https://api.valet.dev/checker/v1/report/eyJhbGciOi…",
    "expires_at": "2026-10-21T15:04:05Z"
  },
  "deadline": "2026-10-21T15:04:05Z"
}

A checker that already started work for an idempotency_key answers as it did the first time. Valet sends the same key when it retries a start that failed or that it can't remember finishing.

The answer is one of four:

{ "status": "done", "findings": [] }

{ "status": "failed", "failure": "Snyk returned 401" }

{ "status": "accepted", "handle": "test-7f3a", "poll_after_seconds": 20 }

{ "status": "accepted" }

• done reports the findings, inline.
• failed reports that the checker couldn't evaluate the release. The failure is shown to people. With no reason, Valet records "the checker reported a failure without a reason". Valet treats the evaluation as errored, not as passed.
• accepted with a handle asks Valet to poll. poll_after_seconds is a hint, held to between 1 second and 5 minutes. A handle is at most 1,024 characters.
• accepted with no handle means the checker will call back.

Any other status is an invalid answer.

Archives#

Each entry in archives is one file set. Fetch it with GET on its url. A HEAD returns the headers. The name and format say what it is:

source is sent only if the checker was approved for source. build and each layer are sent only if it was approved for build, and only for an app or an agent. An artifact has no build to send, so it gets only source.

A link works from any network and needs no Valet credential. It lasts one hour, and never past the evaluation's deadline. Valet signs fresh archive links on every poll. A link serves one file set of one release, and dies when the evaluation finishes: when it is reported on, expires, is removed, or its release is replaced. Every link also dies when the checker is added again with a new URL or new headers, and an archive link when it is added again with fewer inputs. Adding it again with the same URL and headers keeps every link working. A link that doesn't work answers 404, whatever the reason. A failure on Valet's side answers 503 with Retry-After: 5. A file set that fails partway closes the connection, so a checker never reads a short archive as a whole one.

Poll#

poll sends the start body again, with fresh archives and the same callback, plus the handle. The callback lasts until the deadline, so it doesn't change between polls:

{
  "api_version": 1,
  "idempotency_key": "0192f1c4-7b3e-7c1a-9d2e-5a1f3c4b6d7e",
  "handle": "test-7f3a"
}

(The body also holds every other start field. The example shows the fields that differ.) The answer is done, failed, or:

{ "status": "running", "poll_after_seconds": 30 }

Valet polls again after poll_after_seconds. When a checker gives no hint, Valet chooses the gap. A poll that fails is polled again.

Findings#

A done answer carries findings. A finding is:

{
  "status": "done",
  "findings": [
    {
      "severity": "high",
      "rule": "snyk/SNYK-JS-LODASH-1018905",
      "message": "Prototype Pollution in lodash@4.17.20",
      "package": "pkg:npm/lodash@4.17.20",
      "fix": "Upgrade lodash to 4.17.21"
    }
  ]
}

A severity outside the five blocks as critical. Valet keeps the 200 most severe findings, so dropping the rest never changes a decision, and cuts a field longer than its limit. It removes control characters and invisible characters from every text field. A done answer with no findings passes the release.

The report callback#

A checker that answered accepted with no handle reports with a POST to callback.url. The body is a done or failed answer, the same shape start returns:

{ "status": "done", "findings": [] }

Valet checks whether the evaluation finished before it checks whether the link was revoked. A callback revoked because the checker moved to a new URL or headers answers 404 while its evaluation runs, and 409 once the evaluation finishes.

A callback reports one evaluation once. Retry on 503 and on a connection failure, and treat 409 as done.

Timeouts, retries and limits#

A call that doesn't answer HTTP 200 with a valid body within 30 seconds and the size limit fails. A checker should accept the work and answer accepted, not make Valet wait while it scans.

Valet retries a failed start after 5 seconds, then 10, doubling to at most 5 minutes, up to 8 attempts in all, about 10 minutes. With the same idempotency_key, so a checker that did start work doesn't repeat it. After the last attempt the evaluation errors with a message such as "the checker org:snyk answered HTTP 503" or "the checker org:snyk didn't answer within 30 seconds". An invalid answer, an address that isn't public, and an unreachable endpoint fail the same way.

An evaluation that no report finishes within 24 hours errors with "did not finish in time", and its links stop working.

Valet allows an organization two calls to checkers in flight at once, so a checker sees at most two concurrent start or poll calls from one organization.

Authentication#

Valet to your checker. Valet sends the headers you configured on every call, over HTTPS. The checker verifies them as it would an API key and answers 401 or 403 to a call that lacks them. Valet adds no signature.

A credential in a header is a {{NAME}} reference to a secret env var of your organization, such as Authorization: Bearer {{SNYK_TOKEN}}. Valet stores the reference, never the value. It resolves the reference at each call, so rotating the env var needs no change to the checker. Headers are shown as written, so a reference is safe to show. A reference can name any secret org env var, and any member can add a checker, so any member can send any org secret to a URL they choose.

• A header whose name looks like a credential must hold a reference: Authorization, Proxy-Authorization, Cookie, X-Api-Key, Api-Key, or a name that ends in -Token, -Secret or -Key, in any case.
• Any other header may hold a literal value. Valet stores it and shows it, as written, to every member of the organization, so never write a credential there.
• A reference must name a secret env var at the organization's scope. A plain env var, an agent's or app's env var, and a name that doesn't exist are all refused. A plain org env var is named as one, such as "SNYK_TOKEN, which is a plain env var in acme; set it as a secret with valet env set SNYK_TOKEN=... --org acme".
• valet env unset --org refuses to remove an env var that a checker's header references, unless you pass --force. A removed variable makes each call fail with a message such as "the header Authorization names SNYK_TOKEN, which isn't a secret env var in acme", until you set it again.
• A checker has at most 16 headers. A name is at most 256 bytes, and a value at most 8 KiB. Valet sets Host, Content-Length, Content-Type, Transfer-Encoding and Accept itself, so a checker can't configure them.

Your checker to Valet. The archive links and the callback are the only calls a checker makes to Valet. Each link carries its own signed token, so the checker needs no other credential.

A checker that is a Valet app#

A checker can be an app in your organization. Valet calls its public address like any outside caller, with your headers and no Valet session. A private app answers 401, so a checker app must use public access and check the header itself. See Apps for access.

Add a checker#

Add a checker by URL and headers. Saving it calls describe. A checker whose endpoint doesn't answer isn't saved.

A URL must:

• use https;
• hold no user name, password, query or fragment;
• be at most 2,048 bytes;
• name a public address.

A checker's name uses lowercase letters, digits and hyphens, and is at most 63 characters. Policies name it org:<name>. Adding a name that exists replaces the checker, so every policy that names it uses the new URL and headers from its next call. A new URL or new headers also restart the checker's running evaluations at the new endpoint, before their unchanged deadline, because the old endpoint's links stop working. Pass every header again.

In the CLI and MCP#

Set the secret first, as an organization env var. Set it yourself, so the value never passes through a conversation. Only the CLI sets an organization env var:

Then add the checker. Ask your agent:

Add a Valet checker named snyk at https://snyk-checker.acme.valet.run. Send it the header Authorization: Bearer {{SNYK_CHECKER_TOKEN}}.

set_checker
{
  "checker": "snyk",
  "url": "https://snyk-checker.acme.valet.run",
  "headers": { "Authorization": "Bearer {{SNYK_CHECKER_TOKEN}}" }
}

The command prints the checker's display name and the files it receives. Confirm them before you rely on the checker: they are the files Valet will send.

Check the endpoint later, without sending a release. verify calls describe with the saved headers and reports how long the endpoint took. Within 10 seconds of the last check, it reuses that result and says so. Pass --set to check the settings a policy would give the checker too. A setting the checker doesn't declare is refused with policy.invalid_settings, as valet policies add refuses it. It exits 1 when the endpoint fails or refuses the settings.

MCP has no verify tool. list_policies shows each checker's URL, headers, approved files and last_error, and the dashboard verifies too.

Then add a policy that uses it. Ask your agent:

Add a Valet release policy called snyk that uses the org:snyk checker with snyk_org acme-prod. Start it in Monitor.

set_policy
{
  "policy": "snyk",
  "checker": "org:snyk",
  "settings": { "snyk_org": "acme-prod" },
  "enforcement": "monitor"
}

Start in Monitor and watch what the checker finds before you let it block. See Release policies.

valet policies checkers lists every checker, with an organization checker's URL, headers, files, last description and last error.

In the dashboard#

Open Governance › Policies and select Add policy. The catalog has two groups: From Valet and From your organization. Under From your organization, select Add a checker. Enter the name and URL, and add a header row for each header. A header's value field offers your organization's secret env vars and inserts the one you pick as {{NAME}}. The dashboard has no page for organization env vars, so set the variable first with valet env set --org.

Select Check endpoint. Valet describes the endpoint and shows its name, description, settings and the files it will receive, with the agreement to send them. Save checker is that agreement, and it approves only the files shown. If the endpoint asks for more files before you save, the checker is saved unavailable until you save it again.

Each checker in the group shows its name, its org: id, the URL's host, its approved files, and its unavailable reason or last error. An unavailable checker can't be picked for a policy. Manage opens the checker, with Verify, Edit and Remove.

Remove a checker#

Remove a policy before the checker it names. Ask your agent:

Remove the Valet checker org:snyk.

destroy_checker
{ "checker": "snyk" }

While a policy names the checker, Valet refuses with policy.checker_in_use and names the policy. Removing a checker that is already gone changes nothing. Valet then stops sending release files to its URL.

When a checker is broken#

A checker can fail in three ways, and each shows up where policies already report trouble.

• Unavailable. The checker's description asks for files you haven't approved. The catalog shows the reason, such as "its endpoint asks for source, which acme hasn't approved; save the checker again to approve it". Adding a policy that names it fails with policy.checker_unavailable, and a running evaluation errors with the reason. Add the checker again to approve the new files.
• Errored evaluations. An endpoint that is down, slow or answers 401 errors its evaluations after the start retries. A blocking policy then refuses the release with deploy.policy_errored, quoting the failure. A Monitor policy records the error and lets the release go. The Policies tab counts the errors and shows the last one.
• last_error. A failed describe or validate records its reason on the checker, before any release fails. valet policies checkers, list_policies and the dashboard show it.

A blocking policy whose checker is down holds each new release until the start retries run out, about 10 minutes, and then refuses it.

Ways out#

• Point the checker at a working endpoint. Add it again with valet policies checkers add. The next call uses the new URL and headers, so a held release whose start is still being retried goes through. A held release that waits for a callback from the old endpoint can't finish that way. Remove or change its policy to release it.
• Remove the policy, or change its checker. Either cancels the policy's running evaluations, and a release that waited only on it goes ahead.
• Fix the endpoint. If the checker is a Valet app and a blocking policy evaluates the app's own releases with that checker, the fix waits on the broken checker. Remove the policy, deploy the fix, and add the policy back.
• Switch the policy to Monitor. This reaches only new evaluations. A running evaluation keeps the enforcement it started with, so a held release stays held and is refused when its retries run out. Publish again, and Valet evaluates the new release under Monitor and lets it go live.

See What a change does for the rule behind each.

Limits#

• An organization has at most 20 checkers. Adding a 21st fails with policy.too_many_checkers. Replacing a checker always fits.
• Until Valet has organization roles, any member of the organization can add, replace, verify and remove its checkers, as any member can change its policies. Saving a checker sends release files to a URL the member chose.
• A checker's headers can reference any secret org env var, so any member can send any of the organization's secrets to a URL they choose. A literal header value is shown, as written, to every member. Add checkers with care.
• A checker has at most 16 headers and 32 settings fields. See Timeouts, retries and limits for the rest.
• Policies evaluate every release, and a policy can't be limited to some services. A checker that wants fewer can skip a service and report no findings.
• Valet sends a checker the source and the build. It sends no list of packages, so a checker that wants one reads the lockfiles in the build.

For the errors a checker can produce, see Error codes.