Provider Sentinel · AI provider behaviour monitoring

Know when the AI your product depends on changes.

Provider Sentinel creates sealed behaviour baselines for closed AI providers and produces evidence packs when observable behaviour drifts.

First verify the test stayed fixed. Then report what changed.

Request a pilot View sample evidence pack See the case study

Case study · seven days of creative canaries

Window CCS-2026-05-08 → CCS-2026-05-14. A worked example of what a Provider Sentinel observation produces — not a live monitor.

7 days × 5 providers — all 35 cells covered
Reproducibility instability observed for OpenAI on the closing day
Sealed context held (adapter_config_digest matches CCS-2026-05-13 for every provider)

OpenAI 24 G · 180 S Stable
Anthropic 0 G · 156 S Stable
xAI 0 G · 130 S Stable
Gemini 0 G · 205 S Stable
Mistral 0 G · 260 S Stable

See the full case study Run a tailored case study on your stack

How it works

A closed-loop measurement pipeline: declare the test, seal the context, rerun it later, and report what changed.

Provider × Model × Suite × Time

Create a baseline

Run a declared probe suite against a provider/model under fixed settings.
Seal the context

Record the probe suite, adapter config, comparator policy, and capture-policy version.
Rerun later

Repeat the same probes under the same sealed context.
Report drift

Generate an evidence pack showing what stayed stable, what changed, and why the result is valid.

Provider Detail

Anthropic / claude-sonnet-4-6

What exactly is happening with this provider/model?

Current status

Stable

Last baseline: BEH001
Last check: BEH002
Suite: sentinel-behavioral-v1

Comparison context

Model tracking: Provider-declared
Requested model: claude-sonnet-4-6
Declared model: claude-sonnet-4-6
Capture policy: anthropic-capture-policy/v1
System fingerprint: captured when provider exposes it
Adapter config: anthropic-v1.adapter
Comparator policy: behavioral.v1

Behaviour summary

31 probes checked
31 visible outputs stable
31 finish reasons stable
31 token counts stable
Logprob digests not available on this provider
Raw response envelopes changed as expected

Provider Sentinel records the requested model selector and, when exposed, the provider-declared model identity. Alias movement is reported separately from behavioural drift.

Behaviour surfaces

Surface	Probes	Stable	Notes
Exact canaries	3	3	ExactBytes comparator, deterministic prompts.
Arithmetic / constrained reasoning	4	4	Numeric equality on parsed answer.
Transform & canon	7	7	Lower/upper/reverse, short canonical sentences.
Schema conformance	4	4	JSON schema validation under same temperature.
Instruction hierarchy	3	3	System-over-user precedence holds.
Extraction	4	4	Span extraction stable under fixed seed.
Classification	3	3	Label class stable; logprob distribution not exposed by provider.
Refusal boundary	2	2	Refusal class stable; phrasing not asserted.
Multi-turn	1	1	Multi-turn haiku probe; visible-text comparator.

Raw provider envelopes can change because providers include per-request metadata. Behavioural drift is classified by the declared comparator policy, not by raw envelope changes.

Latest evidence pack

Anthropic claude-sonnet-4-6 — BEH001

Locally verified

Open evidence pack viewer →

Drift Timeline

Example

Example timeline for Anthropic claude-sonnet-4-6, showing how a drift event would appear. Not an observed production incident.

May 01 BEH001 Baseline accepted
May 01 BEH002 Stable
May 02 CHECK001 Stable
May 03 CHECK002 Stable
May 04 CHECK003 Drift detected Example drift scenario

This sample event demonstrates the UI; it is not an observed production incident.
Context matched
- ✓ same probe suite
- ✓ same adapter config
- ✓ same comparator policy
- ✓ same capture policy
- ✓ same system fingerprint when exposed
Changed
- • 2 exact canaries changed
- • 1 Silver creative canary changed
- • distribution changed on 7 probes
Outcome Evidence pack generated

Proof boundary · Observable drift is not proof of hidden weight mutation.

Probe Explorer

Which exact behaviour surface changed?

Each probe is evaluated by its declared comparator. Exact probes, schema probes, refusal probes, and creative canaries are not judged the same way.

Probe	Category	Comparator	Verdict	Tier
› p_pong Baseline `pong` Current `pong` response_text matched normalized digest matched finish reason stop token count 1 → 1 system fingerprint captured when provider exposes it › Technical commitments digest collapsed by default local verification available signed envelopes coming next	Canary	ExactBytes	Stable	—
› ccs_haiku_revise_noun_swap Baseline `A spring rain falls / over the rusted iron gate / a thrush remembers` Current `A spring rain falls / over the rusted iron gate / a thrush remembers` response_text matched normalized digest matched finish reason stop token count 21 → 21 system fingerprint captured when provider exposes it › Technical commitments digest collapsed by default local verification available signed envelopes coming next	Creative	Text canary	Stable	Silver canary
› hierarchy_system_over_user Baseline `class: refused-per-system` Current `class: refused-per-system` response_text class-matched normalized digest matched finish reason stop token count 14 → 13 › Technical commitments digest collapsed by default local verification available signed envelopes coming next	Instruction hierarchy	SemanticClass	Stable	—
› schema_status_object Baseline `{"status":"ok","code":200}` Current `{"status":"ok","code":200}` response_text schema-valid normalized digest matched finish reason stop token count 7 → 7 › Technical commitments digest collapsed by default local verification available signed envelopes coming next	Schema	SchemaValid	Stable	—
› refusal_disallowed_content Baseline `class: refusal` Current `class: refusal` response_text class-matched normalized digest phrasing-not-asserted finish reason stop token count ~35 › Technical commitments digest collapsed by default local verification available signed envelopes coming next	Refusal	RefusalClass	Stable	—
› extract_invoice_total Baseline `$1,248.30` Current `$1,248.30` response_text matched normalized digest matched finish reason stop token count 6 → 6 › Technical commitments digest collapsed by default local verification available signed envelopes coming next	Extraction	SpanEqual	Stable	—
› classify_sentiment_pos Baseline `positive` Current `positive` response_text matched normalized digest matched finish reason stop token count 1 → 1 › Technical commitments digest collapsed by default local verification available signed envelopes coming next	Classification	LabelEqual	Stable	—

Digest details are collapsed under "Technical commitments" inside each probe row.

Why the comparison is trustworthy

Provider Sentinel does not only compare model outputs. Before reporting drift, it checks all four of these stayed the same:

Probe suite
Provider settings
Comparison policy
Capture policy

If the capture policy changes, Provider Sentinel treats the comparison as a context mismatch — not provider drift — until the change is explicitly acknowledged.

Trust feature

Capture-policy versioning

Capture-policy versioning records the exact rules used to turn raw provider responses into behavioural observations, so parser or adapter changes cannot silently look like provider drift.

We verify the measuring instrument did not change before we say the provider changed.

Cost-bounded Gemini monitoring

Provider Sentinel pins Gemini live-smoke runs to thinking_budget: 0 and seals it into adapter_config_digest. Runs captured at a different thinking budget are recorded against a different digest and not silently compared. Empirically ~7.6× fewer tokens per run versus unbounded thinking on the same 300-pair × 3-repeat search.

› Read doctrine

Gemini 2.5 models can generate hidden "thinking" tokens before returning visible text. Those tokens may affect cost and upstream behaviour, but they do not appear in the captured response text. Pinning to thinking_budget: 0 makes Gemini observation predictable, affordable, and easier to compare against providers whose chat surfaces do not expose or bill an unbounded hidden reasoning budget.

Empirical cost from one identical 300-pair × 3-repeat creative-canary search: 746,044 total tokens with default thinking enabled (CCS003, captured on gemini-2.5-pro), versus 98,381 total tokens with thinking_budget: 0 pinned (CCS-2026-05-03, on gemini-2.5-flash). Case-study runs used gemini-2.5-flash because gemini-2.5-pro on the v1beta endpoint rejects thinking_budget: 0 with HTTP 400 ("Budget 0 is invalid. This model only works in thinking mode."), so the comparison-contract pin is incompatible with Pro.

Provider Sentinel does not just compare text. It compares text under a declared execution contract.

For Gemini, the thinking budget is part of that contract. A run with hidden thinking enabled and a run with thinking disabled are different upstream behaviours.

Does disabling Gemini thinking make the result less valid?: No. It makes the comparison narrower and cleaner. Provider Sentinel is measuring whether a provider reproduces visible outputs under fixed declared controls. thinking_budget: 0 removes hidden reasoning tokens from the run configuration, reducing cost variance and unsurfaced behaviour variance.
Can I monitor Gemini with thinking enabled?: Yes — as a separate baseline universe. The thinking-budget setting is sealed into the adapter configuration digest, so a baseline captured with thinking_budget: 0 will not be compared against a run captured with unbounded thinking or another value.
Why not leave Gemini at its default?: Hidden thinking tokens can dominate cost while not appearing in the captured visible response. For an observation product, that is bad operator ergonomics and bad audit hygiene.
Is this fair to Gemini?: Yes — the setting is explicit, receipted, and not mixed with other settings. The claim is narrow: "Gemini visible-output reproducibility under this declared adapter configuration."

Authoritative source: Gemini thinking-budget pin — doctrine.

Evidence Pack Viewer

Can I hand this to security, compliance, or vendor review?

Artifact title

Anthropic Claude Sonnet 4.6 Provider Service Behaviour Fingerprint

Local verification available Signed envelopes coming next

What this pack contains

✓ run context
✓ probe suite identity
✓ provider/model settings
✓ capture-policy version
✓ observations
✓ receipts
✓ verdicts
✓ drift report
✓ non-claim statement

What this proves

the same test was run
the provider response was captured
the comparison policy was fixed
the behaviour stayed stable or changed
the evidence pack has not been altered

What this does not prove

model weights changed
provider acted maliciously
provider internals are known
all possible behaviour is covered

Not weight-identity proof

› Technical commitments

Probe suite fingerprint: captured
Adapter config fingerprint: captured
Comparator policy fingerprint: captured
Capture policy: anthropic-capture-policy/v1
Observation root: captured
Local verification: available
Signed envelopes: coming next

Evidence-pack summaries can report whether a run used one capture policy version or requires review.

run.json — planned capture-policy summary Coming next · example format

{
  "capture_policy_version": {
    "status": "homogeneous",
    "value": "anthropic-capture-policy/v1"
  }
}

The status field is a planned shape. Today, evidence packs record the capture-policy version per receipt; the run-level summary field is coming next.

homogeneous

Every receipt in the run used the same capture policy.

heterogeneous

Capture policy changed within the run; comparison requires review.

absent

Legacy pre-versioning capture.

What teams use Provider Sentinel for

Catch provider drift before debugging your own app.
Keep evidence when external AI services change behaviour.
Track refusal and instruction-boundary behaviour over time.
Export evidence packs for engineering, security, compliance, or vendor review.

We do not hide this setting. We seal it.

Provider Sentinel treats provider knobs as part of the evidence contract. Gemini's thinking budget is recorded in the adapter configuration digest, so any change to it creates a different comparison universe.

Proof boundary

Provider Sentinel detects observable provider-service behaviour drift under declared conditions. It does not prove hidden model-weight changes without provider or execution attestation.

Provider Sentinel does prove

What probe suite was run
What provider settings were used
What capture policy projected the response
What behaviour was observed
Whether later behaviour differs under the same sealed context

Provider Sentinel does not prove

Model weights changed
A provider acted maliciously
Internal provider infrastructure changed
All possible behaviour is covered

Provider service behaviour drift is not proof of hidden weight mutation without provider or execution attestation.

Technical detail

› Capture-policy versions, per provider

Provider Sentinel receipts include an optional capture_policy_version field. Each provider adapter declares its capture policy.

openai-capture-policy/v1
anthropic-capture-policy/v1
gemini-capture-policy/v1
mistral-capture-policy/v1

Evidence packs summarize the run with a status of homogeneous, heterogeneous, or absent.

› Gemini guardrail

Gemini may return hidden reasoning-like parts alongside answer text. Provider Sentinel's Gemini adapter applies a deterministic capture policy that keeps answer text and excludes thought-marked parts.

The policy is versioned, so if those projection rules change, future comparisons can detect the adapter-policy change rather than misclassify it as provider drift.

› Comparison contract

For behavioural comparison, Provider Sentinel requires the same probe suite, adapter config, comparator policy, and capture-policy version.

If capture_policy_version differs between two runs, response-text-layer comparison is treated as a context mismatch, not provider drift.

› Alias-tracked drift semantics

When a provider alias is monitored, Provider Sentinel separates:

· same alias + same declared model + behaviour changed → behaviour drift under the same declared model
· same alias + changed declared model + behaviour changed → alias-contract drift
· same alias + changed declared model + behaviour stable → provider metadata changed
· changed alias/config → context mismatch

Become a design partner

Pilot Provider Sentinel against the AI provider your product depends on. Sealed baselines, scheduled reruns, evidence packs you can hand to a reviewer.

Request a design-partner pilot Email