> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensor9.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting Explanations

> Diagnose controls, receipts, reports and SDK behavior without replaying an ambiguous mutation.

Troubleshoot the service operation and the explanation system as separate paths. A target operation can succeed while its report is temporarily unavailable. A complete report can accurately describe an operation that was refused. A missing report never proves that a write had no effect.

Start by preserving the original request time, exact endpoint, operation, mode, response status, origin request IDs, Explain receipt headers and observed target state. Remove credentials and sensitive payload data.

## Symptom and next-action matrix

| Symptom                                     | Evidence to inspect                                                                  | Safe next action                                                                                 |
| ------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| Malformed `x-t9-explain` refusal            | Exact single header value; size; JSON version, mode, format and unknown fields       | Correct the control before signing; do not weaken it to ordinary execution                       |
| Unsigned Explain control                    | Final signed-header set and middleware order                                         | Insert the control before SigV4 signing                                                          |
| Duplicate Explain controls                  | SDK interceptors, default headers and request override                               | Keep exactly one field; do not choose one arbitrarily                                            |
| Explicit mode unsupported for the operation | Error response and directed service profile                                          | Use supported coverage or another test operation; do not expect fallback                         |
| `x-t9-explain-unexplained` response         | Adapter default named by the header; operation                                       | Treat this request as ordinary execution with no report; investigate coverage                    |
| No report receipt                           | Status/body, `x-t9-explain-unexplained`, transport state and proxy behavior          | Determine whether an HTTP response arrived and whether the request was explicitly explained      |
| Permission mismatch                         | Signing identity, endpoint, target runtime identity and report-read identity         | Fix the failing boundary; do not assume origin and target permissions are interchangeable        |
| `409 NotYetVisible`                         | Report ID, requested `minimumVersion` and current stored revision in the disposition | Keep the previously recorded snapshot and poll the same report after bounded backoff             |
| `503 Unavailable`                           | Native result, report ID and read route                                              | Retry only the report read                                                                       |
| `410 Expired`                               | Preserved bundle and operation state                                                 | Use preserved evidence; repeat the operation only under its own safety rules                     |
| `410 Consumed`                              | Owner acknowledgment and exported evidence                                           | Use the previously exported bundle; an identical owner acknowledgment is idempotent              |
| `404 NotFound`                              | Exact report ID and owner identity                                                   | Retry the exact read with bounded backoff; do not infer persistence, access or operation outcome |
| Incomplete capture                          | Lifecycle, assurance and limitations                                                 | State what remains unknown and choose one observation that resolves it                           |
| Missing backend observation                 | Planned/attempted evidence, provider ID and native status                            | Inspect exact target state or use a declared status read                                         |
| SDK decoding or retry behavior              | Raw HTTP metadata, retry count and response content type                             | Prevent explicit Explain documents from entering normal service retries                          |
| Endpoint or signing mismatch                | Service endpoint, region, signing name and credential provider                       | Correct client construction before changing Explain mode                                         |
| Wrong report format                         | Existing report ID and desired renderer                                              | Read the same report with another format; do not rerun the origin operation                      |

## Failure 1: the control was added after signing

### Symptom

The request is refused as an unsigned Explain control, even though the application appears to send a valid value.

### Why it happens

`x-t9-explain` changes whether the request executes and must be integrity-protected. If an HTTP interceptor appends the header after SigV4 signing, the adapter cannot trust it. A reverse proxy can cause the same problem if it adds or rewrites the control after the client signs.

### Confirm it

Inspect request construction without printing the authorization credential:

1. Identify where endpoint selection, request override and signing occur.
2. Confirm `x-t9-explain` is present before the signer runs.
3. Check that the signed-header set covers it.
4. Confirm there is exactly one value and that no proxy rewrites it.

### Fix it

In AWS SDK for Java 2.x, use request override configuration while building the service request:

```java theme={null}
PutObjectRequest request = PutObjectRequest.builder()
    .bucket(bucket)
    .key(key)
    .overrideConfiguration(c -> c.putHeader(
        "x-t9-explain",
        "{\"version\":1,\"mode\":\"Explain\",\"fmt\":\"Json\"}"))
    .build();
```

The SDK applies the override before signing. Do not add a second copy in a generic HTTP interceptor.

### What not to do

Do not remove the header and send the mutation normally just to see whether the endpoint works. Use a non-mutating operation or fix signing first.

## Failure 2: the operation succeeded but the report is unavailable

### Symptom

An `ExecuteAndExplain` call returned its native success and `x-t9-explain-id`, but reading the report returns `404 NotFound`, `503 Unavailable` or remains partial.

### Interpretation

The origin operation and report have separate lifecycles. Native success is evidence about the origin API response. It does not seal the report. A report receipt identifies correlation intent; it does not prove that a report row was persisted. Report unavailability is evidence about the diagnostic read path, not a rollback of target effects.

### Safe response

1. Preserve the native response, origin request ID, report ID and target resource identity.
2. Verify the intended target state using the service's normal read path.
3. Retry only the exact signed report GET with bounded backoff and the same owner identity.
4. If a declared origin status poll exists, select the same report ID and exact resource identity.
5. Record the latest report revision; ignore a late older revision.
6. Do not replay the mutating operation merely to create another report.

If the report never becomes available, the operation's native evidence and target-state observation remain the source for deciding cleanup or remediation. Report absence must be recorded as a diagnostic limitation. Do not guess another report ID.

## Failure 3: an explicit mode is unsupported

### Symptom

The application explicitly requests `Explain` or `ExecuteAndExplain`, and the adapter refuses before the operation runs.

### Interpretation

An explicit request is fail-closed. The adapter does not silently ignore it, fall back to the adapter default or execute normally. This protects a preview from becoming a mutation.

### Safe response

1. Read the exact refusal and origin operation.
2. Inspect the directed service profile and operation-specific coverage.
3. Confirm that the deployed adapter and selected target match the profile you inspected.
4. Choose a supported non-mutating operation for connectivity testing, or collect ordinary service evidence without claiming an explanation.

Do not switch to `x-t9-explain: true` as a workaround: that value means `ExecuteAndExplain`, not a more compatible preview.

## Malformed, duplicate and false controls

The adapter refuses these controls before ordinary service processing:

```text theme={null}
x-t9-explain: false
x-t9-explain: {"mode":"Explain"}
x-t9-explain: {"version":1,"mode":"explain"}
x-t9-explain: {"version":1,"mode":"Explain","unknown":true}
```

The first has no off spelling; omit the header for ordinary execution. The second lacks `version`. The third uses the wrong enum case. The fourth contains an unknown field. Empty, non-ASCII, over-4096-byte and repeated fields are also refused.

Log a redacted representation of the control and its byte length, not the complete signed request.

## Default-caused unexplained requests

When an adapter default asks for explanation but a declared status read must execute ordinarily, the response can include:

```text theme={null}
x-t9-explain-unexplained: Explain
```

This says the default was not applied and no report was opened. It is not an error-provenance header and not a report receipt. An explicit unsupported Explain request is refused instead of receiving this fallback.

Check the operation's coverage and whether your client intended an ordinary status read or a report-aware poll.

## Missing receipt

No `x-t9-explain-id` can have several causes:

* the request never reached an HTTP service;
* no explicit control or adapter default applied;
* the operation was served under default fallback and has `x-t9-explain-unexplained`;
* the request was refused before report creation, including some authorization paths;
* middleware removed response headers;
* the client inspected only a decoded error and discarded raw HTTP metadata.

Do not collapse these cases. First establish whether an HTTP response arrived, then preserve its raw status and headers. For example, an S3 authorization denial occurs before a report opens; the inspected VPC path can represent an authenticated denial. Report behavior is service-specific, not a universal policy-trace promise.

## Report read dispositions

Use the status and disposition together:

* `409 NotYetVisible` means the stored revision is below the direct GET's requested `minimumVersion`; retain the previously recorded snapshot, record the current stored revision returned by the disposition, and retry the read.
* `409 NotSealed` or `RevisionMismatch` applies to acknowledgment; read the current report and use its supplied acknowledgeable revision.
* `410 Consumed` means the owner already acknowledged the sealed revision. Repeating the same owner acknowledgment at the same revision is idempotent; another revision is a mismatch.
* `410 Expired` is terminal while the expired record remains; after retention collection, the same report can appear as `404`.
* `503 Unavailable` permits a later read attempt, not a mutation replay.
* `404 NotFound` can mean nonexistent, concealed from this principal, not persisted, unsupported schema or already collected after expiry. Changing the guessed ID is not a discovery mechanism.

Changing `fmt=Human` to `fmt=Json` renders the same report. It does not start a new report and cannot fill missing evidence.

## SDK response decoding and retries

Explicit `Explain` returns a diagnostic document instead of the service's normal result. A generated SDK may report a decoding error if it expects an S3 or DynamoDB response. That document is not a transient malformed provider response.

An explanation-aware client should:

* classify the explained response as non-retryable;
* retain the document and report ID;
* avoid converting it to a successful resource result;
* ensure the request control is signed on every attempt;
* preserve every report ID if an executing request is genuinely retried.

Use `ExecuteAndExplain` only when performing the operation is intended and the ordinary native response is required. See [Explaining SDK Requests](/cloud-adapter/debugging/explain-sdk).

## Endpoint, region and credential mistakes

An origin SDK still signs for its origin service and region while sending to the adapter endpoint. Check these independently:

* process-wide `AWS_ENDPOINT_URL_<SERVICE>` or programmatic endpoint override;
* origin region and signing name;
* origin credential provider used by the SDK;
* adapter's configured target identity;
* network and TLS path to the adapter.

Fixing target-cloud IAM does not correct an invalid origin signature. Fixing origin credentials does not grant the adapter's target identity permission. A connection failure before an HTTP response produces no report headers.

## Build a support bundle

Include:

* product, adapter and SDK versions;
* origin, target and operation;
* exact mode and sanitized control;
* timestamp, endpoint category and request identifiers;
* native status/error and Explain receipt headers;
* report ID, revisions, lifecycle, assurance and limitations;
* the smallest redacted request shape;
* observed target state and how it was checked;
* whether any retry or cleanup ran;
* what remains unknown.

Exclude credentials, authorization headers, signed URLs, secret payloads and unrelated customer data. If using a coding agent, follow the [bounded agent workflow](/cloud-adapter/debugging/agents).