Troubleshooting Explanations

Troubleshoot the service operation and the explanation system as separate paths. A target operation can succeed while its report is temporarily unavailable. A complete report can accurately describe an operation that was refused. A missing report never proves that a write had no effect. Start by preserving the original request time, exact endpoint, operation, mode, response status, origin request IDs, Explain receipt headers and observed target state. Remove credentials and sensitive payload data.

Symptom and next-action matrix

Symptom	Evidence to inspect	Safe next action
Malformed `x-t9-explain` refusal	Exact single header value; size; JSON version, mode, format and unknown fields	Correct the control before signing; do not weaken it to ordinary execution
Unsigned Explain control	Final signed-header set and middleware order	Insert the control before SigV4 signing
Duplicate Explain controls	SDK interceptors, default headers and request override	Keep exactly one field; do not choose one arbitrarily
Explicit mode unsupported for the operation	Error response and directed service profile	Use supported coverage or another test operation; do not expect fallback
`x-t9-explain-unexplained` response	Adapter default named by the header; operation	Treat this request as ordinary execution with no report; investigate coverage
No report receipt	Status/body, `x-t9-explain-unexplained`, transport state and proxy behavior	Determine whether an HTTP response arrived and whether the request was explicitly explained
Permission mismatch	Signing identity, endpoint, target runtime identity and report-read identity	Fix the failing boundary; do not assume origin and target permissions are interchangeable
`409 NotYetVisible`	Report ID, requested `minimumVersion` and current stored revision in the disposition	Keep the previously recorded snapshot and poll the same report after bounded backoff
`503 Unavailable`	Native result, report ID and read route	Retry only the report read
`410 Expired`	Preserved bundle and operation state	Use preserved evidence; repeat the operation only under its own safety rules
`410 Consumed`	Owner acknowledgment and exported evidence	Use the previously exported bundle; an identical owner acknowledgment is idempotent
`404 NotFound`	Exact report ID and owner identity	Retry the exact read with bounded backoff; do not infer persistence, access or operation outcome
Incomplete capture	Lifecycle, assurance and limitations	State what remains unknown and choose one observation that resolves it
Missing backend observation	Planned/attempted evidence, provider ID and native status	Inspect exact target state or use a declared status read
SDK decoding or retry behavior	Raw HTTP metadata, retry count and response content type	Prevent explicit Explain documents from entering normal service retries
Endpoint or signing mismatch	Service endpoint, region, signing name and credential provider	Correct client construction before changing Explain mode
Wrong report format	Existing report ID and desired renderer	Read the same report with another format; do not rerun the origin operation

Failure 1: the control was added after signing

Symptom

The request is refused as an unsigned Explain control, even though the application appears to send a valid value.

Why it happens

x-t9-explain changes whether the request executes and must be integrity-protected. If an HTTP interceptor appends the header after SigV4 signing, the adapter cannot trust it. A reverse proxy can cause the same problem if it adds or rewrites the control after the client signs.

Confirm it

Inspect request construction without printing the authorization credential:

Identify where endpoint selection, request override and signing occur.
Confirm x-t9-explain is present before the signer runs.
Check that the signed-header set covers it.
Confirm there is exactly one value and that no proxy rewrites it.

Fix it

In AWS SDK for Java 2.x, use request override configuration while building the service request:

PutObjectRequest request = PutObjectRequest.builder()
    .bucket(bucket)
    .key(key)
    .overrideConfiguration(c -> c.putHeader(
        "x-t9-explain",
        "{\"version\":1,\"mode\":\"Explain\",\"fmt\":\"Json\"}"))
    .build();

The SDK applies the override before signing. Do not add a second copy in a generic HTTP interceptor.

What not to do

Do not remove the header and send the mutation normally just to see whether the endpoint works. Use a non-mutating operation or fix signing first.

Failure 2: the operation succeeded but the report is unavailable

Symptom

An ExecuteAndExplain call returned its native success and x-t9-explain-id, but reading the report returns 404 NotFound, 503 Unavailable or remains partial.

Interpretation

The origin operation and report have separate lifecycles. Native success is evidence about the origin API response. It does not seal the report. A report receipt identifies correlation intent; it does not prove that a report row was persisted. Report unavailability is evidence about the diagnostic read path, not a rollback of target effects.

Safe response

Preserve the native response, origin request ID, report ID and target resource identity.
Verify the intended target state using the service’s normal read path.
Retry only the exact signed report GET with bounded backoff and the same owner identity.
If a declared origin status poll exists, select the same report ID and exact resource identity.
Record the latest report revision; ignore a late older revision.
Do not replay the mutating operation merely to create another report.

If the report never becomes available, the operation’s native evidence and target-state observation remain the source for deciding cleanup or remediation. Report absence must be recorded as a diagnostic limitation. Do not guess another report ID.

Failure 3: an explicit mode is unsupported

Symptom

The application explicitly requests Explain or ExecuteAndExplain, and the adapter refuses before the operation runs.

Interpretation

An explicit request is fail-closed. The adapter does not silently ignore it, fall back to the adapter default or execute normally. This protects a preview from becoming a mutation.

Safe response

Read the exact refusal and origin operation.
Inspect the directed service profile and operation-specific coverage.
Confirm that the deployed adapter and selected target match the profile you inspected.
Choose a supported non-mutating operation for connectivity testing, or collect ordinary service evidence without claiming an explanation.

Do not switch to x-t9-explain: true as a workaround: that value means ExecuteAndExplain, not a more compatible preview.

Malformed, duplicate and false controls

The adapter refuses these controls before ordinary service processing:

x-t9-explain: false
x-t9-explain: {"mode":"Explain"}
x-t9-explain: {"version":1,"mode":"explain"}
x-t9-explain: {"version":1,"mode":"Explain","unknown":true}

The first has no off spelling; omit the header for ordinary execution. The second lacks version. The third uses the wrong enum case. The fourth contains an unknown field. Empty, non-ASCII, over-4096-byte and repeated fields are also refused. Log a redacted representation of the control and its byte length, not the complete signed request.

Default-caused unexplained requests

When an adapter default asks for explanation but a declared status read must execute ordinarily, the response can include:

x-t9-explain-unexplained: Explain

This says the default was not applied and no report was opened. It is not an error-provenance header and not a report receipt. An explicit unsupported Explain request is refused instead of receiving this fallback. Check the operation’s coverage and whether your client intended an ordinary status read or a report-aware poll.

Missing receipt

No x-t9-explain-id can have several causes:

the request never reached an HTTP service;
no explicit control or adapter default applied;
the operation was served under default fallback and has x-t9-explain-unexplained;
the request was refused before report creation, including some authorization paths;
middleware removed response headers;
the client inspected only a decoded error and discarded raw HTTP metadata.

Do not collapse these cases. First establish whether an HTTP response arrived, then preserve its raw status and headers. For example, an S3 authorization denial occurs before a report opens; the inspected VPC path can represent an authenticated denial. Report behavior is service-specific, not a universal policy-trace promise.

Report read dispositions

Use the status and disposition together:

409 NotYetVisible means the stored revision is below the direct GET’s requested minimumVersion; retain the previously recorded snapshot, record the current stored revision returned by the disposition, and retry the read.
409 NotSealed or RevisionMismatch applies to acknowledgment; read the current report and use its supplied acknowledgeable revision.
410 Consumed means the owner already acknowledged the sealed revision. Repeating the same owner acknowledgment at the same revision is idempotent; another revision is a mismatch.
410 Expired is terminal while the expired record remains; after retention collection, the same report can appear as 404.
503 Unavailable permits a later read attempt, not a mutation replay.
404 NotFound can mean nonexistent, concealed from this principal, not persisted, unsupported schema or already collected after expiry. Changing the guessed ID is not a discovery mechanism.

Changing fmt=Human to fmt=Json renders the same report. It does not start a new report and cannot fill missing evidence.

SDK response decoding and retries

Explicit Explain returns a diagnostic document instead of the service’s normal result. A generated SDK may report a decoding error if it expects an S3 or DynamoDB response. That document is not a transient malformed provider response. An explanation-aware client should:

classify the explained response as non-retryable;
retain the document and report ID;
avoid converting it to a successful resource result;
ensure the request control is signed on every attempt;
preserve every report ID if an executing request is genuinely retried.

Use ExecuteAndExplain only when performing the operation is intended and the ordinary native response is required. See Explaining SDK Requests.

Endpoint, region and credential mistakes

An origin SDK still signs for its origin service and region while sending to the adapter endpoint. Check these independently:

process-wide AWS_ENDPOINT_URL_<SERVICE> or programmatic endpoint override;
origin region and signing name;
origin credential provider used by the SDK;
adapter’s configured target identity;
network and TLS path to the adapter.

Fixing target-cloud IAM does not correct an invalid origin signature. Fixing origin credentials does not grant the adapter’s target identity permission. A connection failure before an HTTP response produces no report headers.

Build a support bundle

Include:

product, adapter and SDK versions;
origin, target and operation;
exact mode and sanitized control;
timestamp, endpoint category and request identifiers;
native status/error and Explain receipt headers;
report ID, revisions, lifecycle, assurance and limitations;
the smallest redacted request shape;
observed target state and how it was checked;
whether any retry or cleanup ran;
what remains unknown.

Exclude credentials, authorization headers, signed URLs, secret payloads and unrelated customer data. If using a coding agent, follow the bounded agent workflow.

Tensor9 Cloud Adapter

How It Works

Getting Started

Configuration

Deployment

Operations

Tuning

Debugging

Testing

AWS

Google Cloud

Azure

Reference

Guides

Troubleshooting Explanations

Symptom and next-action matrix

Failure 1: the control was added after signing

Symptom

Why it happens

Confirm it

Fix it

What not to do

Failure 2: the operation succeeded but the report is unavailable

Symptom

Interpretation

Safe response

Failure 3: an explicit mode is unsupported

Symptom

Interpretation

Safe response

Malformed, duplicate and false controls

Default-caused unexplained requests

Missing receipt

Report read dispositions

SDK response decoding and retries

Endpoint, region and credential mistakes

Build a support bundle

​Symptom and next-action matrix

​Failure 1: the control was added after signing

​Symptom

​Why it happens

​Confirm it

​Fix it

​What not to do

​Failure 2: the operation succeeded but the report is unavailable

​Symptom

​Interpretation

​Safe response

​Failure 3: an explicit mode is unsupported

​Symptom

​Interpretation

​Safe response

​Malformed, duplicate and false controls

​Default-caused unexplained requests

​Missing receipt

​Report read dispositions

​SDK response decoding and retries

​Endpoint, region and credential mistakes

​Build a support bundle

Symptom and next-action matrix

Failure 1: the control was added after signing

Symptom

Why it happens

Confirm it

Fix it

What not to do

Failure 2: the operation succeeded but the report is unavailable

Symptom

Interpretation

Safe response

Failure 3: an explicit mode is unsupported

Symptom

Interpretation

Safe response

Malformed, duplicate and false controls

Default-caused unexplained requests

Missing receipt

Report read dispositions

SDK response decoding and retries

Endpoint, region and credential mistakes

Build a support bundle