> ## Documentation Index > Fetch the complete documentation index at: https://docs.tensor9.com/llms.txt > Use this file to discover all available pages before exploring further. # Debugging AWS S3 > Diagnose S3 signing, object reads, streaming failures and multipart uncertainty on the selected backend. Start with one bucket, key and request attempt. Record the adapter version, selected backend, origin endpoint, AWS region, SDK version, addressing style and whether the request names an object version. The [S3 profile](https://docs.tensor9.com/cloud-adapter/service-catalog/aws/databases-storage/s3.md) describes operation coverage separately for each target. Working `PutObject` behavior on GCS does not establish the same multipart, checksum or version behavior on Azure Blob or an S3-compatible backend. Keep the origin response, target object state and explanation separate. An upload timeout can leave an object, and a readable report does not prove that the upload completed. ## Classify the symptom | Symptom | Inspect first | Useful next observation | | - | - | - | | DNS, TLS or connection failure | Printed endpoint, proxy path and certificate hostname | Confirm that the SDK actually used that endpoint. | | Signature or authorization refusal | IAM identity, session token, signing region, Host and signed path | Preserve the final request metadata without credentials. | | `NoSuchBucket`, `NoSuchKey` or unexpected `403` | Account, exact bucket/key, case, encoding and version | Perform a permitted metadata read for that exact identity. | | HTTP success followed by a read failure | Body consumption, cancellation and SDK retries | Establish whether the complete response body was received. | | Different length, ETag or checksum | Upload form, algorithm and target mapping | Compare the same object version using a supported integrity check. | | Multipart completion timed out | Upload ID, part list and possible final object | Inspect state before another completion attempt. | | Cached or conditional response looks wrong | Version, Range, conditions and cache headers | Compare the response with those request conditions. | ## Check addressing before changing permissions Path-style requests put the bucket in the path; virtual-hosted requests put it in the hostname. The adapter decodes the bucket and key from that actual request. Select the addressing style documented for your deployment and make the choice before signing. A private endpoint without suitable wildcard DNS and certificates may require a different client setup from a public S3 hostname. Check the final Host, path and query after endpoint resolution. Preserve escaped characters and duplicate or ordered query values where they matter to the signature. A proxy that rewrites the host or path after SigV4 signing can create a signature failure even when the access key is correct. Do not fix it by disabling signing or TLS verification. Origin IAM authorization and target storage authorization are separate. A valid origin signature does not prove the adapter's Google, Azure or S3-compatible runtime identity can access the mapped object. ## Read the exact object's metadata With the workload's authorized IAM credentials available through the normal AWS credential chain, set `S3_ENDPOINT`, `AWS_REGION`, `BUCKET` and `OBJECT_KEY` to the original request's values: ```bash theme={null} aws s3api head-object \ --endpoint-url "$S3_ENDPOINT" \ --region "$AWS_REGION" \ --bucket "$BUCKET" \ --key "$OBJECT_KEY" \ --output json ``` This is a metadata read, not an upload. It still follows the adapter's configured Explain default; a preview refusal is not an object-metadata result. If the original request used a known `VersionId`, add `--version-id "$VERSION_ID"` only where that operation is supported. A read of the latest object cannot establish the contents of an earlier version. A delete marker and a physically absent version also require different interpretations. A failed HEAD normally has no explanatory object body. On AWS, permissions can affect whether a missing object yields `403` or `404`; do not interpret either code without the selected adapter's authorization behavior. See the [AWS HeadObject reference](https://docs.aws.amazon.com/cli/latest/reference/s3api/head-object.html). Ask the target operator to inspect the mapped physical bucket and object independently when origin metadata and target state disagree. Do not derive the physical name from the logical name. ## Verify bodies and integrity separately For `GetObject`, receiving headers is not the same as receiving the complete stream. Consume the body to its supported completion condition, handle a later error, and close it when finished or abandoned. Retain whether cancellation, a socket failure or a retry happened after headers arrived. A report can distinguish an attempted response from an observed completed read only within its captured scope. Record the requested checksum algorithm and whether it was sent in a header or trailer. An SDK upgrade can change that wire form. Check the backend's supported checksum and streaming forms before changing client settings. Do not disable integrity checks merely to make a failing test pass. ETag interpretation depends on the upload and backend mapping; do not assume every ETag is the MD5 of the final object. Multipart uploads need their own comparison. For an approved read of a bounded test object, compare its complete bytes or a separately computed digest with the known fixture. Keep Range and conditional reads distinct from full reads. A `206`, `304` or failed condition can be the correct result. Preserve `Content-Length`, `Content-Range`, ETag, version and native `Cache-Control`/`Vary` values when supplied. Explanation retrieval has its own cache policy; it is not an object-cache response. ## Compare the same read on three backends These are illustrative investigations of an existing `HeadObject` failure, not results from your deployment. Keep the origin command above unchanged except for the installed endpoint and the exact logical bucket/key. Ask the target operator to obtain physical identifiers from the installed mapping; a logical S3 bucket is not necessarily the native bucket or container name. Native checks use separately authorized provider identities, not the caller's AWS credentials. ### GCS: distinguish origin authorization from access to the mapped object For **S3 → Google Cloud Storage**, core object reads use Cloud Storage's S3-interoperable XML API. Suppose origin `HeadObject` returns an authorization error while an operator believes the object exists. With the mapped physical bucket and key, the operator can make this metadata-only check: ```bash theme={null} gcloud storage objects describe "gs://$GCS_BUCKET/$GCS_OBJECT" \ --project "$GCP_PROJECT" --format=json ``` The [native describe command](https://docs.cloud.google.com/sdk/gcloud/reference/storage/objects/describe) should identify the object and its generation and size when authorized. Compare that identity with the original request's version and mapping. An operator's successful read proves that operator's access, not the adapter runtime identity's access. If the exact object exists but the adapter cannot read it, inspect the adapter's configured Google identity and bucket permissions. Do not change the origin IAM policy solely because the target refused its own credential. For multipart uncertainty, inspect the final object before retrying: this mapping stages parts and completes through GCS compose, rather than forwarding an AWS completion verb unchanged. ### Azure Blob: compare identity and length, not opaque ETags For **S3 → Azure Blob**, the adapter translates the request to Blob APIs. For the same logical read symptom, use the mapped storage account, container and blob name: ```bash theme={null} az storage blob show --account-name "$STORAGE_ACCOUNT" \ --container-name "$CONTAINER" --name "$BLOB_NAME" \ --auth-mode login --output json ``` This uses an already authorized Microsoft Entra login; see [Blob CLI authorization](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-data-operations-cli). Compare the exact blob identity, version where applicable, and content length with the origin observation. Blob's opaque ETag is not an S3 MD5 checksum. If the bytes and identity match but an application rejects the ETag as a digest, investigate that assumption before retransmitting data. A successful native read under the operator identity still does not establish the adapter identity's permissions. ### MinIO: separate an adapter route from the store itself For the catalog's **S3 → MinIO** mapping, the adapter signs and forwards supported operations to the customer's S3-compatible store. A target operator can reuse the same `aws s3api head-object` command against the configured MinIO HTTPS endpoint, physical bucket/key and a separate protected MinIO credential profile. Keep the original command against the Tensor9 origin endpoint as the comparison. Never copy MinIO keys into the workload's origin IAM profile. With `MINIO_PROFILE` already configured in protected credential storage and the physical identifiers confirmed, the read is: ```bash theme={null} aws s3api head-object --endpoint-url "$MINIO_ENDPOINT" \ --region "$MINIO_SIGNING_REGION" --profile "$MINIO_PROFILE" \ --bucket "$MINIO_BUCKET" --key "$MINIO_OBJECT_KEY" --output json ``` If native HEAD succeeds and origin HEAD fails, compare logical mapping, signing region, addressing style, TLS trust and origin authorization. If both fail for the exact object, inspect native object state and the store's read error first. Disk, node and quorum diagnostics belong to the MinIO operator; an adapter retry cannot repair unavailable storage. These three cases require different target evidence even though the application sees one S3 operation. ## Worked case: completion timed out Suppose an existing test upload to `reports/example.bin` lost its `CompleteMultipartUpload` response. 1. Preserve its upload ID, ordered part numbers and ETags, request time, retries and any explanation receipt. Do not send another completion merely to collect diagnostics. 2. Read metadata for the exact bucket/key and, if known and supported, the expected version. Compare length and application metadata with the intended object. 3. Ask the target operator to confirm the mapped object's version or generation and content. Finding an object is stronger evidence than the caller timeout, but its identity and bytes must match this upload. 4. If the final object is absent, use a supported `ListParts` read for the known upload ID to distinguish retained parts from an unknown upload. Do not substitute an unsupported `ListMultipartUploads` operation or abandon unrelated uploads. 5. Decide any completion retry or cleanup using the observed state and the application's recovery policy. The metadata read itself neither completes nor aborts the upload. ## Use Explain without changing the investigation For a supported request, retain `x-t9-explain-id` and retrieve the existing report through the dedicated `t9` service using the original owner's IAM identity. Follow [the SDK workflow](https://docs.tensor9.com/cloud-adapter/debugging/explain-sdk.md) and its installed-version requirements. `Explain` preview does not perform the proposed action; `ExecuteAndExplain` is a real operation. Neither returns an inline plan in place of an S3 response. The [planned PostgreSQL plan integration](https://docs.tensor9.com/cloud-adapter/debugging/postgres-query-plans.md) can describe instrumented PostgreSQL metadata work; it does not explain a GCS or Blob byte transfer merely because some adapter state uses PostgreSQL. Full plan JSON can contain application literals. Sanitize an exported copy before sharing it. Include the exact version, addressing style, operation, version/upload identity, native result, stream completion, supported integrity checks and target observations in a [support bundle](https://docs.tensor9.com/cloud-adapter/support/getting-help.md). Use [S3 tuning](https://docs.tensor9.com/cloud-adapter/tuning/aws-s3.md) only after the failing behavior is understood. A [Support Agent](https://docs.tensor9.com/cloud-adapter/support/agents.md) can help compare the origin response with the exact mapped object and target identity, then carry unresolved storage findings to the responsible operator. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.