Debugging AWS OpenSearch

Separate domain provisioning from search execution. An AWS OpenSearch management request describes or changes a domain. Indexing and search use an HTTP endpoint with engine-specific authentication and APIs. A healthy deployment status does not prove that a particular index, mapping, query or credential works. Read the directed service profile before interpreting a failure. An OpenSearch-engine target, an Elasticsearch-family mapping and a retained AWS domain have different request routes and operating responsibilities. Follow the profile’s documented application route: native engine traffic can go directly to the target, while an Elastic-family mapping can require an origin-facing translation path. Changing an AWS management endpoint does not automatically redirect the application’s search client.

Establish the exact context

Record the adapter version, client version, actual backend engine/version, selected mapping, index or alias, endpoint, authentication method and topology. Record both the logical AWS region used by a signed client and the target deployment’s physical region. Include replica settings, storage class and managed-versus-operated ownership when investigating availability. Keep the original HTTP status and engine error type. For bulk ingestion, inspect individual item results as well as the outer HTTP response; a request-level success does not establish that every document was indexed. Avoid replaying a timed-out write before determining its possible effects and the application’s document-ID/idempotency rules.

Symptom	Smallest useful check	Likely next investigation
Domain creation or resize remains pending	Read the selected provider’s operation status, failure details and deployment events	Quota, capacity, permissions, storage or an incomplete configuration change
TLS failure or connection timeout	Check the application’s actual search hostname, trusted CA, port and network access	Wrong route, private networking or certificate mismatch
HTTP 401 or 403	Identify whether the response comes from the origin-facing adapter or native engine	IAM identity/signing, backend credentials or index permissions
Index not found or unexpected empty results	Read the exact index/alias and the application’s query target	A different index, alias, routing value, tenant or visibility interval
Cluster health is yellow or red	Read cluster health and the affected index’s shard allocation state	Replica placement, missing primary shards or resource pressure
HTTP 429 or rejected operations	Inspect bounded workload concurrency, rejected work, CPU, heap and disk	Overload, uneven shard placement or insufficient capacity
Plugin or endpoint error	Compare that exact operation with the selected engine/version’s profile	An unsupported plugin, translation or client API assumption

On AWS, configuration changes have their own processing and change-progress states. For a retained AWS domain, inspect those states. For another managed service or an operated Kubernetes cluster, inspect its actual operation or rollout events. An AWS-style label cannot replace target-state evidence.

Start with a bounded read-only engine request

Using the application’s authenticated search client, request GET /_cluster/health with a short timeout. If that succeeds, narrow the check to the affected index and read its settings or mapping. Health reports shard allocation: green means primaries and replicas are allocated, yellow means some replicas are unassigned, and red means at least one primary is unassigned. It does not validate query answers. See the native Cluster Health API. For a target that actually uses HTTP basic authentication, this optional Python example prints only a small health summary. Install requests first. Use the authenticated origin-facing endpoint for an Elastic-family mapping when the profile requires it; use the real signing client instead if the application path requires IAM signing.

import getpass
import requests

base = input("Configured HTTPS search endpoint: ").rstrip("/")
ca_bundle = input("Trusted CA bundle path: ")
with requests.Session() as session:
    session.auth = (input("Username: "), getpass.getpass("Password: "))
    response = session.get(
        base + "/_cluster/health",
        params={"level": "cluster", "timeout": "5s"},
        timeout=10, verify=ca_bundle,
    )
    response.raise_for_status()
    health = response.json()
    print({key: health.get(key) for key in
           ("status", "number_of_nodes", "unassigned_shards", "timed_out")})

A permission refusal to a cluster-wide endpoint does not establish that index searches are broken. Repeat the smallest authorized read for the affected index using the application’s identity. Preserve TLS verification and distinguish authentication from authorization instead of changing credentials, network rules and index settings together.

Compare a native OpenSearch cluster with Elastic Cloud

These are illustrative investigations for two catalog mappings. Use the application’s selected backend, engine/client versions, index or alias and configured authentication route. The following HTTP paths are read-only requests to send through that authenticated client; they are not unauthenticated curl instructions.

Kubernetes OpenSearch: explain an unassigned replica

For AWS OpenSearch → OpenSearch on Kubernetes, search traffic goes directly to the native cluster. Suppose the bounded health check reports yellow, the affected index’s primaries are allocated, and one replica is unassigned. Narrow the diagnostic to that index, then inspect allocation:

GET /products/_settings?flat_settings=true
GET /_cluster/allocation/explain

Replace products with the affected permitted index. The native allocation API chooses an unassigned shard when no body is supplied; confirm the returned index and shard before using its explanation for this incident. Its decider, decision and explanation identify the applicable allocation constraints. Illustrative result: the only eligible data node already holds the primary, so the replica cannot be assigned there. Compare replica count with eligible nodes, failure-domain placement and storage availability. This result calls for a topology or allocation review, not a credential reset or re-import. Yellow health does not by itself prove that product searches failed. If the allocation explanation concerns a different index, request the affected shard explicitly through the native API rather than applying its conclusion to products.

Elastic Cloud: identify which permission or API was refused

For AWS OpenSearch → Elastic Cloud Hosted, the application uses the origin-facing translator, which verifies AWS SigV4; backend calls use Elastic credentials or API keys. It is not the same direct OpenSearch route as the Kubernetes example. Suppose the same application’s index metadata read succeeds but another operation returns a refusal:

GET /products/_mapping
GET /products/_settings?flat_settings=true

These successful reads establish only their own index access and supported translation. Preserve the failing operation’s exact path and error type. For a supported index/search operation refused by Elastic, ask the backend owner to inspect the configured Elastic identity’s privileges. A valid origin IAM signature does not grant an Elastic API key extra rights. If the failed path instead belongs to OpenSearch PPL or the OpenSearch Security API, the catalog mapping excludes it. Broader credentials will not add that API. Elasticsearch compatibility headers address selected REST-version differences, as described in Elastic’s compatibility documentation; they do not install OpenSearch plugins. Compare the required feature with this directed profile before changing the application or target. Keep native allocation explanations, query profiles and Tensor9 reports distinct; these native reads do not promise a Tensor9 explanation receipt.

Worked case: an import succeeds but products are missing

A nightly product import receives HTTP responses, yet several products cannot be found. First inspect each bulk item’s status and error. Suppose some items were rejected during a concurrency spike while other items succeeded. Compare rejected work with CPU, memory pressure and shard distribution before adding retries. AWS documents these causes in its OpenSearch troubleshooting guide; inspect equivalent native metrics on other targets. Reduce or pause additional import concurrency while preserving the failed-item record. Retry only the identified failures under the application’s idempotency policy. For successfully indexed documents, compare direct document reads with searches against the exact alias, routing value and refresh interval. An accepted write and immediate search visibility are separate observations. If the issue instead follows an engine change, compare mappings, analyzers, plugin endpoints and client response parsing with the directed profile. OpenSearch _plugins APIs do not become universally compatible with Elasticsearch by changing a hostname. Follow OpenSearch tuning for measured shard, refresh, bulk-concurrency and capacity changes after establishing compatibility.

Explain is separate from native search diagnostics

Use Tensor9 Explain only where the installed adapter version and the exact operation advertise support. Native search traffic does not automatically produce Tensor9 receipts. OpenSearch’s own query-explanation or profiling APIs describe engine behavior; they are separate from Tensor9’s explanation of an adapted request and can have their own cost and permission requirements. When an admitted operation supplies x-t9-explain-id, retrieve that existing report through the install’s synthetic AWS t9 service with the original owner’s IAM credentials and SigV4 signing name t9. Preserve the report’s revision and assurance. Reads do not rerun indexing or search. The default retention is 24 hours from creation; the published expiry is fixed and reads do not renew it. Check installed protocol support before using retrieval examples. PostgreSQL query plans remain a planned integration with operation-specific qualification. A PostgreSQL control store does not turn a Lucene/OpenSearch query into PostgreSQL SQL. Where qualified, full SQL plan JSON can contain customer constants, predicates and relation names and is restricted to the exact report owner. For support, collect redacted configuration, versions, selected profile, endpoint route, index/alias, timestamps, native status/error, a small failed-item summary and relevant provider or cluster events. Attach an existing receipt and expiry when available. Exclude credentials, document bodies and unrestricted query logs. A Support Agent can help separate translation coverage, backend permissions and shard evidence before recommending a cluster change.

Tensor9 Cloud Adapter

How It Works

Getting Started

Configuration

Deployment

Costs

Support

Amazon Web Services

Google Cloud

Microsoft Azure

Making Trade-offs

Operations with Agents

Tuning

Debugging

Testing

Guides

Reference

Debugging AWS OpenSearch

Establish the exact context

Start with a bounded read-only engine request

Compare a native OpenSearch cluster with Elastic Cloud

Kubernetes OpenSearch: explain an unassigned replica

Elastic Cloud: identify which permission or API was refused

Worked case: an import succeeds but products are missing

Explain is separate from native search diagnostics

​Establish the exact context

​Start with a bounded read-only engine request

​Compare a native OpenSearch cluster with Elastic Cloud

​Kubernetes OpenSearch: explain an unassigned replica

​Elastic Cloud: identify which permission or API was refused

​Worked case: an import succeeds but products are missing

​Explain is separate from native search diagnostics

Establish the exact context

Start with a bounded read-only engine request

Compare a native OpenSearch cluster with Elastic Cloud

Kubernetes OpenSearch: explain an unassigned replica

Elastic Cloud: identify which permission or API was refused

Worked case: an import succeeds but products are missing

Explain is separate from native search diagnostics