Tuning AWS SQS

Before running a tensor9 command, install the CLI, set T9_API_KEY, and verify your customer account. Online commands are authenticated and their adapter detail is limited to mappings your account is entitled to use.

SQS tuning starts with the work your consumers perform. A Classic queue of independent notifications, a long export and a FIFO account ledger need different delivery and capacity settings, even in the same target cloud. This article covers backend selection, a Pub/Sub endpoint experiment, PostgreSQL queues, and a measurement and change procedure. Read the Classic profile or FIFO profile alongside this article. Each describes an intended directed mapping and its real limits. Confirm that the exact design, origin type and configuration are supported by the selected installed release before tuning it. An authored design is not evidence of deployment or measured capacity. See SQS trade-offs for the complete portfolio and total cost.

Decide what you are changing

There are three places to tune a queue workload:

Setting	Where it belongs	Example
Queue placement	`t9:backend` on `CreateQueue`	Place a Standard queue on Google Pub/Sub or Cloud SQL for PostgreSQL.
SQS behavior	SQS queue attributes and operation parameters	Set visibility, retention, receive wait and batch size.
Shared backend capacity	The installation or infrastructure that owns the backend	Size the shared PostgreSQL server or configure the Pub/Sub resources owned by the installation.

VisibilityTimeout is already an SQS attribute. Supply it through the SQS API. A queue’s PostgreSQL server size is a different kind of choice: several queues can share that server, so changing one queue’s tags must not resize all of its neighbors. The t9:tuning:<path> mechanism is for documented target-native fields. A target cloud’s API field is not automatically an accepted queue tuning path. Use the exact paths published for the selected mapping; do not turn a native console setting into a guessed tag. See Tuning Your Adapters.

Choose a backend

Choose Classic for independent at-least-once messages or FIFO for per-group order and the selected deduplication contract. Extra capacity cannot turn a Classic broker mapping into FIFO, remove group serialization or supply a missing receipt authority.

Backend	Environments	Why choose it?	Tuning consideration
Pub/Sub Classic	Google Cloud	Native message custody and StreamingPull	10–600-second queue visibility; no delayed first delivery; native deadline/redelivery receipt rules; shared control PostgreSQL remains.
Direct PostgreSQL	Cloud SQL, Azure Flexible Server, or the selected cluster/managed host	Durable claims for Classic/FIFO	Direct FIFO allows one in-flight message per group; logical databases still share server capacity.
Azure native and document choices	Azure	Service Bus Standard/Premium, Cosmos capacity/scopes, Queue Storage small Classic	Check native locks, partition scope, encoded size and receipt-update restrictions before tuning capacity.
Google transactional/composite designs	Google Cloud	Firestore/Spanner; PostgreSQL plus full-body Pub/Sub or wakeups	Distinct authority resources and conservative batch/replay restrictions; confirm selected-release availability.

The presence of a native queue service in a cloud does not make it an available backend. Choose a current catalog direction returned for your account by tensor9 adapt svc list. Backend selection is made when the queue is created. For example:

{
  "QueueName": "notifications-tuning",
  "Attributes": {
    "VisibilityTimeout": "120",
    "ReceiveMessageWaitTimeSeconds": "20",
    "MessageRetentionPeriod": "345600"
  },
  "tags": {
    "t9:backend": "google::1.0.0::pubsub",
    "purpose": "tuning-test"
  }
}

The lower-case tags property is the AWS CLI’s CreateQueue input shape. t9:backend selects where this queue is placed. It is not a setting that moves an existing queue when updated with TagQueue.

Try SQS on Pub/Sub

This experiment requires an already-running, owned test install that is licensed and configured to serve Classic SQS on native Pub/Sub. It does not install or start Cloud Adapter, select a FIFO composite or establish another release/backend’s availability. Use its actual authenticated SQS endpoint, signing Region and existing origin workload credentials. This example creates a disposable Standard queue with two minutes of visibility, twenty-second long polling and four-day retention. Use a test project and a fresh queue name.

Confirm the existing install and permissions

Obtain the test install’s SQS endpoint, signing Region, existing origin workload profile, selected runtime/backend and native resource owner. Confirm permission to create a fresh disposable queue and no other consumers on that queue. Read the mapping’s visibility, delay and dead-letter contract before the experiment.The install owner supplies its valid license, persistent control state, actual backend configuration and target credentials. Google lifecycle permissions differ from publish/consume permissions. Merely installing the Tensor9 CLI and configuring Application Default Credentials does not establish a serving adapter. Use installation and authentication and configuration for setup outside this experiment.

Point the AWS CLI at the adapter

In a shell with the existing AWS workload profile, configure the actual endpoint and signing Region:

export T9_CLOUD_ADAPTER_ENDPOINT="<owned test install SQS endpoint>"
export AWS_ENDPOINT_URL_SQS="$T9_CLOUD_ADAPTER_ENDPOINT"
export AWS_PROFILE="<existing origin workload profile>"
export AWS_REGION="<install signing Region>"

AWS_ENDPOINT_URL_SQS applies to SQS commands in this shell. You can instead put --endpoint-url "$T9_CLOUD_ADAPTER_ENDPOINT" on each command. The existing profile supplies the origin-facing identity; Google Cloud credentials and target configuration belong to the running install. A Tensor9 account API key does not replace these SQS workload credentials.

Create and inspect the queue

Save the preceding JSON as notifications-tuning.json, then run:

export SQS_TUNING_QUEUE_URL="$(aws sqs create-queue \
  --cli-input-json file://notifications-tuning.json \
  --query QueueUrl --output text)"

aws sqs get-queue-attributes \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --attribute-names VisibilityTimeout ReceiveMessageWaitTimeSeconds MessageRetentionPeriod

aws sqs list-queue-tags --queue-url "$SQS_TUNING_QUEUE_URL"

Confirm the three declared attributes and the selected backend. Also inspect the native Pub/Sub subscription’s acknowledgement deadline and retention. A successful queue create is not a measurement of workload performance.

Send, receive and settle one message

Send a small representative payload:

aws sqs send-message \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --message-body '{"job":"send-email","order":"order-1001"}'

aws sqs receive-message \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --max-number-of-messages 1 \
  --wait-time-seconds 20 \
  --visibility-timeout 120

Copy the returned receipt handle. After processing the message, settle that exact receive:

aws sqs delete-message \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --receipt-handle '<receipt handle from this receive>'

Record the successful delete acknowledgement and the processed job ID. Observe native subscription backlog and redelivery over an appropriate interval. Neither an empty receive nor an immediate metric alone proves settlement: metrics can lag, and a message can still be hidden by its visibility timeout.

Finish the trial

Confirm all test messages are settled. Remove only the disposable queue and resources created for this trial, using the lifecycle owner that created them. Deleting a logical queue does not always delete its pre-provisioned native infrastructure.Leave the existing install running under its owner. Clear the endpoint override if you are returning to native AWS work:

unset AWS_ENDPOINT_URL_SQS

Pub/Sub: tune visibility, batching and retries

Pub/Sub is a good starting point for independent, short-running jobs. Its acknowledgement deadline constrains the SQS visibility lease.

Example: workers taking 45–90 seconds

Start with a 120-second queue visibility timeout:

aws sqs set-queue-attributes \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --attributes VisibilityTimeout=120,ReceiveMessageWaitTimeSeconds=20

Measure time from receipt to successful deletion, including application retries and downstream calls. If workers sometimes exceed two minutes, decide whether to shorten their work, extend the current lease within the backend’s range, or select a backend suitable for longer claims.

aws sqs change-message-visibility \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --receipt-handle '<current receipt handle>' \
  --visibility-timeout 300

The adapter accepts Pub/Sub visibility values of 10–600 seconds. A per-message value of zero releases the claim. This does not guarantee a lease for an arbitrarily long job. Receipts are native acknowledgement IDs for the delivery on the selected subscription; settlement uses that subscription and ID. Stream closure alone does not establish that the token is invalid. Restart interrupts local buffering without a qualified SQS-style durable-receipt guarantee. Use the current delivery’s receipt and plan for redelivery. An acknowledgement after deadline expiry may return success even though redelivery can still occur.

Example: receive ten messages without hiding unstarted work

aws sqs receive-message \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --max-number-of-messages 10 \
  --wait-time-seconds 20 \
  --visibility-timeout 120

Ten is a maximum, not a promise that ten messages will be returned. If one worker processes the batch sequentially and each item takes 30 seconds, the later items can lose their visibility lease before processing starts. Use enough processing concurrency for the received batch, reduce its size, or choose an appropriate lease. Count time waiting inside your worker as part of the processing budget.

Example: tolerate several attempts before dead-letter delivery

Pub/Sub accepts a dead-letter threshold of 5–100 delivery attempts. For a threshold of ten, the SQS policy has this shape:

{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:000000000000:notifications-dead-letter\",\"maxReceiveCount\":\"10\"}"
}

Replace the account, region and destination with your installation’s origin-facing values. Prepare the destination and the permissions needed for forwarding before applying the policy. Pub/Sub forwards to a dead-letter topic, and delivery-attempt thresholds are best-effort. Verify the destination subscription as well as the topic so that operators can actually inspect forwarded messages. ApproximateReceiveCount is meaningful on this backend only when the dead-letter policy and its permissions are configured. A queue without that policy reports 1; that value must not drive an application poison-message cutoff.

Example: delayed jobs

A nonzero DelaySeconds does not map to Pub/Sub. If the job must become available five minutes after publication, choose the PostgreSQL backend and use a per-message delay. A longer visibility timeout is not a substitute: visibility begins when a consumer receives a message, while delivery delay applies before its first receive.

PostgreSQL for delayed or long-running work

Direct PostgreSQL serves Classic and FIFO using the SQS-owned message database on the selected host, including Cloud SQL, Azure Flexible Server and CloudNativePG. Host placement changes recovery and operation: a managed database outside the cluster survives replacement of that cluster; cluster-hosted volumes need explicit recovery. Performance depends on physical capacity, storage, network placement and workload. The direct FIFO engine permits one message in flight per group; ordered-batch alternatives are separate designs.

Prepare the shared database

Prepare the SQS-owned message database and its authenticated runtime connection through the installation’s resource owner. It is distinct from the persistent shared PostgreSQL control catalog. A separate logical database/login provides data and permission isolation, but shares CPU, IOPS, WAL and server connection limits when it uses the same instance. Dedicated physical server capacity is an additional allocation. See Configuration Files for the existing binding contract; do not invent a native tuning tag for credentials or infrastructure. For the following Cloud SQL experiment, obtain an already-running owned test endpoint licensed and configured for Classic SQS on Cloud SQL. Its owner must have prepared the actual SQS message binding, target database credentials and persistent control state. Use the endpoint/profile/Region steps above with this install’s values. Do not assume a previous endpoint or queue URL selects the new backend. This experiment supplies no first-install launcher command or configuration document.

Example: export jobs with a fifteen-minute processing budget

{
  "QueueName": "export-jobs-tuning",
  "Attributes": {
    "VisibilityTimeout": "900",
    "ReceiveMessageWaitTimeSeconds": "20",
    "MessageRetentionPeriod": "604800"
  },
  "tags": {
    "t9:backend": "google::1.0.0::cloudsql::postgresql",
    "purpose": "tuning-test"
  }
}

This gives consumers a fifteen-minute claim and keeps messages for seven days. It does not allocate fifteen minutes of compute, reserve database capacity or guarantee that a worker finishes on time. Retention can expire a message while it is in flight; extending visibility does not extend retention. Current receipt handles survive an adapter restart because their claim state is stored in PostgreSQL. After a message is claimed again, an older consumer must not assume that its previous handle still owns the message. Make export completion idempotent, for example by recording the job’s result under a stable job ID.

Example: defer a job for five minutes

After creating the export queue, save its returned URL as SQS_EXPORT_QUEUE_URL:

aws sqs send-message \
  --queue-url "$SQS_EXPORT_QUEUE_URL" \
  --message-body '{"jobId":"export-1001","account":"acme"}' \
  --delay-seconds 300

Verify that the message is not returned before its delay elapses, then receive and delete it normally. Test the timing with a small workload before relying on it for a production scheduling requirement. This per-message delay example is for Classic. FIFO rejects per-message delay; a delayed or unavailable predecessor must not be skipped to deliver a later group member.

Example: size a server for two queues

Suppose email-jobs receives small messages all day and export-jobs produces a large evening burst. Both queues share one PostgreSQL server. Measure them together: a test of email-jobs alone cannot show whether the export burst will delay it.

Resource	What to measure	Change to consider
CPU	Sustained utilization and query latency during the combined peak	Increase the database’s compute allocation or reduce excessive polling.
Storage	Write latency, IOPS, growth and free space	Increase the storage allocation or choose an appropriate storage performance level.
Connections	Active/waiting connections across all adapter processes	Budget connections for the complete deployment before increasing worker or adapter counts.
Availability	Recovery time and queue accessibility during failover	Choose the host’s HA and backup configuration that meets the workload’s requirements.
Placement	Network latency between adapter, consumers and database	Keep the request path close where residency and availability requirements permit.

Make these changes through the infrastructure that owns the database. A CreateQueue tag does not resize the shared server. Likewise, adding more adapter processes does not add database capacity; it can increase the number of competing database connections. For direct FIFO, measure the busiest group’s receive-to-settlement time. A group taking T seconds per settled head is bounded by roughly 1/T useful completions, regardless of idle capacity elsewhere. More independent ready groups can improve aggregate concurrency. A larger receive collects multiple group heads rather than a pipeline within one group. Test stalled groups and bounded scheduler/receiver slots; do not infer fairness from average throughput.

Tune Azure and transactional alternatives by their actual authority

Design	Capacity/setting owner	Constraint that more capacity does not remove
Native Service Bus Standard/Premium	Provisioned namespace/entity settings; Premium messaging-unit allocation	Five-minute visibility profile, link-bound receipts, finite fresh-entity inventory and native FIFO response/replay limits
Service Bus with SQL FIFO authority	SQS-owned Azure PostgreSQL message database plus broker wakeups	SQL group claims, conservative ordered-batch/replay restrictions and two-service cost
Cosmos serverless/manual/autoscale	Explicit account/container RU and storage allocation	Queue- versus MessageGroup-dedup scope, logical-partition storage and transaction size; more RU does not raise storage limits
Queue Storage small Classic	Complete encoded admission and initial receive visibility	64-KiB encoded ceiling; at most 48 KiB raw before envelope/attributes; visibility-token rotation needs qualified retained aliases
Firestore/Spanner	Transactional queue storage, indexes and selected regional capacity	Hot-group serialization, claim/replay state and transaction work
Pub/Sub full-body FIFO plus SQL	SQL message authority plus broker payload transport	First delivery waits for broker observation; full body/attribute/envelope per-key bytes matter
SQL FIFO with Pub/Sub wakeups	SQL authority; small hints and repair scans	SQL remains payload/claim authority; compare with LISTEN/NOTIFY before adding broker cost

Native Service Bus ReceiveRequestAttemptId replay is unsupported; broker duplicate detection is not a durable receive-attempt record. Group-partitioned Cosmos also rejects explicit attempt IDs. Queue-scope Cosmos and the transactional ordered-batch designs replay unchanged five-minute attempts, with explicit restrictions after claims are modified. Do not assume those behaviors from direct PostgreSQL’s simpler one-head engine. Cosmos queue-wide deduplication keeps the entire queue in one authoritative partition. MessageGroup scope distributes groups only when explicitly selected and changes which duplicate IDs are suppressed. The selected profile’s 256-KB message admission is separate from larger native document/transaction limits; those limits do not automatically raise accepted SQS message size. Include payload, attributes/index overhead, dedup markers, attempts, terminal records and cleanup lag in retained bytes. The 20-GB logical-partition ceiling applies to the whole queue or busiest group respectively. A capacity upgrade does not change the scope or raise that storage limit. Serverless-to-provisioned conversion is account-wide and irreversible; it needs an explicit resource decision. Queue Storage’s unchanged current native receipt permits Receive→Delete across adapter restart. Successful native visibility changes rotate that token without returning a replacement SQS handle. The native design does not specify the retained alias authority needed to preserve post-update SQS operations over arbitrary replicas/restarts. Treat visibility updates and subsequent settlement as conditional; without qualified mapping, the supported contract requires refusal before effects. No Table or SQL alias resource is implicitly selected. Atomic dead-letter moves require source and destination in the same supported SQS-owned message transaction. Separate databases on one server, different stores or the shared control catalog do not provide that atomicity. Cosmos cross-partition/store transfers are unsupported in the transactional FIFO design. Classic copy/delete paths and application poison routing have different guarantees.

Respect backend limits

For Pub/Sub, nonzero queue- or message-level delay, retention below ten minutes, visibility outside the documented range and a dead-letter threshold outside 5–100 are rejected. They are not adjusted into equivalent behavior. Native subscription/delivery receipt effectiveness follows acknowledgement deadlines and redelivery, with no qualified restart-continuity guarantee. Google Cloud backlog metrics are operational observations with their own timing and definitions; they are not SQS visible/in-flight/delayed availability counts. Pure Pub/Sub has no authoritative source for those SQS attributes in this design. For PostgreSQL, use the configured profile’s message and retention limits. Cleanup can remove messages in flight; stale receipts cannot settle a newer claim. Requested counts read one message-state snapshot: eligible rows are visible, future-visible rows with receipts are in flight, and future-visible rows without receipts are initially delayed. They can change immediately afterward. Unavailable requested counts return an explicit unsupported result, not a fabricated zero. Read the Cloud SQL PostgreSQL profile before designing alerts around these responses. For CloudNativePG, a full cluster rebuild loses queue data unless a separate recovery procedure restores it. Test with the exact PostgreSQL image/version, adapter version, queue schema and topology you will run. The available Standard-queue benchmark does not establish another host’s capacity or verify message and claim recovery after a database outage. Test those recovery paths separately before production.

Measure the workload before changing capacity

Use a bounded workload with known message count, payload size and processing duration. Keep those inputs the same when comparing two settings.

Observation	Likely questions to investigate
Backlog grows while workers are busy	Is processing slower than arrival? Is a downstream dependency limiting throughput?
Many duplicate receives	Does work exceed visibility? Did the adapter or stream restart? Are batches waiting unprocessed?
High empty-receive volume	Are too many workers polling an idle queue? Does this backend emulate long polling with repeated native reads?
High PostgreSQL latency	Are other queues sharing the server? Are CPU, disk, connections or lock waits saturated?
Dead-letter messages appear unexpectedly	Are delivery attempts caused by lease expiry or worker failures? Does the backend use a best-effort threshold?
Queue counts are absent or stale	Does the mapping expose live counts? Should the dashboard read the target cloud’s metrics instead?

Record send success, receive-to-delete latency, end-to-end completion latency, retries, failed batch entries and duplicate processing. Observe native backend metrics as well as origin-shaped responses. A successful batch HTTP response can still contain failed entries. Measure body, attributes and transport-envelope bytes; busiest-group fraction; active ready groups; settlement duration; retained backlog; conflicts; retries and cleanup. Missing sizes, observation windows or distributions remain unknown, rather than taking a benchmark’s assumed payload. The historical Classic offered-load benchmark does not establish another host, FIFO, HA, capacity mode or composite’s ceiling. Count polling is a separate cost: PostgreSQL scans the message population even when one counter is requested, competing for pool and I/O capacity as backlog grows. Metadata-only reads avoid count work. Track count-query frequency separately from successful sends, attempts and completed cycles. Queue depth can fall after purge, expiry or dead-letter transfer without an application completion. Include the adapter and allocated persistent shared control PostgreSQL compute/storage/backup/HA in total cost, even for native/serverless message storage. Add any separate SQL message authority, selected capacity floors, broker traffic, repair/outbox retries, retained state and cleanup. Use generated SQS costs only within the displayed scenario’s scope; see total cost trade-offs.

Change one setting and verify its effect

Capture the baseline

Save the queue attributes, tags, backend selection, native resource settings and workload results. Include the adapter version and the shared database capacity or Pub/Sub resource settings.

Choose the owner of the change

Use SQS attributes for queue behavior, the documented target tuning contract for a resource-specific override, or installation infrastructure for shared capacity. A backend move requires a new queue and a cutover procedure.

Apply a bounded change

Use a disposable queue or controlled test workload first. Changing visibility affects how long another worker waits after a failure; changing retention affects how long work remains recoverable. Keep those consequences in the experiment.

Read back both sides

Read the declared SQS attribute and the relevant native setting. Run the behavior that setting controls: delay a message, let a claim expire, retry a send or exercise the dead-letter path. Use tensor9 explain and diagnostic headers when the observed behavior differs from the mapping.

Keep or reverse the change

Restore the previous mutable setting through the same owner if results are worse. A larger capacity allocation does not necessarily roll back without operational effects, and restoring retention cannot recover messages that have already expired.For a backend change, drain or otherwise account for the old queue’s messages before redirecting producers and consumers. Changing t9:backend on an existing queue is not a message migration.

For an unexplained result, preserve the request and target request IDs, relevant x-t9-explain-* response headers, queue attributes and the message’s attempt history. See Debugging Your Adapters.

Tensor9 Cloud Adapter

How It Works

Getting Started

Configuration

Deployment

Costs

Support

Amazon Web Services

Google Cloud

Microsoft Azure

Making Trade-offs

Operations with Agents

Tuning

Debugging

Local Development and Testing

Guides

Reference

Decide what you are changing

Choose a backend

Try SQS on Pub/Sub

Pub/Sub: tune visibility, batching and retries

Example: workers taking 45–90 seconds

Example: receive ten messages without hiding unstarted work

Example: tolerate several attempts before dead-letter delivery

Example: delayed jobs

PostgreSQL for delayed or long-running work

Prepare the shared database

Example: export jobs with a fifteen-minute processing budget

Example: defer a job for five minutes

Example: size a server for two queues

Tune Azure and transactional alternatives by their actual authority

Respect backend limits

Measure the workload before changing capacity

Change one setting and verify its effect

​Decide what you are changing

​Choose a backend

​Try SQS on Pub/Sub

​Pub/Sub: tune visibility, batching and retries

​Example: workers taking 45–90 seconds

​Example: receive ten messages without hiding unstarted work

​Example: tolerate several attempts before dead-letter delivery

​Example: delayed jobs

​PostgreSQL for delayed or long-running work

​Prepare the shared database

​Example: export jobs with a fifteen-minute processing budget

​Example: defer a job for five minutes

​Example: size a server for two queues

​Tune Azure and transactional alternatives by their actual authority

​Respect backend limits

​Measure the workload before changing capacity

​Change one setting and verify its effect

Decide what you are changing

Choose a backend

Try SQS on Pub/Sub

Pub/Sub: tune visibility, batching and retries

Example: workers taking 45–90 seconds

Example: receive ten messages without hiding unstarted work

Example: tolerate several attempts before dead-letter delivery

Example: delayed jobs

PostgreSQL for delayed or long-running work

Prepare the shared database

Example: export jobs with a fifteen-minute processing budget

Example: defer a job for five minutes

Example: size a server for two queues

Tune Azure and transactional alternatives by their actual authority

Respect backend limits

Measure the workload before changing capacity

Change one setting and verify its effect