Tuning AWS SQS - Tensor9

SQS tuning starts with the work your consumers perform. A queue of small notifications, a queue of twenty-minute export jobs and a FIFO queue of account updates need different delivery settings. They may also need different backends, even when they run in the same target cloud. This article covers backend selection, a local Pub/Sub trial, PostgreSQL queues, Azure backends, FIFO workloads, and a measurement and change procedure. Read the SQS Standard and SQS FIFO service profiles alongside this article. They describe the operations and behavior for each directed mapping.

Decide what you are changing

There are three places to tune a queue workload:

Setting	Where it belongs	Example
Queue placement	`t9:backend` on `CreateQueue`	Place a Standard queue on Google Pub/Sub or Cloud SQL for PostgreSQL.
SQS behavior	SQS queue attributes and operation parameters	Set visibility, retention, receive wait, batch size and message-group IDs.
Shared backend capacity	The installation or infrastructure that owns the backend	Size a PostgreSQL server, choose a Service Bus namespace tier or select a storage account’s redundancy.

VisibilityTimeout is already an SQS attribute. Supply it through the SQS API. A queue’s PostgreSQL server size is a different kind of choice: several queues can share that server, so changing one queue’s tags must not resize all of its neighbors. The t9:tuning:<path> mechanism is for documented target-native fields. A target cloud’s API field is not automatically an accepted queue tuning path. Use the exact paths published for the selected mapping; do not turn a native console setting into a guessed tag. See Tuning Your Adapters.

Choose a backend

Start by deciding whether the application needs Standard or FIFO behavior. A Standard queue allows duplicate delivery and does not promise ordering. FIFO preserves order within a message group; it still requires consumers to handle redelivery.

Backend	Queue families	Why choose it?	Tuning consideration
Google Pub/Sub	Standard	Managed message delivery without sizing a database	Visibility is 10–600 seconds; delayed delivery is unavailable.
Google Cloud SQL for PostgreSQL	Standard and FIFO	Durable queue state, delayed delivery and long-running claims	All queues on a server share its compute, storage and connection budget.
Azure Service Bus	Standard and FIFO	Managed broker; sessions support FIFO message groups	Visibility requests are limited to five minutes; namespace capacity is shared.
Azure Queue Storage	Standard	Small messages and simple work queues	Budget for encoded payload size, polling transactions and application-managed poison messages.
Azure Cosmos DB queue backend	Standard and FIFO	Durable claims and queue state on Cosmos DB	The queue mapping uses serverless Cosmos DB; measure request units and hot groups.
Azure Database for PostgreSQL Flexible Server	Standard and FIFO	PostgreSQL queue behavior on an Azure-managed database	Size and monitor the shared database independently of individual queues.
OCI Queue	Standard	Native managed queue service in OCI	Review the 128 KiB payload and seven-day retention limits.
Scaleway Queues	Standard	Native SQS-compatible queue service	Choose PostgreSQL when the workload needs FIFO or delayed delivery.
Scaleway Managed Database for PostgreSQL	Standard and FIFO	PostgreSQL queue semantics in Scaleway	Capacity depends on the selected database and other queues using it.
PostgreSQL in your deployment	Standard and FIFO	A PostgreSQL backend in a private environment	Include database durability, backups, failover and storage in the deployment design.

Backend selection is made when the queue is created. For example:

{
  "QueueName": "notifications-tuning",
  "Attributes": {
    "VisibilityTimeout": "120",
    "ReceiveMessageWaitTimeSeconds": "20",
    "MessageRetentionPeriod": "345600"
  },
  "tags": {
    "t9:backend": "google::1.0.0::pubsub",
    "purpose": "tuning-test"
  }
}

The lower-case tags property is the AWS CLI’s CreateQueue input shape. t9:backend selects where this queue is placed. It is not a setting that moves an existing queue when updated with TagQueue.

Try SQS on Pub/Sub

This example creates a disposable Standard queue with two minutes of visibility, twenty-second long polling and four-day retention. Use a test project and a fresh queue name.

Install the CLI and inspect the mapping

Install the Tensor9 CLI, or upgrade an existing installation:

brew tap tensor9ine/tensor9
brew install tensor9
tensor9 --version

tensor9 explain \
  -origin aws::1.0.0::sqs::classic \
  -target google::1.0.0::pubsub \
  -fmt Human

Use brew upgrade tensor9 instead of brew install tensor9 if it is already installed. Read the mapping’s visibility, delay and dead-letter behavior before running a workload.

Prepare target access

Configure Google Cloud Application Default Credentials for the identity the adapter will use. Select a dedicated test project and provide its target coordinates through your Cloud Adapter configuration.The runtime identity needs the permissions required by the selected queue lifecycle. Publishing and consuming messages are different permissions from creating topics and subscriptions. For an installation using pre-provisioned resources, prepare those resources before starting the trial.

Start the single-binary adapter

In the first terminal, run:

tensor9 adapt svc run \
  --origin aws::1.0.0::sqs::classic \
  --backend google::1.0.0::pubsub

Leave this process running. Copy the listening endpoint it prints. The local process accepts SQS requests; Pub/Sub holds the messages.

Point the AWS CLI at the adapter

In a second terminal, configure the endpoint and an origin-side profile accepted by the adapter:

export T9_CLOUD_ADAPTER_ENDPOINT="<endpoint printed by tensor9>"
export AWS_ENDPOINT_URL_SQS="$T9_CLOUD_ADAPTER_ENDPOINT"
export AWS_PROFILE="adapter-test"
export AWS_REGION="us-east-1"

AWS_ENDPOINT_URL_SQS applies to SQS commands in this shell. You can instead put --endpoint-url "$T9_CLOUD_ADAPTER_ENDPOINT" on each command. The profile supplies the origin-facing identity; Google Cloud credentials belong to the adapter process.

Create and inspect the queue

Save the preceding JSON as notifications-tuning.json, then run:

export SQS_TUNING_QUEUE_URL="$(aws sqs create-queue \
  --cli-input-json file://notifications-tuning.json \
  --query QueueUrl --output text)"

aws sqs get-queue-attributes \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --attribute-names VisibilityTimeout ReceiveMessageWaitTimeSeconds MessageRetentionPeriod

aws sqs list-queue-tags --queue-url "$SQS_TUNING_QUEUE_URL"

Confirm the three declared attributes and the selected backend. Also inspect the native Pub/Sub subscription’s acknowledgement deadline and retention. A successful queue create is not a measurement of workload performance.

Send, receive and settle one message

Send a small representative payload:

aws sqs send-message \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --message-body '{"job":"send-email","order":"order-1001"}'

aws sqs receive-message \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --max-number-of-messages 1 \
  --wait-time-seconds 20 \
  --visibility-timeout 120

Copy the returned receipt handle. After processing the message, settle that exact receive:

aws sqs delete-message \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --receipt-handle '<receipt handle from this receive>'

Record the successful delete acknowledgement and the processed job ID. Observe native subscription backlog and redelivery over an appropriate interval. Neither an empty receive nor an immediate metric alone proves settlement: metrics can lag, and a message can still be hidden by its visibility timeout.

Finish the trial

Confirm all test messages are settled. Remove only the disposable queue and resources created for this trial, using the lifecycle owner that created them. Deleting a logical queue does not always delete its pre-provisioned native infrastructure.Press Control-C in the adapter terminal. Clear the endpoint override if you are returning to native AWS work:

unset AWS_ENDPOINT_URL_SQS

Pub/Sub: tune visibility, batching and retries

Pub/Sub is a good starting point for independent, short-running jobs. Its acknowledgement deadline constrains the SQS visibility lease.

Example: workers taking 45–90 seconds

Start with a 120-second queue visibility timeout:

aws sqs set-queue-attributes \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --attributes VisibilityTimeout=120,ReceiveMessageWaitTimeSeconds=20

Measure time from receipt to successful deletion, including application retries and downstream calls. If workers sometimes exceed two minutes, decide whether to shorten their work, extend the current lease within the backend’s range, or select a backend suitable for longer claims.

aws sqs change-message-visibility \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --receipt-handle '<current receipt handle>' \
  --visibility-timeout 300

The adapter accepts Pub/Sub visibility values of 10–600 seconds. A per-message value of zero releases the claim immediately. It does not automatically keep a message leased for an arbitrarily long job. Receipt handles are tied to the receiving stream; after a stream restart, receive the message again and use its new handle.

Example: receive ten messages without hiding unstarted work

aws sqs receive-message \
  --queue-url "$SQS_TUNING_QUEUE_URL" \
  --max-number-of-messages 10 \
  --wait-time-seconds 20 \
  --visibility-timeout 120

Ten is a maximum, not a promise that ten messages will be returned. If one worker processes the batch sequentially and each item takes 30 seconds, the later items can lose their visibility lease before processing starts. Use enough processing concurrency for the received batch, reduce its size, or choose an appropriate lease. Count time waiting inside your worker as part of the processing budget.

Example: tolerate several attempts before dead-letter delivery

Pub/Sub accepts a dead-letter threshold of 5–100 delivery attempts. For a threshold of ten, the SQS policy has this shape:

{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:000000000000:notifications-dead-letter\",\"maxReceiveCount\":\"10\"}"
}

Replace the account, region and destination with your installation’s origin-facing values. Prepare the destination and the permissions needed for forwarding before applying the policy. Pub/Sub forwards to a dead-letter topic, and delivery-attempt thresholds are best-effort. Verify the destination subscription as well as the topic so that operators can actually inspect forwarded messages. ApproximateReceiveCount is meaningful on this backend only when the dead-letter policy and its permissions are configured. A queue without that policy reports 1; that value must not drive an application poison-message cutoff.

Example: delayed jobs

A nonzero DelaySeconds does not map to Pub/Sub. If the job must become available five minutes after publication, choose the PostgreSQL backend and use a per-message delay. A longer visibility timeout is not a substitute: visibility begins when a consumer receives a message, while delivery delay applies before its first receive.

PostgreSQL for delayed or long-running work

The PostgreSQL queue implementation runs on Cloud SQL, Azure Flexible Server, Scaleway PostgreSQL or PostgreSQL in your deployment. The queue behavior is shared; performance depends on database size, storage, placement and workload.

Prepare the shared database

Provision the database and its runtime connection before starting the adapter. Keep its secret-backed binding in the installation’s sparse service configuration. This is the kind of prerequisite that needs a service entry; ordinary catalog mappings do not need to be repeated in the file. See Configuration Files. For a Cloud SQL trial, use:

tensor9 adapt svc run \
  --origin aws::1.0.0::sqs::classic \
  --backend google::1.0.0::cloudsql::postgresql

Follow the same endpoint steps used for Pub/Sub, copying the endpoint from this process. Do not assume the previous endpoint or queue URL addresses the new backend.

Example: export jobs with a fifteen-minute processing budget

{
  "QueueName": "export-jobs-tuning",
  "Attributes": {
    "VisibilityTimeout": "900",
    "ReceiveMessageWaitTimeSeconds": "20",
    "MessageRetentionPeriod": "604800"
  },
  "tags": {
    "t9:backend": "google::1.0.0::cloudsql::postgresql",
    "purpose": "tuning-test"
  }
}

This gives consumers a fifteen-minute claim and keeps messages for seven days. It does not allocate fifteen minutes of compute, reserve database capacity or guarantee that a worker finishes on time. Retention can expire a message while it is in flight; extending visibility does not extend retention. Current receipt handles survive an adapter restart because their claim state is stored in PostgreSQL. After a message is claimed again, an older consumer must not assume that its previous handle still owns the message. Make export completion idempotent, for example by recording the job’s result under a stable job ID.

Example: defer a job for five minutes

After creating the export queue, save its returned URL as SQS_EXPORT_QUEUE_URL:

aws sqs send-message \
  --queue-url "$SQS_EXPORT_QUEUE_URL" \
  --message-body '{"jobId":"export-1001","account":"acme"}' \
  --delay-seconds 300

Verify that the message is not returned before its delay elapses, then receive and delete it normally. Test the timing with a small workload before relying on it for a production scheduling requirement.

Example: size a server for two queues

Suppose email-jobs receives small messages all day and export-jobs produces a large evening burst. Both queues share one PostgreSQL server. Measure them together: a test of email-jobs alone cannot show whether the export burst will delay it.

Resource	What to measure	Change to consider
CPU	Sustained utilization and query latency during the combined peak	Increase the database’s compute allocation or reduce excessive polling.
Storage	Write latency, IOPS, growth and free space	Increase the storage allocation or choose an appropriate storage performance level.
Connections	Active/waiting connections across all adapter processes	Budget connections for the complete deployment before increasing worker or adapter counts.
Availability	Recovery time and queue accessibility during failover	Choose the host’s HA and backup configuration that meets the workload’s requirements.
Placement	Network latency between adapter, consumers and database	Keep the request path close where residency and availability requirements permit.

Make these changes through the infrastructure that owns the database. A CreateQueue tag does not resize the shared server. Likewise, adding more adapter processes does not add database capacity; it can increase the number of competing database connections.

Azure Service Bus

Service Bus offers Standard queues and session-based FIFO queues. Separate message-processing settings from the namespace and entity settings provisioned for them.

Example: short jobs with scheduled delivery

For a prepared Standard queue, choose a two-minute default visibility timeout:

{
  "QueueName": "billing-jobs-tuning",
  "Attributes": {
    "VisibilityTimeout": "120",
    "ReceiveMessageWaitTimeSeconds": "20"
  },
  "tags": {
    "t9:backend": "azure::1.0.0::servicebus"
  }
}

Create this logical queue against an installation whose Service Bus namespace and queue entity are ready. Set native retention and dead-letter routing through the entity’s infrastructure configuration. Naming MessageRetentionPeriod on a runtime queue create or update is not how this backend changes the entity’s TTL. After capturing the returned queue URL:

aws sqs send-message \
  --queue-url "$SQS_BILLING_QUEUE_URL" \
  --message-body '{"invoice":"invoice-1001"}' \
  --delay-seconds 30

The thirty-second per-message delay becomes scheduled delivery. Do not turn it into a nonzero queue-level DelaySeconds setting. For this mapping, those are different capabilities.

Plan for lease duration and restarts

Service Bus visibility requests cannot exceed 300 seconds. Deletion and visibility changes use the receiver that acquired the broker lock. If the adapter restarts, old receipt handles expire; consumers must accept redelivery and use new handles. For jobs lasting much longer than five minutes, evaluate a PostgreSQL or Cosmos queue backend with durable claim state. Do not assume a five-minute Service Bus setting automatically provides a twenty-minute processing window.

Size the namespace and entities

Namespace tier and capacity serve multiple queues. Queue storage limits and partitioning are entity decisions. Record both when comparing results: a queue-level experiment cannot attribute a change in latency if namespace capacity changed at the same time. For a sustained workload, measure broker throttling, message-operation volume, active messages, dead-letter growth and consumer processing time. Include receive, delete and lease-related operations when comparing operating cost. A queue backlog caused by a slow downstream database will not necessarily improve when the broker’s capacity increases.

Azure Queue Storage

Queue Storage is useful for small independent work items where the application can handle poison messages explicitly.

Example: a small image-processing job

{
  "QueueName": "thumbnail-jobs-tuning",
  "Attributes": {
    "VisibilityTimeout": "180",
    "ReceiveMessageWaitTimeSeconds": "20",
    "MessageRetentionPeriod": "86400",
    "MaximumMessageSize": "16384"
  },
  "tags": {
    "t9:backend": "azure::1.0.0::queue-storage"
  }
}

The job body should contain the source object’s location and a stable job ID. Store the image itself in object storage. Queue Storage’s 64 KiB encoded limit leaves roughly 48 KiB for a raw body before message-attribute overhead; choosing a 16 KiB application limit leaves room for the envelope and makes oversized jobs fail early.

Tune retries and account durability separately

Queue Storage exposes its dequeue count as ApproximateReceiveCount. If the application gives up after five attempts, have it copy the work item to a separately provisioned poison queue and delete the source only after that copy succeeds. The two operations are separate. A crash between them can produce a duplicate poison message, so preserve a stable job ID there too. The account’s LRS, ZRS, GRS or GZRS choice controls storage redundancy for every queue in that account. Select it through the account’s infrastructure configuration. Changing a queue’s visibility or retention does not alter the account’s redundancy. Long polling on this backend uses repeated checks, approximately 300 milliseconds apart when empty. Measure empty receives and transaction volume. Twenty-second long polling reduces repeated application calls, but does not turn every receive into a single native storage operation.

Azure Cosmos DB queue backend

The SQS Cosmos mapping stores claim state in Cosmos DB and uses a container per queue. Its account is serverless. This differs from DynamoDB tuning, where a provisioned Cosmos container can have an explicit RU/s allocation.

Example: a ten-minute document-processing claim

{
  "QueueName": "document-jobs-tuning",
  "Attributes": {
    "VisibilityTimeout": "600",
    "ReceiveMessageWaitTimeSeconds": "20",
    "MessageRetentionPeriod": "172800"
  },
  "tags": {
    "t9:backend": "azure::1.0.0::cosmosdb::queue"
  }
}

Prepare the account, database and queue container through the installation’s provisioning workflow. The queue gets a ten-minute claim and two days of retention. Native item TTL handles expiration; visibility does not protect a message from its retention deadline. Measure request units for a complete send/receive/delete cycle, including retries, empty polls and dead-letter work. A count read may lag by about five seconds, so a single immediately repeated count is not a reliable throughput measurement. Do not copy container.properties.options.autoscaleSettings.maxThroughput from the provisioned DynamoDB example into this queue’s tags. A serverless queue backend and a provisioned DynamoDB backend have different resource and capacity contracts.

FIFO concurrency and deduplication

Choose an ordering-capable backend before creating a FIFO queue. For a separate local trial, prepare the Cloud SQL binding described above, then start the FIFO mapping:

tensor9 adapt svc run \
  --origin aws::1.0.0::sqs::fifo \
  --backend google::1.0.0::cloudsql::postgresql

In your client terminal, set AWS_ENDPOINT_URL_SQS to this process’s printed endpoint. Do not keep using the Pub/Sub Standard trial’s endpoint. Save the following request as create-fifo-queue.json:

{
  "QueueName": "account-events-tuning.fifo",
  "Attributes": {
    "FifoQueue": "true",
    "VisibilityTimeout": "120",
    "ReceiveMessageWaitTimeSeconds": "20"
  },
  "tags": {
    "t9:backend": "google::1.0.0::cloudsql::postgresql"
  }
}

Example: independent accounts, ordered updates

Create the queue and capture its URL:

export SQS_FIFO_QUEUE_URL="$(aws sqs create-queue \
  --cli-input-json file://create-fifo-queue.json \
  --query QueueUrl --output text)"

Send two updates for one account:

aws sqs send-message \
  --queue-url "$SQS_FIFO_QUEUE_URL" \
  --message-body '{"account":"acme","version":41}' \
  --message-group-id account-acme \
  --message-deduplication-id acme-version-41

aws sqs send-message \
  --queue-url "$SQS_FIFO_QUEUE_URL" \
  --message-body '{"account":"globex","version":12}' \
  --message-group-id account-globex \
  --message-deduplication-id globex-version-12

Updates for one account share a group and retain order. Independent accounts can use different groups. Putting every message in a single group serializes the entire workload; increasing worker count cannot remove that ordering constraint. Retry a send with the same deduplication ID when it represents the same event. Use a different ID for a distinct event, even if its body happens to match. Deduplication suppresses repeated sends within the backend’s documented window; it does not remove the need for idempotent consumers after a receive, crash or lease expiry. Content-based deduplication belongs in the queue’s infrastructure definition where supported. Do not assume runtime ContentBasedDeduplication, DeduplicationScope or FifoThroughputLimit edits are available merely because explicit per-message deduplication IDs work.

Example: one slow account

If account-acme takes three minutes per event but every other account takes one second, examine latency by group. A deployment-wide average can hide the slow group. Increasing the backend’s capacity may help resource contention, but it cannot allow the second Acme event to overtake the first while preserving the queue’s ordering contract. Measure group skew, processing time and visibility expiry. Split groups only when the application’s consistency requirements allow independent processing.

OCI and Scaleway choices

For OCI Queue, design within the 128 KiB message limit and seven-day retention ceiling. Its built-in dead-letter queue is not an arbitrary SQS destination. If the application depends on fourteen-day retention or a named dead-letter topology, read the OCI mapping before choosing it. For Scaleway, native Queues is a Standard-queue option. A delayed job or an ordered message group is a reason to choose Scaleway PostgreSQL. Size that database for the total queue workload and verify delay, visibility and redelivery using the same application test as the other PostgreSQL hosts. These are backend choices with observable application consequences. A tuning tag cannot add ordering or delayed delivery to a target that does not supply the required behavior.

Measure the workload before changing capacity

Use a bounded workload with known message count, payload size, group distribution and processing duration. Keep those inputs the same when comparing two settings.

Observation	Likely questions to investigate
Backlog grows while workers are busy	Is processing slower than arrival? Is a downstream dependency limiting throughput?
Many duplicate receives	Does work exceed visibility? Did the adapter or stream restart? Are batches waiting unprocessed?
High empty-receive volume	Are too many workers polling an idle queue? Does this backend emulate long polling with repeated native reads?
High PostgreSQL latency	Are other queues sharing the server? Are CPU, disk, connections or lock waits saturated?
High Cosmos RU consumption	Are retries, empty polls, payload size or hot message groups driving extra operations?
One FIFO group falls behind	Is ordered work within that group inherently serial? Is its processing unusually slow?
Dead-letter messages appear unexpectedly	Are delivery attempts caused by lease expiry or worker failures? Does the backend use a best-effort threshold?
Queue counts are absent or stale	Does the mapping expose live counts? Should the dashboard read the target cloud’s metrics instead?

Record send success, receive-to-delete latency, end-to-end completion latency, retries, failed batch entries and duplicate processing. Observe native backend metrics as well as origin-shaped responses. A successful batch HTTP response can still contain failed entries.

Change one setting and verify its effect

Capture the baseline

Save the queue attributes, tags, backend selection, native resource settings and workload results. Include the adapter version and the shared database or namespace capacity.

Choose the owner of the change

Use SQS attributes for queue behavior, the documented target tuning contract for a resource-specific override, or installation infrastructure for shared capacity. A backend move requires a new queue and a cutover procedure.

Apply a bounded change

Use a disposable queue or controlled test workload first. Changing visibility affects how long another worker waits after a failure; changing retention affects how long work remains recoverable. Keep those consequences in the experiment.

Read back both sides

Read the declared SQS attribute and the relevant native setting. Run the behavior that setting controls: delay a message, let a claim expire, retry a send or exercise the dead-letter path. Use tensor9 explain and diagnostic headers when the observed behavior differs from the mapping.

Keep or reverse the change

Restore the previous mutable setting through the same owner if results are worse. A larger capacity allocation does not necessarily roll back without operational effects, and restoring retention cannot recover messages that have already expired.For a backend change, drain or otherwise account for the old queue’s messages before redirecting producers and consumers. FIFO cutover must preserve ordering within each group. Changing t9:backend on an existing queue is not a message migration.

For an unexplained result, preserve the request and target request IDs, relevant x-t9-explain-* response headers, queue attributes and the message’s attempt history. See Debugging Your Adapters.

​Decide what you are changing

​Choose a backend

​Try SQS on Pub/Sub

​Pub/Sub: tune visibility, batching and retries

​Example: workers taking 45–90 seconds

​Example: receive ten messages without hiding unstarted work

​Example: tolerate several attempts before dead-letter delivery

​Example: delayed jobs

​PostgreSQL for delayed or long-running work

​Prepare the shared database

​Example: export jobs with a fifteen-minute processing budget

​Example: defer a job for five minutes

​Example: size a server for two queues

​Azure Service Bus

​Example: short jobs with scheduled delivery

​Plan for lease duration and restarts

​Size the namespace and entities

​Azure Queue Storage

​Example: a small image-processing job

​Tune retries and account durability separately

​Azure Cosmos DB queue backend

​Example: a ten-minute document-processing claim

​FIFO concurrency and deduplication

​Example: independent accounts, ordered updates

​Example: one slow account

​OCI and Scaleway choices

​Measure the workload before changing capacity

​Change one setting and verify its effect