Tuning AWS S3

For S3 on Google Cloud Storage, tune the request workload and bucket policy separately. Client concurrency controls how much work arrives at the adapter. Lifecycle rules control how stored objects age into another class or expire. Neither is a substitute for choosing suitable bucket placement. Read the S3 service profile before using an operation. This guide focuses on S3 to GCS; a storage-class name or lifecycle rule does not imply the same native behavior on Azure Blob, MinIO or another S3 backend.

Start an isolated S3-to-GCS experiment

Prepare a test bucket and identity

Follow the S3-to-GCS configuration example. Use a disposable bucket, not a production bucket. Prepare the adapter’s Google identity and a separate identity for native inspection. Record the origin-facing and physical bucket names; do not assume they are identical.

Run one adapter

tensor9 adapt svc run \
  --origin aws::1.0.0::s3 \
  --backend google::1.0.0::gcs

Leave that terminal open. Backend operations use real Google Cloud resources even though the adapter runs locally.

Route S3 from a second terminal

Copy the listening endpoint printed by the command:

export AWS_ENDPOINT_URL_S3="<endpoint printed by tensor9>"
export AWS_REGION="us-east-1"
export AWS_REQUEST_CHECKSUM_CALCULATION="WHEN_REQUIRED"

Keep the origin credentials appropriate to your endpoint. The AWS client credentials and adapter’s Google credentials are separate. The examples use tuning-media as the origin-facing bucket and tuning-media-gcs as its physical GCS bucket. Replace these with your dedicated test bucket names throughout; native bucket names must be globally unique.

Record a baseline

Measure a representative mix of writes, reads and metadata requests before changing settings. Include small objects, large objects, retries and failed requests. Verify object contents independently of HTTP success.

Example: compare bounded upload concurrency

This Python example uses a programmatically configured S3 client instead of relying on the service-wide endpoint setting. Save it as s3-upload-test.py and run it in an environment with boto3 installed.

import hashlib
import argparse
import time
import uuid
from concurrent.futures import ThreadPoolExecutor

import boto3
from botocore.config import Config

parser = argparse.ArgumentParser()
parser.add_argument("--workers", type=int, default=4)
workers = parser.parse_args().workers
if not 1 <= workers <= 32:
    raise ValueError("Use 1 to 32 workers for this bounded experiment")

bucket = "tuning-media"
prefix = f"tuning-test/{uuid.uuid4()}/"
body = b"adapter-transfer-test\n" * 32768
expected = hashlib.sha256(body).digest()
s3 = boto3.client(
    "s3",
    endpoint_url="<endpoint printed by tensor9>",
    region_name="us-east-1",
    config=Config(
        max_pool_connections=workers,
        retries={"mode": "standard", "total_max_attempts": 3},
    ),
)

def upload(index):
    key = f"{prefix}{index:04d}.bin"
    started = time.perf_counter()
    s3.put_object(Bucket=bucket, Key=key, Body=body)
    return key, time.perf_counter() - started

print(f"Test prefix: {prefix}", flush=True)
started = time.perf_counter()
with ThreadPoolExecutor(max_workers=workers) as pool:
    results = list(pool.map(upload, range(64)))
elapsed = time.perf_counter() - started
durations = sorted(duration for _, duration in results)
print({"workers": workers, "objects": len(results), "seconds": elapsed,
       "objects_per_second": len(results) / elapsed,
       "p95_seconds": durations[int(0.95 * (len(durations) - 1))]})

# Verify separately so reads do not change the upload measurement.
for key, _ in results:
    response = s3.get_object(Bucket=bucket, Key=key)
    stream = response["Body"]
    try:
        actual = hashlib.sha256(stream.read()).digest()
    finally:
        stream.close()
    if actual != expected:
        raise RuntimeError(f"Content mismatch: {key}")
print("All object bodies verified; retain the test prefix for inspection.")

Compare three runs:

python s3-upload-test.py --workers 1
python s3-upload-test.py --workers 4
python s3-upload-test.py --workers 8

These are test points, not recommended production limits. Keep object size, placement and retry settings constant. Repeat the runs in a different order to check warm-up effects. If a request fails, keep the failed run in the result set; do not compare only successful runs and claim a speedup. Increasing concurrency can improve aggregate transfer rate while worsening tail latency or target throttling. If throughput plateaus, inspect adapter CPU and memory, network throughput, target errors and SDK retries before adding workers. A larger client connection pool permits more connections; it does not allocate native storage capacity. The script leaves test objects for inspection. Remove only the printed test prefixes after validating the result. Versioned buckets can retain old versions after deletion, and retention policies can prevent immediate cleanup.

Example: move infrequently read objects to Nearline

Suppose exported reports are frequently read for the first month and seldom read afterward. In the S3-to-GCS mapping, the supported STANDARD_IA lifecycle transition maps to GCS Nearline. Save this policy as reports-lifecycle.json:

{
  "Rules": [
    {
      "ID": "reports-after-30-days",
      "Status": "Enabled",
      "Prefix": "reports/",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" }
      ]
    }
  ]
}

Applying a lifecycle configuration replaces the bucket’s lifecycle rules. Use the disposable bucket, or merge this rule with the existing complete policy before an approved production change. Lifecycle rules can affect existing objects as well as future writes.

Apply and read it through S3:

aws s3api put-bucket-lifecycle-configuration \
  --bucket tuning-media \
  --lifecycle-configuration file://reports-lifecycle.json

aws s3api get-bucket-lifecycle-configuration --bucket tuning-media

Inspect the target bucket independently:

gcloud storage buckets describe gs://tuning-media-gcs --format=json

Check the rule’s prefix, age and resulting storage class in the native policy. A policy readback does not prove an object has already transitioned. Native lifecycle processing is asynchronous; inspect eligible objects separately. See Google Cloud lifecycle behavior. Nearline trades lower storage cost for access charges and a minimum storage duration. Include retrievals, operations and early deletion alongside stored bytes in the comparison. See Cloud Storage classes. Do not generalize this example into an unsupported Glacier or Deep Archive transition. The service profile lists the accepted rule forms; a valid AWS policy can still ask for behavior unavailable in this mapping.

Example: expire disposable exports

For a separate test bucket containing only disposable exports, use a prefix-scoped expiration rule:

{
  "Rules": [
    {
      "ID": "temporary-exports",
      "Status": "Enabled",
      "Prefix": "temporary-exports/",
      "Expiration": { "Days": 7 }
    }
  ]
}

Save it as exports-lifecycle.json and use the same apply/readback procedure, substituting that filename. This policy deletes eligible objects. It is not a reversible performance setting: removing a rule does not restore deleted content. Verify retention, holds and versioning requirements before using expiration. Do not combine short expiration with a colder storage class just because both individually reduce retained bytes. A minimum storage duration can make early deletion more expensive than leaving short-lived data in Standard storage.

Example: placement before connection counts

For latency-sensitive small-object traffic, record the regions of the application, adapter and bucket. Measure their current path before changing client concurrency. A local laptop test includes the laptop-to-cloud network and is not a measurement of a co-located production deployment. If comparing regional and multi-region placement, create separate test deployments under the provisioning owner’s configuration. Compare latency and recovery requirements together. A bucket placement change is not an S3 object tuning tag, and switching production data to a differently placed bucket requires a migration plan.

Debug and reverse a change

Inspect the mapping with:

tensor9 explain \
  -origin aws::1.0.0::s3 \
  -target google::1.0.0::gcs \
  -operation PutObject \
  -fmt Human

Use Explain SDK integration to capture selected live requests with x-t9-explain: true. That header executes the request. Preview a mutation in Explain mode when you do not want to apply it.

Change	Verify	Reverse carefully
Client concurrency	Completed bytes/objects, latency, retries and errors	Restore the client setting; allow in-flight requests to finish.
Lifecycle transition	Native policy and eligible objects’ actual classes	Restore the policy; already transitioned objects may require separate changes and incur charges.
Lifecycle expiration	Prefix scope, retention and object history	Remove the rule to stop future eligibility; deleted data needs a separate recovery source.
Placement	Native location, latency and recovery requirements	Switch through a tested migration/cutover procedure.

For each experiment, preserve both the origin request and native observation. An S3 response or explanation alone is not proof that an asynchronous native lifecycle action has completed.

Tensor9 Cloud Adapter

How It Works

Getting Started

Configuration

Deployment

Making Trade-offs

Operations

Tuning

Debugging

Testing

Amazon Web Services

Google Cloud

Microsoft Azure

Reference

Guides

Start an isolated S3-to-GCS experiment

Example: compare bounded upload concurrency

Example: move infrequently read objects to Nearline

Example: expire disposable exports

Example: placement before connection counts

Debug and reverse a change

​Start an isolated S3-to-GCS experiment

​Example: compare bounded upload concurrency

​Example: move infrequently read objects to Nearline

​Example: expire disposable exports

​Example: placement before connection counts

​Debug and reverse a change

Start an isolated S3-to-GCS experiment

Example: compare bounded upload concurrency

Example: move infrequently read objects to Nearline

Example: expire disposable exports

Example: placement before connection counts

Debug and reverse a change