> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensor9.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Tuning AWS S3

> Tune S3-to-GCS transfer workloads and storage lifecycle policies without changing the application's object API.

For S3 on Google Cloud Storage, tune the request workload and bucket policy separately. Client concurrency controls how much work arrives at the adapter. Lifecycle rules control how stored objects age into another class or expire. Neither is a substitute for choosing suitable bucket placement.

Read the [S3 service profile](/cloud-adapter/service-catalog/aws/databases-storage/s3#on-google-cloud) before using an operation. This guide focuses on S3 to GCS; a storage-class name or lifecycle rule does not imply the same native behavior on Azure Blob, MinIO or another S3 backend.

## Start an isolated S3-to-GCS experiment

<Steps>
  <Step title="Prepare a test bucket and identity">
    Follow the <a href="/cloud-adapter/configuration/examples#s3-to-google-cloud-storage">S3-to-GCS configuration example</a>. Use a disposable bucket, not a production bucket. Prepare the adapter's Google identity and a separate identity for native inspection. Record the origin-facing and physical bucket names; do not assume they are identical.
  </Step>

  <Step title="Run one adapter">
    ```bash theme={null}
    tensor9 adapt svc run \
      --origin aws::1.0.0::s3 \
      --backend google::1.0.0::gcs
    ```

    Leave that terminal open. Backend operations use real Google Cloud resources even though the adapter runs locally.
  </Step>

  <Step title="Route S3 from a second terminal">
    Copy the listening endpoint printed by the command:

    ```bash theme={null}
    export AWS_ENDPOINT_URL_S3="<endpoint printed by tensor9>"
    export AWS_REGION="us-east-1"
    export AWS_REQUEST_CHECKSUM_CALCULATION="WHEN_REQUIRED"
    ```

    Keep the origin credentials appropriate to your endpoint. The AWS client credentials and adapter's Google credentials are separate. The examples use `tuning-media` as the origin-facing bucket and `tuning-media-gcs` as its physical GCS bucket. Replace these with your dedicated test bucket names throughout; native bucket names must be globally unique.
  </Step>

  <Step title="Record a baseline">
    Measure a representative mix of writes, reads and metadata requests before changing settings. Include small objects, large objects, retries and failed requests. Verify object contents independently of HTTP success.
  </Step>
</Steps>

## Example: compare bounded upload concurrency

This Python example uses a programmatically configured S3 client instead of relying on the service-wide endpoint setting. Save it as `s3-upload-test.py` and run it in an environment with `boto3` installed.

```python theme={null}
import hashlib
import argparse
import time
import uuid
from concurrent.futures import ThreadPoolExecutor

import boto3
from botocore.config import Config

parser = argparse.ArgumentParser()
parser.add_argument("--workers", type=int, default=4)
workers = parser.parse_args().workers
if not 1 <= workers <= 32:
    raise ValueError("Use 1 to 32 workers for this bounded experiment")

bucket = "tuning-media"
prefix = f"tuning-test/{uuid.uuid4()}/"
body = b"adapter-transfer-test\n" * 32768
expected = hashlib.sha256(body).digest()
s3 = boto3.client(
    "s3",
    endpoint_url="<endpoint printed by tensor9>",
    region_name="us-east-1",
    config=Config(
        max_pool_connections=workers,
        retries={"mode": "standard", "total_max_attempts": 3},
    ),
)

def upload(index):
    key = f"{prefix}{index:04d}.bin"
    started = time.perf_counter()
    s3.put_object(Bucket=bucket, Key=key, Body=body)
    return key, time.perf_counter() - started

print(f"Test prefix: {prefix}", flush=True)
started = time.perf_counter()
with ThreadPoolExecutor(max_workers=workers) as pool:
    results = list(pool.map(upload, range(64)))
elapsed = time.perf_counter() - started
durations = sorted(duration for _, duration in results)
print({"workers": workers, "objects": len(results), "seconds": elapsed,
       "objects_per_second": len(results) / elapsed,
       "p95_seconds": durations[int(0.95 * (len(durations) - 1))]})

# Verify separately so reads do not change the upload measurement.
for key, _ in results:
    response = s3.get_object(Bucket=bucket, Key=key)
    stream = response["Body"]
    try:
        actual = hashlib.sha256(stream.read()).digest()
    finally:
        stream.close()
    if actual != expected:
        raise RuntimeError(f"Content mismatch: {key}")
print("All object bodies verified; retain the test prefix for inspection.")
```

Compare three runs:

```bash theme={null}
python s3-upload-test.py --workers 1
python s3-upload-test.py --workers 4
python s3-upload-test.py --workers 8
```

These are test points, not recommended production limits. Keep object size, placement and retry settings constant. Repeat the runs in a different order to check warm-up effects. If a request fails, keep the failed run in the result set; do not compare only successful runs and claim a speedup.

Increasing concurrency can improve aggregate transfer rate while worsening tail latency or target throttling. If throughput plateaus, inspect adapter CPU and memory, network throughput, target errors and SDK retries before adding workers. A larger client connection pool permits more connections; it does not allocate native storage capacity.

The script leaves test objects for inspection. Remove only the printed test prefixes after validating the result. Versioned buckets can retain old versions after deletion, and retention policies can prevent immediate cleanup.

## Example: move infrequently read objects to Nearline

Suppose exported reports are frequently read for the first month and seldom read afterward. In the S3-to-GCS mapping, the supported `STANDARD_IA` lifecycle transition maps to GCS Nearline. Save this policy as `reports-lifecycle.json`:

```json theme={null}
{
  "Rules": [
    {
      "ID": "reports-after-30-days",
      "Status": "Enabled",
      "Prefix": "reports/",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" }
      ]
    }
  ]
}
```

<Warning>
  Applying a lifecycle configuration replaces the bucket's lifecycle rules. Use the disposable bucket, or merge this rule with the existing complete policy before an approved production change. Lifecycle rules can affect existing objects as well as future writes.
</Warning>

Apply and read it through S3:

```bash theme={null}
aws s3api put-bucket-lifecycle-configuration \
  --bucket tuning-media \
  --lifecycle-configuration file://reports-lifecycle.json

aws s3api get-bucket-lifecycle-configuration --bucket tuning-media
```

Inspect the target bucket independently:

```bash theme={null}
gcloud storage buckets describe gs://tuning-media-gcs --format=json
```

Check the rule's prefix, age and resulting storage class in the native policy. A policy readback does not prove an object has already transitioned. Native lifecycle processing is asynchronous; inspect eligible objects separately. See [Google Cloud lifecycle behavior](https://docs.cloud.google.com/storage/docs/lifecycle).

Nearline trades lower storage cost for access charges and a minimum storage duration. Include retrievals, operations and early deletion alongside stored bytes in the comparison. See [Cloud Storage classes](https://docs.cloud.google.com/storage/docs/storage-classes).

Do not generalize this example into an unsupported Glacier or Deep Archive transition. The service profile lists the accepted rule forms; a valid AWS policy can still ask for behavior unavailable in this mapping.

## Example: expire disposable exports

For a separate test bucket containing only disposable exports, use a prefix-scoped expiration rule:

```json theme={null}
{
  "Rules": [
    {
      "ID": "temporary-exports",
      "Status": "Enabled",
      "Prefix": "temporary-exports/",
      "Expiration": { "Days": 7 }
    }
  ]
}
```

Save it as `exports-lifecycle.json` and use the same apply/readback procedure, substituting that filename. This policy deletes eligible objects. It is not a reversible performance setting: removing a rule does not restore deleted content. Verify retention, holds and versioning requirements before using expiration.

Do not combine short expiration with a colder storage class just because both individually reduce retained bytes. A minimum storage duration can make early deletion more expensive than leaving short-lived data in Standard storage.

## Example: placement before connection counts

For latency-sensitive small-object traffic, record the regions of the application, adapter and bucket. Measure their current path before changing client concurrency. A local laptop test includes the laptop-to-cloud network and is not a measurement of a co-located production deployment.

If comparing regional and multi-region placement, create separate test deployments under the provisioning owner's configuration. Compare latency and recovery requirements together. A bucket placement change is not an S3 object tuning tag, and switching production data to a differently placed bucket requires a migration plan.

## Debug and reverse a change

Inspect the mapping with:

```bash theme={null}
tensor9 explain \
  -origin aws::1.0.0::s3 \
  -target google::1.0.0::gcs \
  -operation PutObject \
  -fmt Human
```

Use [Explain SDK integration](/cloud-adapter/debugging/explain-sdk) to capture selected live requests with `x-t9-explain: true`. That header executes the request. Preview a mutation in Explain mode when you do not want to apply it.

| Change               | Verify                                               | Reverse carefully                                                                                |
| -------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| Client concurrency   | Completed bytes/objects, latency, retries and errors | Restore the client setting; allow in-flight requests to finish.                                  |
| Lifecycle transition | Native policy and eligible objects' actual classes   | Restore the policy; already transitioned objects may require separate changes and incur charges. |
| Lifecycle expiration | Prefix scope, retention and object history           | Remove the rule to stop future eligibility; deleted data needs a separate recovery source.       |
| Placement            | Native location, latency and recovery requirements   | Switch through a tested migration/cutover procedure.                                             |

For each experiment, preserve both the origin request and native observation. An S3 response or explanation alone is not proof that an asynchronous native lifecycle action has completed.