> ## Documentation Index > Fetch the complete documentation index at: https://docs.tensor9.com/llms.txt > Use this file to discover all available pages before exploring further. # MSK (Kafka) > MSK (Kafka) APIs with Cloud Adapter. This page describes how MSK (Kafka) maps to services in the environment where the application runs. Some profiles adapt origin API calls; others translate infrastructure or document target-native behavior. ## Supported environments | Environment | Mapping | | ------------------ | -------------- | | Akamai | Infrastructure | | Azure | Infrastructure | | DigitalOcean | Infrastructure | | Google Cloud | Infrastructure | | OCI | Infrastructure | | Private Kubernetes | Infrastructure | | Scaleway | Infrastructure | API means the profile adapts origin API behavior. Infrastructure means the profile changes provisioned resources or documents a target-native alternative without promising an origin API endpoint. Check the operation and capability tables for the behavior your application depends on. ## How the targets compare Each row compares a capability of MSK (Kafka) with its adaptation on each target. A dash means this profile does not state the capability for that target. ### Cloud Adapter | Capability | MSK (Kafka) | Akamai, DigitalOcean, Google Cloud, Private Kubernetes, and Scaleway · Strimzi Kafka | Akamai, Google Cloud, Private Kubernetes, and Scaleway · Bitnami Kafka | Akamai, Private Kubernetes, and Scaleway · Apache Kafka | Azure | DigitalOcean · DigitalOcean Managed Kafka | Google Cloud · Managed Service for Apache Kafka | OCI | Scaleway · Scaleway Clusters for Apache Kafka | | -------------------------------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | Kafka wire protocol | Apache Kafka | native: producer/consumer unchanged | native: producer/consumer unchanged | native: producer/consumer unchanged | native: producer/consumer unchanged | native: producer/consumer unchanged | native: producer/consumer unchanged | native: producer/consumer unchanged | native: producer/consumer unchanged | | Durability / replication / HA · who operates the broker | AWS-managed (MSK) | you (self-managed on the cluster) | you (self-managed on the cluster) | you (self-managed on the cluster) | Microsoft (managed Event Hubs) | DigitalOcean (managed Kafka, currently 3.8) | Google (managed Apache Kafka) | Oracle (OCI-managed stream pool) | Scaleway (Clusters for Apache Kafka, a real Apache Kafka engine) | | Authentication · broker auth | SASL/SCRAM, mTLS, IAM | SASL/SCRAM + mTLS (MSK IAM has no analog) | SASL/SCRAM + mTLS (MSK IAM has no analog) | SASL/SCRAM + mTLS (MSK IAM has no analog) | Shared Access Signatures / Microsoft Entra (not Kafka SASL or MSK IAM) | DigitalOcean-issued SASL/TLS (MSK IAM has no analog) | Google Cloud IAM over SASL\_SSL/OAUTHBEARER (MSK IAM, SASL/SCRAM and mTLS have no analog) | OCI auth tokens (not MSK SASL/SCRAM/mTLS/IAM) | Scaleway-issued username/password on the cluster (MSK's SASL/IAM has no analog) | | Retention / storage | broker log + tiered storage | the broker's own local storage (no tiered storage) | the broker's own local storage (no tiered storage) | the broker's own local storage (no tiered storage) | Event Hubs managed retention (capacity sized in throughput/processing units, separate from the partition count) | the managed broker's own storage | the managed broker's own storage, tuned per topic | the stream pool's managed storage, under OCI quotas | the managed broker's own storage, allocated in 5 GB steps from a 10 GB floor | | Encryption in transit | Yes | Yes - client-broker TLS terminates on the target listener's TLS | Yes - client-broker TLS terminates on the target listener's TLS | Yes - client-broker TLS terminates on the target listener's TLS | Yes - client-broker TLS terminates on the target listener's TLS | Yes - client-broker TLS terminates on the target listener's TLS | Yes - client-broker TLS terminates on the target listener's TLS | Yes - client-broker TLS terminates on the target listener's TLS | Yes - client-broker TLS terminates on the target listener's TLS | | Encryption at rest | KMS key ARN | the target's own storage / managed encryption (the KMS key ARN has no equivalent) | the target's own storage / managed encryption (the KMS key ARN has no equivalent) | the target's own storage / managed encryption (the KMS key ARN has no equivalent) | the target's own storage / managed encryption (the KMS key ARN has no equivalent) | the target's own storage / managed encryption (the KMS key ARN has no equivalent) | the target's own storage / managed encryption (the KMS key ARN has no equivalent) | the target's own storage / managed encryption (the KMS key ARN has no equivalent) | the target's own storage / managed encryption (the KMS key ARN has no equivalent) | | API coverage | full | high | high | high | partial | high | high | partial | high | | SASL/IAM clients need a client-side change · a mechanism, not a credential | AWS\_MSK\_IAM via aws-msk-iam-auth | - | - | - | - | - | - | - | SASL/PLAIN or SASL/SCRAM with Scaleway's credentials | | The Kafka version is chosen, not copied · and this is why | your cluster's kafka\_version | - | - | - | - | - | - | - | 4.1.1, the newest Scaleway offers | | Every broker size is marked beta · Scaleway's own flag, not ours | the MSK broker instance type you chose | - | - | - | - | - | - | - | brokernode-shared-2c-8g (2 vCPU / 8 GB) | | The same stack references work on either cloud · a deliberate constraint | bootstrap\_brokers\_tls, bootstrap\_brokers\_sasl\_scram, … | - | - | - | - | - | - | - | the identical set on Scaleway and DigitalOcean | ## On Akamai, DigitalOcean, Google Cloud, Private Kubernetes, and Scaleway ### Strimzi Kafka | Operation | Area | Support | Depth | Notes | | ----------------------------------------------------------------- | ------------------ | ------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitoring / broker-log delivery / at-rest KMS / VPC placement | AWS control plane | Out of scope | Full surface | Configure target monitoring and broker logs in place of MSK's CloudWatch, Prometheus, S3 and Firehose settings. Storage encryption uses the target's key and storage configuration. Select target subnets and failure domains separately from the supplied bootstrap endpoint. | | Broker config (custom broker-config revision) | Admin | Partial | Most usage | the target broker natively accepts custom broker config (via the Strimzi or Bitnami chart's config values, or the broker's own config for a raw Apache deployment); MSK's custom broker-config revision has no counterpart, so broker config is set on the target directly | | Partition semantics | Admin | Supported | Most usage | partitions are the real broker's own; set and grow them via the AdminClient as on Kafka | | Topic admin (create / delete / configs) | Admin | Supported | Most usage | the Kafka AdminClient talks to the real broker: create/delete topics, alter configs, and manage partitions natively | | ACLs / SASL auth | Auth | Partial | Full surface | SASL/SCRAM and mutual TLS are supported. Replace MSK IAM authentication and configure access-control lists on the broker. | | Offset commit (\_\_consumer\_offsets) | Consume | Supported | Most usage | offsets commit to the broker's own \_\_consumer\_offsets, exactly as on Kafka | | Glue Schema Registry | Ecosystem | Out of scope | Full surface | AWS Glue Schema Registry has no target-side analog; run a Kafka-native schema registry (Confluent/Apicurio) alongside the broker if you need one | | MSK Connect | Ecosystem | Out of scope | Full surface | MSK Connect (managed Kafka Connect) is not provisioned; run Kafka Connect yourself against the broker if you need connectors | | acks / idempotence / transactions | Producer semantics | Supported | Most usage | acks, idempotence, and transactions are the real broker's own, with full exactly-once semantics | | Broker sizing (MSK broker instance type → CPU / memory / storage) | Sizing | Supported | Most usage | the MSK broker instance type is normalized to CPU/memory and emitted as the broker pod's resource requests/limits (kafka.m5.large → 2 vCPU / 8 GiB), and the storage volume size becomes the broker's persistent-volume size, so sizing follows through to a broker you own | | Tiered storage | Storage | Out of scope | Full surface | MSK's tiered storage is not configured on the target; a self-hosted Kafka broker has native tiered storage (KIP-405, production-ready in Kafka 3.9+) you can enable yourself with a remote-store backend, and the managed targets expose no equivalent knob; retention is the target broker's own local or managed storage | | Consume (consumer groups) | Wire protocol | Supported | Common | your Kafka consumer and consumer-group membership run unchanged over the wire | | Produce (Kafka wire) | Wire protocol | Supported | Common | your Kafka producer runs unchanged; the adapter supplies the target bootstrap-broker endpoint to the target | #### Connecting Kafka clients to Apache Kafka on Kubernetes cluster (via the Strimzi operator) Producers and consumers connect directly to Apache Kafka on Kubernetes cluster (via the Strimzi operator) using the Kafka protocol. Tensor9 supplies the target bootstrap endpoint; there is no Tensor9 message proxy between client and broker. Broker metadata can return additional advertised addresses, so clients need network access and TLS trust for those addresses as well as the bootstrap address. Authentication changes with the broker: SASL/SCRAM and mutual TLS are supported. Replace MSK IAM authentication and manage the broker's access-control lists. Check producer acknowledgment, idempotence and transaction settings against the target-specific support table. A Kafka-compatible endpoint does not by itself guarantee every Kafka administrative or transactional feature. #### State and operating responsibility You configure the replication factor and minimum in-sync replicas (ISR), monitor replica health, and operate storage, recovery and upgrades. Topic partitions hold the retained log and consumer-group offsets record progress. A new broker does not contain the source records, offsets or consumer-group state. Coordinate producer and consumer cutover, and use a separately planned replication process if retained history must move. Source offsets cannot be applied blindly to a different log. Review topic creation, retention, partition counts and broker configuration on the target. AWS IAM authentication, MSK Connect and Glue Schema Registry require separate decisions; changing bootstrap servers does not replace those dependencies. Configure the target's monitoring, backup or recovery procedures and certificate rotation before production cutover. Performance depends on the chosen broker, storage and workload; this profile has no Tensor9 broker benchmark. ## On Akamai, Google Cloud, Private Kubernetes, and Scaleway ### Bitnami Kafka | Operation | Area | Support | Depth | Notes | | ----------------------------------------------------------------- | ------------------ | ------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitoring / broker-log delivery / at-rest KMS / VPC placement | AWS control plane | Out of scope | Full surface | Configure target monitoring and broker logs in place of MSK's CloudWatch, Prometheus, S3 and Firehose settings. Storage encryption uses the target's key and storage configuration. Select target subnets and failure domains separately from the supplied bootstrap endpoint. | | Broker config (custom broker-config revision) | Admin | Partial | Most usage | the target broker natively accepts custom broker config (via the Strimzi or Bitnami chart's config values, or the broker's own config for a raw Apache deployment); MSK's custom broker-config revision has no counterpart, so broker config is set on the target directly | | Partition semantics | Admin | Supported | Most usage | partitions are the real broker's own; set and grow them via the AdminClient as on Kafka | | Topic admin (create / delete / configs) | Admin | Supported | Most usage | the Kafka AdminClient talks to the real broker: create/delete topics, alter configs, and manage partitions natively | | ACLs / SASL auth | Auth | Partial | Full surface | SASL/SCRAM and mutual TLS are supported. Replace MSK IAM authentication and configure access-control lists on the broker. | | Offset commit (\_\_consumer\_offsets) | Consume | Supported | Most usage | offsets commit to the broker's own \_\_consumer\_offsets, exactly as on Kafka | | Glue Schema Registry | Ecosystem | Out of scope | Full surface | AWS Glue Schema Registry has no target-side analog; run a Kafka-native schema registry (Confluent/Apicurio) alongside the broker if you need one | | MSK Connect | Ecosystem | Out of scope | Full surface | MSK Connect (managed Kafka Connect) is not provisioned; run Kafka Connect yourself against the broker if you need connectors | | acks / idempotence / transactions | Producer semantics | Supported | Most usage | acks, idempotence, and transactions are the real broker's own, with full exactly-once semantics | | Broker sizing (MSK broker instance type → CPU / memory / storage) | Sizing | Supported | Most usage | the MSK broker instance type is normalized to CPU/memory and emitted as the broker pod's resource requests/limits (kafka.m5.large → 2 vCPU / 8 GiB), and the storage volume size becomes the broker's persistent-volume size, so sizing follows through to a broker you own | | Tiered storage | Storage | Out of scope | Full surface | MSK's tiered storage is not configured on the target; a self-hosted Kafka broker has native tiered storage (KIP-405, production-ready in Kafka 3.9+) you can enable yourself with a remote-store backend, and the managed targets expose no equivalent knob; retention is the target broker's own local or managed storage | | Consume (consumer groups) | Wire protocol | Supported | Common | your Kafka consumer and consumer-group membership run unchanged over the wire | | Produce (Kafka wire) | Wire protocol | Supported | Common | your Kafka producer runs unchanged; the adapter supplies the target bootstrap-broker endpoint to the target | #### Connecting Kafka clients to Apache Kafka on Kubernetes cluster (via the Bitnami Kafka chart) Producers and consumers connect directly to Apache Kafka on Kubernetes cluster (via the Bitnami Kafka chart) using the Kafka protocol. Tensor9 supplies the target bootstrap endpoint; there is no Tensor9 message proxy between client and broker. Broker metadata can return additional advertised addresses, so clients need network access and TLS trust for those addresses as well as the bootstrap address. Authentication changes with the broker: SASL/SCRAM and mutual TLS are supported. Replace MSK IAM authentication and manage the broker's access-control lists. Check producer acknowledgment, idempotence and transaction settings against the target-specific support table. A Kafka-compatible endpoint does not by itself guarantee every Kafka administrative or transactional feature. #### State and operating responsibility You configure the replication factor and minimum in-sync replicas (ISR), monitor replica health, and operate storage, recovery and upgrades. Topic partitions hold the retained log and consumer-group offsets record progress. A new broker does not contain the source records, offsets or consumer-group state. Coordinate producer and consumer cutover, and use a separately planned replication process if retained history must move. Source offsets cannot be applied blindly to a different log. Review topic creation, retention, partition counts and broker configuration on the target. AWS IAM authentication, MSK Connect and Glue Schema Registry require separate decisions; changing bootstrap servers does not replace those dependencies. Configure the target's monitoring, backup or recovery procedures and certificate rotation before production cutover. Performance depends on the chosen broker, storage and workload; this profile has no Tensor9 broker benchmark. ## On Akamai, Private Kubernetes, and Scaleway ### Apache Kafka | Operation | Area | Support | Depth | Notes | | ----------------------------------------------------------------- | ------------------ | ------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitoring / broker-log delivery / at-rest KMS / VPC placement | AWS control plane | Out of scope | Full surface | Configure target monitoring and broker logs in place of MSK's CloudWatch, Prometheus, S3 and Firehose settings. Storage encryption uses the target's key and storage configuration. Select target subnets and failure domains separately from the supplied bootstrap endpoint. | | Broker config (custom broker-config revision) | Admin | Partial | Most usage | the target broker natively accepts custom broker config (via the Strimzi or Bitnami chart's config values, or the broker's own config for a raw Apache deployment); MSK's custom broker-config revision has no counterpart, so broker config is set on the target directly | | Partition semantics | Admin | Supported | Most usage | partitions are the real broker's own; set and grow them via the AdminClient as on Kafka | | Topic admin (create / delete / configs) | Admin | Supported | Most usage | the Kafka AdminClient talks to the real broker: create/delete topics, alter configs, and manage partitions natively | | ACLs / SASL auth | Auth | Partial | Full surface | SASL/SCRAM and mutual TLS are supported. Replace MSK IAM authentication and configure access-control lists on the broker. | | Offset commit (\_\_consumer\_offsets) | Consume | Supported | Most usage | offsets commit to the broker's own \_\_consumer\_offsets, exactly as on Kafka | | Glue Schema Registry | Ecosystem | Out of scope | Full surface | AWS Glue Schema Registry has no target-side analog; run a Kafka-native schema registry (Confluent/Apicurio) alongside the broker if you need one | | MSK Connect | Ecosystem | Out of scope | Full surface | MSK Connect (managed Kafka Connect) is not provisioned; run Kafka Connect yourself against the broker if you need connectors | | acks / idempotence / transactions | Producer semantics | Supported | Most usage | acks, idempotence, and transactions are the real broker's own, with full exactly-once semantics | | Broker sizing (MSK broker instance type → CPU / memory / storage) | Sizing | Supported | Most usage | the MSK broker instance type is normalized to CPU/memory and emitted as the broker pod's resource requests/limits (kafka.m5.large → 2 vCPU / 8 GiB), and the storage volume size becomes the broker's persistent-volume size, so sizing follows through to a broker you own | | Tiered storage | Storage | Out of scope | Full surface | MSK's tiered storage is not configured on the target; a self-hosted Kafka broker has native tiered storage (KIP-405, production-ready in Kafka 3.9+) you can enable yourself with a remote-store backend, and the managed targets expose no equivalent knob; retention is the target broker's own local or managed storage | | Consume (consumer groups) | Wire protocol | Supported | Common | your Kafka consumer and consumer-group membership run unchanged over the wire | | Produce (Kafka wire) | Wire protocol | Supported | Common | your Kafka producer runs unchanged; the adapter supplies the target bootstrap-broker endpoint to the target | #### Connecting Kafka clients to Apache Kafka on Kubernetes cluster (as a raw Apache Kafka deployment) Producers and consumers connect directly to Apache Kafka on Kubernetes cluster (as a raw Apache Kafka deployment) using the Kafka protocol. Tensor9 supplies the target bootstrap endpoint; there is no Tensor9 message proxy between client and broker. Broker metadata can return additional advertised addresses, so clients need network access and TLS trust for those addresses as well as the bootstrap address. Authentication changes with the broker: SASL/SCRAM and mutual TLS are supported. Replace MSK IAM authentication and manage the broker's access-control lists. Check producer acknowledgment, idempotence and transaction settings against the target-specific support table. A Kafka-compatible endpoint does not by itself guarantee every Kafka administrative or transactional feature. #### State and operating responsibility You configure the replication factor and minimum in-sync replicas (ISR), monitor replica health, and operate storage, recovery and upgrades. Topic partitions hold the retained log and consumer-group offsets record progress. A new broker does not contain the source records, offsets or consumer-group state. Coordinate producer and consumer cutover, and use a separately planned replication process if retained history must move. Source offsets cannot be applied blindly to a different log. Review topic creation, retention, partition counts and broker configuration on the target. AWS IAM authentication, MSK Connect and Glue Schema Registry require separate decisions; changing bootstrap servers does not replace those dependencies. Configure the target's monitoring, backup or recovery procedures and certificate rotation before production cutover. Performance depends on the chosen broker, storage and workload; this profile has no Tensor9 broker benchmark. ## On Azure ### Azure Event Hubs | Operation | Area | Support | Depth | Notes | | ----------------------------------------------------------------- | ------------------ | ------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitoring / broker-log delivery / at-rest KMS / VPC placement | AWS control plane | Out of scope | Full surface | Configure target monitoring and broker logs in place of MSK's CloudWatch, Prometheus, S3 and Firehose settings. Storage encryption uses the target's key and storage configuration. Select target subnets and failure domains separately from the supplied bootstrap endpoint. | | Broker config (custom broker-config revision) | Admin | Out of scope | Most usage | Event Hubs does not expose Kafka broker config, so MSK's custom broker-config revision has no analog | | Partition semantics | Admin | Partial | Most usage | on Standard the partition count is set at creation and cannot be changed; Premium and Dedicated allow increasing (but not decreasing) it after creation, as Kafka itself allows; capacity is sized on a separate axis (throughput units on Standard, processing units on Premium), independent of the partition count | | Topic admin (create / delete / configs) | Admin | Partial | Most usage | topics are Event Hubs, created up front; the Kafka broker-admin/ACL surface differs, so it is not full AdminClient parity | | ACLs / SASL auth | Auth | Partial | Full surface | authentication is Shared Access Signatures or Microsoft Entra, not Kafka SASL/SCRAM or MSK IAM; your Kafka clients configure the Event Hubs connection string instead | | Offset commit (\_\_consumer\_offsets) | Consume | Partial | Most usage | Event Hubs keeps its own consumer-group offset store; the semantics differ slightly from Kafka's \_\_consumer\_offsets | | Glue Schema Registry | Ecosystem | Out of scope | Full surface | AWS Glue Schema Registry has no target-side analog; run a Kafka-native schema registry (Confluent/Apicurio) alongside the broker if you need one | | MSK Connect | Ecosystem | Out of scope | Full surface | MSK Connect (managed Kafka Connect) is not provisioned; run Kafka Connect yourself against the broker if you need connectors | | acks / idempotence / transactions | Producer semantics | Partial | Most usage | produce over the Kafka wire is unchanged; full transactions/idempotence are not complete on the Event Hubs Kafka surface | | Broker sizing (MSK broker instance type → CPU / memory / storage) | Sizing | Partial | Most usage | capacity is throughput units (Standard) or processing units (Premium), sized on a separate axis from the MSK broker instance type; the partition count is fixed at creation on Standard, while Premium and Dedicated allow increasing it | | Tiered storage | Storage | Out of scope | Full surface | MSK's tiered storage is not configured on the target; a self-hosted Kafka broker has native tiered storage (KIP-405, production-ready in Kafka 3.9+) you can enable yourself with a remote-store backend, and the managed targets expose no equivalent knob; retention is the target broker's own local or managed storage | | Consume (consumer groups) | Wire protocol | Supported | Common | your Kafka consumer and consumer-group membership run unchanged over the wire | | Produce (Kafka wire) | Wire protocol | Supported | Common | your Kafka producer runs unchanged; the adapter supplies the target bootstrap-broker endpoint to the target | #### How it works Your Kafka client connects directly to Azure Event Hubs at `.servicebus.windows.net:9093`. Standard and Premium tiers expose this Kafka endpoint. Produce and consume code remain unchanged; configure the new bootstrap address and credentials. Tensor9 does not proxy these connections. Event Hubs differs from Kafka in administration, transactions, authentication and capacity settings. Review those differences below before selecting this target.
Before: on AWS the application's Kafka client connects to MSK bootstrap brokers over SASL_SSL. After: in the target Azure subscription the same client and the same Kafka protocol connect to an Event Hubs namespace's Kafka endpoint on port 9093, with no Tensor9 adapter in the data path. Before: on AWS the application's Kafka client connects to MSK bootstrap brokers over SASL_SSL. After: in the target Azure subscription the same client and the same Kafka protocol connect to an Event Hubs namespace's Kafka endpoint on port 9093, with no Tensor9 adapter in the data path.
#### Estimating namespace capacity MSK capacity is declared as a broker count and instance type, with storage and placement per broker. Event Hubs capacity is set on a namespace using throughput units or processing units. Tensor9 estimates namespace capacity from broker count multiplied by vCPUs per broker. Brokers with at least 16 vCPUs each, such as `kafka.m5.4xlarge`, select Premium processing units. Smaller brokers select Standard throughput units. Basic is excluded because it has no Kafka endpoint. The conversion is reported as a lossy translation because machine size does not determine your workload's messaging throughput. Load-test the generated capacity and adjust it using measured traffic. #### Provision topics as Event Hubs The generated Terraform creates the namespace. It creates no Event Hubs because an MSK cluster resource does not declare topics: Kafka topics are created by producers or administrators at runtime. `aws_msk_configuration` contains `server.properties` settings, not a topic list. Each Kafka topic needs an Event Hub inside the namespace. Provision these before producing messages; this target does not create them automatically on first use. #### Configure authentication Configure clients to use an Event Hubs Shared Access Signature (SAS) policy or Microsoft Entra identity. MSK's SASL/SCRAM, mutual TLS and IAM configurations do not transfer directly. Namespace authentication is configured separately from the generated stack. Credentials use the Cloud Adapter deployment's secrets mechanism; the generated Terraform contains no SAS key or connection string. Event Hubs requires TLS and encrypts stored data by default, so MSK's `encryption_info` is not translated. Configure a customer-managed encryption key separately on Azure if required. #### Limitations △ Where MSK and Event Hubs diverge, read before you adopt * **Transactions and exactly-once are not complete.** Produce and consume over the Kafka wire are unchanged, but the Event Hubs Kafka surface does not offer full Kafka transaction and idempotent-producer semantics. An application that relies on exactly-once delivery through Kafka transactions needs these limitations resolved before it can use this target. * **Partition count is fixed at creation on Standard.** On Standard, an Event Hub's partition count is set when it is created and cannot be changed afterwards. Premium and Dedicated allow increasing it (but never decreasing), which is what Kafka itself allows. Size partitions for the consumer parallelism you expect to need, because on Standard the only way to change your mind is a new hub. * **Capacity and partitions are sized independently.** On Kafka, partitions set both parallelism and much of your throughput headroom. On Event Hubs, throughput is bought separately as throughput units (Standard) or processing units (Premium), independent of partition count. Sizing one does not size the other, and a partition count copied across from MSK does not bring its throughput with it. * **Consumer-group offsets live in a different store.** Event Hubs keeps its own consumer-group offset store rather than Kafka's `__consumer_offsets` topic. Committing and reading offsets works through the Kafka client as usual; tooling that reads the offsets topic directly, or that reasons about its retention and compaction, does not port. * **Kafka administration is only partly supported.** Topics are Event Hubs, created up front, and the broker-administration and ACL surface differs from Kafka's. Code that provisions topics or manages ACLs through AdminClient at runtime should be treated as needing rework rather than assumed to work. * **There is no Kafka version to pin.** MSK's `kafka_version` has no counterpart: Event Hubs tracks its own supported Kafka protocol version and you do not choose or freeze it. A stack that pins a version for compatibility reasons is relying on something this target cannot promise. * **Custom server.properties do not transfer.** MSK's `configuration_info` (retention hours, default partition counts and the rest of `server.properties`) has no namespace-level equivalent. The settings that do exist live per Event Hub and are tuned after provisioning, so a carefully tuned cluster configuration is re-expressed rather than migrated. * **Broker placement across availability zones has no analog.** An Event Hubs namespace is a regional service with no brokers to place, so MSK's per-broker subnet and availability-zone distribution has nothing to map onto. If your MSK topology was chosen for zone-level placement, that reasoning does not transfer. #### Other considerations * **Update client endpoint and authentication.** Repoint the bootstrap servers at the namespace's Kafka endpoint on port 9093 and switch the SASL configuration to a SAS policy or a Microsoft Entra identity. Produce and consume paths, serializers and partitioning logic remain unchanged. * **Data does not migrate; the namespace starts empty.** Provisioning creates a fresh namespace. Messages still in MSK do not move, and Kafka's retention means the window is finite anyway, so plan a cutover in which producers switch over and consumers drain the old cluster, rather than expecting a copy. * **Capacity is the number to revisit after launch.** The emitted SKU and capacity are estimated from your broker fleet, not measured from your traffic. Watch throughput-unit utilisation once real load arrives; Standard's throughput units and Premium's processing units are both adjustable. * **Naming is deterministic, so re-planning is stable.** The namespace name derives from the stack's persisted logical identity and the original cluster name, so the same stack compiles to the same namespace every time. A re-plan does not propose replacing infrastructure that has not changed. ## On DigitalOcean ### DigitalOcean Managed Kafka | Operation | Area | Support | Depth | Notes | | ----------------------------------------------------------------- | ------------------ | ------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitoring / broker-log delivery / at-rest KMS / VPC placement | AWS control plane | Out of scope | Full surface | Configure target monitoring and broker logs in place of MSK's CloudWatch, Prometheus, S3 and Firehose settings. Storage encryption uses the target's key and storage configuration. Select target subnets and failure domains separately from the supplied bootstrap endpoint. | | Broker config (custom broker-config revision) | Admin | Partial | Most usage | broker-config tuning is more limited than a broker you administer directly; MSK's custom broker-config revision is not reproduced | | Partition semantics | Admin | Supported | Most usage | partitions are a real Kafka broker's own | | Topic admin (create / delete / configs) | Admin | Supported | Most usage | the Kafka AdminClient talks to a real managed Kafka broker, so topic and partition admin work natively | | ACLs / SASL auth | Auth | Partial | Full surface | DigitalOcean issues its own SASL/TLS credentials out of band; MSK's IAM auth has no analog, and broker-config tuning is more limited than a broker you administer directly | | Offset commit (\_\_consumer\_offsets) | Consume | Supported | Most usage | offsets commit to the real broker's \_\_consumer\_offsets, as on Kafka | | Glue Schema Registry | Ecosystem | Out of scope | Full surface | AWS Glue Schema Registry has no target-side analog; run a Kafka-native schema registry (Confluent/Apicurio) alongside the broker if you need one | | MSK Connect | Ecosystem | Out of scope | Full surface | MSK Connect (managed Kafka Connect) is not provisioned; run Kafka Connect yourself against the broker if you need connectors | | acks / idempotence / transactions | Producer semantics | Supported | Most usage | acks, idempotence, and transactions are a real managed Kafka engine's own | | Broker sizing (MSK broker instance type → CPU / memory / storage) | Sizing | Partial | Most usage | the managed plan / node size is chosen out of band on DigitalOcean, not derived from the MSK broker instance type or storage volume size | | Tiered storage | Storage | Out of scope | Full surface | MSK's tiered storage is not configured on the target; a self-hosted Kafka broker has native tiered storage (KIP-405, production-ready in Kafka 3.9+) you can enable yourself with a remote-store backend, and the managed targets expose no equivalent knob; retention is the target broker's own local or managed storage | | Consume (consumer groups) | Wire protocol | Supported | Common | your Kafka consumer and consumer-group membership run unchanged over the wire | | Produce (Kafka wire) | Wire protocol | Supported | Common | your Kafka producer runs unchanged; the adapter supplies the target bootstrap-broker endpoint to the target | #### Connecting Kafka clients to DigitalOcean Managed Kafka Producers and consumers connect directly to DigitalOcean Managed Kafka using the Kafka protocol. Tensor9 supplies the target bootstrap endpoint; there is no Tensor9 message proxy between client and broker. Broker metadata can return additional advertised addresses, so clients need network access and TLS trust for those addresses as well as the bootstrap address. Authentication changes with the broker: DigitalOcean issues its own SASL/TLS credentials out of band; MSK's IAM auth has no analog. Check producer acknowledgment, idempotence and transaction settings against the target-specific support table. A Kafka-compatible endpoint does not by itself guarantee every Kafka administrative or transactional feature. #### State and operating responsibility DigitalOcean operates durability and HA; broker-config tuning is more limited than a broker you administer directly. Topic partitions hold the retained log and consumer-group offsets record progress. A new broker does not contain the source records, offsets or consumer-group state. Coordinate producer and consumer cutover, and use a separately planned replication process if retained history must move. Source offsets cannot be applied blindly to a different log. Review topic creation, retention, partition counts and broker configuration on the target. AWS IAM authentication, MSK Connect and Glue Schema Registry require separate decisions; changing bootstrap servers does not replace those dependencies. Configure the target's monitoring, backup or recovery procedures and certificate rotation before production cutover. Performance depends on the chosen broker, storage and workload; this profile has no Tensor9 broker benchmark. ## On Google Cloud ### Managed Service for Apache Kafka | Operation | Area | Support | Depth | Notes | | ----------------------------------------------------------------- | ------------------ | ------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitoring / broker-log delivery / at-rest KMS / VPC placement | AWS control plane | Out of scope | Full surface | Configure target monitoring and broker logs in place of MSK's CloudWatch, Prometheus, S3 and Firehose settings. Storage encryption uses the target's key and storage configuration. Select target subnets and failure domains separately from the supplied bootstrap endpoint. | | Broker config (custom broker-config revision) | Admin | Partial | Most usage | Google manages the broker configuration; per-topic settings (retention, partition count) are yours to set, but MSK's custom broker-config revision has no equivalent | | Partition semantics | Admin | Supported | Most usage | partitions are a real Kafka broker's own; create and grow them via the AdminClient as on Kafka | | Topic admin (create / delete / configs) | Admin | Supported | Most usage | the Kafka AdminClient talks to a real Apache Kafka broker, so topic, config, and partition admin all work natively | | ACLs / SASL auth | Auth | Partial | Full surface | clients authenticate through Google Cloud IAM over SASL\_SSL/OAUTHBEARER with Google's login callback handler; MSK's IAM auth has no analog, and SASL/SCRAM credentials and mTLS client certificates have no equivalent, so producers and consumers swap their callback handler and credentials once | | Offset commit (\_\_consumer\_offsets) | Consume | Supported | Most usage | offsets commit to the real broker's \_\_consumer\_offsets, exactly as on Kafka | | Glue Schema Registry | Ecosystem | Out of scope | Full surface | AWS Glue Schema Registry has no target-side analog; run a Kafka-native schema registry (Confluent/Apicurio) alongside the broker if you need one | | MSK Connect | Ecosystem | Out of scope | Full surface | MSK Connect (managed Kafka Connect) is not provisioned; run Kafka Connect yourself against the broker if you need connectors | | acks / idempotence / transactions | Producer semantics | Supported | Most usage | acks, idempotence, and transactions are a real Apache Kafka broker's own, because Google runs Kafka itself rather than a Kafka-compatible surface | | Broker sizing (MSK broker instance type → CPU / memory / storage) | Sizing | Partial | Most usage | Managed Kafka sizes the cluster (vCPU and memory across the whole fleet) rather than a per-broker instance type, so the source instance capacity is multiplied by broker count to estimate cluster capacity and the per-broker EBS volume becomes the managed per-broker disk | | Tiered storage | Storage | Out of scope | Full surface | MSK's tiered storage is not configured on the target; a self-hosted Kafka broker has native tiered storage (KIP-405, production-ready in Kafka 3.9+) you can enable yourself with a remote-store backend, and the managed targets expose no equivalent knob; retention is the target broker's own local or managed storage | | Consume (consumer groups) | Wire protocol | Supported | Common | your Kafka consumer and consumer-group membership run unchanged over the wire | | Produce (Kafka wire) | Wire protocol | Supported | Common | your Kafka producer runs unchanged; the adapter supplies the target bootstrap-broker endpoint to the target | #### Connecting Kafka clients to Google Managed Service for Apache Kafka Producers and consumers connect directly to Google Managed Service for Apache Kafka using the Kafka protocol. Tensor9 supplies the target bootstrap endpoint; there is no Tensor9 message proxy between client and broker. Broker metadata can return additional advertised addresses, so clients need network access and TLS trust for those addresses as well as the bootstrap address. Authentication changes with the broker: clients authenticate through Google Cloud IAM over SASL\_SSL/OAUTHBEARER; MSK IAM, SASL/SCRAM, and mTLS each need a one-time client-side handler and credential swap. Check producer acknowledgment, idempotence and transaction settings against the target-specific support table. A Kafka-compatible endpoint does not by itself guarantee every Kafka administrative or transactional feature. #### State and operating responsibility Google operates durability and HA; capacity is cluster-wide vCPU/memory rather than a broker instance type you pick, and the cluster is reachable only inside your VPC. Topic partitions hold the retained log and consumer-group offsets record progress. A new broker does not contain the source records, offsets or consumer-group state. Coordinate producer and consumer cutover, and use a separately planned replication process if retained history must move. Source offsets cannot be applied blindly to a different log. Review topic creation, retention, partition counts and broker configuration on the target. AWS IAM authentication, MSK Connect and Glue Schema Registry require separate decisions; changing bootstrap servers does not replace those dependencies. Configure the target's monitoring, backup or recovery procedures and certificate rotation before production cutover. Performance depends on the chosen broker, storage and workload; this profile has no Tensor9 broker benchmark. ## On OCI ### OCI Streaming | Operation | Area | Support | Depth | Notes | | ----------------------------------------------------------------- | ------------------ | ------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitoring / broker-log delivery / at-rest KMS / VPC placement | AWS control plane | Out of scope | Full surface | Configure target monitoring and broker logs in place of MSK's CloudWatch, Prometheus, S3 and Firehose settings. Storage encryption uses the target's key and storage configuration. Select target subnets and failure domains separately from the supplied bootstrap endpoint. | | Broker config (custom broker-config revision) | Admin | Partial | Most usage | OCI exposes a subset of broker configuration settings; MSK's custom broker-config revision has no counterpart | | Partition semantics | Admin | Partial | Most usage | partitions are the stream pool's, under OCI quotas | | Topic admin (create / delete / configs) | Admin | Partial | Most usage | topics are created explicitly (auto-create is off); OCI exposes a subset of Kafka's broker configuration settings, not the full broker admin surface | | ACLs / SASL auth | Auth | Partial | Full surface | authentication is OCI auth tokens at the stream-pool endpoint, not MSK's SASL/SCRAM, mTLS, or IAM; encryption and monitoring move to OCI-managed | | Offset commit (\_\_consumer\_offsets) | Consume | Partial | Most usage | consumer-group offsets use OCI Streaming's managed offset store; the semantics track Kafka's with OCI's quotas | | Glue Schema Registry | Ecosystem | Out of scope | Full surface | AWS Glue Schema Registry has no target-side analog; run a Kafka-native schema registry (Confluent/Apicurio) alongside the broker if you need one | | MSK Connect | Ecosystem | Out of scope | Full surface | MSK Connect (managed Kafka Connect) is not provisioned; run Kafka Connect yourself against the broker if you need connectors | | acks / idempotence / transactions | Producer semantics | Partial | Most usage | produce over the Kafka wire runs unchanged, but full idempotence/transactions are not complete on OCI Streaming's Kafka-compatible API, and acks and throughput are bounded by OCI's quotas rather than a broker you tune | | Broker sizing (MSK broker instance type → CPU / memory / storage) | Sizing | Partial | Most usage | the stream pool is not a broker you size; capacity is OCI's partitions and quotas, not an instance type or a volume you set | | Tiered storage | Storage | Out of scope | Full surface | MSK's tiered storage is not configured on the target; a self-hosted Kafka broker has native tiered storage (KIP-405, production-ready in Kafka 3.9+) you can enable yourself with a remote-store backend, and the managed targets expose no equivalent knob; retention is the target broker's own local or managed storage | | Consume (consumer groups) | Wire protocol | Supported | Common | your Kafka consumer and consumer-group membership run unchanged over the wire | | Produce (Kafka wire) | Wire protocol | Supported | Common | your Kafka producer runs unchanged; the adapter supplies the target bootstrap-broker endpoint to the target | #### Connecting Kafka clients to OCI Streaming Producers and consumers connect directly to OCI Streaming using the Kafka protocol. Tensor9 supplies the target bootstrap endpoint; there is no Tensor9 message proxy between client and broker. Broker metadata can return additional advertised addresses, so clients need network access and TLS trust for those addresses as well as the bootstrap address. Authentication changes with the broker: authentication is OCI auth tokens at the stream-pool endpoint, not MSK's SASL/SCRAM, mTLS, or IAM; your Kafka clients configure OCI's auth instead. Check producer acknowledgment, idempotence and transaction settings against the target-specific support table. A Kafka-compatible endpoint does not by itself guarantee every Kafka administrative or transactional feature. #### State and operating responsibility Oracle operates durability and HA; partitions and retention live under OCI's quotas rather than a broker you tune directly. Topic partitions hold the retained log and consumer-group offsets record progress. A new broker does not contain the source records, offsets or consumer-group state. Coordinate producer and consumer cutover, and use a separately planned replication process if retained history must move. Source offsets cannot be applied blindly to a different log. Review topic creation, retention, partition counts and broker configuration on the target. AWS IAM authentication, MSK Connect and Glue Schema Registry require separate decisions; changing bootstrap servers does not replace those dependencies. Configure the target's monitoring, backup or recovery procedures and certificate rotation before production cutover. Performance depends on the chosen broker, storage and workload; this profile has no Tensor9 broker benchmark. ## On Scaleway ### Scaleway Clusters for Apache Kafka | Operation | Area | Support | Depth | Notes | | ----------------------------------------------------------------- | ------------------ | ------------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitoring / broker-log delivery / at-rest KMS / VPC placement | AWS control plane | Out of scope | Full surface | Configure target monitoring and broker logs in place of MSK's CloudWatch, Prometheus, S3 and Firehose settings. Storage encryption uses the target's key and storage configuration. Select target subnets and failure domains separately from the supplied bootstrap endpoint. | | Broker config (custom broker-config revision) | Admin | Out of scope | Most usage | a custom MSK configuration (aws\_msk\_configuration server.properties - log.retention.hours, num.partitions and the rest) is NOT projected; tune the Scaleway cluster's topic and broker settings after it is provisioned | | Partition semantics | Admin | Supported | Most usage | partitions are a real Kafka broker's own | | Topic admin (create / delete / configs) | Admin | Supported | Most usage | the Kafka AdminClient talks to a real broker, so topic and partition admin work natively | | ACLs / SASL auth | Auth | Partial | Full surface | Scaleway issues its OWN credentials for the cluster (the resource carries user\_name/password) rather than translating MSK's. A cluster whose clients use SASL/SCRAM or mTLS is reconfigured onto those credentials. A cluster whose clients use SASL/IAM is a HARDER change and is called out separately below - that one is a mechanism, not a credential | | Offset commit (\_\_consumer\_offsets) | Consume | Supported | Most usage | offsets commit to the real broker's \_\_consumer\_offsets, as on Kafka | | Glue Schema Registry | Ecosystem | Out of scope | Full surface | AWS Glue Schema Registry has no target-side analog; run a Kafka-native schema registry (Confluent/Apicurio) alongside the broker if you need one | | MSK Connect | Ecosystem | Out of scope | Full surface | MSK Connect (managed Kafka Connect) is not provisioned; run Kafka Connect yourself against the broker if you need connectors | | acks / idempotence / transactions | Producer semantics | Supported | Most usage | acks, idempotence and transactions are a real Kafka engine's own - Scaleway runs Apache Kafka 4.1, not a Kafka-compatible protocol layer | | Broker sizing (MSK broker instance type → CPU / memory / storage) | Sizing | Partial | Most usage | the MSK broker instance type is not carried across: Scaleway sizes by its own node-type catalogue, and the build emits the smallest of the three it offers (brokernode-shared-2c-8g, 2 vCPU / 8 GB). Broker STORAGE does follow - a requested volume is rounded UP to Scaleway's 10 GB floor in 5 GB steps | | Tiered storage | Storage | Out of scope | Full surface | MSK's tiered storage is not configured on the target; a self-hosted Kafka broker has native tiered storage (KIP-405, production-ready in Kafka 3.9+) you can enable yourself with a remote-store backend, and the managed targets expose no equivalent knob; retention is the target broker's own local or managed storage | | Consume (consumer groups) | Wire protocol | Supported | Common | your Kafka consumer and consumer-group membership run unchanged over the wire | | Produce (Kafka wire) | Wire protocol | Supported | Common | your Kafka producer runs unchanged; the adapter supplies the target bootstrap-broker endpoint to the target | ## Existing data and credentials Selecting a backend does not copy existing data, credentials or access policies. Plan and verify migration separately before changing an application's endpoint. Do not assume an identifier, credential or encrypted value from the origin service works unchanged on the target. ## Configure, tune and debug Start with [setup](/cloud-adapter/getting-started/overview) and [configuration](/cloud-adapter/configuration/overview). Use [tuning](/cloud-adapter/tuning/overview) to understand supported request tags, [debugging](/cloud-adapter/debugging/overview) to investigate a request, and [High Fidelity Cloud Emulators](/cloud-adapter/local-testing/overview) to validate a bounded reproduction.