Kubernetes

Kubernetes resources can be used with Tensor9 by embedding them within Terraform or CloudFormation origin stacks. Unlike other infrastructure-as-code formats that Tensor9 supports, Kubernetes manifests cannot be used as standalone origin stacks - they must be embedded within a parent origin stack.

What is a Kubernetes origin stack?

A Kubernetes origin stack consists of Kubernetes resources (Deployments, Services, ConfigMaps, etc.) defined within a Terraform or CloudFormation origin stack using the respective provider’s Kubernetes resources. Key characteristic: Kubernetes resources are always embedded within another origin stack format. The parent origin stack (Terraform or CloudFormation) serves as the container, while Kubernetes manifests define the workload orchestration.

Your Kubernetes resources should be part of your existing Terraform or CloudFormation configuration. Tensor9 is designed to work with the infrastructure-as-code you already have - you don’t need to rewrite your Kubernetes deployments just for Tensor9. The goal is to maintain a single stack that works for both your cloud deployment and private customer deployments.

Why embed Kubernetes?

Kubernetes manifests define how your application runs (pods, deployments, services), but they don’t provision the underlying infrastructure (clusters, networks, load balancers). By embedding Kubernetes within Terraform or CloudFormation, you get:

Complete infrastructure: The parent stack provisions the cluster (EKS, GKE, AKS) and supporting infrastructure
Unified deployment: One release process deploys both infrastructure and workloads
Automatic artifact handling: Tensor9 automatically copies container images to customer environments
Form factor adaptation: Kubernetes workloads adapt to different cloud environments seamlessly

How Kubernetes origin stacks work

Define Kubernetes resources in Terraform/CloudFormation

In your Terraform or CloudFormation origin stack, use the Kubernetes provider to define your Kubernetes resources. For Terraform, this typically means using kubernetes_manifest or kubernetes_deployment resources:Terraform example:

resource "kubernetes_manifest" "my_app_deployment" {
  manifest = {
    apiVersion = "apps/v1"
    kind       = "Deployment"
    metadata = {
      name      = "my-app"
      namespace = "default"
    }
    spec = {
      replicas = 3
      selector = {
        matchLabels = {
          app = "my-app"
        }
      }
      template = {
        metadata = {
          labels = {
            app = "my-app"
          }
        }
        spec = {
          containers = [
            {
              name  = "my-app"
              image = "myregistry.io/my-app:v1.0.0"
              ports = [
                { containerPort = 8080 }
              ]
            }
          ]
        }
      }
    }
  }
}

Publish and create a release

Publish your parent origin stack (Terraform or CloudFormation) to your control plane. When you create a release for an appliance, your control plane:

Finds Kubernetes resources: Scans the origin stack for kubernetes_manifest, kubernetes_deployment, and other Kubernetes provider resources
Extracts container images: Identifies all container image references in Kubernetes specs
Prepares image copying: Configures the deployment stack to copy images to the appliance’s container registry (specific to the cloud provider)
Rewrites image references: Updates container image fields to point to the locally-copied images
Compiles the deployment stack: Generates a ready-to-deploy stack with all Kubernetes resources intact

The result is a deployment stack that includes both your infrastructure and Kubernetes workloads. When deployed, the deployment stack will copy the container images into the customer’s appliance.

Deploy to the appliance

Deploy the compiled deployment stack using the parent stack’s tooling:For Terraform deployment stacks:

cd acme-corp-production
tofu init
tofu apply

For CloudFormation deployment stacks: Your control plane automatically creates the CloudFormation stack in your control plane’s AWS account. Monitor deployment using:

tensor9 report -customerName acme-corp

During deployment, the Kubernetes resources are applied to the customer’s cluster with all container images pointing to the locally-copied versions.

Supported Kubernetes resources

Tensor9 automatically handles artifact copying for these Kubernetes resource types:

kubernetes_manifest (Terraform)

The kubernetes_manifest resource accepts any Kubernetes manifest as a map. Tensor9 automatically detects and processes:

Deployments (kind: "Deployment"): Extracts container images from spec.template.spec.containers[].image
Other workload types: Support for additional resource types is expanding

Example:

resource "kubernetes_manifest" "nginx" {
  manifest = {
    apiVersion = "apps/v1"
    kind       = "Deployment"
    metadata = {
      name = "nginx"
    }
    spec = {
      template = {
        spec = {
          containers = [
            {
              name  = "nginx"
              image = "docker.io/nginx:1.21"  # Automatically copied
            }
          ]
        }
      }
    }
  }
}

kubernetes_deployment (Terraform)

The kubernetes_deployment resource provides typed Kubernetes Deployment support. Tensor9 extracts container images from the deployment spec. Example:

resource "kubernetes_deployment" "app" {
  metadata {
    name = "my-app"
  }
  spec {
    template {
      spec {
        container {
          name  = "app"
          image = "ghcr.io/myorg/app:v1.0.0"  # Automatically copied
        }
      }
    }
  }
}

Other Kubernetes provider resources

Tensor9 supports other Kubernetes provider resources (Services, ConfigMaps, Secrets, etc.). These resources pass through compilation unchanged, as they typically don’t reference external artifacts.

Helm charts

Helm charts can be deployed using the Terraform Helm provider. Helm is a package manager for Kubernetes that bundles multiple Kubernetes resources into a single deployable unit called a “chart.”

Add the Helm provider to your Terraform configuration

Include the Helm provider in your required_providers block:

terraform {
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 3.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.38"
    }
  }
}

Configure the provider to connect to your Kubernetes cluster

Configure the Helm provider to use your cluster’s endpoint and credentials:

provider "helm" {
  kubernetes {
    host                   = aws_eks_cluster.main.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.main.token
  }
}

Deploy charts using helm_release resources

Define helm_release resources to deploy Helm charts:

resource "helm_release" "nginx_ingress" {
  name       = "nginx-ingress"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = "default"
  version    = "4.10.1"

  set {
    name  = "controller.service.type"
    value = "LoadBalancer"
  }

  depends_on = [aws_eks_cluster.main]
}

How Helm charts are compiled

Helm resources pass through

The helm_release resource is included in the deployment stack unchanged.

Charts deploy at runtime

When you run tofu apply on the deployment stack, Terraform installs the Helm chart into the cluster.

Container images in charts

Container images referenced within Helm charts are pulled from their original registries at deployment time.

Currently, Tensor9 does not automatically copy container images referenced within Helm charts to the appliance’s container registry. This means Helm charts must be able to pull images from their original registries (e.g., Docker Hub, GitHub Container Registry). If your appliance cannot reach external registries, consider using Kubernetes manifests directly instead of Helm charts, or pre-pull images and reference them from a local registry.

Best practices for Helm charts

Pin chart versions: Always specify a version in your helm_release to ensure consistent deployments
Use accessible registries: Ensure Helm charts reference images from registries the appliance can reach
Test chart deployments: Validate Helm charts in test appliances before deploying to customer appliances
Configure chart values: Use set or values blocks to customize chart behavior for each environment

Container image handling

Tensor9 automatically handles container images referenced in Kubernetes resources:

Automatic image copying

When Tensor9 finds a container image in a Kubernetes resource, it:

Validates the image reference: Ensures the image has a registry (e.g., docker.io, ghcr.io, myregistry.io)
Configures image copying: Prepares the deployment stack to copy the image to the appliance’s container registry
Rewrites the reference: Updates the Kubernetes manifest to point to the locally-copied image

Before compilation (origin stack):

spec = {
  containers = [
    {
      image = "docker.io/nginx:1.21"
    }
  ]
}

After compilation (deployment stack):

spec = {
  containers = [
    {
      # Now points to the customer's ECR or appliance registry
      image = "123456789.dkr.ecr.us-west-2.amazonaws.com/t9-app-images:nginx-1.21-abc123"
    }
  ]
}

Image registry requirements

For Tensor9 to copy a container image, it must include a registry in the image reference:

✅ Supported: docker.io/nginx:latest, ghcr.io/myorg/app:v1.0, myregistry.io/image:tag
⚠️ Skipped: nginx:latest (no registry - assumed to be publicly available in the appliance)

Images without a registry are assumed to be publicly available from Docker Hub and are not copied. If your appliance cannot reach Docker Hub, make sure to include the registry prefix: docker.io/nginx:latest instead of nginx:latest.

Where images are stored

Copied container images are stored in the appliance’s container registry. The storage location is determined by the appliance’s form factor and is handled automatically by Tensor9:

Appliance Environment	Container Registry
AWS	Amazon ECR
Google Cloud	Google Artifact Registry
Azure	Azure Container Registry
DigitalOcean	DigitalOcean Container Registry
Private Kubernetes	Appliance-local container registry

Prerequisites

Before using Kubernetes resources in your origin stack:

For Terraform origin stacks

Kubernetes provider configured: Include the Kubernetes Terraform provider in your required_providers
Cluster access configured: The Kubernetes provider must be configured to connect to your cluster (typically via EKS, GKE, or AKS data sources)
Valid Kubernetes manifests: Your Kubernetes resources must be valid according to the Kubernetes API

Example Terraform configuration:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.38"
    }
  }
}

# Configure Kubernetes provider using EKS cluster
data "aws_eks_cluster" "cluster" {
  name = aws_eks_cluster.main.name
}

data "aws_eks_cluster_auth" "cluster" {
  name = aws_eks_cluster.main.name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

For CloudFormation origin stacks

Support for Kubernetes resources in CloudFormation origin stacks is under development. Currently, Terraform is the recommended approach for embedding Kubernetes resources.

If you have a CloudFormation + Kubernetes use case, please reach out to [email protected] to discuss your requirements.

Example: Complete Terraform + Kubernetes origin stack

This example shows a complete Terraform origin stack that provisions an EKS cluster and deploys a Kubernetes application:

# Configure providers
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.38"
    }
  }
}

provider "aws" {
  region = "us-west-2"
}

# Create EKS cluster and supporting infrastructure
# (VPC, subnets, IAM roles, etc. - see AWS documentation)
resource "aws_eks_cluster" "main" {
  name     = "my-app-cluster"
  role_arn = aws_iam_role.eks_cluster.arn

  vpc_config {
    subnet_ids = [aws_subnet.private_1.id, aws_subnet.private_2.id]
  }
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "main-nodes"
  node_role_arn   = aws_iam_role.eks_nodes.arn
  subnet_ids      = [aws_subnet.private_1.id, aws_subnet.private_2.id]

  scaling_config {
    desired_size = 2
    max_size     = 4
    min_size     = 1
  }
}

# Configure Kubernetes provider
data "aws_eks_cluster" "main" {
  name = aws_eks_cluster.main.name
}

data "aws_eks_cluster_auth" "main" {
  name = aws_eks_cluster.main.name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

# Deploy application to Kubernetes
resource "kubernetes_manifest" "app_deployment" {
  manifest = {
    apiVersion = "apps/v1"
    kind       = "Deployment"
    metadata = {
      name      = "my-app"
      namespace = "default"
    }
    spec = {
      replicas = 3
      selector = {
        matchLabels = {
          app = "my-app"
        }
      }
      template = {
        metadata = {
          labels = {
            app = "my-app"
          }
        }
        spec = {
          containers = [
            {
              name  = "app"
              # This image will be automatically copied to the customer's ECR
              image = "ghcr.io/myorg/my-app:v1.0.0"
              ports = [
                { containerPort = 8080 }
              ]
              env = [
                {
                  name  = "DATABASE_URL"
                  value = "postgresql://..."
                }
              ]
            }
          ]
        }
      }
    }
  }
}

resource "kubernetes_manifest" "app_service" {
  manifest = {
    apiVersion = "v1"
    kind       = "Service"
    metadata = {
      name      = "my-app"
      namespace = "default"
    }
    spec = {
      type = "LoadBalancer"
      selector = {
        app = "my-app"
      }
      ports = [
        {
          port       = 80
          targetPort = 8080
        }
      ]
    }
  }
}

Publishing and deploying

Publishing and deploying a Kubernetes origin stack follows the same workflow as any Terraform origin stack:

1. Publish your origin stack

tensor9 stack publish \
  -stackType TerraformWorkspace \
  -stackS3Key my-kubernetes-app \
  -dir /path/to/terraform

This uploads your entire Terraform workspace (including Kubernetes resources) to your control plane.

2. Bind the stack to your app

tensor9 stack bind \
  -appName my-app \
  -nativeStackId "s3://bucket/my-kubernetes-app.tf.tgz"

Only needed once per app.

3. Create a release

tensor9 stack release create \
  -appName my-app \
  -testApplianceName my-test-appliance \
  -vendorVersion "1.0.0"

Your control plane compiles the origin stack, automatically copying container images and rewriting references.

4. Deploy the deployment stack

cd my-test-appliance
tofu init
tofu apply

This creates the EKS cluster, node groups, and deploys your Kubernetes workloads. The deployment stack will copy container images into the customer’s appliance.

Best practices

Always specify image registries

Include the full registry in your container image references (docker.io/nginx:latest instead of nginx:latest). This ensures Tensor9 can copy the images to customer environments.

Use versioned image tags

Avoid using :latest tags in production. Use specific version tags (v1.0.0, sha-abc123) to ensure consistent deployments across customer appliances.

Configure provider authentication carefully

When configuring the Kubernetes provider to connect to your cluster, use data sources (like aws_eks_cluster) rather than hardcoded values. This ensures the provider configuration adapts to each appliance’s cluster.

Use environment variables for secrets

Store secrets in AWS Secrets Manager in your origin stack, then pass them to your Kubernetes pods as environment variables. Avoid using Kubernetes Secrets for sensitive data, as Tensor9 does not automatically map runtime SDK calls to fetch secrets across different cloud environments.

# Define secret in AWS Secrets Manager
resource "aws_secretsmanager_secret" "db_password" {
  name = "${var.instance_id}/prod/db/password"
}

# Reference in Kubernetes Deployment as environment variable
resource "kubernetes_deployment" "app" {
  spec {
    template {
      spec {
        container {
          env {
            name = "DB_PASSWORD"
            value_from {
              secret_key_ref {
                name = aws_secretsmanager_secret.db_password.name
                key  = "password"
              }
            }
          }
        }
      }
    }
  }
}

For non-sensitive configuration, Kubernetes ConfigMaps are appropriate.

Mind resource limits and requests

Always specify resource requests and limits for containers. This ensures proper scheduling and prevents resource contention in customer clusters.

Limitations and considerations

Standalone Kubernetes manifests not supported

Kubernetes YAML files cannot be used as standalone origin stacks. They must be embedded within Terraform or CloudFormation. This is by design - Kubernetes defines workloads, not the underlying infrastructure.

Cluster must exist before Kubernetes resources

The parent origin stack (Terraform/CloudFormation) must provision or reference the Kubernetes cluster before defining Kubernetes resources. Use proper dependency management (depends_on in Terraform) to ensure correct ordering.

Kubernetes provider versions

Ensure your Kubernetes provider version is compatible with your target cluster version. Different Kubernetes versions may have different API schemas for resources.

Image pull secrets

If your container images require authentication to pull, you’ll need to configure image pull secrets in your Kubernetes manifests. Tensor9 copies the images but doesn’t automatically create pull secrets.

Troubleshooting

Container image not copied

Symptom: Deployment fails because container image cannot be pulled from the original registry.Cause: Image reference doesn’t include a registry, or Tensor9 couldn’t detect the image reference.Solution:

Ensure your image reference includes the full registry: docker.io/nginx:latest
Check that the image is referenced in a supported field (spec.containers[].image)
Review compilation logs for warnings about skipped images

Kubernetes provider authentication fails

Symptom: Terraform apply fails with “unable to connect to Kubernetes cluster” error.Cause: Kubernetes provider is not correctly configured to connect to the cluster.Solution:

Verify the cluster exists before applying Kubernetes resources
Use data sources to dynamically configure the provider
Check that IAM roles/permissions allow cluster access

Deployment applies but pods don't start

Symptom: Kubernetes deployment is created but pods fail to start with ImagePullBackOff.Cause: Pods cannot pull the container image from the appliance registry.Solution:

Verify that the deployment stack shows rewritten image references
Check that the node group has permissions to pull from ECR (for AWS)
Ensure the image was successfully copied during compilation

Terraform/OpenTofu: Learn about Terraform origin stacks
CloudFormation: Learn about CloudFormation origin stacks
Deployments: Understand the deployment workflow
Form Factors: How Kubernetes workloads adapt to different environments

Tensor9

Getting Started

How It Works

Deploying

Your Customer's Environment

Auto-Customizations

Security

Connectivity & Networking

Observability & Operations

tensor9 CLI

What is a Kubernetes origin stack?

Why embed Kubernetes?

How Kubernetes origin stacks work

Supported Kubernetes resources

kubernetes_manifest (Terraform)

kubernetes_deployment (Terraform)

Other Kubernetes provider resources

Helm charts

How Helm charts are compiled

Best practices for Helm charts

Container image handling

Automatic image copying

Image registry requirements

Where images are stored

Prerequisites

For Terraform origin stacks

For CloudFormation origin stacks

Example: Complete Terraform + Kubernetes origin stack

Publishing and deploying

1. Publish your origin stack

2. Bind the stack to your app

3. Create a release

4. Deploy the deployment stack

Best practices

Limitations and considerations

Troubleshooting

​What is a Kubernetes origin stack?

​Why embed Kubernetes?

​How Kubernetes origin stacks work

​Supported Kubernetes resources

​kubernetes_manifest (Terraform)

​kubernetes_deployment (Terraform)

​Other Kubernetes provider resources

​Helm charts

​How Helm charts are compiled

​Best practices for Helm charts

​Container image handling

​Automatic image copying

​Image registry requirements

​Where images are stored

​Prerequisites

​For Terraform origin stacks

​For CloudFormation origin stacks

​Example: Complete Terraform + Kubernetes origin stack

​Publishing and deploying

​1. Publish your origin stack

​2. Bind the stack to your app

​3. Create a release

​4. Deploy the deployment stack

​Best practices

​Limitations and considerations

​Troubleshooting

​Related topics

What is a Kubernetes origin stack?

Why embed Kubernetes?

How Kubernetes origin stacks work

Supported Kubernetes resources

kubernetes_manifest (Terraform)

kubernetes_deployment (Terraform)

Other Kubernetes provider resources

Helm charts

How Helm charts are compiled

Best practices for Helm charts

Container image handling

Automatic image copying

Image registry requirements

Where images are stored

Prerequisites

For Terraform origin stacks

For CloudFormation origin stacks

Example: Complete Terraform + Kubernetes origin stack

Publishing and deploying

1. Publish your origin stack

2. Bind the stack to your app

3. Create a release

4. Deploy the deployment stack

Best practices

Limitations and considerations

Troubleshooting

Related topics