Skip to main content
A submitted ops command moves through a state machine on its way from “you want to run this” to “you have the released output.” This page walks your side of that flow: how to submit, how to watch progress, how customer review and output release work, and how to verify the audit chain. Commands created through the support-request flow also provide a /support/<token> link for customer review; plain CLI submission does not itself attach that request metadata.
One command has separate customer decisions for execution and output release. The vendor submits, the customer signs command approval, the appliance verifies and executes when a worker is available, and the customer reviews and signs output release. Plaintext follows authorized release. Arrows show sequence, not network traffic.One command has separate customer decisions for execution and output release. The vendor submits, the customer signs command approval, the appliance verifies and executes when a worker is available, and the customer reviews and signs output release. Plaintext follows authorized release. Arrows show sequence, not network traffic.One command has separate customer decisions for execution and output release. The vendor submits, the customer signs command approval, the appliance verifies and executes when a worker is available, and the customer reviews and signs output release. Plaintext follows authorized release. Arrows show sequence, not network traffic.One command has separate customer decisions for execution and output release. The vendor submits, the customer signs command approval, the appliance verifies and executes when a worker is available, and the customer reviews and signs output release. Plaintext follows authorized release. Arrows show sequence, not network traffic.
The approved customer-signed path separates command approval from output release. On an available worker, the appliance verifies approval, executes the command and encrypts the normal result record with a key kept in the customer vault. After execution, the customer reviews output and signs release; authorized release makes plaintext available to you. Arrows show logical sequence. Rejection, deadlines and other paths are covered below.

​
Submitting a command

Required flags: One of the following picks the body of the command: Common modifier flags: The --commandType flag selects between command shapes. It’s almost always inferred from --template or auto-defaults; only set explicitly if you’re authoring tooling that needs to be specific:

​
A note on --vars escaping

--vars is a single comma-separated string, so values cannot contain commas or = characters. Use templates whose variable values are constrained to simple alphanumerics + path characters. The action prints the assigned commandName and the initial state, then returns. The command is now Submitted; your customer’s review experience begins next.

​
Lifecycle at a glance

The lifecycle has 14 states: eight inflight states and six terminal states. Completed is the successful terminal state. OutputApproved remains inflight until the appliance has supplied plaintext and your control plane has mirrored output. Cancelling records cancellation intent; the state alone does not prove that the process stopped or its effects were rolled back.

​
Command approval

Submitted moves through configured approval to CmdApproved, or ends at CmdRejected; elapsed deadlines can end any inflight state at Timeout.Submitted moves through configured approval to CmdApproved, or ends at CmdRejected; elapsed deadlines can end any inflight state at Timeout.Submitted moves through configured approval to CmdApproved, or ends at CmdRejected; elapsed deadlines can end any inflight state at Timeout.Submitted moves through configured approval to CmdApproved, or ends at CmdRejected; elapsed deadlines can end any inflight state at Timeout.

The appliance asks its configured approver, which can wait, approve or reject. Existing signed support approval or rejection can act directly from Submitted, so CmdApproving is not a mandatory stop. Pre-approval can supply the decision without a new per-command human review. Solid arrows advance lifecycle; dashed arrows show rejection or deadline alternatives.

​
Execution and executing cancellation

CmdApproved advances through Executing to Executed; exception or staleness can fail execution, while an executing cancel takes Cancelling to Cancelled.CmdApproved advances through Executing to Executed; exception or staleness can fail execution, while an executing cancel takes Cancelling to Cancelled.CmdApproved advances through Executing to Executed; exception or staleness can fail execution, while an executing cancel takes Cancelling to Cancelled.CmdApproved advances through Executing to Executed; exception or staleness can fail execution, while an executing cancel takes Cancelling to Cancelled.

After approval verification and worker admission, the appliance runs the command and stores its result. Exceptions or stale execution can lead to ExecutionFailed; a returned nonzero exit code is not the same as a reactor exception. The Executing → Cancelling → Cancelled path records lifecycle changes, without guaranteeing process termination, rollback or absence of partial effects. Solid arrows show normal progression; dashed arrows show failure, executing cancellation or elapsed deadlines.

​
Output release and completion

Executed can progress through OutputApproving and OutputApproved to Completed; output rejection, release failures and deadlines have separate paths, and failure output decisions do not revive the lifecycle.Executed can progress through OutputApproving and OutputApproved to Completed; output rejection, release failures and deadlines have separate paths, and failure output decisions do not revive the lifecycle.Executed can progress through OutputApproving and OutputApproved to Completed; output rejection, release failures and deadlines have separate paths, and failure output decisions do not revive the lifecycle.Executed can progress through OutputApproving and OutputApproved to Completed; output rejection, release failures and deadlines have separate paths, and failure output decisions do not revive the lifecycle.

OutputApproved stays inflight while plaintext and mirroring are completed; Completed is the successful terminal state. The configured approver or the existing signed support route can approve or reject release; that route can act directly from Executed. Failure diagnostics can be released or rejected by changing outputApproval while lifecycle stays ExecutionFailed. Solid arrows advance lifecycle, dashed arrows show alternatives, and the dotted blue arrow changes metadata only.

​
Happy path in words

1

Submitted

The command exists; your customer’s appliance has not picked it up yet. You see this immediately after tensor9 ops command create.
2

CmdApproving

Your customer’s appliance has the command in its inbox and is waiting for its configured approver. The decision may come from customer review or a verified pre-approval. A command created through the support-request flow has a /support/<token> link; a plain CLI submission does not itself attach that request metadata.
3

CmdApproved

Execution is approved and the appliance is preparing to run the command. Approval verification and an available execution worker still precede Executing. An approval state does not add resource permissions.
4

Executing

The command body is running inside the appliance’s sandboxed working directory. The appliance captures stdout / stderr / exit code, uploads each output stream to your blob store (S3 in your customer’s account) and stores a small [blob: ...]\\n<presigned-url> payload on the command record. That payload is then encrypted with a per-command key.
5

Executed

Execution finished. The appliance is now waiting for your customer to review the output and decide whether to release it to you.
6

OutputApproved

Your customer signed an Ed25519 release manifest. The control plane surfaces the decrypted blob-payload to you; calling tensor9 ops command retrieve returns the payload, and you fetch the actual bytes by curling the presigned URL it contains. State advances to Completed next.

​
Terminal unhappy states

​
Encryption mechanism

Output is encrypted with a per-command AES-256-GCM key that the appliance generates on-the-fly. The ciphertext includes a SHA-256 fingerprint of the key (in the AAD), so the decrypt path can find the right key in the appliance’s secret store even after a re-execution overwrites the per-scope slot. Keys live in the appliance’s secret store under /t9-private/projection/.../ops-cmd/... and are deleted after successful release or output rejection. When you see a “decryption failed” alert from the appliance, the likely causes are: (a) a re-execution clobbered the scope-keyed slot before your customer released the previous run’s output, (b) the secret store is unreachable, (c) your customer rotated keys mid-flight. The fingerprint addressing is the defense for (a); see the appliance audit log for the specific failure mode.

​
Watching progress

retrieve shows the current state, full audit chain (who approved what, when), and (once Completed) the released stdout / stderr / exit code. Both commands accept --output json for scripting; pipe into jq .lifecycle to poll a single state value.

​
What your customer sees

For commands created through the support-request flow, your customer can use the /support/<token> link for this four-step approval UI:
For support-request commands, if your customer hasn’t subscribed a notification channel for ops-command events, submission silently produces no notification. The command still appears in your tensor9 ops command list (state: Submitted), and your customer would only see it if they happened to visit the support portal directly. Confirm channel subscription with each customer at onboarding; otherwise an “unanswered” command is more likely to mean “your customer doesn’t know about it” than “your customer is ignoring it.”

​
What “review” actually shows

For Terraform templates, your customer sees the literal HCL source plus the values they’re submitting for each variable. They do not see a tofu plan output: a plan would require evaluating data sources, which can only run after the appliance is authorized to do so. Your customer is reviewing “the shape of what will execute” plus the declared data_access, side_effects, and permission tier, not a fully-resolved diff. Things the HCL surface does NOT pre-evaluate for the reviewer:
  • ${var.X} interpolations stay as literal strings in the displayed HCL. Your customer sees ${var.MOUNT_PREFIX} in the command body and the submitted value of MOUNT_PREFIX separately; they have to substitute mentally at review time. The approval UI shows the submitted variable values next to the HCL.
  • for_each cardinality is invisible at review time. A for_each = toset(data.aws_s3_buckets.all.buckets[*].name) does not show whether it will iterate over 3 buckets or 30,000. Bound the potential blast radius via data_access + side_effects declarations and use description to explain the cardinality semantics in plain English.
  • local-exec heredocs are reviewed as shell. A multi-line command = <<-EOT ... EOT is shown verbatim. Customers reviewing a kubectl drain ... && kubectl ... heredoc are reviewing a shell program, not a Terraform plan. Keep heredocs short and named in the description.
  • null_resource.triggers are not re-evaluated against prior state (there is no prior state; see Authoring templates). A triggers = { mount_prefix = var.MOUNT_PREFIX } block makes the resource look like it fires only on change, which is misleading. Document the behavior in the template’s description or omit the triggers block.
Because the review surface is the HCL, the description field on tensor9_command is what your customer actually reads before approving. Treat it as the plain-English equivalent of the HCL: name the exact APIs called, the expected output shape, the cardinality of any fan-out, and the intended side effects. For Script and Kubectl templates the review surface is the literal script body or kubectl invocation. Same caveats: ${VAR} references are unsubstituted; the customer reads the script and the variable values side-by-side.

​
Trust properties for the release step

Release has two properties to explain to your customer:
  • The plaintext output passes through the appliance your customer already controls before you see it. Decryption happens on the appliance (which lives in your customer’s cloud account under their IAM); the control plane only ever sees the ciphertext before release. Once your customer signs release, the control plane forwards the plaintext to you.
  • The release decision is non-repudiable. Your customer’s signed release manifest is preserved in the audit chain and can be verified independently with tensor9 ops command audit verify.
Pre-approved templates skip the per-command Review and Approve steps; see Pre-approvals.

​
Resource limits and queueing

Operations is for diagnostics and short-lived interventions, not for bulk data extraction. The appliance enforces a few limits that you should size your templates against: The 5 GiB cap is high enough that you generally don’t think about it. The appliance uploads each stream to your S3 bucket and stores only a small [blob: bucket=..., key=..., size=..., sha256=...]\\n<presigned-url> payload (encrypted) on the command record. The customer’s release script fetches the actual bytes, sha256-verifies them, and shows them in the local preview. Your tensor9 ops command retrieve returns the same payload; curling the URL gives you the bytes back. Bulk log dumps (journalctl --since 24h, kubectl logs deployment/...) flow through without the per-template | tail -c self-capping templates used to need. On Kubernetes-form-factor appliances whose blob store does not support presigned URLs, such as MinIO, the appliance falls back to an inline 4 MiB cap with a marker like [stdout truncated - 4 MB cap]; the release script’s preview still works, but the customer and you see only the first 4 MiB of any stream over the cap. If you’re on-call and you see a command stuck in Executing for more than five minutes, you can either wait for the 20-minute staleness recovery (automatic) or request cancellation with tensor9 ops command cancel --commandName <name>. For the existing executing path, cancellation records Cancelling and the appliance then records Cancelled. Those state changes do not prove process termination or rollback. See Cancelling for entry-point differences.

​
Cancelling

The current cancellation entry points apply different state selection rules. The table describes the existing implementation, not a promise that every entry point has the same late-state behavior: For the existing executing path, the appliance advances Cancelling to Cancelled. This describes command-record state, not a process-interruption or rollback guarantee. Inspect the customer’s environment after cancellation of a mutating command; partial effects may remain. If you submitted many commands by accident and need to cancel them all in one shot:
cancel-bulk lists the commands it’s about to cancel and prompts for confirmation; pass --yes to skip the prompt for scripts. The initial selection includes only Submitted and CmdApproving, with other states reported as skipped. The version is refreshed before dispatch; this is not an atomic selection-and-cancel operation.

​
Batches

When you need to fan a command out across multiple customers or appliances, use the batch surface:
A batch creates one underlying ops command per appliance. The lifecycle tracks each child command independently, so different customers approving at different times is normal. batch retrieve rolls the children up into a single status summary. A single batch is capped at 50 child commands (one per appliance). For larger fleets, submit multiple batches with a small delay between them to avoid a thundering-herd against the notification path. We are working on a higher cap; let us know what your steady-state fan-out looks like.

​
Audit and forensics

Three Ed25519 signatures protect every ops command. Together they form a non-repudiation chain your customer can verify independently: The signatures are stored on the command’s audit record and survive the encrypt/decrypt cycle (the integrity signature is computed over plaintext before encryption, then preserved in the ciphertext metadata). Either you or your customer can verify the full chain on a specific command:
The action retrieves the command record + the appliance’s pinned signing public key, reconstructs the canonical signed-data for each of the three signatures, and verifies. The three checks use two different trust anchors:
  • commandApproval and outputApproval verify the customer’s signature on the approval / release manifest, using the public key embedded in the manifest itself. The output reports the signerPublicKeyFingerprint, which you (or the customer) should cross-reference against the customer-signing pubkey currently pinned on the appliance vault, which is the actual trust anchor.
  • outputIntegrity verifies the appliance’s signature over the plaintext output, against the customer’s pinned opsCmdPubKey.
Output (in the healthy case):
Any failure is surfaced as [FAIL] with a one-line reason, and the process exits non-zero. Customers running compliance audits should script this against their full ops-command history; you should run it whenever a customer reports “you ran something I didn’t approve” so the disagreement turns into a verifiable record very quickly. UNSIGNED_LEGACY records (commands authored before the signature chain was required, with no customer-signed manifest attached) are reported but do not cause a failure exit by default; pass --strict to fail on those too. For programmatic use, pass --output json. Each entry in the JSON checks array has a trustAnchor, signerPublicKeyFingerprint, signedBy, and signedAt, which let you distinguish auto-approved-by-pre-approval from manually-approved commands and archive the chain independently.