Sandboxing Agent Runtimes with NVIDIA OpenShell

Most AI agent runtimes trust the agent. They hand it a terminal, a filesystem, maybe a network socket, and hope the model does what the prompt asked. OpenShell, released by NVIDIA this week as an open-source Rust project, takes the opposite position: the agent runs inside a sandboxed process with explicit capability grants, and the host kernel decides what it can touch. That security model differs from standard setups, and the architecture shows both clear advantages and practical boundaries.

Step 1 – Install OpenShell and verify the runtime

OpenShell is a standalone Rust binary with zero daemon dependencies. Download the release binary for your platform and put it in your executable PATH. The project builds cleanly on Linux and macOS, while Windows remains experimental under current releases.

curl -LO https://github.com/NVIDIA/OpenShell/releases/latest/download/openshell-linux-x86_64.tar.gz
tar -xzf openshell-linux-x86_64.tar.gz
sudo mv openshell /usr/local/bin/
openshell --version

Before launching an agent, run openshell audit inside an empty directory. The command inspects system capabilities and prints what syscalls the sandbox permits, what filesystem paths are writable, and what network egress rules take effect. The defaults stay conservative. Arbitrary outbound traffic is blocked, and writes outside the declared workspace fail immediately. Reviewing this output confirms the sandbox matches your expectations before running untrusted model commands.

Step 2 – Define a capability manifest

OpenShell uses a TOML manifest to declare execution boundaries. Instead of granting the agent your full login session, you specify allowed binaries, filesystem paths, and network destinations. Anything outside the declared manifest receives an immediate deny signal at the syscall layer.

# agent.toml
[agent]
name = "build-agent"
model_command = ["claude", "--no-prompt"]

[capabilities]
fs_write = ["/tmp/build", "/home/ci/artifacts"]
fs_read  = ["/home/ci/src", "/usr/local/bin"]
network  = ["github.com", "registry.npmjs.org"]
exec     = ["git", "npm", "cargo", "make"]

[limits]
max_wall_sec   = 300
max_output_kb  = 2048

The exec filter is critical. An agent permitted to spawn arbitrary processes can bypass higher-level filters by running binaries that perform raw socket connections. OpenShell traps execution requests at the kernel boundary rather than inspecting command strings after launch. Binaries not explicitly registered in the array fail with EPERM before process initialization begins.

Step 3 – Run a session and inspect the trace

Start a sandboxed agent run with structured tracing enabled so every runtime call is captured to disk:

openshell run --config agent.toml --trace /tmp/agent-trace.jsonl

The trace file records each attempted syscall, whether the sandbox allowed it, and the model turn that triggered the operation. Standard container setups associate actions only with a process ID. OpenShell links syscall events directly to the prompt turn that caused them. Filter denied events using jq:

jq 'select(.verdict == "deny")' /tmp/agent-trace.jsonl

Denied events indicate either an overly restrictive manifest or an errant agent prompt. If an agent tries to modify system binaries under /etc, that points to an architectural prompt scoping bug rather than a permission failure. Inspecting the JSON lines output helps pinpoint the exact moment the model drifted off course during the task.

Step 4 – Tighten the manifest based on trace data

After running multiple test sessions, generate a minimal policy from actual execution records. OpenShell provides a dedicated analyzer for this workflow:

openshell suggest-manifest --trace /tmp/agent-trace.jsonl --output agent-tight.toml

The analyzer discards declared paths that the agent never accessed and retains only observed active dependencies. Permissive manifests usually happen when operators add broad paths during debugging and never clean them up. The suggest subcommand keeps policies small by constructing manifests from verified activity logs.

Capability manifests enforce consequence, not intent. An agent granted read access to private keys and outbound network access can exfiltrate secrets without breaking sandbox rules. Syscall filtering and prompt constraints serve separate purposes; effective security requires both layers working together.

Step 5 – Integrate with CI and set hard failure modes

OpenShell exits with a non-zero status code whenever a capability check fails. This makes it straightforward to use as an automated quality gate inside continuous integration pipelines:

# .github/workflows/agent-run.yml (excerpt)
- name: Run agent in sandbox
  run: openshell run --config agent.toml --strict --trace /tmp/trace.jsonl
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

- name: Upload trace on failure
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: agent-trace
    path: /tmp/trace.jsonl

The --strict flag instructs OpenShell to abort execution on the first denied syscall instead of logging a warning. In continuous deployment environments, strict termination prevents partially completed runs from polluting build workspaces. Retaining the trace file via CI artifacts provides an immutable audit log for post-mortem analysis whenever an automated job fails.

Keep in mind that the current release focuses on single-node execution. If you deploy multi-agent workflows where agents communicate over inter-process buses, OpenShell does not yet isolate agent-to-agent message queues. Treat incoming agent RPC payloads as untrusted input until protocol-level filtering lands in upstream releases.

Next steps

Inspect the example manifests in the OpenShell repository before constructing your own configuration files. Run openshell audit against your current build environment to measure what system privileges your agent pipelines currently consume. If you are building security tooling, read the sandbox implementation in the src/sandbox/ directory of the codebase to understand how Linux seccomp and Landlock policies are bound to the execution process.