witr: Tracing Process Start Chains

Every production box holds a process nobody can fully explain. You meet it the hard way: a port that will not free up, a CPU spike at 03:12 you did not schedule. First instinct is ps aux | grep. That gives you a PID. It does not give you who started it, which supervisor owns it, or what breaks if you kill it. This post covers the gap between what is running and why it is running, and a tool called witr that closes the gap by tracing the start chain.

Problem

The Unix toolbox answers what with mechanical precision and why with silence. ss -ltnp reports that PID 5034 listens on port 5001. ps reports that the same PID is PM2. systemctl reports that PM2 runs as a user service. No single command reports the chain: PM2 sits under a deploy script, the script ran from a screen session, the screen session has survived three reboots, and the app restarts unattended every time the box comes back. That chain is the answer to almost every serious incident question.

The trouble compounds when supervisors nest. systemd starts Docker. Docker starts the container. The container init starts the app, and the app forks a worker pool. Kill a worker and the supervisor restarts it, so the load does not go away. Stop the wrong user unit and staging goes down with it. That is the moment people start asking each other “who owns this?” which is a documentation failure showing up at the worst possible time.

Giving a real answer means correlating ps output, the process tree, cgroup membership, container runtime state and shell history. The correlation is manual, it differs on every host, and it never gets written down.

Existing solutions

ps, top, lsof, ss, systemctl and the container runtimes each expose one slice of live state. Accurate and current, yes. Static, yes. Someone still has to stitch the slices together by hand in the middle of an incident, and the stitching is the step that never makes it into a runbook.

witr approaches the problem from the other end. It treats everything as a process question. Ports, containers, files and services all end at a PID, and once witr finds the PID it walks the ancestry: what started it, what keeps it alive, plus the context it runs in. The output is a chain, not another table of snapshots.

brew install witr
witr --port 5000 --tree

That prints the full lineage, roughly this shape: systemd (pid 1) -> pm2 (pid 5034) -> node (pid 14233), with working directory, git repo, sockets and source supervisor attached. Same lookup works for PIDs, container names and open files, and the container paths resolve Docker, Podman, nerdctl, crictl, Incus, LXC, LXD and FreeBSD jails back to host processes. That matters on boxes where the container runtime itself is the thing under investigation.

They show what is running, but leave you to infer why, by manually correlating outputs across tools.

The project README states the gap in one line. The TUI mode splits into processes, ports, containers and file locks, with the ancestry tree in a side panel, and you can send signals or renice from the same screen. Scripts get JSON output, a one-line chain comes via --short, and the full tree via --tree. I also like what the tool does not do: it reads /proc, cgroups and runtime metadata, the same sources as every other utility, then orders them into a chain and prints it in one pass. No hidden state, no reimplementation of the kernel.

Future with AI

The chain witr prints is exactly the structured input an assistant needs, and most current assistants never see it. They answer from conversation history or a knowledge base, neither of which knows that PID 5034 was forked by PM2 at 11:42 inside the /opt/apps/expense-manager directory. Hand an agent the live chain and it can answer the questions that take a human twenty minutes of tab switching: who owns this port, does it survive a reboot, what restarts it when it dies.

A concrete version: why did postgres restart at 03:12. Feed the agent witr JSON, a journald window around the restart and the config diff from the last deploy. That turns a guess into a query. The acceptance test is replay: the agent proposes the kill order and the rollback, and a human checks both against the same output. If the answer does not reproduce, the agent is wrong.

The boundary stays where it belongs. Provenance records who started what and when. It cannot say whether starting it was a mistake. Useful agents will stay inside that line: present the chain, run the verification, leave the judgment to whoever owns the outage.

My expectation, stated plainly: runbook sections about tracing a stray process get replaced by assistants holding a live chain plus journald access. The tracing half is mechanical, so the machine takes it. The judgment half is not, so it stays with the operator. The enabler is not a smarter model. It is clean, machine-readable provenance delivered at query time.