You start a long job on a remote machine: a training, a build, a data download. Ten minutes later your Wi-Fi drops, the SSH connection dies, and when you reconnect the job is gone. Run the same job inside tmux, and it is still running, as if nothing had happened.
Most people learn the rule (“use tmux on a server”) before the reason. This post gives the reason. It first looks at how a process is attached to its parent and to a terminal, then at what happens to it when an SSH connection ends, and finally at how tmux is built to not be part of that chain: a server that owns your processes, and a client that talks to it through a Unix socket.
The post assumes that you know what a process and a signal are. It does not cover tmux key bindings or configuration.
A process is a node in a tree
How a process is created: fork, then exec
Every process on Linux is created by another one, in two steps:
fork()creates a clone of the calling process. The clone has the same code, the same memory (copied lazily, copy-on-write) and the same open files. Only the PID and the PPID differ: the clone has a new PID, and its PPID is the PID of the original.exec()replaces the code and the memory of the calling process by another program. The PID does not change.
When you type ls in a shell, this is what happens:
parent: waits"] S2["clone of the shell (PID 101, PPID 100)
child: still shell code"] E["exec('ls')"] LS["ls (PID 101, PPID 100)
same PID, new program"] W["wait() returns
exit status of ls"] S --> F F --> S1 F --> S2 S2 -- "set up redirections,
pipes, cwd, setsid..." --> E E --> LS S1 --> W LS -. "exits" .-> W classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class S2 run
The reason for two steps is the highlighted box. Between fork() and exec(), the child is still the shell’s code, so it can prepare its own environment with ordinary system calls: redirect its files (ls > out.txt), plug a pipe (a | b), change directory or user, or call setsid() to leave the session. Then exec() starts the real program, which inherits all of this without knowing it. A single “launch a program” call would have to take all these options as parameters.
The memory is isolated (the child works on its own copy), but open files and the environment are inherited on purpose. This is what makes pipes and redirections work.
The numbers that describe a process
The result of all the forks is a tree, where each process knows its parent. Three more numbers add other structures on top of it:
- PID: the identifier of the process.
- PPID: the PID of its parent.
- PGID: the process group, a set of processes that are one job: typically one shell pipeline, such as
a | b | c. A signal can be sent to the whole group at once. - SID: the session, a set of process groups that depend on the same terminal. A session has one leader, usually the shell started by a login.
You can see these numbers with ps:
ps -o pid,ppid,pgid,sid,tty,cmd
Why the tree is not enough
If everything descends from a parent, why do we need groups and sessions? Because the tree cannot answer the questions that matter for a terminal:
- The tree comes apart. When a parent dies, its children are re-attached to PID 1. “Everything that came from my shell” does not exist in the tree any more, and a daemon even detaches itself from its parent on purpose.
- The tree does not show jobs. In
a | b | c, the three processes are siblings: three children of the shell. Nothing in the tree says that they are one job, yetCtrl+Cmust reach all of them and nothing else.
So there are three structures at the same time, each one with its own question:
| Structure | Question it answers | Typical use |
|---|---|---|
| Tree (PPID) | Who created me? Who collects my exit status? | wait(), zombies, orphans |
| Process group (PGID) | Which processes form one job? | kill -- -PGID, Ctrl+C, fg / bg |
| Session (SID) | Which jobs depend on the same terminal? | SIGHUP when the terminal disappears |
The groups and sessions are nested (process ⊂ group ⊂ session), and they cut across the tree:
PID 100"] end subgraph G2["Group 200: the job a | b | c"] A["a
PID 200"] B["b
PID 201"] C["c
PID 202"] end end SH --> A SH --> B SH --> C classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class A,B,C run
With this, you can handle a subset of your processes:
kill -TERM -- -200sends the signal to the whole group 200 (note the minus sign before the PGID), and to nothing else.setpgid()moves a process to another group, andsetsid()moves it to a new session with no terminal. This is whatsetsid cmddoes, and what tmux does for each pane.
One shell is not one session
A session is created by setsid(), which is called by whoever prepares a terminal, not by each shell. So starting N shells does not give N sessions:
(typed inside the shell)"] SUB --> J1["your job"] end subgraph S2["Session 2: SSH connection #2"] direction TB SH2["login shell"] --> J2["your job"] end subgraph S3["Session 3: tmux pane, created by the tmux server"] direction TB SH3["pane shell"] --> J3["your job"] end classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class SH3,J3 run
| Situation | Sessions |
|---|---|
| You open a terminal window, or an SSH connection | One new session (the terminal program or sshd calls setsid) |
| You open N windows or N connections | N sessions |
You type bash inside an open shell | Same session: the sub-shell is just a child process |
You run a | b | c | Same session, one new process group for the pipeline |
| Each tmux pane | One session per pane, created by the tmux server |
Inside a window, your shell and the programs it starts share the same SID, and in a second window the SID is different. The SID is also the PID of the session leader: a shell that leads its session has PID = SID, and it is this process that receives the SIGHUP when the terminal disappears.
Two more notions complete the picture:
- The controlling terminal. A session can be attached to one terminal (a TTY or a pseudo-terminal, a pty). The terminal is what turns your keyboard into signals (
Ctrl+CsendsSIGINTto the foreground process group) and what tells the session that the user has gone away. - The foreground process group. Among the groups of a session, only one at a time can read from the terminal and receive its keyboard signals. The others are background jobs.
What happens when you kill a process
kill does not kill anything: it sends a signal, and the kernel or the target decides what to do with it.
| Signal | Number | Can be caught | Default effect |
|---|---|---|---|
SIGHUP | 1 | Yes | Terminate. Means “the terminal hung up” |
SIGINT | 2 | Yes | Terminate (this is Ctrl+C) |
SIGTERM | 15 | Yes | Terminate. The polite request, the default of kill |
SIGKILL | 9 | No | Terminate immediately, the process cannot react |
SIGSTOP | 19 | No | Pause (this is the base of Ctrl+Z, with SIGTSTP) |
Two consequences are often misunderstood:
Killing a parent does not kill its children. A signal is sent to a PID (or to a whole group, with kill -- -PGID), not to a subtree. If a parent dies, its children are orphaned: the kernel re-attaches them to PID 1 (or to the nearest subreaper, such as a systemd --user instance) and they keep running.
The reverse is also true: a child that ends must be collected. When a process exits, it stays in the table as a zombie until its parent calls wait() to read its exit status. If the parent died, PID 1 does this job.
So if the parent’s death does not kill the children, why do jobs usually die when a terminal closes? Because there is a second link, the one with the terminal.
What a pty is
A TTY (from teletypewriter) was a physical terminal, a keyboard and a screen wired to the computer. The kernel kept its abstraction: a terminal device that turns Ctrl+C into a signal, echoes the keys you type, and tells a session when the user has gone. A pseudo-terminal (pty) is the same thing made of software, with no hardware. Almost every terminal today is a pty: a terminal window, an SSH connection, a tmux pane.
A pty is a pair of two ends, connected inside the kernel:
| End | Held by | Role |
|---|---|---|
| master | The program that provides the terminal: a terminal window, sshd, the tmux server | Writes the keys, reads what must be displayed |
slave (/dev/pts/N) | The shell and its jobs | Looks like a real terminal. It is their controlling terminal |
The master is not a file with a name. It is a file descriptor obtained by opening /dev/ptmx, which is only a factory: each open("/dev/ptmx") creates a new pair, and its slave appears as /dev/pts/N. One master has exactly one slave, but many processes (the shell, ls, watch) can share the same slave.
One descriptor, two queues
Each end is a single descriptor that supports both read and write, and behind it the kernel keeps two separate queues, one per direction:
or network"] --> TP["terminal program
holds the MASTER
(window, sshd, tmux server)"] TP -- "write(master)" --> QA(["queue A: keys"]) QA -- "read(0)" --> SH["shell and jobs
hold the SLAVE
fd 0, 1, 2 = /dev/pts/7"] SH -- "write(1) or write(2)" --> QB(["queue B: display"]) QB -- "read(master)" --> TP TP --> SC["screen
or network"] classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class QA,QB run
Between the two ends, the kernel applies the line discipline: echo of the typed keys, \n turned into \r\n on output, Ctrl+C turned into SIGINT for the foreground process group. A plain pipe or socketpair has none of this, which is why programs such as ls or vim, which ask isatty() and set the terminal mode, need a pty.
Why two queues, and not one shared block? With one queue, the program that writes and then reads would pop its own bytes: the terminal would read back the key it has just sent, and the shell would never see it. Nothing in a queue says who wrote a byte. Two queues put the direction in the structure, and both directions can be active at the same time (a watch that refreshes while you type).
Following a command: ls
Here is the whole round trip when you type ls and press Enter, from the terminal window’s point of view:
write(master, key)"] C["3. Queue A: the shell's read(0) returns the key"] D["4. The shell echoes the key
write(slave): this is the letter you see"] E["5. The shell reads the line 'ls'
fork(), exec('ls'), wait()"] F["6. ls inherits fd 0, 1, 2 (the slave)
write(1, names)"] G["7. Queue B: the terminal program
read(master) and draws"] A --> B --> C --> D --> E --> F --> G classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class C,F run
Two details:
- The letters you see while typing are the shell’s echo, which made a round trip through the pty. They are not your keystroke displayed directly.
lsasksisatty(1). On a terminal it prints columns and colors, and into a pipe (ls | cat) it prints one name per line, because its stdout is no longer a terminal.
Who can read what
Reading the slave (cat /dev/pts/7) reads queue A, the keys meant for the shell. It does not show what the shell displays, and it steals keys, since a byte is popped by only one reader. Writing to the slave (echo hi > /dev/pts/7) displays text on that terminal, which is how wall and write work. The display is on queue B, and only the holder of the master can read it. This is the asymmetry that tmux relies on: its server holds the master of each pane, and sees everything.
When all copies of a master are closed, the kernel signals a hangup to the session on the slave side. This is the SIGHUP of the next sections.
What an SSH connection builds
When you run ssh host, the remote sshd does not give you a shell directly. It builds a small chain:
(one process per connection)"] SD -- "pty master" --> PTY(["pty"]) PTY -- "pty slave
= controlling terminal" --> SH["login shell
(session leader)"] SH --> J["your job"] classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class PTY run
sshdallocates a pty, a pair of connected endpoints. It keeps the master end and reads or writes it to relay data to your TCP connection.- It starts your login shell with the slave end as its controlling terminal, in a new session (the shell becomes the session leader).
- Everything you launch from this shell is in this session, and so attached to this terminal.
The key point is that the pty is the only thing that connects your job to you. When it disappears, the session is told.
What happens when the connection ends
The connection can end because you typed exit, closed the laptop, or because the network broke. The result is the same: after a clean close, or after a timeout (ClientAliveInterval on the server, TCP timeouts), sshd closes the pty master. The kernel then reacts, and this is the sequence:
(default action of SIGHUP)"] A --> B --> C --> D --> E --> F classDef bad fill:#F5D0D0,stroke:#C98A8A,color:#6B1A1A class F bad
Three details matter:
- The kernel hangs up the session leader, and the shell relays it. An interactive shell (bash, zsh) that receives
SIGHUPforwards it to all the jobs in its job table, including the background ones started with&, and then exits. - When the session leader exits, the foreground process group also gets
SIGHUP, from the kernel. - A job that survived the signal still has a problem. Its terminal is gone, so a later
writeto it fails (EIO, orSIGPIPE), and many programs die at their nextprint.
So the job is not killed because its parent died. It is killed because its session lost its terminal.
The usual workarounds, and their limit
| Tool | What it does | Its limit |
|---|---|---|
nohup cmd & | Starts cmd with SIGHUP ignored, and its output redirected to nohup.out | You can never get back interactively to the process: no stdin, no screen |
disown | Removes a job from the shell’s table, so the shell does not forward SIGHUP | The job is still in the old session, and it still loses its terminal. Its output is still tied to it |
setsid cmd | Starts cmd in a new session, with no controlling terminal | Same as nohup: it runs, but nothing is attached to it |
screen, tmux | Keep the process under a terminal that does not die | They are one more program to learn |
The first three make the process survive. None of them lets you come back and see it as it was: the program’s screen, its prompt, a Python REPL, a htop. This is what a terminal multiplexer adds.
What tmux does differently
The architecture of tmux is the whole trick: it is two programs, not one.
- The server is a long-running background process. It owns the sessions, the windows, the panes, and the processes running inside them.
- The client is the
tmuxcommand you type. It is short-lived: it connects to the server, shows what the server draws, and forwards your keys.
The first time you run tmux, no server exists. The client starts one, and the server daemonizes: it detaches from your terminal and from your session, and it is not a child of your shell any more. This is visible with ps: the server has no controlling terminal of the SSH connection, and its parent is PID 1 (or a user manager).
Then, for each pane, the server does what sshd did for you: it creates its own pty, and starts a shell in a new session with this pty as controlling terminal.
pty #1 (SSH)"] SH --> CL["tmux client"] INIT["PID 1"] --> SRV["tmux server
(no terminal)"] SRV --> P1["pane shell
pty #2 (owned by the server)"] P1 --> JOB["your job"] CL -. "Unix socket" .-> SRV classDef run fill:#FFEFC2,stroke:#E5D28A,color:#6B4A00 class SRV,P1,JOB run
Look at the two sides of this diagram:
- On the left, the usual chain:
sshd, a shell, the pty #1 of the SSH connection, and only the tmux client hangs from it. - On the right, a second chain that has nothing to do with SSH. Your job’s controlling terminal is the pty #2, and the master of pty #2 is held by the server, which does not go away.
When the SSH connection dies, the left side gets its SIGHUP, and the client dies. The right side never receives any signal: the pty #2 is still open, since its master is still held by the server. For the job, nothing happened.
How the client and the server talk
The link between the two is a Unix domain socket, a file-like endpoint for local inter-process communication. On this machine, with the default settings, it is:
/tmp/tmux-1000/default # /tmp/tmux-<uid>/<socket-name>
The directory is private to your user (mode 700), and it is the permission on this socket that decides who can connect to a session. A few options change it: -L name picks another socket name in the same directory (an independent tmux server), and -S path gives an explicit path, which is the way to share a session between users.
On this socket, tmux does not send raw text. It uses a small message protocol (imsg, a library from OpenBSD) where each message has a type and a payload. When you start a client, the exchange looks like this:
- Identify. The client sends who it is: its terminal type (
TERM), its working directory, its environment, its terminal features. - Hand over the terminal. The client sends the file descriptor of its own terminal to the server, using a feature of Unix sockets (
SCM_RIGHTS) that can pass an open file from one process to another. After that, the server can write to the terminal of your SSH session by itself, which is how it draws the screen. - Command. The client sends the command to execute (
attach,new-session,ls, …) and the server runs it. Manytmuxcommands are only this: a client asking the server to do something, and exiting. - Events. While attached, the client mostly waits. It reports terminal resizes (
SIGWINCH), and tells the server when to detach or exit.
What happens in each case
| Event | What happens to the client | What happens to the server and the job |
|---|---|---|
Ctrl+b d (detach) | Exits cleanly | Nothing changes |
| SSH connection lost | Receives SIGHUP, exits | Nothing changes. The server later notices that the client is gone |
tmux attach from a new SSH connection | A new client starts | The server hands the screen to it. Same panes, same programs |
| Two clients attached to the same session | Both are views of it | One session, shown twice |
exit in the last pane of the last window | The client exits | The shell exits, the session closes, and the server stops if no session remains |
kill-server, or kill on the server PID | Exits | Everything dies: the server is the parent of all the pane processes |
The last line is the real limit of tmux: the server is a single point of failure. If it is killed, so are its panes.
What tmux does not protect you from
- A reboot or a crash of the machine. The server is a normal process: it does not survive a reboot. (Plugins can restore the layout and some programs, but not the running state.)
- Your user’s processes being cleaned at logout. On systems that use
systemd-logind, the settingKillUserProcesses=yeskills everything that belongs to your login session when it ends, including a tmux server started from it. The default depends on the distribution, and many servers disable it, but if tmux does not survive a logout, this is the first thing to check. The user-level fix isloginctl enable-linger <user>, or starting the server in its own scope (systemd-run --user --scope tmux). - Out-of-memory kills. The OOM killer chooses a process, and a big job in a pane is a good candidate. tmux keeps the job alive when the network drops, not when the machine lacks memory.
- The job itself. If it reads from a network connection of its own (a database, an SSH tunnel), that connection may break independently.
The minimum to use it
tmux new -s work # start a session named "work"
# ... start your job, then press Ctrl+b then d to detach
tmux ls # list the sessions of the server
tmux attach -t work # come back (alias: tmux a -t work)
tmux attach -d -t work # come back, and detach any other client first
tmux kill-session -t work # end the session and everything in it
Summary
tmux does not “protect” a process from SIGHUP. It puts it somewhere SIGHUP never arrives: under a pty that belongs to a server that is not tied to your connection. The SSH connection becomes a window that you can open and close, and the work happens behind it.