← all posts

Today I learned · SSH

SSH never sends your key. It sends a signature only your key could make.

When you log in to a server with an SSH key, the private key stays on your computer. The server keeps a list of public keys, and you prove you hold the matching private key by signing this one connection. This guide follows a login from the first knock to the shell, then covers the files and rules that decide who gets in.

Press Play to watch a key login in five steps.

  1. The laptop holds a private key. It never leaves the laptop.
  2. The server holds a copy of the matching public key, on the list for your account.
  3. When you connect, both sides work out a session ID for this one connection. The laptop signs it with the private key.
  4. Only the signature crosses the network.
  5. The server checks the signature with the public key. It fits, so you're in.

Every figure has its own controls: Play, Back, Next and Restart. When a figure has focus, ← → step through it and Space plays or pauses. Nothing plays until you press Play. The machines, names, addresses and numbers on this page are made up.

The twelve words this guide uses

Each one has its own colour or shape in the figures, and keeps it all the way down.

Private key

The secret half of a key pair. It stays on the computer that made it and is never sent anywhere.

Public key

The shareable half. It can check signatures made by its private key, but it can't make them.

Signature

A short proof made with a private key over some data. The matching public key checks it.

authorized_keys

A file on the server: the public keys allowed into one account, one per line.

Host key

The server's own key pair. It proves the server is the one you meant to reach.

known_hosts

A file on your computer: the host keys of servers you have already decided to trust.

ssh and sshd

ssh is the client you type into. sshd is the server program that answers, usually on port 22.

Session ID

A number both sides work out when a connection opens. Every connection gets a different one.

Fingerprint

A short ID for a key, written SHA256:…. The same fingerprint means the same key.

sshd_config

The server's rules: which accounts may log in, and with which methods.

~/.ssh/config

Your own address book: short names for servers, with the user and the key to use.

ssh-agent

A helper that holds your unlocked private key in memory and makes signatures on request.

Part 1

How a login works

Seven chapters, from the two programs involved to the full handshake.

1

Two programs, one port

SSH is two programs: ssh, on the computer you sit at, and sshd, on the computer you log in to.

You type ssh me@server on your laptop. me is the account on the server and server is its name or address. The laptop's ssh program connects to the server. There, a program called sshd has been waiting for connections, by default on TCP port 22.

sshd decides whether to let you in. If it does, you get a shell: a prompt on the server. The commands you type now run on the server, and their output comes back to your laptop, encrypted the whole way.

Press Play to follow one connection.

  1. You type ssh me@server on the laptop. ssh is the client program.
  2. ssh opens a connection across the network to the server, on port 22 unless you say otherwise.
  3. On the server, sshd is listening on that port. It answers, and it decides whether you get in.
  4. If it lets you in, you get a shell: a prompt where your commands run on the server, not on the laptop.
2

Reaching the port is not getting in

Anyone who can reach port 22 can knock; only proof of identity gets them in.

A server's SSH port can be reachable from many places. People on the same home network can reach it. So can friends on a VPN you share, and, for a server on the internet, the whole internet. Automated scanners try every address they find, all day.

So the network is not the lock. The lock is the login itself, and that is what the rest of this guide is about.

Press Play to see who gets in.

  1. Four visitors can reach the server's port 22. Reaching the port only lets them ask.
  2. A friend's laptop on the same VPN reaches the port too. It has no key on the server's list, so sshd refuses it.
  3. A scanner on the internet knocks on every address it finds. It has no key either: refused.
  4. A guesser tries common passwords. With password login turned off, there is nothing to guess: refused.
  5. Your laptop offers a key that is on the list. It is the only visitor let in.
3

A key pair: one secret, one copy for everyone

ssh-keygen makes two files: a private key that never leaves your computer, and a public key you can hand to anyone.

The two halves are made together and only work together. The public key can be worked out from the private key, but never the other way round. That one-way link is what makes it safe to give the public key away.

Press Play to make a key pair.

  1. You run ssh-keygen once, on the laptop.
  2. It writes the private key to ~/.ssh/id_ed25519. That file stays on the laptop.
  3. It writes the public key beside it, with the same name plus .pub.
  4. You copy only the public key to the computers that should let the laptop in.

What making one looks like

Since OpenSSH 9.5, ssh-keygen makes an Ed25519 key by default, so -t ed25519 only matters on older versions. -C sets the comment, a label that ends up at the end of the public key.

$ ssh-keygen -t ed25519 -C "me@laptop"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/me/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/me/.ssh/id_ed25519
Your public key has been saved in /home/me/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:q3Xv…(43 characters)… me@laptop

Reading a public key

A public key file is one line of text with three parts.

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG…(68 characters) me@laptop
  1. The key type. Ed25519 is the modern default: short and fast. RSA keys (ssh-rsa) still work everywhere.
  2. The key itself, in base64. Every Ed25519 public key starts with the same characters, because they spell out the type again. An Ed25519 key body is always 68 characters.
  3. The comment. Free text, ignored by sshd. Use it as a name tag, so you can tell later which device a key belongs to.
  • The private key must be readable by you alone. If the file grants any permission to group or others (anything but 600 or 400), ssh prints WARNING: UNPROTECTED PRIVATE KEY FILE! and ignores the key.
  • Make one key pair per device. If a device is lost, you then remove exactly one key and nothing else.
4

Signing: proof without showing the key

A private key makes signatures; the matching public key can check them but can never make them.

A signature is a short number computed from two inputs: some data, and the private key. Anyone with the public key can check that the signature fits that data. Nobody without the private key can produce one that fits.

This is how the server knows your laptop holds the private key, without the key ever being sent.

Press Play to sign and check.

  1. The laptop takes a piece of data. In a login, it is this connection's session ID.
  2. It computes a signature from the data and its private key: a short number only this private key could make for this exact data.
  3. The server checks the signature and the data with the laptop's public key. They fit, so the laptop must hold the private key.
  4. Checked with a different public key, the same signature doesn't fit.
  5. Checked against different data, it doesn't fit either. A signature made for one connection is useless on any other.

The values are illustrative and shortened. A real session ID is a hash of 32 bytes or more, and an Ed25519 signature is 64 bytes.

5

authorized_keys: the server's list for one account

A key may log in to an account when its public key is a line in that account's ~/.ssh/authorized_keys.

Each account on the server has its own list, in its own home directory. For the account me, that is /home/me/.ssh/authorized_keys. Giving a device access means adding its public key to that list as one more line.

Press Play to add a key and log in with it.

  1. On the server, the account me has a list with two lines: the desktop's public key and the phone's.
  2. You add the laptop's public key as a third line.
  3. The laptop connects and offers its key. sshd compares it with line 1: no match.
  4. Line 2: no match either.
  5. Line 3 matches. One matching line is enough, as long as no other server rule forbids the login.

Adding a line

Most systems ship ssh-copy-id, which logs in once (with whatever method still works) and appends the key for you. By hand, it is one line on the server:

# from the laptop, while some other way of logging in still works
ssh-copy-id -i ~/.ssh/id_ed25519.pub me@server

# or, on the server, paste the .pub line in
echo 'ssh-ed25519 AAAAC3Nz… me@laptop' >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys

A line can carry rules of its own

In front of the key, a line may hold options that limit what that one key can do.

ssh-ed25519 AAAAC3Nz…k2Q me@desktop
ssh-ed25519 AAAAC3Nz…7Fw me@phone
from="192.0.2.0/24",restrict ssh-ed25519 AAAAC3Nz…Xb7 me@laptop
  1. Options, optional, separated by commas with no spaces. from= accepts this key only from those addresses. restrict switches off port, agent and X11 forwarding and the terminal, for keys that only run commands. Others include command= and expiry-time=.
  2. Type and key, exactly as in the .pub file.
  3. The comment. It is how you'll find this line again when the laptop is retired.

Access goes one way

The laptop's key on the server's list lets the laptop into the server. It does not let the server into the laptop. For that, the server's public key has to go on the laptop's own list. Every direction is a separate line.

  • sshd checks the list only if the file, ~/.ssh and the home directory can't be written by other users (StrictModes, on by default). A list that others could edit is ignored.
  • Blank lines and lines starting with # are ignored, so you can annotate the list.
6

The server proves who it is first

Before you prove who you are, the server proves who it is with its host key, and your known_hosts file remembers that key.

Every server running sshd has its own key pair, the host key, made when SSH was installed. Without a check on it, an attacker in the middle of the network could pretend to be your server. Your laptop would then talk to the attacker, not the server.

So ssh remembers each server's host key in ~/.ssh/known_hosts and compares on every visit. People call the first-visit approach "trust on first use": you confirm a key once, and after that any change is treated as a warning sign.

Press Play to visit a server three times.

  1. First visit: the server shows its host key. Your known_hosts has no entry for it, so ssh shows you the key's fingerprint and asks.
  2. You compare the fingerprint with one you got another way, and type yes. ssh saves the key in ~/.ssh/known_hosts.
  3. Next visit: the server shows the same host key. It matches the saved one, so ssh goes on without asking.
  4. One day the server shows a different host key.
  5. It doesn't match the saved one. ssh prints a loud warning and refuses to connect, because something may be pretending to be the server.

The fingerprints are made up and shortened.

What you'll see

The first visit asks. The address is from a range reserved for examples.

The authenticity of host 'server (192.0.2.10)' can't be established.
ED25519 key fingerprint is SHA256:Ab3f…
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'server' (ED25519) to the list of known hosts.

A changed key stops you cold:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
…
Host key verification failed.
  • To check a fingerprint properly, read it on the server itself (at its console, or from your hosting provider's panel): ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub.
  • A server that was reinstalled gets new host keys, so the warning is expected. Once you've confirmed that, remove the old entry with ssh-keygen -R server and connect again.
  • StrictHostKeyChecking sets how ssh treats new keys. ask, the default, asks. accept-new saves new keys silently. Every setting except no refuses a changed key.
7

One login, message by message

A key login runs in a fixed order: connect, agree on secret keys while the server proves itself, then sign the session ID.

Everything from the earlier chapters happens in about a second, in this order. Watch the yellow private key: it never moves off the laptop.

Press Play to watch the ten messages of a login.

  1. The laptop opens a TCP connection to the server's port 22.
  2. Both sides announce their SSH version and the algorithms they support, and pick the best ones they share.
  3. Key exchange begins. The laptop sends a public value it made up just for this connection.
  4. The server answers with its own one-time value, its host key, and a signature made with the host key over the session ID. From the two one-time values, both sides now compute the same shared secret, which someone watching the network can't work out. The session ID is a hash of this exchange.
  5. The laptop checks that the host key is the one in known_hosts, and that the signature fits. Now it knows it's talking to the real server.
  6. Both sides switch on encryption with keys made from the shared secret. Every message from here on is encrypted, including your login.
  7. The laptop asks whether the account me may log in with this public key. It sends the public key only.
  8. sshd checks its rules for me, then finds the key on me's authorized_keys list, and answers: yes, prove you hold the private key.
  9. The laptop signs the session ID together with this login request, using its private key, and sends the signature.
  10. sshd checks the signature with the public key from the list. It fits, so the login succeeds and you get a shell.

Why each detail matters

  • The server proves itself before you say anything secret. If the host key check fails, ssh stops before your login starts.
  • The session ID is inside your signature. Someone who recorded one login can't replay its signature: the next connection has a different session ID.
  • The "may I?" question is optional. It lets ssh skip keys the server won't accept, without spending a signature. The client may also send the signature straight away.
  • The key exchange method has moved on. OpenSSH 10.0 and later use a hybrid post-quantum method (mlkem768x25519-sha256) by default, with Curve25519 as the fallback. The shape of the handshake is the same.

See it on your own machine

Run ssh -v server and the debug lines follow this order. Here is a shortened excerpt:

debug1: Connecting to server [192.0.2.10] port 22.
debug1: Local version string SSH-2.0-OpenSSH_…
debug1: Server host key: ssh-ed25519 SHA256:Ab3f…
debug1: Host 'server' is known and matches the ED25519 host key.
debug1: Offering public key: /home/me/.ssh/id_ed25519 ED25519 SHA256:q3Xv…
debug1: Server accepts key: /home/me/.ssh/id_ed25519 ED25519 SHA256:q3Xv…
Authenticated to server ([192.0.2.10]:22) using "publickey".

Part 2

Setting it up well

Five chapters on the files and rules you control, and what to do when something changes.

8

The server's rules: sshd_config

sshd_config decides who may even try: which accounts, which methods, and whether root can log in.

A key on the list is necessary, not enough. sshd applies its own rules first. Three of them turn "a key works" into "only keys work, and only for the accounts you meant".

Press Play to run four login attempts.

  1. Four login attempts reach sshd. Each one names an account and offers a way to prove itself.
  2. root is refused, even with a valid key: PermitRootLogin no.
  3. me with a password is refused: PasswordAuthentication no and KbdInteractiveAuthentication no turn passwords off.
  4. guest is refused before any key is looked at: AllowUsers me names the only account that may log in.
  5. me with a key passes every rule. Only now does sshd look at me's authorized_keys.

The rules, in a file of their own

Most systems read extra files from /etc/ssh/sshd_config.d/. Put your rules there instead of editing the main file.

# /etc/ssh/sshd_config.d/10-keys-only.conf
AllowUsers me
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
  1. Only these accounts. Everyone else is refused before keys are checked. It takes several names, and patterns like me@192.0.2.*.
  2. Never root. The default, prohibit-password, already blocks root passwords; no also blocks root keys. Log in as yourself and use sudo.
  3. Keys only. Both lines are needed: each is a separate way of asking for a password. Both default to yes upstream.

The first value wins

For each setting, sshd uses the first value it reads and ignores later ones. Many distributions put Include /etc/ssh/sshd_config.d/*.conf at the top of the main file, and the files are read in name order. So a drop-in beats the main file, and 10-… beats 50-…. Cloud images often ship a 50-cloud-init.conf that turns passwords back on; a 10- file wins over it. AllowUsers is an exception: every line adds names to the list.

Change rules without locking yourself out

# 1. keep your current session open the whole time
sudo sshd -t                          # 2. check the files for mistakes
sudo sshd -T | grep -Ei 'allowusers|permitroot|passwordauth|kbdinteractive'
                                      # 3. see the values that actually won
sudo systemctl reload ssh             # 4. apply (the service is called sshd on some systems)
# 5. test from a second terminal before closing the first
  • A reload makes sshd re-read its rules for new connections. Connections already open keep running, because each one has its own process.
  • Ubuntu starts sshd through socket activation. After changing Port or ListenAddress, a reload is not enough: run sudo systemctl daemon-reload, then sudo systemctl restart ssh.socket.
  • On a Mac, sshd is "Remote Login": System Settings, General, Sharing. Its ⓘ button has "Allow access for", where "Only these users" works like AllowUsers. Recent macOS reads /etc/ssh/sshd_config.d/ too.
9

Your address book: ~/.ssh/config

~/.ssh/config turns a short name into full connection details, so ssh server is all you type.

Without it you type ssh -i ~/.ssh/id_ed25519 me@server.example.net every time. With an entry, ssh server does the same. Give every one of your machines the same entries, and the same short names work from anywhere.

Press Play to look up one name.

  1. You type ssh server. ssh reads ~/.ssh/config from the top, looking for Host lines that match the name server.
  2. Host server matches. Its lines fill in the real address, the account, the key file and IdentitiesOnly.
  3. Host desktop doesn't match the name, so ssh skips it.
  4. Host * matches every name. It can only fill settings that are still empty, because the first value found always wins.
  5. So specific entries go at the top and Host * goes last. Put it first and its values would win for every server.
Host server srv
  HostName server.example.net
  User me
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes

Host *
  ServerAliveInterval 60
  1. The names you can type. Several aliases may share one entry; patterns like *.lan work too.
  2. Where to connect: a DNS name, a VPN name or an address.
  3. Which private key to use for this server.
  4. Offer only that key. Without it, ssh also offers every key in your agent, one by one.

"Too many authentication failures"

Each key the server rejects counts against its MaxAuthTries limit, 6 by default. With many keys in your agent, the right one may come too late and the server hangs up. IdentitiesOnly yes fixes it by offering only the right key. Servers on OpenSSH 10.6 and later no longer count the "may I?" questions, so this happens less there.

10

Protect the key itself: passphrase and agent

A passphrase encrypts the private key file; ssh-agent unlocks it once and signs for you without handing it out.

Everything so far assumes nobody else has your private key file. A passphrase protects it if the file is copied: without the passphrase, the copy is useless. The cost is typing the passphrase, which is where ssh-agent comes in.

Press Play to unlock once and log in three times.

  1. With a passphrase, the private key file on disk is encrypted. Someone who copies the file can't use it.
  2. You type the passphrase once, when you add the key to ssh-agent with ssh-add.
  3. ssh-agent keeps the unlocked key in memory. It never hands the key itself to anyone.
  4. Each login asks the agent for a signature. The agent makes it, and ssh sends it on.
  5. So you type the passphrase once per session, not once per login.
ssh-keygen -p -f ~/.ssh/id_ed25519     # add or change the passphrase
ssh-add ~/.ssh/id_ed25519              # unlock it into the agent
ssh-add -l                             # list the keys the agent holds

# macOS: keep the passphrase in the Keychain (in ~/.ssh/config)
Host *
  AddKeysToAgent yes
  UseKeychain yes
  • There is no way to recover a lost passphrase. Make a new key pair instead.
  • On macOS, the Keychain stores the passphrase, not the key. ssh-add --apple-use-keychain saves it there.

Be careful with agent forwarding

ForwardAgent yes lets a server you're logged in to use your agent for further logins. Anyone with root on that server can then ask your agent for signatures while you're connected. They can't copy the key, but they can use it. To reach a server through another one, prefer ssh -J jumphost server (ProxyJump), which keeps your keys on your own machine.

11

Draw your access as arrows

Each arrow from A to B is one line on B's list: A's public key, in the account A logs in to.

With a few machines, access becomes a small map. Drawing it makes mistakes easy to spot: an arrow you didn't mean, or one that's missing.

Press Play to draw the arrows one machine at a time.

  1. Three machines run sshd: the laptop, the desktop and the server. The phone has an SSH app but no sshd.
  2. The laptop's public key is on the desktop's list and on the server's list: two arrows out of the laptop.
  3. The desktop's public key is on the laptop's list and on the server's list.
  4. The server's public key is on the laptop's list and on the desktop's list.
  5. The phone's public key is on the server's list only.
  6. No arrow points at the phone: without sshd, nothing can log in to it. Seven arrows, seven lines, each on the list at an arrow's tip.

The same map as a table. Read a row as "this machine's key is on these lists".

MachineIts private keyIts public key is onRuns sshd
laptop~/.ssh/id_ed25519desktop, serveryes
desktop~/.ssh/id_ed25519laptop, serveryes
server~/.ssh/id_ed25519laptop, desktopyes
phoneinside its SSH appserverno
12

When a device is added, lost or replaced

Revoking a device means deleting its line from every list it is on, which is easy only if every key has a name tag.

Because each arrow is one line, every change to your access is a change to some lines. The comment at the end of each line is what lets you find the right one.

Press Play to revoke a lost laptop.

  1. The laptop is lost. Its public key is a line on two lists: the server's and the desktop's.
  2. Find the lines by their comment, me@laptop.
  3. Delete the line on the server. The laptop's key no longer opens the server.
  4. Delete it on the desktop too. Now the lost laptop opens nothing.

The recipes

# see what a list holds: one line per key, ending in its comment
ssh-keygen -lf ~/.ssh/authorized_keys

# revoke a device: keep every line except the ones with its comment
grep -v 'me@laptop' ~/.ssh/authorized_keys > ~/.ssh/ak.new \
  && chmod 600 ~/.ssh/ak.new && mv ~/.ssh/ak.new ~/.ssh/authorized_keys

# add a device: make its pair on the device, then copy the .pub line
ssh-keygen -C "me@new-laptop"
ssh-copy-id -i ~/.ssh/id_ed25519.pub me@server
  • Replacing a key is adding the new line, testing it, then deleting the old line. Never the other way round.
  • Check every account. A key may also sit in another account's list, root's included.
  • Keep a way in that isn't SSH. A physical console, or your hosting provider's web console, fixes a list or a rule that locked you out.
  • For many machines, deleting lines by hand stops scaling. sshd's RevokedKeys file refuses listed keys everywhere it is configured, and SSH certificates (TrustedUserCAKeys) replace the lists with a single trusted signer.
✓

What to do with this