Introduction
We use the Secure Shell Protocol (SSH) so often in the modern IT landscape, that it is often hard to avoid its ubiquity. Logging into your home machine from abroad or a supercomputer in a massive data-center from your home? SSH. Pushing to a Git remote? SSH. Forwarding a port with a remote machine? SSH. Instituting network, site, and organization-wide policies using tools like Ansible? SSH. Accessing a graphical X11 application remotely? SSH. It’s very convenient to setup and use, and the additional ubiquity that public-key authentication has brought is excellent.
But, how secure is this seemingly universal remote access protocol? Pretty secure as it turns out, very secure in fact. The many SSH RFCs and implementations available today have been excellent examples of good, secure, and long-lasting protocol design. However, this excellent protocol isn’t without its fair share of vulnerabilities, all of which can be solved with very simple modifications to the server’s configuration file. Those very simple fixes and additions are the focus of this post, with the eventually hope that you will follow through and hopefully make your own configurations a bit more secure, but given you’re reading this you probably already have. You may work in a setting where certain ciphers and settings must be preserved for no reasons other than compatibility with legacy software. Obviously those goals take priority over the somewhat minimal security improvements gained by hardening cipher support from the already (somewhat) reasonable default. That said, if you’re not encumbered by legacy applications, you should probably consider modifying the default configuration, assuming you haven’t done so already.
Terrapin and Emerging Threats
One of the many very useful features of SSH includes protection against man-in-the-middle (MiTM) attacks. However, a novel method defeating some of these protections was recently discovered and publicly disclosed, known collectively as Terrapin. The Terrapin attack is a attack against the SSH protocol’s method of prefix truncation. It lets an attacker downgrade secure signature algorithms and disable security features designed to prevent keystroke timing attacks in SSH. This could potentially open the door to nastier MiTM attacks, however the attacker would need to be able to intercept the connection first. An additional caveat is that the bug specifically requires that the session be protected by either ChaCha20-Poly1305 or CBC with Encrypt-then-MAC encryption for the attack to succeed.
The vulnerability is particularly concerning due to its broad applicability. Both ChaCha20-Poly1305 and CBC with Encrypt-then-MAC encryption modes are used fairly extensibility. Some researchers have found that about 77% of internet-accessible SSH servers support at least one of these modes1, thus making it vulnerable to Terrapin.
While the Terrapin attack is a practical threat and vulnerability, it requires MITM capabilities along with specific conditions to be effective. In other words, your network must already have been compromised so that an attacker can intercept and modify the connection’s traffic before Terrapin can cause issues.
In light of this discovery, many developers responsible for SSH implementations have been contacted and almost all have updated their SSH implementations to support an optional strict key exchange. This optional countermeasure introduces sequence number resets, removing an attacker’s ability to inject packets during the initial, unencrypted handshake1.
That said, you should probably check your server’s accepted ciphers and see what you find. You never really know what’s available until you check first-hand.
Additional undiscovered bugs that could pose an emerging threat to your systems could exist in currently available ciphers, therefore reducing the overall number of accepted ciphers and other methods of attack surface reduction may be important to consider in addition to Terrapin mitigation.
Tinfoil-hattery and Allegations of NSA Collusion
There have been a few suspicions, particularly among the more paranoid, that that the National Security Agency (NSA) could have the ability to decrypt certain SSH sessions along with the suspicion that they may have backdoored the NIST P-curves used in elliptic-curve cryptography. These curves are used in algorithms such as ecdsa-sha2-nistp256, ecdh-sha2-nistp521, ecdh-sha2-nistp384, and ecdh-sha2-nistp256.
Whilst there’s some very truthful elements to this theory, involving some very complicated math, it’s more complicated than backdoored vs not-backdoored. A project called SafeCurves demonstrated fairly well that there’s been some rather confusing and concerning elements involved with NIST P-curves, most namely unexplained (and unexplainable) seed values used, along with some graduate-level applied mathematics that I have zero chance of explaining or understanding all that well.
Much of these concerns have been exacerbated from historic incidents of malfeasance regarding NIST cryptographic standards combined with the aforementioned unexplained seeds. Most particularly, Dual_EC_DRBG, a supposedly secure pseudorandom number generator proposed by NIST (and allegedly the NSA). Publicly disclosed leaks from Snowden, along with major scrutiny from expert cryptographers led to it’s eventual phasing-out. The incidents surrounding Dual_EC_DRBG are quite a story in their own right, and given my limited cryptographic knowledge, I likely wouldn’t do it much justice if I fully covered it here and now.
However, this theory regarding SSH ciphers and P-curves has a few confusing elements. The US government would and could never use such a curve for their own sensitive data, because they’d expect one of the NSA’s unfriendly foreign counterparts to crack it rather quickly.
As a solution, it’s always possible to disable the offending ciphers and keep the known trusted ones. If you’re using a modern client on just about any operating system, you’ll have no issue using a non-P-curve algo.
It’s worth noting that all this stuff was approved for and used for, protecting highly sensitive and classified communications in the US government. You’d think that NIST/NSA would be smarter than to recommend a weak cipher for their own use, but tinfoil-hatters are going to tinfoil-hat nonetheless. However, the cryptographers are right to be suspicious and wary of unexplained constants, so it doesn’t hurt to disable a potentially-negative feature that you don’t require. Attack surface reduction and all that (more on that later).
Traditional Attacks Against Passwords
Before now, I have yet to make a distinction between SSH sessions using passwords as authentication versus public-key authenticated sessions. SSH sessions are significantly more secure, and easier to use and maintain if public-keys are the sole factor used to authenticate. They’re easy to setup, easy to use, and provide an additional factor of authentication if used properly.
For starters, it’s no secret that passwords have and can be broken by attackers. Passwords are a common element of weakness among attacks against both individuals and large corporations. Billions of leaked passwords have been publicly exposed on the internet, and many millions more will join their ranks in years to come. Password reuse is fairly common among end-users, so attacks like credential stuffing are fairly successful among ransomware gangs and internet fraudsters. Additionally, given the existing corpus of commonly used passwords on the internet and the wide adoption of computers and internet accounts among non-technical specialties, it’s not too difficult to guess that a few individuals out of any given organization will use a password that’s widely known to the world as being weak. Password spraying and similar techniques prey on this weakness, and do so without triggering account lockouts and WAF restrictions.
SSH, whilst very secure in protocol and implementation, is only as secure as the weakest link, the humans involved. Given the previously mentioned attacks against users and their passwords, it would make some sense to avoid those vulnerabilities all together by avoiding passwords entirely. That said, long, unique, and securely-stored passwords will pose no immediate threat to an organizations security posture, nor are a major concern to vulnerability reports or internal security teams. However, if a potential threat can be avoided entirely, that’d certainly be preferred.
Public keys also allow for simpler SSH access and are far more convenient to generate, change or deploy. It’s really a win-win for security and simplicity all-in-all.
Fixing the Aforementioned Flaws
For starters, a good idea might be to check your existing SSH servers with a general vulnerability assessment tool or a specialized SSH auditing tool like ssh-audit2. It’s not much of an undertaking, as SSH is fairly simplistic and secure-by-default. From there, you’ll probably get a list of ciphers that are either discouraged/unpreferred or straight-up deprecated/insecure depending on how old the server/software is. From there, you can always disable a few or add a few settings manually to whatever servers may concern you.
However, manually editing configurations to meet compliance really isn’t the greatest idea, especially if you manage many servers at once. First, make a clean backup of your existing configuration, and store it in a safe place. Then, I’d recommend putting a copy of that existing configuration file under version control, then gradually modifying settings and testing them on a test machine you have physical access to.
If you want to take a more streamlined approach, I’d recommend taking inspiration from Mozilla’s official configuration, and modify it to fit your needs from there. It’s pretty sane and secure, and fairly compatible with older versions. That’s how I ended up with my configuration, of course with some minor tweaks to fit my use case. Primarily, disabling all the features I didn’t need (X11 forwarding, TCP forwarding, et cetera), along with disabling P-curves (don’t need them per-se, so why not mitigate the risk, even if it’s small?). A few crashed daemons later, I ended up with the following:
Include /etc/ssh/sshd_config.d/*.conf
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
KexAlgorithms [email protected],diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
HostKeyAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256
MACs [email protected],[email protected],[email protected]
LogLevel VERBOSE
MaxAuthTries 5
MaxSessions 8
PubkeyAuthentication yes
UsePAM yes
PermitRootLogin no
PasswordAuthentication no
PermitEmptyPasswords no
KbdInteractiveAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no
PrintMotd no
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
# override default of no subsystems
Subsystem sftp /usr/lib/openssh/sftp-server
Once you’ve found a good balance of security and usability that works for you, you can roll the changes out to your servers. You can do this manually using SCP or Rsync, but automating the process using configuration management tools like Chef or Ansible is certainly a lot easier and will save you time in the long-run. I personally like to use Ansible for tasks such as these, so here’s my example OpenSSH configuration playbook:
- hosts: linux
remote_user: bhays
become: true
become_user: root
tasks:
- name: Update/install OpenSSH
apt:
name: openssh-server
state: latest
update_cache: yes
- name: Add 'bhays' user
user:
name: bhays
groups: sudo,adm
append: yes
shell: /bin/bash
comment: Benjamin Hays
- name: Update/install Sudo
apt:
name: sudo
state: latest
- name: Ensure .ssh user folder exists
file:
path: "/home/bhays/.ssh/"
state: directory
- name: Copy public key
copy:
src: ~/.ssh/authorized_keys
dest: /home/bhays/.ssh/authorized_keys
- name: Copy Secure Configuration File
copy:
src: ../Configs/sshd_config
dest: /etc/ssh/sshd_config
- name: Restart OpenSSH
systemd_service:
name: sshd
state: restarted
You can find more complete versions of these playbooks, configurations, and other assorted scripts/utilities at: https://github.com/benjaminhays/homelab-automation/
P.S. Sorry for the long delay in articles being posted, curricular responsibilities gave me little time to work on my blog. Should hopefully be more regular in coming weeks/months