How to Fix SSH ‘X11 connection rejected because of wrong authentication’ Error on Linux

X11 forwarding is a feature of the SSH (Secure Shell) protocol that enables graphical applications found on a remote computer to be accessed over an SSH connection.

The GUI application is executed on the remote server, but the application’s window is displayed on the local computer where the SSH session was started. This allows us to use the GUI application as if it was installed on the local computer itself.

Backstory

I’ve used X11 forwarding several times in the past, but a few weeks ago, I was unable to launch GUI apps from a freshly provisioned AWS Lightsail instance. Instead of seeing the application’s window, I got this error message:

X11 connection rejected because of wrong authentication.

Below are the steps I took to fix this error.

1. Remote Server Configuration

First, connect to the remote sever via SSH normally (without enabling X11 forwarding by omitting the -X option):

ssh user@xxx.xxx.xxx.xxx

Step 1: Create a Group for X11 Forwarding

Let’s create a group called x11users on the remote server. In Step 2, we’ll configure SSH to allow X11 forwarding for all users who are in this group.

export GROUP_X11="x11users"
sudo groupadd $GROUP_X11

Next, we add our user to the newly created group.

sudo usermod -a -G $GROUP_X11 my_user

Step 2: Enable X11 Forwarding (Server-side)

We now need to configure SSH to enable X11 forwarding for the x11users group. Add the following block at the end of the /etc/ssh/sshd_config file on the remote server:

## Uncomment the line below only if you have disabled IPv6
#AddressFamily inet

Match Group x11users
	X11Forwarding yes
	X11UseLocalhost no
	X11DisplayOffset 10
	AllowTcpForwarding yes

Then, reload the SSH service to pick up the new configuration, and close the SSH session:

sudo sshd -t
sudo systemctl reload ssh
exit

2. Client (Local Computer) Configuration

Step 1: Enable X11 Forwarding (Client-side)

The following directives should be present in the /etc/ssh/ssh_config file on your client (local computer):

Host *
	ForwardAgent yes
	ForwardX11 yes
	ForwardX11Trusted yes

Just like before, the SSH service (this time locally) is reloaded with the updated configuration:

sudo systemctl reload ssh

Step 2: Connect to the SSH Server

Let’s now connect to the remote server with X11 forwarding enabled:

ssh -X user@xxx.xxx.xxx.xxx

Launching any GUI app from the SSH session should display that app’s user interface on your local computer. If you’re still experiencing X11 authentication errors, the steps below explain how to troubleshoot and resolve the error.

Troubleshooting: Check $DISPLAY variable on the SSH Server

If X11 forwarding is working properly, the $DISPLAY variable will be set to some value by default.

echo $DISPLAY

On the other hand, if the value of $DISPLAY is (null) or empty, it means the X11 proxy was not properly started. This could be because xauth is missing on the remote computer and/or the client computer. You should install xauth on both the client and server as follows:

1. On Manjaro and Arch Linux derivatives:

sudo pacman -S --needed xorg-xauth

2. On Ubuntu and Debian derivatives:

sudo apt install xauth

3. On RedHat, RHEL, and Fedora derivarives:

sudo dnf xorg-x11-xauth

Fix X11 Wrong Authentication Errors

The commands in this section are intended for the remote (SSH) server, not the local (client) computer!

Solution 1: Set $XAUTHORITY on the SSH Server

echo $XAUTHORITY

If you get an empty output, the $XAUTHORITY environment variable needs to be set on the SSH server before launching the GUI application:

export XAUTHORITY=$HOME/.Xauthority

Solution 2: Grant Permission to the New User

If you switched users during the SSH session (for example using sudo or su), then that new user (e.g user2) differs from the original user (e.g user1) that started the SSH session. Consequently, the new user (user2) does not have permission to access your X server, thereby causing the X11 forwarding to fail.

This new user (user2) can be granted permission to your X server using xhost. But first you need to switch to the original user that started the SSH session on the server.

su user1

Then, running this command on the SSH server will grant the new user (e.g user2) access to your X server:

xhost +SI:localuser:user2

You should now be able to launch GUI apps over SSH even if you switch to the new user user2.

xhost -

Solution 3: Use the Original User’s Authorization File

As a last resort, you can add the authorization entry of the original (non-root) user that started the SSH session to the current user’s /.Xauthority file on the server:

export USER_ORG="user_who_started_ssh_session"
xauth add $(xauth -f ~${USER_ORG}/.Xauthority list | tail -1)

# Footnotes

https://www.ibm.com/support/pages/x11-forwarding-ssh-connection-rejected-because-wrong-authentication

https://wiki.archlinux.org/title/Xhost

https://linux.die.net/man/1/xauth