A few weeks ago I published a how to guide to configure Apache HTTP as a reverse proxy. On that ocasion I was following what the average guide on the internet does on Linux. A front end server with Apache HTTP on calls a backend server where the real site is sitting. Many backend calls through a proxy are still performed via plain HTTP. When considering the security landscape, the variety and number of assets, add the vague control of those, on any enterprise of medium to big size (even some small) a different approach should be considered as of 2021. And that is a with a TLS connection with the backend.
If you find the articles in Adminbyaccident.com useful to you, please consider making a donation.
Use this link to get $200 credit at DigitalOcean and support Adminbyaccident.com costs.
Get $100 credit for free at Vultr using this link and support Adminbyaccident.com costs.
Mind Vultr supports FreeBSD on their VPS offer.
But, is there any real need to encrypt the connection from the frontal server (the one with the proxy) to the backend one? What advantage does anyone gain in doing that when this connection is made on the inbounds side of the network? This may slow down your web server responses by a really small margin, so little it can perfectly be discarded. What is the security advantage to encrypt traffic on the inward network?
Well… let’s imagine an attacker is already sitting on your LAN. Sniffing traffic on that connection is not that complicated. You may answer that is impossible to do in a network you know or you literally work on. But, have you taken in consideration what a rogue employee may do with such condition? Is that LAN really isolated from the rest of the employee segments of that private network? Are all the engineers in your organization happy to work there? What could happen if one of them wanted to capture some traffic on some special web server and sell such data to, let’s say, any of the organization’s rivals? What if you have a proxy server here on this datacenter and the backend is sitting somewhere else? Are you going to connect in plain text? Really?
No, you won’t. You will configure Apache with a TLS reverse proxy backend so that connection is made safely, in a way that even if someone is able to sniff the traffic, they will not be able to look at the content.
Pre-requisites
Two FreeBSD boxes (or jails) one acting as a fronted web server and the other as the backend.
Those boxes configured following this guide I wrote a few weeks ago.
Step 1. Enable TLS on the backend
The default install of Apache HTTP on FreeBSD has a dedicated default configuration for TLS connectivity. To enable it one must uncomment a couple of statements on the main configuration file named ‘httpd.conf’ that sits on /usr/local/etc/apache24. One will enable the SSL module and the other one triggers the dedicated configuration to be used.
To enable the module just look for the ssl_module entry and uncomment that line.
Before:
root@backend:/usr/local/etc/apache24 # grep -n 'mod_ssl.so' httpd.conf
148:#LoadModule ssl_module libexec/apache24/mod_ssl.so
root@backend:/usr/local/etc/apache24 #
After:
root@backend:/usr/local/etc/apache24 # grep -n 'mod_ssl.so' httpd.conf
148:LoadModule ssl_module libexec/apache24/mod_ssl.so
root@backend:/usr/local/etc/apache24 #
This next command does the same trick and it’s quicker than using grep and editing the file manually. Run it as root or use sudo.
sed -i -e '/mod_ssl.so/s/#LoadModule/LoadModule/' /usr/local/etc/apache24/httpd.conf
Now the module is enabled it’s time to give it a configuration to use. The entry sits on the line 534.
root@backend:/usr/local/etc/apache24 # grep -n 'httpd-ssl.conf' httpd.conf
534:#Include etc/apache24/extra/httpd-ssl.conf
root@backend:/usr/local/etc/apache24 #
Just uncomment that line to get it working. It should read as follows.
root@backend:/usr/local/etc/apache24 # grep -n 'httpd-ssl.conf' httpd.conf
534: Include etc/apache24/extra/httpd-ssl.conf
root@backend:/usr/local/etc/apache24 #
Alternatively this one line does it:
sed -i -e '/httpd-ssl.conf/s/#Include/Include/' /usr/local/etc/apache24/httpd.conf
If we check now if there’s a sane config Apache HTTP will complain about a missing module involed in the ‘httpd-ssl.conf’ file in the ‘extra’ directory. Let’s fix this by enabling the cache for SSL module.
root@backend:/usr/local/etc/apache24 # grep -n 'mod_socache_shmcb' httpd.conf
92:#LoadModule socache_shmcb_module libexec/apache24/mod_socache_shmcb.so
root@backend:/usr/local/etc/apache24 #
It sits on line 92 and you can just uncomment it or use this next command.
sed -i -e '/mod_socache_shmcb.so/s/#LoadModule/LoadModule/' /usr/local/etc/apache24/httpd.conf
Don’t restart Apache HTTP yet, there’s no key nor certificate in place, plus the ‘Listen’ directive in ‘httpd-ssl.conf’ (the configuration file for SSL/TLS) still states to use port 443. That is quite surely the port the server at the front (the proxy) is using, so this needs to be changed. This is done in the next step.
Tip: Before you panic if you restart Apache HTTP at this point, you can first create a temporary pair of certificate and key (later this must be replaced) so you can operate Apache HTTP with TLS in the meantime. Use this one liner, and adjust the necessary options to your needs. Like not using my organization name for example.
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /usr/local/etc/apache24/server.key -out /usr/local/etc/apache24/server.crt -subj "/C=ES/ST=Barcelona/L=Terrassa/O=Adminbyaccident.com/CN=example.com/[email protected]"
If you decide to create the certificate, test your configuration with ‘apachectl configtest’ and check for any errors on the screen. If everything is OK (as it should) you’ll be told so.
root@backend:/usr/local/etc/apache24 # apachectl configtest
Performing sanity check on apache24 configuration:
Syntax OK
root@backend:/usr/local/etc/apache24 #
root@backend:/usr/local/etc/apache24 # apachectl restart
Performing sanity check on apache24 configuration:
Syntax OK
Stopping apache24.
Waiting for PIDS: 16421.
Performing sanity check on apache24 configuration:
Syntax OK
Starting apache24.
root@backend:/usr/local/etc/apache24 #
Step 2. Reconfigure the backend port
As seen in the previous guide the backend connection is made through port 8080 in plain text. That can be left on that port but as an informal convention that is considered a plain text port and the 8443 an encrypted one. So that should be changed in the ‘Listen’ directive.
There are two ways to do this. I prefer to shutdown the ‘Listen’ directive in the main configuration file ‘httpd.conf’ and declare the port with the same directive but on the ‘httpd-ssl.conf’ file.
Alternatively you can just change the port from 8080 to 8443 in the main ‘httpd.conf’ file and leave the default 443 on ‘httpd-ssl.conf’ but any port that is not in use should be shut, so close that on httpd-ssl.conf.
I’ll show my preference here but hopefully you know what you’re doing if you fancy the other way around. Just remember any time the website breaks the ‘Listen’ directive is declared in httpd-ssl.conf inside the ‘extra’ directory and not on the main configuration file.
We look for the ‘Listen’ directive on ‘httpd.conf’ with grep:
root@backend:/usr/local/etc/apache24 # grep -n 'Listen' httpd.conf
44:# Listen: Allows you to bind Apache to specific IP addresses and/or
48:# Change this to Listen on specific IP addresses as shown below to
51:#Listen 12.34.56.78:8080
52:Listen 8080
root@backend:/usr/local/etc/apache24 #
Once found we need to edit line 52 so it finally reads ‘#Listen 8080’ instead. Grab your favourite editor for that. Now it should look like:
root@backend:/usr/local/etc/apache24 # grep -n 'Listen' httpd.conf
44:# Listen: Allows you to bind Apache to specific IP addresses and/or
48:# Change this to Listen on specific IP addresses as shown below to
51:#Listen 12.34.56.78:80
52:#Listen 8080
root@backend:/usr/local/etc/apache24 #
We’re not finished yet. We must move to the ‘extra’ directory and edit the ‘Listen’ directive in the ‘httpd-ssl.conf’ file to use port 8443.
root@backend:/usr/local/etc/apache24/extra # grep -n 'Listen' httpd-ssl.conf
36:Listen 443
root@backend:/usr/local/etc/apache24/extra #
Just replace the 443 for 8443 with your favourite editor or use this one line below.
sed -i -e '/Listen/s/443/8443/' /usr/local/etc/apache24/extra/httpd-ssl.conf
It should now read as follows:
root@backend:/usr/local/etc/apache24/extra # grep -n 'Listen' httpd-ssl.conf
36:Listen 8443
root@backend:/usr/local/etc/apache24/extra #
With that change in place you may also need to addapt the virtualhost entry on the backend server so it no longer reads ‘*80’ but instead ‘_default_:8443’ at the very first entry of the configuration. To tackle that and add the necessary configuration bits to use a key and a certificate for this server let’s move on to the next step.
Step 3. Configure the Virtual Host on the backend
On the original reverse proxy article I didn’t configure any virtualhost entry for the backend server. It’s always a good idea, so one can rewrite there a few configuration bits, for example where the certificate and key sit or the cipher suites to use, etc.
This step is going to have two parts. First wer’re going to dress a virtual host file and later we’re going to set the certificate and key to use in this now encrypted communication between the front end server (the proxy) and the backend.
For the first bit of this step we’re going to remove the default ‘httpd-vhosts.conf’ file sitting in the ‘/usr/local/etc/apache24/extra/’ directory. Don’t panic, there’s an identical file named ‘httpd-vhosts.conf.sample’ so we are not losing anything really.
root@backend:/usr/local/etc/apache24/extra # rm httpd-vhosts.conf
root@backend:/usr/local/etc/apache24/extra #
We now create an empty file named identically.
root@backend:/usr/local/etc/apache24/extra # touch httpd-vhosts.conf
root@backend:/usr/local/etc/apache24/extra #
Once created we will dress it with the following configuration. Obviously change the domain name you are dealing with for your test or site.
<VirtualHost _default_:8443>
ServerName www.example.com:8443
ServerAlias www.example.com
ServerSignature Off
Protocols h2 h2c http/1.1
DocumentRoot /usr/local/www/apache24/data/
Options -Indexes +FollowSymLinks -Includes
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLCertificateFile /usr/local/etc/apache24/server.crt
SSLCertificateKeyFile /usr/local/etc/apache24/server.key
ErrorLog "/var/log/example.com-error_log"
CustomLog "/var/log/example.com-access_log"common
</VirtualHost>
Now, let’s understand what a few of the directives here do.
ServerName. This is quite self explanatory.
Server Alias. This is useful for example when your target domain is ‘example.com’ but need also to server connections looking for ‘www.example.com’ or other ‘aliases’ from the same host.
SeverSignature. This is set to ‘Off’ in here so there’s no indication of what version of Apache HTTP is being used to anyone making a request. This is a security mesure.
Protocols. If not self explanatory enough just read the information given in the link. Just as a reminder, set here the same protocols as you’ve set on the proxy (frontal server) for this site. Otherwise the ‘impaired’ one won’t be used at all or may give you errors. And of course don’t forget to enable the necessary modules for http2 whenever you enable it.
DocumentRoot. Where the data to serve this site is sitting. Mind to point this correctly!
Options -Indexes +FollowSymLinks -Includes. This is a melting pot, but basically the minus in the ‘Indexes’ option does forbid directory listing if the path in the URL doesn’t match any structure in the server. The plus on the ‘FollowSymLinks’ allows links to be followed. And the minus for the ‘Includes’ option disables the option to run ‘server side include’ programs. I’d recommend this exact configuration to enhance the server’s security.
SSLEngine. The star in this configuration bit. Without it set as ‘On’ the SSL/TLS connections wouldn’t work, so a must have.
SSLProtocol. This sets which protocols are allowed to use. A must have to secure the SSL/TLS connections. The configuration here allows all the versions to run except the ones with the minus in front of them. Basically we are allowing to run TLS1.2 and TLS1.3 which are the currently supported ones. The rest of them are deprecated. Don’t run them!
SSLHonorCipherOrder. When set to ‘on’ the server prefered cipher suite for the handshake will be used. When ‘off’ the client’s one will be used instead.
SSLCipherSuite. This configures what cipher suites does the server support. Choosing this by hand requires time, knowledge, patience and will power. Fortunately for us the Mozilla folks have build a page where you can select the ones you desire, depending of the support grade you want to give to your visitors.
SSLCertificateFile. Basically this points to where the cerficate to use in the SSL/TLS connections sits.
SSLCertificateKeyFile. The same as before but pointing to the key. Don’t ever leak this!
ErrorLog. This is a must. You always need to store the error logs from the web server somewhere. This does it.
CustomLog. This sets the file format for the logs.
Once we’ve dressed the ‘httpd-vhosts.conf’ file we now move on to the second part of this step, which is seting up the certificate and key files.
In order for the request to hit the backend through a TLS connection it is necessary for both servers to understand each other. By using the same certificate and key as used at the proxy we’ll have this communication without much hassle.
On the previous guide we set up a TLS termination on the front end (the proxy). We used self signed certificates. We used this command below (don’t use it on the backend!).
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /usr/local/etc/apache24/server.key -out /usr/local/etc/apache24/server.crt -subj "/C=ES/ST=Barcelona/L=Terrassa/O=Adminbyaccident.com/CN=example.com/[email protected]"
So both certificate and key are sitting on the front end server, the proxy. Copy them and place them in the same directory path the ‘SSLCertificateFile’ and ‘SSLCertificateKeyFile’ state respectively on the backend. Print them on your screen and copy-paste them or use the scp command, or any other method you know of. As for example I’d recommend to do something similar to this from the frontend server to the backend:
scp /usr/local/etc/apache24/server.crt username@backendserver:/usr/local/etc/apache24/server.crt
scp /usr/local/etc/apache24/server.key username@backendserver:/usr/local/etc/apache24/server.key
Now we’ve put all the configuration bits we need to restart the Apache HTTP server on the backend, so all these changes take effect.
Before doing so you may want to test it so some clues about any errors that may appear wil be printed on the screen.
root@backend:~ # apachectl configtest
Performing sanity check on apache24 configuration:
Syntax OK
root@backend:~ #
As there isn’t any error being displayed we can proceed to restart the backend’s web server.
root@backend:~ # apachectl restart
Performing sanity check on apache24 configuration:
Syntax OK
Stopping apache24.
Waiting for PIDS: 8690.
Performing sanity check on apache24 configuration:
Syntax OK
Starting apache24.
root@backend:~ #
Once the backend has been all configured it’s time to move on to the front end and apply the necessary bits there so the communication between both is performed under an encrypted and safe TLS connection.
Step 4. Configure the front end’s Virtual Host.
In this step we need to adjust the virtual host entry to the newly configured backend which is now responding to a different port and also using encrypted ‘https’ instead of the plain text ‘http’.
If we previously had this configuration block below, we now need to make a few adjustments.
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
ServerSignature Off
Protocols h2 h2c http/1.1
Redirect / https://www.example.com/
</VirtualHost>
<VirtualHost *:443>
ServerName example.com
ServerAlias www.example.com
ServerSignature Off
Protocols h2 h2c http/1.1
ProxyPass / http://192.168.1.220:8080/
ProxyPassReverse / http://192.168.1.220:8080/
Options -Indexes +FollowSymLinks -Includes
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLCertificateFile /usr/local/etc/apache24/server.crt
SSLCertificateKeyFile /usr/local/etc/apache24/server.key
ErrorLog "/var/log/example.com-error.log"
CustomLog "/var/log/example.com-access.log"common
Include /usr/local/etc/apache24/extra/httpd-headers.conf
</VirtualHost>
We need to change the backend port on the ‘ProxyPass’ and ‘ProxyPassReverse’ directives. We also need to make use of ‘https’ instead of plain ‘http’. And a few more changes, like including this other directive ‘SSLProxyEngine’ in order to forward the original connection to the backend through an encrypted channel.
To summarize this is the front end configuration I’d use.
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
ServerSignature Off
Protocols h2 h2c http/1.1
Redirect / https://www.example.com
</VirtualHost>
<VirtualHost *:443>
ServerName example.com
ServerAlias www.example.com
ServerSignature Off
Protocols h2 h2c http/1.1
SSLEngine on
SSLProxyEngine on
SSLProxyCheckPeerName off
ProxyPreserveHost on
ProxyPass / https://192.168.1.198:8443/
ProxyPassReverse / https://192.168.1.198:8443/
Options -Indexes +FollowSymLinks -Includes
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLCertificateFile /usr/local/etc/apache24/server.crt
SSLCertificateKeyFile /usr/local/etc/apache24/server.key
ErrorLog "/var/log/example.com-error_log"
CustomLog "/var/log/example.com-access_log"common
</VirtualHost>
Let’s look at the directives that have been added to get this TLS reverse proxy backend configuration.
SSLProxyEngine. The other star on this show. Without it forget about passing TLS connections to the backend.
SSLProxyCheckPeerName. This directive is here to check the validity of the certificate and key in use. In my case this is set to ‘off’ since I’ve been using a self-signed certificate.
ProxyPreserveHost. With this directive set to on the host in the request’s header will be checked against the entry on the backend. Useful when having several backends to call to from one proxy.
As in the original article the two ‘ProxyPass’ directives are a must. Mind this time we are calling the backend through https and the port has changed from ‘8080’ to ‘8443’.
ProxyPass. This directive allows the local server to find remote servers.
ProxyPassReverse. This directive helps the reverse proxy server not to be skipped due to any redirect present at the backend server.
Now that all the changes have been applied to the front end server, in other words: the proxy, it’s time to restart it and check if everything works.
root@frontal:~ # apachectl restart
Performing sanity check on apache24 configuration:
Syntax OK
Stopping apache24.
Waiting for PIDS: 9439.
Performing sanity check on apache24 configuration:
Syntax OK
Starting apache24.
root@frontal:~ #
Now we can test the site visiting it with our browser or using curl.
Extra tips.
Remember the backend may need to have slightly different modules enabled when compared to the front end. For example the ‘proxy_fcgi_module’ must be enabled on the backend if you pretend to use the MPM event module, which I strongly recommend for performance reasons. However you don’t need to enable such module on the front end.
Another topic is around the HTTP2 protocol. To test the HTTP2 protocol is working here’s a one liner to help.
[albert@BSDVM ~]$ curl -skI https://rollins.com -o /dev/null -w '%{http_version}\n'
2
[albert@BSDVM ~]$ curl -skI http://rollins.com -o /dev/null -w '%{http_version}\n'
1.1
[albert@BSDVM ~]$
The output of ‘2’ afirms the http2 protocol version is working. However the 1.1 afirms http2 isn’t in use but 1.1 is.
To test partial steps of this guide you can make use of the ‘apachectl -t’ command which will test the current configuration. Any issues will be displayed on the screen and will give you clues of what’s wrong.
Conclusion.
Configuring Apache HTTP with a TLS reverse proxy backend takes a bit of time but once the process is clear and you know the directives to be applied there’s little effort to implement this on any sites that are proxied.
I’m a FreeBSD user mostly, specially on my servers, however Apache HTTP is somewhat the same everywhere, unless on Debian and derivatives such as Ubuntu. Just kidding. There you can a2enmod or a2dismod modules at will without the need to grep for lines or use sed.
If you find the articles in Adminbyaccident.com useful to you, please consider making a donation.
Use this link to get $200 credit at DigitalOcean and support Adminbyaccident.com costs.
Get $100 credit for free at Vultr using this link and support Adminbyaccident.com costs.
Mind Vultr supports FreeBSD on their VPS offer.