Turning Up a Personal Vault Instance

Posted on March 31, 2020 in Sysadmin


Key2.jpg

How I secured Vault for personal use by making it VPN-only (with working DNS and SSL certs!)

I've been using Hashicorp's Vault at my job for well over a year now. At first, we were just using it for static secrets management, but as our infrastructure grew and diversified, so did our needs. Now, we use Vault to manage several PKIs, for general password management, and for things like SSH-OTP. The product is simply amazing, and for my colleagues who just need to grab a password, and other users who aren't as tech-savvy, the UI meets all of the basic needs. We talk about Vault a fair bit, as well as other password managers in this episode of Sysadministrivia.

Prior to using Vault, though, we were using pass with a very private Git repo filled with passwords, which I still highly recommend. I also used pass personally for the last 4 or 5 years. The problem with pass sometimes is its inaccessibility. If you want to use it to store all of your passwords, you'd like them to be easily available across all of your devices, often including mobile devices like phones and tablets. Now, you can get around some of this by using pass in conjunction with Git, but it still isn't the most seamless. Vault solves this problem by having an accessible web interface and offering username/password authentication. But, with a functioning web interface, and by relying on usernames and passwords for authentication, you also have more exposure. So how do we limit this exposure while still allowing ourselves access to our passwords from (more or less) anywhere? Well, you have a few options, including the one I landed on and will detail below:

  • Basic auth at the web server level in front of Vault, or some other webserver plugin to enable 2FA
  • Running Vault at home on a server you have to proxy to in some capacity
  • My solution, requires you to have your own VPN server; run Vault on a machine on your VPN, and only allow access over the VPN

Vault Setup

Regardless of which VPN solution you are using, and whether or not you are going to follow through with my entire setup using DNS and LetsEncrypt-issued SSL certs, it's important that you have a sane Vault setup with secure settings. I am not going to go through the entire Vault deployment, as Hashicorp has awesome deployment guides and there is no sense in recreating the wheel. I will paste my Vault configuration file below to note a few things - namely that I am terminating SSL at the webserver, therefore communication between Nginx and Vault is not encrypted. This is probably fine for a personal deployment, or even many deployments in an enterprise environment. It all depends where Vault is running, who has access to the server, etc. It does allow one to use tcpdump to listen on the loopback interface and capture Vault information in plaintext, which can be immensely useful for debugging, but be cautious, and that's my only warning.

/*
 * Vault configuration. See: https://vaultproject.io/docs/config/
 */

backend "file" {
	path = "/var/lib/vault"
}

listener "tcp" {
	/*
	 * By default Vault listens on localhost only.
	 * Make sure to enable TLS support otherwise.
	 *
	 * Note that VAULT_ADDR=http://127.0.0.1:8200 must
	 * be set in the environment in order for the client
	 * to work because it uses HTTPS by default.
	 */
    address = "127.0.0.1:8200"
	tls_disable = 1
}
ui = true


Webserver Setup

Now, you shouldn't be interacting with Vault directly in production, but rather you should be proxying requests to/from it via Nginx (or Apache, if you want to waste your time). Below is the relevant portions of my nginx.conf on my Vault server:

server {
  listen [::]:80 ipv6only=off default_server;
  server_name your.tld;

  location /.well-known/acme-challenge {
    default_type "text/plain";
    root /var/lib/letsencrypt/;
    allow all;
  }

  location / {
    return 301 https://your.tld$request_uri;
    index index.htm index.html;
    allow 192.168.200.0/25;
    deny   all;
  }

}

server {
  listen [::]:443 ssl http2 ipv6only=off default_server;
  ssl_certificate /etc/letsencrypt/live/your.tld/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/your.tld/privkey.pem;
  ssl_trusted_certificate /etc/letsencrypt/live/your.tld/chain.pem;
  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;
  ssl_session_tickets off;
  ssl_protocols TLSv1.2;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
  ssl_dhparam /etc/nginx/ssl/key/dh4096.pem;
  ssl_prefer_server_ciphers on;
  add_header Strict-Transport-Security max-age=15768000;
  ssl_stapling on;
  ssl_stapling_verify on;
  server_name your.tld;


  location / {
    allow 192.168.200.0/25;
    deny   all;
    proxy_pass http://127.0.0.1:8200;
    proxy_set_header Host $host;
    expires -1;
  }

}


Note - the allow directive is my VPN subnet, change it accordingly for your own subnet if employing these methods. These allow and deny directives ensure that the Vault instance is only accessible over the VPN subnet. I would also recommend that you make similar rules at the firewall level, but be sure to leave port 80 open on the WAN for LetsEncrypt to renew certificates. On that note, please see the relevant portion of the config whereby requests to the LetsEncrypt webroot are not redirected to https.

OpenVPN and dnsmasq Setup

The last portion of this is actually setting up your OpenVPN server to push its own DNS resolver (you must run your own resolver to still use DNS names within the VPN like I am doing). I am not going to detail an entire OpenVPN config, or an entire dnsmasq config, as the options for both vary widely by preference and deployment. However, the OpenVPN option to push a single resolver that is the VPN server itself would look like this: push "dhcp-option DNS 192.168.200.1". As a note, Linux clients will likely NOT automatically add the resolver to their /etc/resolv.conf, so you will need to employ an up/down script like this one. In my testing so far though, macOS and Android clients DO automatically add the resolver and everything works as-expected.

I am also not going to tell you how to configure dnsmasq because there are so many options and they may vary across deployments. What I will tell you is that you need to add an A record for your domain (in my case vault.jthan.io) to your config. The simplest way to do this inline is by adding a line like this, adjusting it for wherever on your VPN vault lives: address=/vault.jthan.io/192.168.200.1. My instance of dnsmasq also only listens on my OpenVPN interface using the interface directive: interface=tun0. To prevent leaking information to upstream resolvers, and reduce load, response times, etc. I also recommend the bogus-priv option which won't pass addresses in rfc1918 space to upstreams.

After you've done all of these things, restart your VPN server, dnsmasq, and VPN clients. What you should see is an nslookup from a client NOT on the VPN or using the VPN DNS resolver should return your public IP:

jonathan@tartufo:~$ nslookup vault.jthan.io 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	vault.jthan.io
Address: 50.116.5.110
Name:	vault.jthan.io
Address: 2600:3c01::f03c:91ff:feaa:ca8f


However, a lookup while on the VPN should return the VPN address:

jonathan@shaco:~$ nslookup vault.jthan.io 
Server:		192.168.200.1
Address:	192.168.200.1#53

Name:	vault.jthan.io
Address: 192.168.200.1


Now from a VPN client, if I open my browser and type vault.jthan.io into the address bar, because its DNS query says it's located at 192.168.200.1, and I'm accessing it over the VPN which falls under the Nginx 'allow' directive, I should see my vault web UI. Similarly, you should verify that from a non-VPN client you can't access that URL. The exception, of course, should be a location under our location block for LetsEncrypt. You can ensure that's working as expected like so (yes, it's a 404 because there isn't anything there, but it's not timing out or anything else, which means LE can access it too):

jonathan@shaco:~$ curl -IL http://vault.jthan.io/.well-known/acme-challenge/foo
HTTP/1.1 404 Not Found
Server: nginx/1.16.1
Date: Mon, 30 Mar 2020 19:23:43 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive

Closing Remarks

This is a great method for securing any Vault (or other service) deployment assuming you want to run it on your VPN server, as I did here. One thing to note, if you are going to run it on a DIFFERENT server than the VPN, you can just use a route pushed by the VPN server to route all traffic to Vault through it, and a firewall rule on the Vault server to only allow WAN access from your VPN server's public IP, thus you don't need to add the Vault server as a client of the VPN. This is the benefit of having your own VPN server, and especially so if it is separate from all of your other services. It also exemplifies what you can do with nginx allow and deny directives - another use I have for them personally is only allowing access to /admin on my blog from my VPN server, so only VPN clients routing traffic over the VPN server can access the admin interface. Security should always be layered so you aren't reliant on a single component in your stack to secure any given environment. In the blog example, my VPN serves as a layer in front of Django's auth.


← Return to previous page