I had wanted to create a hybrid connectivity environment in a Google cloud project for months. I always thought only big corporations can afford the Cisco ASR 9000 series routers to create partner or dedicated interconnect connections. It turns out there are many home lab cases where enthusiasts found a way to create BGP sessions in Cloud VPN to connect from a AMD64 based router from home to Google Cloud. The only downside is lack of static IP from Internet service providers.
Installation of pfSense is typically not hard and full of community contributed articles or videos like LTT. The real challenge is installing pfSense on Ubuntu 22.04 KVM. Avoid any articles about Oracle virtualbox. I followed How to install KVM on Ubuntu for installation. After installing KVM, I had to reboot to make the virtual machine manager GUI application to connect to libvirt daemon. A reboot is recommended.
I bought the TP link gigabit PCIe adapter and set it to LAN IP 192.168.3.2 on Ethernet interface enp4s0. My home router’s LAN is 192.168.1.0/24. I’ve done this on AMD Ryzen pro Threadripper and AMD Ryzen 5 7600x based computers. Execute ip a on the command line to verify both Ethernet interfaces are recognized. For KVM to use the existing 2 Ethernet interfaces and to allow created VM instances to be visible on the LAN, you’d need to configure a yaml file similar to the one below using bridges. Assume the WAN NIC is enp1s0 and the LAN NIC is enp4s0. Move existing files of /etc/netplan/* to a backup directory such as /tmp/. Move the new yaml file to /etc/netplan/02-netcfg.yaml and execute sudo service netplan apply.
network:
version: 2
# you may copy this line under enp1s0, enp4s0, br0, br_lan and delete it
# if there are other netplan yaml files in the folder such as wifi.
# This will enable configuration of WiFi in desktop manager settings
renderer: networkd
ethernets:
# Existing Ethernet port for WAN
enp1s0:
dhcp4: false
dhcp6: false
# The installed TP-Link PCIe card for LAN
enp4s0:
dhcp4: false
dhcp6: false
# configuration for KVM to run pfSense for WAN,LAN
bridges:
# bridge for KVM to use the WAN interface
br0:
interfaces: [enp1s0]
dhcp4: false
addresses: [192.168.1.2/24]
routes:
- to: default
via: 192.168.1.1
metric: 100
on-link: true
routing-policy:
- from: 192.168.1.0/24
nameservers:
addresses: [8.8.8.8]
parameters:
stp: false
dhcp6: false
# bridge for KVM to use the LAN interface
br_lan:
interfaces: [enp4s0]
dhcp4: false
addresses: [192.168.3.2/24]
routes:
- to: 10.0.0.0/8
via: 192.168.3.1
metric: 50
on-link: true
routing-policy:
- from: 192.168.3.0/24
nameservers:
addresses: [8.8.8.8]Assume you want the WAN to be 192.168.1.* and LAN to be 192.168.3.*. The WAN’s router is at 192.168.1.1 and the LAN’s pfSense is at 192.168.3.1. You want the br_lan NIC to be 192.168.3.2. The reason br_lan has metric 50 < metric 100 is because Google Cloud has the default VPC network of IP 10.*. Lower metric has higher priority. You want to route to destination of 10.* via 192.168.3.1 instead of 192.168.1.1 . If your VPC network in Google Cloud is 172.* , you’d put 172.16.0.0/12.
Create a VM with a bridge network interface br0 and choose pfSense-CE-2.6.0-RELEASE-amd64.iso or the latest version. I start with 1 CPU and 1 GB RAM. Don’t start the installation right away but configure the hardware to add another network interface for br_lan.
The KVM on Ubuntu 23.04 has NIC device model default to e1000e which will cause pfSense booting to get stuck at LAN configuration. It wasn’t obvious the Device model caused it. Setting it to Hypervisor default || rtl8139 resolves the halted booting for me.
Follow the installation walkthrough to install pfSense and remember the NIC’s MAC addresses. If the installation fails to find disks, change the KVM storage type to SATA or IDE and retry installation. If both NICs are recognized by pfSense, you’d see the following screen depending on the pfSense version. pfSense will likely to show em0 || re0 , em1 || re1 and tell you the MAC addresses. The installation page suggests not setting up VLANs.
In another example, the MAC ending with 06 is for WAN and the MAC ending with a6 is for LAN. Execute 1) and 2) to assign the interfaces to WAN, LAN, set the LAN’s IP. My WAN gets IP assigned by the home router’s DHCP. The configured pfSense shows the following screen:
The later part of 2) should have the following screen.
entering 100 was a mistake in the image below. I retried with 192.168.3.100 and chose y for DHCP , HTTP. accessing pfSense’s web configurator with default username: admin, password: pfsense.
Configure GCP HA VPN
Part 1
follow Configure Google Cloud HA VPN with BGP on pfSense or my archive up to right before `Setup BGP on pfSense` section. The BGP session in the blog is not valid as the plugin’s package is absent. Refer to the blog for detailed instructions. I am providing merely screenshots here.
Create a HA VPN tunnel
Each interface costs $37 a month. Choosing 1 is good for the test environment. Enter the peer VPN gateway’s public IP address. Execute curl ifconfig.me to see the IP. The peer VPN gateway is pfSense.
Click configure BGP session
Choose an ASN that has not been used in the project
Click save BGP configuration
pfSense configuration
I added some steps missing in the blog such as Firewall rules. You’d need those additional steps for the traffic from GCP to LAN.
System > Advanced > Firewall & NAT select Allow APIPA traffic and save
1. Configure WAN interface via Interfaces > WAN: Uncheck “Block private networks and loopback addresses”
2. Select Interfaces | WAN > Uncheck “Block bogon networks” if selected; Click Save and then Apply
VPN / IPSec / Tunnels > Add P1
For the page below, enter the Pre-Shared Key on later pages
Click show phase 2 entries > Add P2
The local network 169.* IP is usually the higher IP. You can get them from the downloaded configuration at the end of the HA VPN creation.
After saving and applying the changes. The Status page should show green:
At this point, Cloud VPN should have VPN tunnel status as connected and green. However, the BGP session may still be pending or disconnected. Part 2 below have instructions to configure the BGP sessions.
Part 2
Follow Site to Site VPN between Google Cloud and pfSense on VMware at home or my archive starting from section pfSense BGP configuration.
System / Package manager / available packages > install frr
Click GBP. The Router ID has to be the BGP peer IP address in VPN tunnel details page. It’s usually the higher IP. The Local AS is the Peer router ASN in VPN tunnel details page. From Google Cloud’s perspective, peer means the network systems on premises.
Click Neighbors. Set the Name/Address to be Cloud Router BGP IP address which is usually the lower IP. Set Remote AS to be Cloud router ASN in the Cloud VPN tunnel details page.
At this moment, any host on the LAN can ping a Compute engine instance’s internal IP but not the other way around. You need to create a firewall rule to allow network traffic from the Google cloud VPC to the LAN. Suppose the VPC network’s IP range is 10.128.0.0/9. Create a Firewall / Rules / IPSec > Add
After that, ping should succeed:
ping 192.168.3.1
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=63 time=295 ms
64 bytes from 192.168.3.1: icmp_seq=3 ttl=63 time=311 ms
64 bytes from 192.168.3.1: icmp_seq=4 ttl=63 time=334 ms
64 bytes from 192.168.3.1: icmp_seq=5 ttl=63 time=298 msCaution
If you are using consumer grade Internet service, the IP may change every 6 months. Go to the VPN page in the cloud console to re-create the VPN tunnel with the new Peer VPN gateway IP. Otherwise, the VPN will fail. You’d need to update IPs on pfSense such as Remote Gateway IP and the 169.* IPs.
Optional step
Enable upstream DNS resolution by configuring DNS server settings at System / General Setup. Enter 8.8.8.8 for Google’s DNS server.
If you want easier troubleshooting, enable pinging the WAN IP in firewall / Rules / WAN
- Status / Interfaces shows the WAN’s IP. If your home router is using DHCP which is about 99% of the case, you may want to change the home router’s start and stop IP to be from .100 to .200 and reserve IPs < .99 for static IP. Then configure the WAN IP to be static.
- This step is usually not needed; with the configuration above, the LAN with DHCP would have Internet connection via WAN => Enable Internet connection in LAN via NAT.
This creates a site to site HA VPN similar to the interconnect at big corporations. Although the bandwidth isn’t comparable, many migration proof of concept work can be done this way.
