I had wanted to create a hybrid connectivity environment in a Google cloud project for months. I always thought only big corporations can afford the Cisco ASR 9000 series routers to create partner or dedicated interconnect connections. It turns out there are many home lab cases where enthusiasts found a way to create BGP sessions in Cloud VPN to connect from a AMD64 based router from home to Google Cloud. The only downside is lack of static IP from Internet service providers.
Installation of pfSense is typically not hard and full of community contributed articles or videos like LTT. The real challenge is installing pfSense on Ubuntu 22.04 KVM. Avoid any articles about Oracle virtualbox. I followed How to install KVM on Ubuntu for installation. After installing KVM, I had to reboot to make the virtual machine manager GUI application to connect to libvirt daemon. A reboot is recommended.
I bought the TP link gigabit PCIe adapter and set it to LAN IP
192.168.3.2 on Ethernet interface
enp4s0. My home router’s LAN is
192.168.1.0/24. I’ve done this on AMD Ryzen pro Threadripper and Intel Core i5–6500 based computers. Execute
ip a on the command line to verify both Ethernet interfaces are recognized. For KVM to use the existing 2 Ethernet interfaces and to allow created VM instances to be visible on the LAN, you’d need to execute
sudo service netplan apply on a file like /etc/netplan/02-netcfg.yaml as /etc/netplan/01-network-manager-all.yaml may have existed.
# Existing Ethernet port for WAN
# The installed TP-Link PCIe card for LAN
# configuration for KVM to run pfSense for WAN,LAN
# bridge for KVM to use the WAN interface
- to: default
- from: 192.168.1.0/24
# bridge for KVM to use the LAN interface
- to: 10.0.0.0/8
- from: 192.168.3.0/24
Create a VM with a bridge network interface br0 and choose pfSense-CE-2.6.0-RELEASE-amd64.iso or the latest version. Don’t start the installation right away but configure the hardware to add another network interface for br_lan.
Follow the installation walkthrough to install pfSense and remember the NIC’s MAC addresses. In my case, the MAC ending with
06 is for WAN and the MAC ending with
a6 is for LAN. pfSense will likely to show em0 and em1 and tell you the MAC addresses. Execute
2 to assign the interfaces to WAN and LAN; set the LAN’s IP. My WAN gets IP assigned by the home router’s DHCP. The configured pfSense shows the following screen:
The later part of
2) would have the following screen; choose
y for DHCP and HTTP.
Configure GCP HA VPN
follow Configure Google Cloud HA VPN with BGP on pfSense or my archive up to right before `Setup BGP on pfSense` section. The BGP session in the blog is not valid as the plugin’s package is absent. Refer to the blog for detailed instructions. I am providing merely screenshots here.
Create a HA VPN tunnel
Each interface costs $37 a month. Choosing 1 is good for the test environment. Enter the peer VPN gateway’s public IP address. Execute
curl ifconfig.me to see the IP. The peer VPN gateway is pfSense.
Click configure BGP session
Choose an ASN that has not been used in the project
Click save BGP configuration
I added some steps missing in the blog such as Firewall rules. You’d need those additional steps for the traffic from GCP to LAN.
System > Advanced > Firewall & NAT select Allow APIPA traffic and save
1. Configure WAN interface via Interfaces > WAN: Uncheck “Block RFC1918 Private Networks”
2. Select Interfaces | WAN > Uncheck “Block bogon networks” if selected; Click Save and then Apply
VPN / IPSec / Tunnels > Add P1
For the page below, enter the Pre-Shared Key on later pages
Click show phase 2 entries > Add P2
The local network 169.* IP is usually the higher IP. You can get them from the downloaded configuration at the end of the HA VPN creation.
After saving and applying the changes. The Status page should show green:
At this point, Cloud VPN should have VPN tunnel status as connected and green. However, the BGP session may still be pending or disconnected. Part 2 below have instructions to configure the BGP sessions.
Follow Site to Site VPN between Google Cloud and pfSense on VMware at home or my archive starting from section pfSense BGP configuration.
System / Package manager / available packages > install frr
Click GBP. The
Router ID has to be the
BGP peer IP address in VPN tunnel details page. It’s usually the higher IP. The
Local AS is the
Peer router ASN in VPN tunnel details page. From Google Cloud’s perspective, peer means the network systems on premises.
Click Neighbors. Set the
Name/Address to be
Cloud Router BGP IP address which is usually the lower IP. Set
Remote AS to be
Cloud router ASN in the Cloud VPN tunnel details page.
At this moment, any host on the LAN can ping a Compute engine instance’s internal IP but not the other way around. You need to create a firewall rule to allow network traffic from the Google cloud VPC to the LAN. Suppose the VPC network’s IP range is 10.128.0.0/9. Create a Firewall / Rules / IPSec > Add
After that, ping should succeed:
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=63 time=295 ms
64 bytes from 192.168.3.1: icmp_seq=3 ttl=63 time=311 ms
64 bytes from 192.168.3.1: icmp_seq=4 ttl=63 time=334 ms
64 bytes from 192.168.3.1: icmp_seq=5 ttl=63 time=298 ms
If you are using consumer grade Internet service, the IP may change every 6 months. Go to the VPN page in the cloud console to re-create the VPN tunnel with the new Peer VPN gateway IP. Otherwise, the VPN will fail. You’d need to update IPs on pfSense such as Remote Gateway IP and the 169.* IPs.
If you want easier troubleshooting, enable pinging the WAN IP in firewall / Rules / WAN
- Status / Interfaces shows the WAN’s IP. If your home router is using DHCP which is about 99% of the case, you may want to change the home router’s start and stop IP to be from .100 to .200 and reserve IPs < .99 for static IP. Then configure the WAN IP to be static.
- Only attempt if the LAN has no Internet connetion: Enable Internet connection in LAN via NAT.
This creates a site to site HA VPN similar to the interconnect at big corporations. Although the bandwidth isn’t comparable, many migration proof of concept work can be done this way.