BGP Maintenance and Security
We announced earlier in the week that there would be some disruption to services for scheduled maintenance but Twitter isn’t really the place to discuss what we are doing.
As part of the work we’ve been doing on OnionContainers we’ve decided to add a Kubernetes cluster to our Reading POP. This allows us to rapidly deploy development versions of the various GoLang daemons that make up Brass Horn Communications, Ablative Hosting and Onion Containers in an environment that is controlled by us.
This cluster isn’t used for any sensitive workloads such as signing our DNSSEC zones or hosting members data - those are all still handled on OpenBSD servers.
Since we were going to be in the DC to install this hardware we took the opportunity to drop the Reading POP down from a /23 of IPv4 addresses to a /24 (it’s always best to be on-site when messing with network configurations and you don’t have an OOB or iLO setup!).
Subnets and IPv4 Exhaustion
When we first deployed the Reading POP in 2015 we were wary of blacklists that would list the entire /24 containing a Tor exit and since we were planning on hosting webservers and email servers in this POP we didn’t want to risk them getting caught up in those blacklists. This resulted in us opting for a flat /23 with Tor Exits in one /24 and everything else in the other.
Because we were so late to the game we only recieved a /22 allocation from RIPE which works out as 1024 IPs and we were consuming 512 of them in one POP. The thing is 255 IPs is more than enough for what we require in this POP as that is more addresses than the equivilent amount of bandwidth (our servers top out at around 250Mbit of Tor throughput each) so we decided to drop down to a /24.
We’ve set the four /24’s as LIR-PARTITIONED PA and assigned them as follows;
- 188.8.131.52/24 - Reading POP [Tor Exits, General Infrastructure]
- 184.108.40.206/24 - Spare
- 220.127.116.11/24 - Spare
- 18.104.22.168/24 - Ablative Hosting VPS’
Whilst we might only have 1024 IPv4 addresses we’ve got 633,825,300,114,114,700,748,351,602,688 IPv6 addresses and are carving them up into /36’s (4,951,760,157,141,521,099,596,496,896) per POP giving us 4096 /48’s which is more than we’ll need.
All of our Tor Exit servers leverage IPv6 for client and exit connections but we are looking to test some IPv6 only bridges and Exits to see what the take up is like.
One of the mitigations to BGP hijacking is to use the Resource Public Key Infrastructure to digitally sign valid route announcements to say which ASNs are allowed to announce which IP subnets.
Much like HSTS, DNSSEC and anything else to do with Public Keys Infrastructure or signed attestation there is a risk that something could go wrong and we might knock ourselves offline but since we’d already shut most things off for the migration work now was a good a time as any to step up our security game.