At War Legend, we use Kubernetes on cloud instances where no data is persisted, we have dedicated servers for databases, including a ScyllaDB cluster. They can communicate together using a link between two private networks (the Cloud and the Dedicated), this is managed by our hosting provider. This lets us make sure that only our servers can reach the database (even if the databases are still protected by password/keys).
How do Private Networks work in the Cloud ?
Back in the days, private networks where physically separated from the public network, with different switches and routers, but now it’s mostly virtualized. Modern switches can have VLANs, which are virtual networks that can be isolated from each other, so that customers can have their own private network without the need for physical separation.
Not all the switchs can be aware of all the VLANs, so your packets have to be encapsulated using a protocol like VXLAN or GRE to be sent to the other network so that the switch can decapsulate them and send them to the right server. There are also other solutions like OpenVPN or WireGuard that can be used to create a private network between servers, but they suffer from the same issue: they need to encapsulate the packets.
The issue with Private Networks
All of these solutions have a performance cost, but it’s usually not a problem for most use cases. Private networks are usually managed by cloud providers, so it is all easy unless you need to add nodes from another provider or in our case a dedicated server (different infrastructure).
It works well 99.9% of the time, but sometimes the link between the two private networks is down, even if the servers can still communicate using the public network. A few weeks ago, the link went down for a few hours so we had to route all the traffic through the public network, exposing our databases to the internet.
This is in theory not a problem in the short term because you still need to be authenticated to access the databases, but if a vulnerability is found in the database software, it could be exploited by anyone.
The solution
This why I decided to create a distributed firewall system that syncs with external services (only Kubernetes for now) to block all the traffic that is not coming from known public IPs. The main goals of the project are:
- Not to rely on external services (like a firewall managed by the hosting provider)
- Be able to support multiple sources (Kubernetes, Custom Whitelist, Provider API)
- Drop all the traffic that is not coming from a known IP except for port 22
The filtering itself is performed by nftables, a Linux kernel subsystem that provides filtering and classification of network packets.
Architecture
We have 3 main components:
- The client: a small Rust program that runs on every server that needs to be protected and syncs the nftables rules with the server
- The server: an API that stores the rules and syncs them with the clients in real-time
- The kubesync client: a Kubernetes controller that watches for changes in the Kubernetes API and updates the server with the new rules, it must run in the kubernetes system it tracks and have the permissions to list nodes.
This project is still early in development, but you can check the progress on the GitHub repository.
If you want to add other node providers, feel free to open a PR or an issue on the repository.