TLDR
IPSet is an extension of IPTables which can give significant performance gains as well as simplify configuration. perf top showed 10-30% CPU overhead for IPTables in my setup. It was negligible with IPSet.
Background
- An additional IP-layer security was needed for a HTTP/s server
- 4 ports to be restricted
- A pool of Virtual Machines on the LAN (no NAT) were the clients
- Clients had random or non-contiguous IPs
- Chef runs on all the machines
It was almost trivial to have some chef code to get the IPs of all the clients and create IPTable rules. All was good. Immediate goal accomplished.
Configuration
4 ports
- 400 Client IPs => 1600 Accept Rules
- 4 Reject Rules
- Total rules => 1604
This is a configuration nightmare if managed manually. However this is powered by Chef, so we never have to hand edit the rules. It’s not pretty, but it works.
Performance
We started noticing a decent amount of CPU time being spent in iptables as seen by perf top, about 10-30% depending on the load.
In addition, some (~0.01%) requests would take longer than usual.
The demo setup later demonstrates the issue via a load test.
Enter IPSet
From Linux Journal
ipset is an extension to iptables that allows you to create firewall rules that match entire “sets” of addresses at once. Unlike normal iptables chains, which are stored and traversed linearly, IP sets are stored in indexed data structures, making lookups very efficient, even when dealing with large sets.
Essentially, instead of enumerating each source IP, define an ipset which contains all the IPs. Then have an IPTable rule which references this set. This reduces lookup from O(n) to O(1).
Using ipset was not that difficult. The chef code was modified slightly and off it went to production.
End Result
- 1 set containing all the IPs
- 4 Accept Rules (1 per port)
- 4 Reject Rules (1 per port)
- Almost negligible CPU overhead
- No more long tail latencies
The ‘Fun’ part
Using one of my favorite tools this can easily be demo’ed. Here’s a chef cookbook which will setup two servers locally.
.kitchen.yml defines the servers + network
- Centos 6.7 with 2.6.32-573.el6.x86_64 kernel
- iptables-1.4.7-16.el6.x86_64
- ipset-6.11-4.el6.x86_64
Destination (dst)
- code
- runs a http server written in Go-Lang using Runit
- http_server can sleep a desired amount of time based on a get parameter
- loads IPTables or IPSet configuration based on a flag
Source (src)
- code
- installs Vegeta to generate the load
- I tried ‘ab’ but wasn’t able to get high-enough throughput
sudo bash /tmp/runto launch the test script- The test involves POSTing a file of a fixed size to the endpoint
- The goal is to have 750 req/sec concurrently
Testing with curl
[vagrant@src1 ~]$ curl http://172.28.3.10:45000/hello?sleep_ms=100 -v
Run #1 - no IP Filtering on the destination server
[root@dst1 ~]# iptables --flush
[root@dst1 ~]# ipset --flush
Vegeta Report
Notice the 95th, 99th percentile and max latencies, we will come back to these later.
Perf top on destination
Number of sockets in use on destination
Run #2 - 1600+ IPTable rules on destination server
[root@dst1 ~]# iptables -L -n | wc -l
1616
Vegeta Report
There’s a definite uptick in 95th and 99th percentile latencies. We even had a timeout! The test can always be run for a much longer duration as well as multiple cycles to cater for outliers.
Perf top on destination
We are spending a fair amount of CPU cycles dealing with IPTables. In the next step, we replace these 1600 rules with a handful using IPSet.
Run #3 - IPset rules on destination server
Vegeta Report
No more timeouts. Latencies look similar to Run #1 without IP Filtering
Perf top on destination
That’s right, ipt_do_table doesn’t even show up!
Conclusion
IPSet can help simplify IPTables configuration as well boost performance under high load. A demo cookbook sets up two servers and performs a load test to illustrate the issue.