4 min read

Fun with IPSet and IPTables

TLDR

IPSet is an extension of IPTables which can give significant performance gains as well as simplify configuration. perf top showed 10-30% CPU overhead for IPTables in my setup. It was negligible with IPSet.

Background

  • An additional IP-layer security was needed for a HTTP/s server
  • 4 ports to be restricted
  • A pool of Virtual Machines on the LAN (no NAT) were the clients
  • Clients had random or non-contiguous IPs
  • Chef runs on all the machines

It was almost trivial to have some chef code to get the IPs of all the clients and create IPTable rules. All was good. Immediate goal accomplished.

Configuration

4 ports

  • 400 Client IPs => 1600 Accept Rules
  • 4 Reject Rules
  • Total rules => 1604

This is a configuration nightmare if managed manually. However this is powered by Chef, so we never have to hand edit the rules. It’s not pretty, but it works.

Performance

We started noticing a decent amount of CPU time being spent in iptables as seen by perf top, about 10-30% depending on the load.

In addition, some (~0.01%) requests would take longer than usual.

The demo setup later demonstrates the issue via a load test.

Enter IPSet

From Linux Journal

ipset is an extension to iptables that allows you to create firewall rules that match entire “sets” of addresses at once. Unlike normal iptables chains, which are stored and traversed linearly, IP sets are stored in indexed data structures, making lookups very efficient, even when dealing with large sets.

Essentially, instead of enumerating each source IP, define an ipset which contains all the IPs. Then have an IPTable rule which references this set. This reduces lookup from O(n) to O(1).

Using ipset was not that difficult. The chef code was modified slightly and off it went to production.

End Result

  • 1 set containing all the IPs
  • 4 Accept Rules (1 per port)
  • 4 Reject Rules (1 per port)
  • Almost negligible CPU overhead
  • No more long tail latencies

The ‘Fun’ part

Using one of my favorite tools this can easily be demo’ed. Here’s a chef cookbook which will setup two servers locally.

.kitchen.yml defines the servers + network

  • Centos 6.7 with 2.6.32-573.el6.x86_64 kernel
  • iptables-1.4.7-16.el6.x86_64
  • ipset-6.11-4.el6.x86_64

Destination (dst)

  • code
  • runs a http server written in Go-Lang using Runit
  • http_server can sleep a desired amount of time based on a get parameter
  • loads IPTables or IPSet configuration based on a flag

Source (src)

  • code
  • installs Vegeta to generate the load
  • I tried ‘ab’ but wasn’t able to get high-enough throughput
  • sudo bash /tmp/run to launch the test script
  • The test involves POSTing a file of a fixed size to the endpoint
  • The goal is to have 750 req/sec concurrently

Testing with curl

[vagrant@src1 ~]$ curl http://172.28.3.10:45000/hello?sleep_ms=100 -v

Run #1 - no IP Filtering on the destination server

[root@dst1 ~]# iptables --flush
[root@dst1 ~]# ipset --flush

Vegeta Report

Notice the 95th, 99th percentile and max latencies, we will come back to these later.

Perf top on destination

Number of sockets in use on destination

Run #2 - 1600+ IPTable rules on destination server

[root@dst1 ~]# iptables -L -n | wc -l
1616

Vegeta Report

There’s a definite uptick in 95th and 99th percentile latencies. We even had a timeout! The test can always be run for a much longer duration as well as multiple cycles to cater for outliers.

Perf top on destination

We are spending a fair amount of CPU cycles dealing with IPTables. In the next step, we replace these 1600 rules with a handful using IPSet.

Run #3 - IPset rules on destination server

Vegeta Report

No more timeouts. Latencies look similar to Run #1 without IP Filtering

Perf top on destination

That’s right, ipt_do_table doesn’t even show up!

Conclusion

IPSet can help simplify IPTables configuration as well boost performance under high load. A demo cookbook sets up two servers and performs a load test to illustrate the issue.