Debugging IPTABLES rules with pkts and bytes

One of the most common problems with IPTABLES is misconfiguration of its rule sets. You can easily block “good” traffic that should be hitting your server. This typically manifests itself when a dependent service cannot talk to your server with IPTABLES running, but communicates just fine with IPTABLES disabled.

So how do you determine if one of your IPTABLES ACCEPT rules is causing the problem?  If you instruct IPTABLES to list its rules with the “iptables -nvL” command, you should be able to find your answer.  This command returns a list of all chains, but also includes two important columns: “pkts” and “bytes.”  The pkts, or packets, column indicates how many packets have passed through the chain, while the bytes column reveals the total number of bytes that have traversed it.

Here’s how it works in practice:

We recently enabled IPTABLES on two nodes of an Oracle RAC cluster.  As soon as IPTABLES was enabled, the second node would be evicted from the cluster, as the first node could no longer see it.  We surmised that IPTABLES was blocking the traffic, thereby cutting the server off.

To validate this, we ran the “iptables -nvL” command, and looked at the last line returned – the REJECT chain. This showed us a huge number of packets and bytes being rejected by IPTABLES.

pkts bytes target     prot opt in     out     source               destination
311M 571M REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

We also saw no mention of interface bond2 in the listing, which is the private interconnect interface for each RAC node. We therefore added an ACCEPT rule for that interface:

-A RH-Firewall-1-INPUT -i bond2 -j ACCEPT

We then restarted the IPTABLES service to pick up the change, and ran the “iptables -nvL |grep bond2” command to view the traffic:

pkts bytes target     prot opt in     out     source               destination
1032  217K ACCEPT     all  --  bond2  *       0.0.0.0/0            0.0.0.0/0

As you can see, IPTABLES was now processing traffic on the interface, as the pkts and bytes totals had advanced from initial zero values.  After two minutes, we re-ran the command:

pkts bytes target     prot opt in     out     source               destination
5373 9541K ACCEPT     all  --  bond2  *       0.0.0.0/0            0.0.0.0/0

Note the substantial increase in packets and bytes, indicating that our rule was, in fact, working. This was later validate up the application stack, as the RAC clusters could now see each other.  By simply looking at the traffic hitting each IPTABLES rule, we could clearly see where our problem was.