So there I was, idly doing some personal sysadmin on my stateside
virtual machine, when I decided, “It’s been a while since I’ve done a
dist-upgrade. I should look at the out-of-date packages.”
apt-get update, and immediately I noticed a problem. It was kind of
obvious, what with it hanging all over my shell and all.
# apt-get update
0% [Connecting to ftp.us.debian.org] [Connecting to security.debian.org] [Connecting to apt.puppetlabs.com]^C
I quickly tracked a few symptoms back to DNS. I couldn’t get a response
off of my DNS servers.
$ host google.com
;; connection timed out; no servers could be reached
I use
Linode
for vps hosting, and they provide a series of resolving name servers for
customer use. It seemed apparent to me that the problem wasn’t on my
end–I could still ssh into the box, and access my webserver from the
Internet–so I contacted support about their DNS service not working.
Support was immediately responsive; but, after confirming my vm
location, they reported that the nameservers seemed to be working
correctly. Further, the tech pointed out that he couldn’t ping my box.
No ping? Nope. So I can’t DNS out, and I can’t ICMP at all. “I have
just rewritten my iptables config, so it’s possible enough that I’ve
screwed something up, there; but, with a default policy of ACCEPT on
OUTPUT, I don’t know what I could have done there to affect this.” I
passed my new config along, hoping that the tech would see somthing
obvious that I had missed. He admitted that it all looked normal, but
that, in the end, he can’t support the configuration I put on my vm.
“For more hands on help with your iptables rules you may want to reach
out to the Linode community.”
Boo. Time to take a step back.
I use Puppet to manage iptables. More
specifically, until recently, I have been using
`bobsh/iptables <http://forge.puppetlabs.com/bobsh/iptables>`__, a
Puppet module that models individual rules with a native Puppet type.
iptables
{ 'http':
state => 'NEW',
proto => 'tcp',
dport => '80',
jump => 'ACCEPT',
}
There’s a newer, more official module out now, though:
`puppetlabs/firewall <http://forge.puppetlabs.com/puppetlabs/firewall>`__.
This module basically does the same thing as bobsh/iptables, but
it’s maintained by Puppet Labs and is positioned for eventual
portability to other firewall systems. Plus, whereas bobsh/iptables
concatenates all of its known rules and then replaces any existing
configuration, puppetlabs/firewall manages the tables in-place,
allowing other systems (e.g., fail2ban) to add rules out-of-band
without conflict.
In other words: new hotness.
The porting effort was pretty minimal. Soon, I had replaced all of my
rules with the new format.
firewall
{ '100 http':
state => 'NEW',
proto => 'tcp',
dport => '80',
action => 'accept',
}
Not that different. I’m using lexical ordering prefixes now, but I could
have done that before. The big win, though, is the replacement of the
pre and post fixtures with now-explicit pervasive rule order.
file
{ '/etc/puppet/iptables/pre.iptables':
source => 'puppet:///modules/s_iptables/pre.iptables',
owner => 'root',
group => 'root',
mode => '0600',
}
file
{ '/etc/puppet/iptables/post.iptables':
source => 'puppet:///modules/s_iptables/post.iptables',
owner => 'root',
group => 'root',
mode => '0600',
}
See, bobsh/iptables uses a pair of flat files to define a static set
of rule fixtures that should always be present.
# pre.iptables
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# post.iptables
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
So, in my fixtures, loopback connections were always ACCEPTed, as
were any existing connections flagged by connection tracking. Everything
else (that isn’t allowed by a rule between these fixtures) is REJECTed.
This works well enough, but the flat files are a bit of a hack.
firewall
{ '000 accept localhost':
iniface => 'lo',
action => 'accept',
}
firewall
{ '000 accept tracked connections':
state => ['RELATED', 'ESTABLISHED'],
action => 'accept',
}
firewall
{ '999 default deny (input)':
proto => 'all',
action => 'reject',
}
firewall
{ '999 default deny (forward)':
chain => 'FORWARD',
proto => 'all',
action => 'reject',
reject => 'icmp-host-prohibited',
}
That’s much nicer. (I think so, anyway.) Definitely more flexible.
Anyway: I spent thirty minutes or so porting my existing rules over to
puppetlabs/firewall, with no real problems to speak of. Until, of
course, I realize I can’t query DNS.
What could have possibly changed? The new configuration is basically
one-to-one with the old configuration.
:INPUT ACCEPT [0:0]
[...]
-A INPUT -p tcp -m comment --comment "000 accept tracked connections" -m state --state RELATED,ESTABLISHED -j ACCEPT
[...]
-A INPUT -m comment --comment "900 default deny (input)" -j REJECT --reject-with icmp-port-unreachable
Oh.
So, it turns out that puppetlabs/firewall has default values. In
particular, proto defaults to tcp. That’s probably the
reasonably pervasive common case, but it was surprising. End result?
That little -p tcp in my connection tracking rule means that
icmp, udp, and anything else other than tcp can’t establish
real connections. The udp response from the DNS server doesn’t get
picked up, so it’s rejected at the end.
The fix: explicitly specifying proto => 'all'.
firewall
{ '000 accept tracked connections':
state => ['RELATED', 'ESTABLISHED'],
proto => 'all',
action => 'accept',
}
Alternatively, I could reconfigure the default; but it’s fair enought
that, as a result, I’d have to explicitly spectify tcp for the
majority of my rules. That’s a lot more verbose in the end.
Firewall
{
proto => 'all',
}
Once again, all is right with the world (or, at least, with RELATED and
ESTABLISHED udp and icmp packets).