The curc::sysconfig::scinet Puppet module
I’ve been working on a new module, curc::sysconfig::scinet, which
will generally do the Right Thing™ when configuring a host on the CURC
science network, with as little configuration as possible.
Let’s look at some examples.
login nodes
class { 'curc::sysconfig::scinet':
location => 'comp',
mgt_if => 'eth0',
dmz_if => 'eth1',
notify => Class['network'],
}
This is the config used on a new-style login node like login05 and
login07. (What makes them new-style? Mostly just that they’ve had
their interfaces cleaned up to use eth0 for “mgt” and eth1 for
“dmz”.)
Here’s the routing table that this produced on login07:
$ ip route list 10.225.160.0/24 dev eth0 proto kernel scope link src 10.225.160.32 10.225.128.0/24 via 10.225.160.1 dev eth0 192.12.246.0/24 dev eth1 proto kernel scope link src 192.12.246.39 10.225.0.0/20 via 10.225.160.1 dev eth0 10.225.0.0/16 via 10.225.160.1 dev eth0 metric 110 10.128.0.0/12 via 10.225.160.1 dev eth0 metric 110 default via 192.12.246.1 dev eth1 metric 100 default via 10.225.160.1 dev eth0 metric 110
Connections to “mgt” subnets use the “mgt” interface eth0, either by
the link-local route or the static routes via comp-mgt-gw
(10.225.160.1). Connections to the “general” subnet (a.k.a. “vlan
2049”), as well as the rest of the science network (“data” and “svc”
networks) also use eth0 by static route. The default eth0 route
is configured by DHCP, but the interface has a default metric of 110, so
it doesn’t conflict with or supersede eth1’s default route, which
is configured with a lower metric of 100.
Speaking of eth1, the “dmz” interface is configured statically,
using information retrieved from DNS by Puppet.
$ cat /etc/sysconfig/network-scripts/ifcfg-eth1 TYPE=Ethernet DEVICE=eth1 BOOTPROTO=static HWADDR=00:50:56:88:2E:36 ONBOOT=yes IPADDR=192.12.246.39 NETMASK=255.255.255.0 GATEWAY=192.12.246.1 METRIC=100 IPV4_ROUTE_METRIC=100
Usually the routing priority of the “dmz” interface would mean that
inbound connections to the “mgt” interface from outside of the science
network would be blocked when the “dmz”-bound response is filtered by
rp_filter; but curc::sysconfig::scinet also configures routing
policy for eth0, so traffic on that interface always returns from
that interface.
$ ip rule show | grep 'lookup 1' 32764: from 10.225.160.32 lookup 1 32765: from all iif eth0 lookup 1 $ ip route list table 1 default via 10.225.160.1 dev eth0
This allows me to ping login07.rc.int.colorado.edu from my office
workstation.
$ ping -c 1 login07.rc.int.colorado.edu PING login07.rc.int.colorado.edu (10.225.160.32) 56(84) bytes of data. 64 bytes from 10.225.160.32: icmp_seq=1 ttl=62 time=0.507 ms --- login07.rc.int.colorado.edu ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 1ms rtt min/avg/max/mdev = 0.507/0.507/0.507/0.000 ms
Because the default route for eth0 is actually configured, outbound
routing from login07 is resilient to failure of the “dmz” link.
# ip route list | grep -v eth1 10.225.160.0/24 dev eth0 proto kernel scope link src 10.225.160.32 10.225.128.0/24 via 10.225.160.1 dev eth0 10.225.0.0/20 via 10.225.160.1 dev eth0 10.225.0.0/16 via 10.225.160.1 dev eth0 metric 110 10.128.0.0/12 via 10.225.160.1 dev eth0 metric 110 default via 10.225.160.1 dev eth0 metric 110
Traffic destined to leave the science network simply proceeds to the
next preferred (and, in this case, only remaining) default route,
comp-mgt-gw.
DHCP, DNS, and the FQDN
Tangentially, it’s important to note that the DHCP configuration of
eth0 will tend to re-wite /etc/resolv.conf and the search
path it defines, with the effect of causing the FQDN of the host to
change to login07.rc.int.colorado.edu. Because login nodes are
logically (and historically) external hosts, not internal hosts, they
should prefer their external identity to their internal identity. As
such, we override the domain search path on login nodes to cause them to
discover their rc.colorado.edu FQDN’s first.
# cat /etc/dhcp/dhclient-eth0.conf supersede domain-search "rc.colorado.edu", "rc.int.colorado.edu";
PetaLibrary/repl
The Petibrary/repl GPFS NSD nodes replnsd{01,02} are still in the
“COMP” datacenter, but only attach to “mgt” and “data” networks.
class { 'curc::sysconfig::scinet':
location => 'comp',
mgt_if => 'eno2',
data_if => 'enp17s0f0',
other_data_rules => [ 'from 10.225.176.61 table 2',
'from 10.225.176.62 table 2',
],
notify => Class['network_manager::service'],
}
This config produces the following routing table on replnsd01…
$ ip route list default via 10.225.160.1 dev eno2 proto static metric 110 default via 10.225.176.1 dev enp17s0f0 proto static metric 120 10.128.0.0/12 via 10.225.160.1 dev eno2 metric 110 10.128.0.0/12 via 10.225.176.1 dev enp17s0f0 metric 120 10.225.0.0/20 via 10.225.160.1 dev eno2 10.225.0.0/16 via 10.225.160.1 dev eno2 metric 110 10.225.0.0/16 via 10.225.176.1 dev enp17s0f0 metric 120 10.225.64.0/20 via 10.225.176.1 dev enp17s0f0 10.225.128.0/24 via 10.225.160.1 dev eno2 10.225.144.0/24 via 10.225.176.1 dev enp17s0f0 10.225.160.0/24 dev eno2 proto kernel scope link src 10.225.160.59 metric 110 10.225.160.49 via 10.225.176.1 dev enp17s0f0 proto dhcp metric 120 10.225.176.0/24 dev enp17s0f0 proto kernel scope link src 10.225.176.59 metric 120
…with the expected interface-consistent policy-targeted routing tables.
$ ip route list table 1 default via 10.225.160.1 dev eno2 $ ip route list table 2 default via 10.225.176.1 dev enp17s0f0
Static routes for “mgt” and “data” subnets are defined for their
respective interfaces. As on the login nodes above, default routes are
specified for both interfaces as well, with the lower-metric “mgt”
interface eno2 being preferred. (This is configurable using the
mgt_metric and data_metric parameters.)
Perhaps the most notable aspect of the PetaLibrary/repl network config
is the provisioning of the GPFS CES floating IP addresses
10.225.176.{61,62}. These addresses are added to the enp17s0f0
interface dynamically by GPFS, and are not defined with
curc::sysconfig::scinet; but the config must reference these
addresses to implement proper interface-consistent policy-targeted
routing tables. Though version of Puppet deployed at CURC lacks the
semantics to infer these rules from a more semantic data_ip
parameter; so the other_data_rules parameter is used in stead.
other_data_rules => [ 'from 10.225.176.61 table 2',
'from 10.225.176.62 table 2',
],
Blanca/ICS login node
porting the blanca login node would be great because it’s got a “dmz”, “mgt”, and “data” interface; so it would exercise the full gamut of features of the module.