Posts about network

The curc::sysconfig::scinet Puppet module

I've been working on a new module, curc::sysconfig::scinet, which will generally do the Right Thing™ when configuring a host on the CURC science network, with as little configuration as possible.

Let's look at some examples.

login nodes

class { 'curc::sysconfig::scinet':
  location => 'comp',
  mgt_if   => 'eth0',
  dmz_if   => 'eth1',
  notify   => Class['network'],
}

This is the config used on a new-style login node like login05 and login07. (What makes them new-style? Mostly just that they've had their interfaces cleaned up to use eth0 for "mgt" and eth1 for "dmz".)

Here's the routing table that this produced on login07:

$ ip route list
10.225.160.0/24 dev eth0  proto kernel  scope link  src 10.225.160.32 
10.225.128.0/24 via 10.225.160.1 dev eth0 
192.12.246.0/24 dev eth1  proto kernel  scope link  src 192.12.246.39 
10.225.0.0/20 via 10.225.160.1 dev eth0 
10.225.0.0/16 via 10.225.160.1 dev eth0  metric 110 
10.128.0.0/12 via 10.225.160.1 dev eth0  metric 110 
default via 192.12.246.1 dev eth1  metric 100 
default via 10.225.160.1 dev eth0  metric 110

Connections to "mgt" subnets use the "mgt" interface eth0, either by the link-local route or the static routes via comp-mgt-gw (10.225.160.1). Connections to the "general" subnet (a.k.a. "vlan 2049"), as well as the rest of the science network ("data" and "svc" networks) also use eth0 by static route. The default eth0 route is configured by DHCP, but the interface has a default metric of 110, so it doesn't conflict with or supersede eth1's default route, which is configured with a lower metric of 100.

Speaking of eth1, the "dmz" interface is configured statically, using information retrieved from DNS by Puppet.

$ cat /etc/sysconfig/network-scripts/ifcfg-eth1 
TYPE=Ethernet
DEVICE=eth1
BOOTPROTO=static
HWADDR=00:50:56:88:2E:36
ONBOOT=yes
IPADDR=192.12.246.39
NETMASK=255.255.255.0
GATEWAY=192.12.246.1
METRIC=100
IPV4_ROUTE_METRIC=100

Usually the routing priority of the "dmz" interface would mean that inbound connections to the "mgt" interface from outside of the science network would be blocked when the "dmz"-bound response is filtered by rp_filter; but curc::sysconfig::scinet also configures routing policy for eth0, so traffic on that interface always returns from that interface.

$ ip rule show | grep 'lookup 1'
32764:  from 10.225.160.32 lookup 1 
32765:  from all iif eth0 lookup 1

$ ip route list table 1
default via 10.225.160.1 dev eth0

This allows me to ping login07.rc.int.colorado.edu from my office workstation.

$ ping -c 1 login07.rc.int.colorado.edu
PING login07.rc.int.colorado.edu (10.225.160.32) 56(84) bytes of data.
64 bytes from 10.225.160.32: icmp_seq=1 ttl=62 time=0.507 ms

--- login07.rc.int.colorado.edu ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.507/0.507/0.507/0.000 ms

Because the default route for eth0 is actually configured, outbound routing from login07 is resilient to failure of the "dmz" link.

# ip route list | grep -v eth1
10.225.160.0/24 dev eth0  proto kernel  scope link  src 10.225.160.32 
10.225.128.0/24 via 10.225.160.1 dev eth0 
10.225.0.0/20 via 10.225.160.1 dev eth0 
10.225.0.0/16 via 10.225.160.1 dev eth0  metric 110 
10.128.0.0/12 via 10.225.160.1 dev eth0  metric 110 
default via 10.225.160.1 dev eth0  metric 110

Traffic destined to leave the science network simply proceeds to the next preferred (and, in this case, only remaining) default route, comp-mgt-gw.

DHCP, DNS, and the FQDN

Tangentially, it's important to note that the DHCP configuration of eth0 will tend to re-wite /etc/resolv.conf and the search path it defines, with the effect of causing the FQDN of the host to change to login07.rc.int.colorado.edu. Because login nodes are logically (and historically) external hosts, not internal hosts, they should prefer their external identity to their internal identity. As such, we override the domain search path on login nodes to cause them to discover their rc.colorado.edu FQDN's first.

# cat /etc/dhcp/dhclient-eth0.conf 
supersede domain-search "rc.colorado.edu", "rc.int.colorado.edu";

PetaLibrary/repl

The Petibrary/repl GPFS NSD nodes replnsd{01,02} are still in the "COMP" datacenter, but only attach to "mgt" and "data" networks.

class { 'curc::sysconfig::scinet':
  location         => 'comp',
  mgt_if           => 'eno2',
  data_if          => 'enp17s0f0',
  other_data_rules => [ 'from 10.225.176.61 table 2',
                        'from 10.225.176.62 table 2',
                        ],
  notify           => Class['network_manager::service'],
}

This config produces the following routing table on replnsd01...

$ ip route list
default via 10.225.160.1 dev eno2  proto static  metric 110 
default via 10.225.176.1 dev enp17s0f0  proto static  metric 120 
10.128.0.0/12 via 10.225.160.1 dev eno2  metric 110 
10.128.0.0/12 via 10.225.176.1 dev enp17s0f0  metric 120 
10.225.0.0/20 via 10.225.160.1 dev eno2 
10.225.0.0/16 via 10.225.160.1 dev eno2  metric 110 
10.225.0.0/16 via 10.225.176.1 dev enp17s0f0  metric 120 
10.225.64.0/20 via 10.225.176.1 dev enp17s0f0 
10.225.128.0/24 via 10.225.160.1 dev eno2 
10.225.144.0/24 via 10.225.176.1 dev enp17s0f0 
10.225.160.0/24 dev eno2  proto kernel  scope link  src 10.225.160.59  metric 110 
10.225.160.49 via 10.225.176.1 dev enp17s0f0  proto dhcp  metric 120 
10.225.176.0/24 dev enp17s0f0  proto kernel  scope link  src 10.225.176.59  metric 120

...with the expected interface-consistent policy-targeted routing tables.

$ ip route list table 1
default via 10.225.160.1 dev eno2

$ ip route list table 2
default via 10.225.176.1 dev enp17s0f0

Static routes for "mgt" and "data" subnets are defined for their respective interfaces. As on the login nodes above, default routes are specified for both interfaces as well, with the lower-metric "mgt" interface eno2 being preferred. (This is configurable using the mgt_metric and data_metric parameters.)

Perhaps the most notable aspect of the PetaLibrary/repl network config is the provisioning of the GPFS CES floating IP addresses 10.225.176.{61,62}. These addresses are added to the enp17s0f0 interface dynamically by GPFS, and are not defined with curc::sysconfig::scinet; but the config must reference these addresses to implement proper interface-consistent policy-targeted routing tables. Though version of Puppet deployed at CURC lacks the semantics to infer these rules from a more semantic data_ip parameter; so the other_data_rules parameter is used in stead.

  other_data_rules => [ 'from 10.225.176.61 table 2',
                        'from 10.225.176.62 table 2',
                        ],

Blanca/ICS login node

[porting the blanca login node would be great because it's got a "dmz", "mgt", and "data" interface; so it would exercise the full gamut of features of the module]