Posts about puppet

The curc::sysconfig::scinet Puppet module

I've been working on a new module, curc::sysconfig::scinet, which will generally do the Right Thing™ when configuring a host on the CURC science network, with as little configuration as possible.

Let's look at some examples.

login nodes

class { 'curc::sysconfig::scinet':
  location => 'comp',
  mgt_if   => 'eth0',
  dmz_if   => 'eth1',
  notify   => Class['network'],
}

This is the config used on a new-style login node like login05 and login07. (What makes them new-style? Mostly just that they've had their interfaces cleaned up to use eth0 for "mgt" and eth1 for "dmz".)

Here's the routing table that this produced on login07:

$ ip route list
10.225.160.0/24 dev eth0  proto kernel  scope link  src 10.225.160.32 
10.225.128.0/24 via 10.225.160.1 dev eth0 
192.12.246.0/24 dev eth1  proto kernel  scope link  src 192.12.246.39 
10.225.0.0/20 via 10.225.160.1 dev eth0 
10.225.0.0/16 via 10.225.160.1 dev eth0  metric 110 
10.128.0.0/12 via 10.225.160.1 dev eth0  metric 110 
default via 192.12.246.1 dev eth1  metric 100 
default via 10.225.160.1 dev eth0  metric 110

Connections to "mgt" subnets use the "mgt" interface eth0, either by the link-local route or the static routes via comp-mgt-gw (10.225.160.1). Connections to the "general" subnet (a.k.a. "vlan 2049"), as well as the rest of the science network ("data" and "svc" networks) also use eth0 by static route. The default eth0 route is configured by DHCP, but the interface has a default metric of 110, so it doesn't conflict with or supersede eth1's default route, which is configured with a lower metric of 100.

Speaking of eth1, the "dmz" interface is configured statically, using information retrieved from DNS by Puppet.

$ cat /etc/sysconfig/network-scripts/ifcfg-eth1 
TYPE=Ethernet
DEVICE=eth1
BOOTPROTO=static
HWADDR=00:50:56:88:2E:36
ONBOOT=yes
IPADDR=192.12.246.39
NETMASK=255.255.255.0
GATEWAY=192.12.246.1
METRIC=100
IPV4_ROUTE_METRIC=100

Usually the routing priority of the "dmz" interface would mean that inbound connections to the "mgt" interface from outside of the science network would be blocked when the "dmz"-bound response is filtered by rp_filter; but curc::sysconfig::scinet also configures routing policy for eth0, so traffic on that interface always returns from that interface.

$ ip rule show | grep 'lookup 1'
32764:  from 10.225.160.32 lookup 1 
32765:  from all iif eth0 lookup 1

$ ip route list table 1
default via 10.225.160.1 dev eth0

This allows me to ping login07.rc.int.colorado.edu from my office workstation.

$ ping -c 1 login07.rc.int.colorado.edu
PING login07.rc.int.colorado.edu (10.225.160.32) 56(84) bytes of data.
64 bytes from 10.225.160.32: icmp_seq=1 ttl=62 time=0.507 ms

--- login07.rc.int.colorado.edu ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.507/0.507/0.507/0.000 ms

Because the default route for eth0 is actually configured, outbound routing from login07 is resilient to failure of the "dmz" link.

# ip route list | grep -v eth1
10.225.160.0/24 dev eth0  proto kernel  scope link  src 10.225.160.32 
10.225.128.0/24 via 10.225.160.1 dev eth0 
10.225.0.0/20 via 10.225.160.1 dev eth0 
10.225.0.0/16 via 10.225.160.1 dev eth0  metric 110 
10.128.0.0/12 via 10.225.160.1 dev eth0  metric 110 
default via 10.225.160.1 dev eth0  metric 110

Traffic destined to leave the science network simply proceeds to the next preferred (and, in this case, only remaining) default route, comp-mgt-gw.

DHCP, DNS, and the FQDN

Tangentially, it's important to note that the DHCP configuration of eth0 will tend to re-wite /etc/resolv.conf and the search path it defines, with the effect of causing the FQDN of the host to change to login07.rc.int.colorado.edu. Because login nodes are logically (and historically) external hosts, not internal hosts, they should prefer their external identity to their internal identity. As such, we override the domain search path on login nodes to cause them to discover their rc.colorado.edu FQDN's first.

# cat /etc/dhcp/dhclient-eth0.conf 
supersede domain-search "rc.colorado.edu", "rc.int.colorado.edu";

PetaLibrary/repl

The Petibrary/repl GPFS NSD nodes replnsd{01,02} are still in the "COMP" datacenter, but only attach to "mgt" and "data" networks.

class { 'curc::sysconfig::scinet':
  location         => 'comp',
  mgt_if           => 'eno2',
  data_if          => 'enp17s0f0',
  other_data_rules => [ 'from 10.225.176.61 table 2',
                        'from 10.225.176.62 table 2',
                        ],
  notify           => Class['network_manager::service'],
}

This config produces the following routing table on replnsd01...

$ ip route list
default via 10.225.160.1 dev eno2  proto static  metric 110 
default via 10.225.176.1 dev enp17s0f0  proto static  metric 120 
10.128.0.0/12 via 10.225.160.1 dev eno2  metric 110 
10.128.0.0/12 via 10.225.176.1 dev enp17s0f0  metric 120 
10.225.0.0/20 via 10.225.160.1 dev eno2 
10.225.0.0/16 via 10.225.160.1 dev eno2  metric 110 
10.225.0.0/16 via 10.225.176.1 dev enp17s0f0  metric 120 
10.225.64.0/20 via 10.225.176.1 dev enp17s0f0 
10.225.128.0/24 via 10.225.160.1 dev eno2 
10.225.144.0/24 via 10.225.176.1 dev enp17s0f0 
10.225.160.0/24 dev eno2  proto kernel  scope link  src 10.225.160.59  metric 110 
10.225.160.49 via 10.225.176.1 dev enp17s0f0  proto dhcp  metric 120 
10.225.176.0/24 dev enp17s0f0  proto kernel  scope link  src 10.225.176.59  metric 120

...with the expected interface-consistent policy-targeted routing tables.

$ ip route list table 1
default via 10.225.160.1 dev eno2

$ ip route list table 2
default via 10.225.176.1 dev enp17s0f0

Static routes for "mgt" and "data" subnets are defined for their respective interfaces. As on the login nodes above, default routes are specified for both interfaces as well, with the lower-metric "mgt" interface eno2 being preferred. (This is configurable using the mgt_metric and data_metric parameters.)

Perhaps the most notable aspect of the PetaLibrary/repl network config is the provisioning of the GPFS CES floating IP addresses 10.225.176.{61,62}. These addresses are added to the enp17s0f0 interface dynamically by GPFS, and are not defined with curc::sysconfig::scinet; but the config must reference these addresses to implement proper interface-consistent policy-targeted routing tables. Though version of Puppet deployed at CURC lacks the semantics to infer these rules from a more semantic data_ip parameter; so the other_data_rules parameter is used in stead.

  other_data_rules => [ 'from 10.225.176.61 table 2',
                        'from 10.225.176.62 table 2',
                        ],

Blanca/ICS login node

[porting the blanca login node would be great because it's got a "dmz", "mgt", and "data" interface; so it would exercise the full gamut of features of the module]

Discovering Salt Stack

I've been a pretty stalwart Puppet user since I first discovered it in 2009. At that time, my choices were, as I saw them, between the brand-new cfengine3, the I've-seen-how-the-sausage-is-made bcfg2, and Puppet. Of those choices, Puppet seemed like the best choice.

In particular, I liked Puppet's "defined state" style of configuration management, and how simple it was to describe dependencies between the various packages, files, and services to be configured.

Like I said, I've been using Puppet happily for the past 4 years; but now, I think I've been swayed by Salt Stack.

I know I looked at salt stack before; but, at the time, I think I dismissed it as just "remote execution." Salt does, after all, start from a very different place than Puppet. At its most simple, it is a mechanism for shipping Python functions to remote nodes and executing them. It seemed the very opposite of the idempotent state management that I was looking for.

But now that I've taken the time to look deeper into the documentation (or, perhaps, now that the project has grown further) I've found Salt Stack States: the state enforcement configuration management system I was looking for; and with a trivial-to-setup remote execution layer underneath it.

Salt is based on 0MQ. I don't know much about message queues; but I do know that I could never get ActiveMQ working for use with Puppet's MCollective. After only 30 minutes of hacking, I had Salt, with 0MQ, running on two OS X machines and two Debian machines, all taking to the same master, each from behind its own form of inconveniently private network.

$ sudo salt '*' test.ping
ln1.civilfritz.net:
    True
Jonathons-MacBook-Pro.local:
    True
numfar.civilfritz.net:
    True
dabade.civilfritz.net:
    True

Glorious.

Some other things that I like about Salt:

  • States are defined in YAML, so there's no proprietary (cough poorly defined cough) language to maintain.
  • The remote execution layer and state module layer help keep executable code separate from state definitions.
  • Key management is a bit less foolish. (It shows you what you're about to sign before you sign it.)

Of course, no new technology arrives without the pain of a legacy conversion. I have a lot of time and effort invested into the Puppet manifests that drive ln1.civilfritz.net; but converting them to Salt Stack States is serving as a pretty good exercise for evaluating whether I really prefer Salt to Puppet.

I've already discovered a few things I don't like, of course:

  • The abstraction of the underlying Python implementation is a bit thin. This is sometimes a good thing, as it's easier to see how a state definition maps to individual function calls; but it also means that error messages sometimes require an understanding of Python. Sometimes you even get full tracebacks.
  • Defined states don't seem to understand the correlation between uid and uidNumber. In Puppet I started specifying group ownership as 0 when I discovered that AIX uses the gid system rather than root. In Salt, this appears to try to reassign the group ownership every time.
  • All hosts in a Salt config have access to all of the files in the master.
  • YAML formatting can be a bit wonky. (Why are arguments lists of dictionaries? Why is the function being called in the same list as its arguments?)
  • No good firewall (iptables) configuration support. The iptables module isn't even present in the version of Salt I have; but the documentation warns that even it is likely to be deprecated in the future.

That said, I can't ignore the fact that, since Salt happens to be written in Python, I might actually be able to contribute to this project. I've already done some grepping around in the source code, and it seems immediately approachable. Enhancing the roots fileserver, for example, to provide node-restricted access to files, shouldn't be too bad. I might even be able to port Puppet Lab's firewall module from Ruby to Python for use as a set of Salt modules.

Time will tell, I suppose. For now, the migration continues.

Introducing civilfritz Minecraft

I started playing Minecraft with my brother and old college roommate a few weeks ago. My expectations have been proven correct, as I've found it much more compelling to play on a persistent server with a group of real-life friends. In fact, in the context of my personal dedicated server instance, I'm finding the game strikes a compelling chord between my gamer side and my sysadmin side.

There's already some documentation for running a Minecraft server on the Minecraft wiki, but none of it was really in keeping with how I like to administer a server. I don't want to run services in a screen session, even if an init script sets it up for me.

I wrote my own Debian init script that uses start-stop-daemon and named pipes to allow server commands. Beyond that, I made a Puppet module that can install and configure the server. You can clone it from Git at git://civilfritz.net/puppet-minecraft.git.

I also really like maps, so I started looking for software that would let me generate maps of the world. (I was almost pacified when I learned how to craft maps. Almost.) I eventually settled on Minecraft Overviewer, mostly because it seems to be the most polished implementation. They even provide a Debian repository, so I didn't have to do anything special to install it.

I've configured Minecraft Overviewer to update the render once a day (at 04:00 EST, which hopefully won't conflict with actual Minecraft server use), with annotations updated once an hour. You can see it at http://civilfritz.net/minecraft/overview.

I couldn't get Overviewer to display over https for some reason I don't understand yet; so all access is redirected back at http for now.

Why I'm abandoning strict Allman style in Puppet manifests

I pretty much always use Allman style in languages that have braces. I like the symmetry, and the visible separation of identifier from value.

Though Allman style has its roots in C, the only brace language I use these days is Puppet. (Python end-runs around this whole issue by omitting braces altogether, which I ultimately prefer.) Pedantic as I am, my choice of brace style has extended (as closely as I could) to writing Puppet manifests.

class motd

(
  $content = undef
)

{
  file
  { '/etc/motd':
    content => $content,
    owner   => '0',
    group   => '0',
    mode    => '0644',
  }
}

This isn't what most people do, and it's certainly not what the examples in the Puppet style guide do; but it's also not in violation of any of the recommendations in the style guide.

I've been doing this for years, now; but today, I had one of those "aha" moments where I pleasantly realized that I've been doing it wrong.

Allman style works just fine for Puppet class definition; but Puppet resources provide their titles within the braces, rather than outside. This supports the compression of multiple resources into a single declaration.

file {
  '/tmp/a':
    content => 'a';
  '/tmp/b':
    content => 'b';
}

This syntax is explicitly discouraged in the style guide, but it's part of the language's legacy.

The problem with Allman style in this context is that is separates the resource title from the resource type. In most braced languages, the title of an element is written outside of the braces, after the type.

1
2
3
4
5
6
#! /bin/bash

function main
{
    # ...
}

In this example, it would be easy to grep a pile of Bash source files for scripts that declare a main function.

$ grep 'function main' *.sh

Not so with Allman style. I can grep for /etc/motd; but that would match against any reference to the file. Finding the declaration itself becomes a manual exercise with a contextual grep (grep --before-context 1).

All of this becomes much simpler, however, if resource declarations include the resource title (and the interstitial brace) on the same line as the resource type.

class motd

(
  $content = undef
)

{
  file { '/etc/motd':
    content => $content,
    owner   => '0',
    group   => '0',
    mode    => '0644',
  }
}

Even I have to admit that grep "file { '/etc/motd':" *.pp is much simpler.

This is immaterial for class declarations, since the class name is located before the brace.

class motd
{
  ...
}

I'd argue that Puppet should at least support a similar syntax for resources; one that puts the title directly after the type.

file '/etc/motd'
{
  ...
}

That could get a bit confusing, though, when using parameterized classes, as a parameterized class application syntax is somewhat close to regular class definition syntax.

# definition
class motd
{
  # ...
}

# declaration
class motd
{
  content => 'Hello, world!',
}