Posts about puppet

Discovering Salt Stack

I've been a pretty stalwart Puppet user since I first discovered it in 2009. At that time, my choices were, as I saw them, between the brand-new cfengine3, the I've-seen-how-the-sausage-is-made bcfg2, and Puppet. Of those choices, Puppet seemed like the best choice.

In particular, I liked Puppet's "defined state" style of configuration management, and how simple it was to describe dependencies between the various packages, files, and services to be configured.

Like I said, I've been using Puppet happily for the past 4 years; but now, I think I've been swayed by Salt Stack.

I know I looked at salt stack before; but, at the time, I think I dismissed it as just "remote execution." Salt does, after all, start from a very different place than Puppet. At its most simple, it is a mechanism for shipping Python functions to remote nodes and executing them. It seemed the very opposite of the idempotent state management that I was looking for.

But now that I've taken the time to look deeper into the documentation (or, perhaps, now that the project has grown further) I've found Salt Stack States: the state enforcement configuration management system I was looking for; and with a trivial-to-setup remote execution layer underneath it.

Salt is based on 0MQ. I don't know much about message queues; but I do know that I could never get ActiveMQ working for use with Puppet's MCollective. After only 30 minutes of hacking, I had Salt, with 0MQ, running on two OS X machines and two Debian machines, all taking to the same master, each from behind its own form of inconveniently private network.

$ sudo salt '*' test.ping
ln1.civilfritz.net:
    True
Jonathons-MacBook-Pro.local:
    True
numfar.civilfritz.net:
    True
dabade.civilfritz.net:
    True

Glorious.

Some other things that I like about Salt:

  • States are defined in YAML, so there's no proprietary (cough poorly defined cough) language to maintain.
  • The remote execution layer and state module layer help keep executable code separate from state definitions.
  • Key management is a bit less foolish. (It shows you what you're about to sign before you sign it.)

Of course, no new technology arrives without the pain of a legacy conversion. I have a lot of time and effort invested into the Puppet manifests that drive ln1.civilfritz.net; but converting them to Salt Stack States is serving as a pretty good exercise for evaluating whether I really prefer Salt to Puppet.

I've already discovered a few things I don't like, of course:

  • The abstraction of the underlying Python implementation is a bit thin. This is sometimes a good thing, as it's easier to see how a state definition maps to individual function calls; but it also means that error messages sometimes require an understanding of Python. Sometimes you even get full tracebacks.
  • Defined states don't seem to understand the correlation between uid and uidNumber. In Puppet I started specifying group ownership as 0 when I discovered that AIX uses the gid system rather than root. In Salt, this appears to try to reassign the group ownership every time.
  • All hosts in a Salt config have access to all of the files in the master.
  • YAML formatting can be a bit wonky. (Why are arguments lists of dictionaries? Why is the function being called in the same list as its arguments?)
  • No good firewall (iptables) configuration support. The iptables module isn't even present in the version of Salt I have; but the documentation warns that even it is likely to be deprecated in the future.

That said, I can't ignore the fact that, since Salt happens to be written in Python, I might actually be able to contribute to this project. I've already done some grepping around in the source code, and it seems immediately approachable. Enhancing the roots fileserver, for example, to provide node-restricted access to files, shouldn't be too bad. I might even be able to port Puppet Lab's firewall module from Ruby to Python for use as a set of Salt modules.

Time will tell, I suppose. For now, the migration continues.

Introducing civilfritz Minecraft

I started playing Minecraft with my brother and old college roommate a few weeks ago. My expectations have been proven correct, as I've found it much more compelling to play on a persistent server with a group of real-life friends. In fact, in the context of my personal dedicated server instance, I'm finding the game strikes a compelling chord between my gamer side and my sysadmin side.

There's already some documentation for running a Minecraft server on the Minecraft wiki, but none of it was really in keeping with how I like to administer a server. I don't want to run services in a screen session, even if an init script sets it up for me.

I wrote my own Debian init script that uses start-stop-daemon and named pipes to allow server commands. Beyond that, I made a Puppet module that can install and configure the server. You can clone it from Git at git://civilfritz.net/puppet-minecraft.git.

I also really like maps, so I started looking for software that would let me generate maps of the world. (I was almost pacified when I learned how to craft maps. Almost.) I eventually settled on Minecraft Overviewer, mostly because it seems to be the most polished implementation. They even provide a Debian repository, so I didn't have to do anything special to install it.

I've configured Minecraft Overviewer to update the render once a day (at 04:00 EST, which hopefully won't conflict with actual Minecraft server use), with annotations updated once an hour. You can see it at http://civilfritz.net/minecraft/overview.

I couldn't get Overviewer to display over https for some reason I don't understand yet; so all access is redirected back at http for now.

Why I'm abandoning strict Allman style in Puppet manifests

I pretty much always use Allman style in languages that have braces. I like the symmetry, and the visible separation of identifier from value.

Though Allman style has its roots in C, the only brace language I use these days is Puppet. (Python end-runs around this whole issue by omitting braces altogether, which I ultimately prefer.) Pedantic as I am, my choice of brace style has extended (as closely as I could) to writing Puppet manifests.

class motd

(
  $content = undef
)

{
  file
  { '/etc/motd':
    content => $content,
    owner   => '0',
    group   => '0',
    mode    => '0644',
  }
}

This isn't what most people do, and it's certainly not what the examples in the Puppet style guide do; but it's also not in violation of any of the recommendations in the style guide.

I've been doing this for years, now; but today, I had one of those "aha" moments where I pleasantly realized that I've been doing it wrong.

Allman style works just fine for Puppet class definition; but Puppet resources provide their titles within the braces, rather than outside. This supports the compression of multiple resources into a single declaration.

file {
  '/tmp/a':
    content => 'a';
  '/tmp/b':
    content => 'b';
}

This syntax is explicitly discouraged in the style guide, but it's part of the language's legacy.

The problem with Allman style in this context is that is separates the resource title from the resource type. In most braced languages, the title of an element is written outside of the braces, after the type.

1
2
3
4
5
6
#! /bin/bash

function main
{
    # ...
}

In this example, it would be easy to grep a pile of Bash source files for scripts that declare a main function.

$ grep 'function main' *.sh

Not so with Allman style. I can grep for /etc/motd; but that would match against any reference to the file. Finding the declaration itself becomes a manual exercise with a contextual grep (grep --before-context 1).

All of this becomes much simpler, however, if resource declarations include the resource title (and the interstitial brace) on the same line as the resource type.

class motd

(
  $content = undef
)

{
  file { '/etc/motd':
    content => $content,
    owner   => '0',
    group   => '0',
    mode    => '0644',
  }
}

Even I have to admit that grep "file { '/etc/motd':" *.pp is much simpler.

This is immaterial for class declarations, since the class name is located before the brace.

class motd
{
  ...
}

I'd argue that Puppet should at least support a similar syntax for resources; one that puts the title directly after the type.

file '/etc/motd'
{
  ...
}

That could get a bit confusing, though, when using parameterized classes, as a parameterized class application syntax is somewhat close to regular class definition syntax.

# definition
class motd
{
  # ...
}

# declaration
class motd
{
  content => 'Hello, world!',
}

In which Puppet broke DNS queries

So there I was, idly doing some personal sysadmin on my stateside virtual machine, when I decided, "It's been a while since I've done a dist-upgrade. I should look at the out-of-date packages."

apt-get update, and immediately I noticed a problem. It was kind of obvious, what with it hanging all over my shell and all.

# apt-get update
0% [Connecting to ftp.us.debian.org] [Connecting to security.debian.org] [Connecting to apt.puppetlabs.com]^C

I quickly tracked a few symptoms back to DNS. I couldn't get a response off of my DNS servers.

$ host google.com
;; connection timed out; no servers could be reached

I use Linode for vps hosting, and they provide a series of resolving name servers for customer use. It seemed apparent to me that the problem wasn't on my end--I could still ssh into the box, and access my webserver from the Internet--so I contacted support about their DNS service not working.

Support was immediately responsive; but, after confirming my vm location, they reported that the nameservers seemed to be working correctly. Further, the tech pointed out that he couldn't ping my box.

No ping? Nope. So I can't DNS out, and I can't ICMP at all. "I have just rewritten my iptables config, so it's possible enough that I've screwed something up, there; but, with a default policy of ACCEPT on OUTPUT, I don't know what I could have done there to affect this." I passed my new config along, hoping that the tech would see somthing obvious that I had missed. He admitted that it all looked normal, but that, in the end, he can't support the configuration I put on my vm.

"For more hands on help with your iptables rules you may want to reach out to the Linode community."

Boo. Time to take a step back.

I use Puppet to manage iptables. More specifically, until recently, I have been using bobsh/iptables, a Puppet module that models individual rules with a native Puppet type.

iptables
{ 'http':
  state => 'NEW',
  proto => 'tcp',
  dport => '80',
  jump  => 'ACCEPT',
}

There's a newer, more official module out now, though: puppetlabs/firewall. This module basically does the same thing as bobsh/iptables, but it's maintained by Puppet Labs and is positioned for eventual portability to other firewall systems. Plus, whereas bobsh/iptables concatenates all of its known rules and then replaces any existing configuration, puppetlabs/firewall manages the tables in-place, allowing other systems (e.g., fail2ban) to add rules out-of-band without conflict.

In other words: new hotness.

The porting effort was pretty minimal. Soon, I had replaced all of my rules with the new format.

firewall
{ '100 http':
  state  => 'NEW',
  proto  => 'tcp',
  dport  => '80',
  action => 'accept',
}

Not that different. I'm using lexical ordering prefixes now, but I could have done that before. The big win, though, is the replacement of the pre and post fixtures with now-explicit pervasive rule order.

file
{ '/etc/puppet/iptables/pre.iptables':
  source => 'puppet:///modules/s_iptables/pre.iptables',
  owner  => 'root',
  group  => 'root',
  mode   => '0600',
}

file
{ '/etc/puppet/iptables/post.iptables':
  source => 'puppet:///modules/s_iptables/post.iptables',
  owner  => 'root',
  group  => 'root',
  mode   => '0600',
}

See, bobsh/iptables uses a pair of flat files to define a static set of rule fixtures that should always be present.

# pre.iptables

-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# post.iptables

-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

So, in my fixtures, loopback connections were always ACCEPTed, as were any existing connections flagged by connection tracking. Everything else (that isn't allowed by a rule between these fixtures) is REJECTed. This works well enough, but the flat files are a bit of a hack.

firewall
{ '000 accept localhost':
  iniface => 'lo',
  action  => 'accept',
}

firewall
{ '000 accept tracked connections':
  state  => ['RELATED', 'ESTABLISHED'],
  action => 'accept',
}

firewall
{ '999 default deny (input)':
  proto  => 'all',
  action => 'reject',
}

firewall
{ '999 default deny (forward)':
  chain  => 'FORWARD',
  proto  => 'all',
  action => 'reject',
  reject => 'icmp-host-prohibited',
}

That's much nicer. (I think so, anyway.) Definitely more flexible.

Anyway: I spent thirty minutes or so porting my existing rules over to puppetlabs/firewall, with no real problems to speak of. Until, of course, I realize I can't query DNS.

What could have possibly changed? The new configuration is basically one-to-one with the old configuration.

:INPUT ACCEPT [0:0]
[...]
-A INPUT -p tcp -m comment --comment "000 accept tracked connections" -m state --state RELATED,ESTABLISHED -j ACCEPT 
[...]
-A INPUT -m comment --comment "900 default deny (input)" -j REJECT --reject-with icmp-port-unreachable

Oh.

So, it turns out that puppetlabs/firewall has default values. In particular, proto defaults to tcp. That's probably the reasonably pervasive common case, but it was surprising. End result? That little -p tcp in my connection tracking rule means that icmp, udp, and anything else other than tcp can't establish real connections. The udp response from the DNS server doesn't get picked up, so it's rejected at the end.

The fix: explicitly specifying proto => 'all'.

firewall
{ '000 accept tracked connections':
  state  => ['RELATED', 'ESTABLISHED'],
  proto  => 'all',
  action => 'accept',
}

Alternatively, I could reconfigure the default; but it's fair enought that, as a result, I'd have to explicitly spectify tcp for the majority of my rules. That's a lot more verbose in the end.

Firewall
{
  proto => 'all',
}

Once again, all is right with the world (or, at least, with RELATED and ESTABLISHED udp and icmp packets).