Migrating from Apache to Nginx

I've had this hanging out as a draft for a while, and never got it polished up as well as I'd like; but in case it's useful to anyone, here's some notes on my recent migration to nginx.


I run civilfritz.net on a VPS, but I do my best to keep as low a monthly payment as possible. That means running on the smallest (lowest-memory) VM available from my provider: a 1GB Linode.

1GB used to seem like a lot of memory; but when I'm trying to run a Minecraft server alongside a preforking Apache server alongside a Salt master, it fills up quickly.

I've wanted to try moving to a lighter-weight webserver for a while; so today I'm porting my Apache config to Nginx.

sites-available/civilfritz.net

civilfritz.net runs as a pair of Apache virtual hosts to support http and https. I want the majority of the configuration between the vhosts to be identical, so I include a separate common configuration file in each.

The http vhost includes the common config, as well as a rewrite for the ikiwiki /auth section. (Authentication should only happen over https, but attempts to authenticate over http should be redirected there.)

# apache http vhost

<VirtualHost *:80>
    RewriteEngine on
    RewriteRule ^/auth(|/.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [R,L]

    Include sites-available/civilfritz.net-common
</VirtualHost>

The transition to nginx was pretty simple. The ikiwiki /auth section is a virtually equivalent rewrite rule, and the include directive is also similar.

# nginx http vhost

server
{
        listen 80;

        rewrite ^/auth(|/.*)$ https://$server_name:443$request_uri? permanent;

        include sites-available/civilfritz.net-common;
}

The https vhost also includes the common config, as well as the requisite ssl config. To support http basic authentication, an instance of pwauth is configured as an external authentication module, which proxies to PAM.

# apache https vhost

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/civilfritz.net.pem
    SSLCertificateKeyFile /etc/ssl/private/civilfritz.net.key

    AddExternalAuth pwauth /usr/sbin/pwauth
    SetExternalAuthMethod pwauth pipe

    <Location />
        AuthType Basic
        AuthBasicProvider external
        AuthExternal pwauth
        AuthName "civilfritz.net"
    </Location>

    Include sites-available/civilfritz.net-common

    <Location /auth>
        Require valid-user
    </Location>
</VirtualHost>

Again, the nginx vhost starts out similarly. Listen on tcp 443, initialize the requisite certificate and key, and include the common config.

pwauth is an Apache-specific interface, so I wasn't able to use it to proxy to pam in nginx; but the auth_pam module works well enough and, since I'm not trying to use PAM to auth directly against local unix files (I'm using sssd to access kerberos), I still don't have to run the server as root.

# nginx ssl vhost

server
{
        listen 443 ssl;

        ssl_certificate /etc/ssl/certs/civilfritz.net.pem;
        ssl_certificate_key /etc/ssl/private/civilfritz.net.key;

        include sites-available/civilfritz.net-common;

        location /auth
        {
                auth_pam "civilfritz.net";
                include fastcgi_params;
                fastcgi_pass unix:/var/run/fcgiwrap.socket;
                fastcgi_index ikiwiki.cgi;
                fastcgi_param REMOTE_USER $remote_user;
        }
}

The semantics of Nginx basic authentication differ from Apache. In Apache I was able to set AuthName globally (at /) and then require authentication arbitrarily at lower points in the tree. Here, the inclusion of the auth_pam directive implies an auth requirement; so I'll have to repeat the authentication realm ("civilfritz.net") anywhere I want to authenticate.

The biggest difference, though, is how Nginx handles cgi. Whereas Apache builds-in cgi execution for nominated files or directories, Nginx proxies all cgi execution through an external interface: here, fastcgi. A packed-in fastcgi_params file contains some useful default cgi environment variables, but omits REMOTE_USER. I set here so that ikiwiki can determine what user has authenticated.

sites-available/civilfritz.net-common

The vast majority of my local config is in the common file included by both vhosts.

# Apache initial config

ServerAdmin anderbubble@gmail.com
DirectoryIndex index.html
ServerName civilfritz.net
ServerAlias www.civilfritz.net

LogLevel warn
ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combined

DocumentRoot /srv/www/wiki

RewriteEngine on

Alias /robots.txt /srv/www/robots.txt

Alias /minecraft/overview /srv/www/minecraft-overviewer

<Location /users/janderson/private>
    Require user janderson
</Location>

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Directory /srv/www>
    Order allow,deny
    Allow from all
</Directory>

<Directory /srv/www/wiki>
    AddHandler cgi-script .cgi
    Order allow,deny
    Allow from all
    Options +ExecCGI
    ErrorDocument 404 /ikiwiki.cgi
    ExpiresActive on
    ExpiresDefault "access plus 0 seconds"
    Header set Cache-Control "no-store, no-cache, must-revalidate, max-age=0"
    Header set Pragma "no-cache"
</Directory>

<Location /gitweb>
    Order allow,deny
    Allow from all
    DirectoryIndex index.cgi
</Location>

<Directory /home/*/public_html/>
    AllowOverride FileInfo AuthConfig Limit Indexes Options=ExecCGI
</Directory>

WSGIApplicationGroup %{GLOBAL}

New Nginx config

sites-available/civilfritz.net-common

index index.html;
server_name civilfritz.net www.civilfritz.net;

root /srv/www/wiki/;

location /
{
        error_page 404 /ikiwiki-404.cgi;
        expires -1;
}

location /robots.txt
{
        alias /srv/www/robots.txt;
}

location /minecraft/overview
{
        alias /srv/www/minecraft-overviewer;
}

location /ikiwiki.cgi
{
        include fastcgi_params;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
        fastcgi_index ikiwiki.cgi;
}

location /ikiwiki-404.cgi
{
        internal;
        include fastcgi_params;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
        fastcgi_param REDIRECT_URL $request_uri;
        # also needed to remove explicit 200
        fastcgi_param REDIRECT_STATUS 404;
}

location ~ /gitweb/(index|gitweb).cgi
{
        root /usr/share/;
        gzip off;
        include fastcgi_params;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
}

location /gitweb/
{
        root /usr/share/;
        gzip off;
        index index.cgi;
}

location ~ ^/~(.+?)(/.*)?$
{
        alias /home/$1/public_html$2;
        autoindex on;
}

Salt state

nginx.sls

nginx:

  pkg:
    - installed

  service:
    - running
    - enable: True
    - reload: True
    - watch:
      - pkg: nginx

civilfritz/www.sls

include:
  - nginx

[...]

/etc/nginx/sites-enabled/default:
  file:
    - absent
    - watch_in:
      - service: nginx

/etc/nginx/sites-enabled/civilfritz.net:
  file:
    - symlink
    - target: /etc/nginx/sites-available/civilfritz.net
    - require:
      - file: /etc/nginx/sites-available/civilfritz.net
    - watch_in:
      - service: nginx

/etc/nginx/sites-available/civilfritz.net:
  file:
    - managed
    - source: salt://civilfritz/nginx-sites/civilfritz.net
    - user: root
    - group: root
    - mode: 0644
    - require:
      - file: /etc/nginx/sites-available/civilfritz.net-common
    - watch_in:
      - service: nginx

/etc/nginx/sites-available/civilfritz.net-common:
  file:
    - managed
    - source: salt://civilfritz/nginx-sites/civilfritz.net-common
    - user: root
    - group: root
    - mode: 0644
    - watch_in:
      - service: nginx

/srv/www/wiki/ikiwiki-404.cgi:
  file:
    - symlink
    - target: /srv/www/wiki/ikiwiki.cgi

Raspberry Pi and Arduino | Christmas 2013

I spent a fair amount of time this Christmas hacking around on the Raspberry Pi with my electrical-engineer brother-in-law. On top of that, he gave me an Arduino Uno Starter Pack for Christmas. So now we have a great foundation of common ground between our neighboring disciplines.

His original project goal for the Raspberry Pi was, "A webpage I can use from my phone to click a button and make lights turn on and off." With the help of Pylons Pyramid and an existing GPIO Python module I was able to make a rudimentary GPIO web interface. It allows you to designate pins as input or output, and read or set the state of each pin. You can even (thanks to Twitter's Bootstrap) visit it from your phone, and it won't look all that terrible.

The Arduino is my first real foray into embedded programming (unless you count the rudimentary CPU I built in my undergraduate Computer Organization and Design course). At the suggestion of my brother-in-law and the Internet I set out to create the "Hello, world!" of electronics: a blinking LED. That said, the Arduino IDE on Debian was so easy to use (it's literally an apt-get install arduino away) that I wasn't terribly satisfied with the relatively minimal work it took to achieve my blinking light. It had been a while since I'd written much C, so I took the opportunity to make a more full-featured "Hello, world!" with the addition of Morse code, serial console logging, and analog encoding control.

When he returns from a visit with his own respective in-laws, we're going to brush up on my soldering skills by building the Adafruit proto shield. Here's hoping that projects like these keep us collaborating, despite the physical distance between us after the holidays are over.

arduino-hello-world.thumbnail.jpg

Documentation as a habit

People seemed to like this comment that I made at LISA 2013:

It’s important to see documentation as a habit, rather than as a task to be completed. Don’t ask how you can catch up on the backlog of documentation that isn’t written; just document what you do and answers to questions you’ve had to ask from here on out.

org-mode iPhone Agenda

I generally keep on top of my to-do list at work, thanks to my “keep everything in org-mode habit.” Since I spend all day at work in front of a computer, org-mode is always a few keys away, and it’s easy to review my agenda throughout the day and keep my lists up-to-date.

There was a time when that was true of my personal lists as well; but, for whatever reason, I don’t tend to have org-mode open at home any more. There will be times (like this instant) where I remember, and will open emacs, and will review my agendas; but, most often, that just leads to me continuing to procrastinate, given how far behind I already am.

I’m pretty sure that what I need is better access to my agenda; and my best guess for how to implement this is to have my agenda available on my iPhone.

I’ve tried to use mobile-org before, but just couldn’t integrate it into my workflow. Maybe that’s still the right thing, and I’ll eventually figure out how to do it. Until then, I want to just configure my agenda to automatically export to html on the web whenever I push to my central repo, and be able to review that whenever I’m out-and-about.

Securing Salt file_roots

My only real problem with Salt vs Puppet is its security model for files stored in the manifest. Puppet’s fileserver supports per-node export configuration, allowing for node-private file distribution. Salt, on the other hand, exposes all files to all nodes at all times.

How Puppet does it

# fileserver.conf
[mount_point]
path /path/to/files
allow *.example.com
deny *.wireless.example.com

[private]
path /data/private/%h
allow *

How salt could do it

# file_roots.sls
file_roots:
  base:
    - /srv/salt
    - /srv/salt-example.com:
      - allow: *.example.com
      - deny: *.wireless.example.com
    -
  dev:
    - /srv/salt/dev/services
    - /srv/salt/dev/states
  prod:
    - /srv/salt/prod/services
    - /srv/salt/prod/states

Proposal

file_roots:
  base:
    - /srv/salt
  *.example.com:
    - /srv/salt-example.com

A New Firewall Salt State

My evaluation of Salt Stack is going pretty well. I’ve moved my main vps over to it with no ill effect, and was able to transcribe its Puppet manifest almost in its entirety. In many instances, I think the Salt version is more readable, and feels lighter than the Puppet version.

One glaring hole, though, is Salt’s support for firewall configuration. I was using the Puppet Labs firewall module to maintain iptables rules for my vps. That worked pretty well; but all Salt has right now is the ability to append new rules to a chain. The existing iptables state is documented at-risk for deprecation, too, so it’s a bad place to start.

It is expected that this state module, and other system-specific firewall states, may at some point be deprecated in favor of a more generic firewall state.

(Salt does have good support for iptables at the functional layer; it’s just the configuration management part that’s lacking.)

Since the firewall module I used before worked well enough, and I have a bunch of config based on it already, I’ve started reimplementing its interface in a Salt state module.

"100 salt-master":
  firewall_rule:
    - managed
    - protocol: tcp
    - ports: 4505:4506
    - action: accept

I’ve found developing a Salt state to be a pretty simple process so far. I really like how Salt’s effective layers cleanly separate between functionality, state management, and configuration. (My firewall state makes liberal use of the existing iptables module, for example.)

I’ve just published the module so far on github. This module at least recognizes that my existing config exists, and would be able to rebuild it in the proper order (sorted lexically by comment) if necessary. There’s a lot of functionality missing, but it’s a place to start. If anyone else uses it, that will just be an excuse to make it better!

Discovering Salt Stack

I’ve been a pretty stalwart Puppet user since I first discovered it in 2009. At that time, my choices were, as I saw them, between the brand-new cfengine3, the I’ve-seen-how-the-sausage-is-made bcfg2, and Puppet. Of those choices, Puppet seemed like the best choice.

In particular, I liked Puppet’s “defined state” style of configuration management, and how simple it was to describe dependencies between the various packages, files, and services to be configured.

Like I said, I’ve been using Puppet happily for the past 4 years; but now, I think I’ve been swayed by Salt Stack.

I know I looked at salt stack before; but, at the time, I think I dismissed it as just “remote execution.” Salt does, after all, start from a very different place than Puppet. At its most simple, it is a mechanism for shipping Python functions to remote nodes and executing them. It seemed the very opposite of the idempotent state management that I was looking for.

But now that I’ve taken the time to look deeper into the documentation (or, perhaps, now that the project has grown further) I’ve found Salt Stack States: the state enforcement configuration management system I was looking for; and with a trivial-to-setup remote execution layer underneath it.

Salt is based on 0MQ. I don’t know much about message queues; but I do know that I could never get ActiveMQ working for use with Puppet’s MCollective. After only 30 minutes of hacking, I had Salt, with 0MQ, running on two OS X machines and two Debian machines, all taking to the same master, each from behind its own form of inconveniently private network.

$ sudo salt '*' test.ping
ln1.civilfritz.net:
    True
Jonathons-MacBook-Pro.local:
    True
numfar.civilfritz.net:
    True
dabade.civilfritz.net:
    True

Glorious.

Some other things that I like about Salt:

  • States are defined in YAML, so there’s no proprietary (cough poorly defined cough) language to maintain.

  • The remote execution layer and state module layer help keep executable code separate from state definitions.

  • Key management is a bit less foolish. (It shows you what you’re about to sign before you sign it.)

Of course, no new technology arrives without the pain of a legacy conversion. I have a lot of time and effort invested into the Puppet manifests that drive ln1.civilfritz.net; but converting them to Salt Stack States is serving as a pretty good exercise for evaluating whether I really prefer Salt to Puppet.

I’ve already discovered a few things I don’t like, of course:

  • The abstraction of the underlying Python implementation is a bit thin. This is sometimes a good thing, as it’s easier to see how a state definition maps to individual function calls; but it also means that error messages sometimes require an understanding of Python. Sometimes you even get full tracebacks.

  • Defined states don’t seem to understand the correlation between uid and uidNumber. In Puppet I started specifying group ownership as 0 when I discovered that AIX uses the gid system rather than root. In Salt, this appears to try to reassign the group ownership every time.

  • All hosts in a Salt config have access to all of the files in the master.

  • YAML formatting can be a bit wonky. (Why are arguments lists of dictionaries? Why is the function being called in the same list as its arguments?)

  • No good firewall (iptables) configuration support. The iptables module isn’t even present in the version of Salt I have; but the documentation warns that even it is likely to be deprecated in the future.

That said, I can’t ignore the fact that, since Salt happens to be written in Python, I might actually be able to contribute to this project. I’ve already done some grepping around in the source code, and it seems immediately approachable. Enhancing the roots fileserver, for example, to provide node-restricted access to files, shouldn’t be too bad. I might even be able to port Puppet Lab’s firewall module from Ruby to Python for use as a set of Salt modules.

Time will tell, I suppose. For now, the migration continues.

Introducing civilfritz Minecraft

I started playing Minecraft with my brother and old college roommate a few weeks ago. My expectations have been proven correct, as I’ve found it much more compelling to play on a persistent server with a group of real-life friends. In fact, in the context of my personal dedicated server instance, I’m finding the game strikes a compelling chord between my gamer side and my sysadmin side.

There’s already some documentation for running a Minecraft server on the Minecraft wiki, but none of it was really in keeping with how I like to administer a server. I don’t want to run services in a screen session, even if an init script sets it up for me.

I wrote my own Debian init script that uses start-stop-daemon and named pipes to allow server commands. Beyond that, I made a Puppet module that can install and configure the server. You can clone it from Git at git://civilfritz.net/puppet-minecraft.git.

I also really like maps, so I started looking for software that would let me generate maps of the world. (I was almost pacified when I learned how to craft maps. Almost.) I eventually settled on Minecraft Overviewer, mostly because it seems to be the most polished implementation. They even provide a Debian repository, so I didn’t have to do anything special to install it.

I’ve configured Minecraft Overviewer to update the render once a day (at 04:00 EST, which hopefully won’t conflict with actual Minecraft server use), with annotations updated once an hour. You can see it at http://civilfritz.net/minecraft/overview.

I couldn’t get Overviewer to display over https for some reason I don’t understand yet; so all access is redirected back at http for now.