unsorted thoughts from Jonathon Anderson (old posts, page 9)

Discovering Salt Stack

I’ve been a pretty stalwart Puppet user since I first discovered it in 2009. At that time, my choices were, as I saw them, between the brand-new cfengine3, the I’ve-seen-how-the-sausage-is-made bcfg2, and Puppet. Of those choices, Puppet seemed like the best choice.

In particular, I liked Puppet’s “defined state” style of configuration management, and how simple it was to describe dependencies between the various packages, files, and services to be configured.

Like I said, I’ve been using Puppet happily for the past 4 years; but now, I think I’ve been swayed by Salt Stack.

I know I looked at salt stack before; but, at the time, I think I dismissed it as just “remote execution.” Salt does, after all, start from a very different place than Puppet. At its most simple, it is a mechanism for shipping Python functions to remote nodes and executing them. It seemed the very opposite of the idempotent state management that I was looking for.

But now that I’ve taken the time to look deeper into the documentation (or, perhaps, now that the project has grown further) I’ve found Salt Stack States: the state enforcement configuration management system I was looking for; and with a trivial-to-setup remote execution layer underneath it.

Salt is based on 0MQ. I don’t know much about message queues; but I do know that I could never get ActiveMQ working for use with Puppet’s MCollective. After only 30 minutes of hacking, I had Salt, with 0MQ, running on two OS X machines and two Debian machines, all taking to the same master, each from behind its own form of inconveniently private network.

$ sudo salt '*' test.ping
ln1.civilfritz.net:
    True
Jonathons-MacBook-Pro.local:
    True
numfar.civilfritz.net:
    True
dabade.civilfritz.net:
    True

Glorious.

Some other things that I like about Salt:

  • States are defined in YAML, so there’s no proprietary (cough poorly defined cough) language to maintain.

  • The remote execution layer and state module layer help keep executable code separate from state definitions.

  • Key management is a bit less foolish. (It shows you what you’re about to sign before you sign it.)

Of course, no new technology arrives without the pain of a legacy conversion. I have a lot of time and effort invested into the Puppet manifests that drive ln1.civilfritz.net; but converting them to Salt Stack States is serving as a pretty good exercise for evaluating whether I really prefer Salt to Puppet.

I’ve already discovered a few things I don’t like, of course:

  • The abstraction of the underlying Python implementation is a bit thin. This is sometimes a good thing, as it’s easier to see how a state definition maps to individual function calls; but it also means that error messages sometimes require an understanding of Python. Sometimes you even get full tracebacks.

  • Defined states don’t seem to understand the correlation between uid and uidNumber. In Puppet I started specifying group ownership as 0 when I discovered that AIX uses the gid system rather than root. In Salt, this appears to try to reassign the group ownership every time.

  • All hosts in a Salt config have access to all of the files in the master.

  • YAML formatting can be a bit wonky. (Why are arguments lists of dictionaries? Why is the function being called in the same list as its arguments?)

  • No good firewall (iptables) configuration support. The iptables module isn’t even present in the version of Salt I have; but the documentation warns that even it is likely to be deprecated in the future.

That said, I can’t ignore the fact that, since Salt happens to be written in Python, I might actually be able to contribute to this project. I’ve already done some grepping around in the source code, and it seems immediately approachable. Enhancing the roots fileserver, for example, to provide node-restricted access to files, shouldn’t be too bad. I might even be able to port Puppet Lab’s firewall module from Ruby to Python for use as a set of Salt modules.

Time will tell, I suppose. For now, the migration continues.

Introducing civilfritz Minecraft

I started playing Minecraft with my brother and old college roommate a few weeks ago. My expectations have been proven correct, as I’ve found it much more compelling to play on a persistent server with a group of real-life friends. In fact, in the context of my personal dedicated server instance, I’m finding the game strikes a compelling chord between my gamer side and my sysadmin side.

There’s already some documentation for running a Minecraft server on the Minecraft wiki, but none of it was really in keeping with how I like to administer a server. I don’t want to run services in a screen session, even if an init script sets it up for me.

I wrote my own Debian init script that uses start-stop-daemon and named pipes to allow server commands. Beyond that, I made a Puppet module that can install and configure the server. You can clone it from Git at git://civilfritz.net/puppet-minecraft.git.

I also really like maps, so I started looking for software that would let me generate maps of the world. (I was almost pacified when I learned how to craft maps. Almost.) I eventually settled on Minecraft Overviewer, mostly because it seems to be the most polished implementation. They even provide a Debian repository, so I didn’t have to do anything special to install it.

I’ve configured Minecraft Overviewer to update the render once a day (at 04:00 EST, which hopefully won’t conflict with actual Minecraft server use), with annotations updated once an hour. You can see it at http://civilfritz.net/minecraft/overview.

I couldn’t get Overviewer to display over https for some reason I don’t understand yet; so all access is redirected back at http for now.

Installing Netflix on an import PS3

I purchased my first HD console—a used PlayStation 3 Slim—from a friend as he was leaving KAUST. Tokyo Games (the most trustworthy games retailer in the kingdom) imports its PS3s from somewhere in Europe: I knew when I purchased it that I would not be able to play region-locked Blue-Ray disks. I really only care about games, though, and PS3 games are region-free.

When I returned to the US, import PS3 in tow, I added a use-case for the system as a streaming media center. Hulu Plus worked well enough while I had it; but, when I dropped the service in favor of Netflix, I couldn’t get the Netflix application to install.

NetfliX is only licensed for use from the US. Though I was in the US, and logged-into an explicitly US PSN account, the XMB never presented me with the option to install the Netflix application. It’s not listed in the PSN store, either; so if the XMB doesn’t advertise Netflix to you on it’s own, there’s no way to get it to install.

I went back and forth for a while with Sony support; but, ultimately, they were completely unhelpful. I was sure that, if only I could get the application to install, it would work, but the technician didn’t seem to understand the technical side of the issue I was having.

Thankfully, the people at unblock-us did understand the problem, as I discovered on my own while waiting for the Sony technician to catch up. Ultimately, all I had to do was visit http://ps3.unblock-us.com/ from the internal PS3 web browser. An HTTP redirect points to the installer on the Sony download server, prompting the system to download and install.

As expected, once the application had been installed, streaming worked perfectly.

It’s disappointing that something intended to simplify the user experience (don’t advertise applications that don’t work in your region) ended up severely complicating mine. I had the stubborn persistence to find a workaround myself; but there’s no reason Sony shouldn’t have a similar URL available and documented for people who take the time to contact support. It’d be even better if Netflix was just listed in the PSN store, too, as that’s already filtered by the region of your PSN account, irrespective of the console’s origin.

At least it “works for me” now.

I wrote about Astro Warrior for GameSpite Journal 12

My article on Astro Warrior was actually published on Telebunny.net, so feel free to go read it over there. As with my Spore article, it was included in an issue of GameSpite, Journal 12. Unlike GameSpite Journal 11, though, I don’t have a physical copy. I’ll probably order one some day; but I’m already having a difficult time dealing with the glut of stuff (a large portion of which is books) I’ve had to cram into our new Manhattan apartment.

This article kinda came out of nowhere—who would have thought that a random Master System vertical space shooter could serve as a memorial for my mother?

Astro Warrior | GameSpite Journal 12

Astro Warrior box art

By all accounts, Astro Warrior is unremarkable. It’s a competent 8-bit shooter; but being, as it is, a product of the 1980’s, it falls victim to many of the nostalgia-shattering tropes that plagued early consoles in their first strides away from their quarter-hungry arcade brethren. Unforgiving controls are the worst offender, with speed powerups making it simultaneously easier to evade enemies and more difficult to avoid crashing into them. Add to that the kind of relentless difficulty that justifies a game with only three zones. Every powerup is greedily and hastily accumulated, culminating in a multi-option death machine that slices through entire fleets of enemy fighters… until any single mistake strips you back down to the base ship, stranded in a now-overwhelming sea of enemy units and fire.

But I was young and oblivious to concepts like design and quality. I played the first stage over and over, each frustrated rage-quit lasting only a few minutes before my next attempt. (After all, my only other option was Altered Beast.) Glossed over as my memories are of the game, I’m sure I never conquered Zanoni, the first of the game’s three bosses; but that didn’t keep me from playing the game, nor did it keep me from remembering it as awesome twenty years later.

See, my mother worked part-time at the local hospital. A few days a week, after picking me and my brother up from school, she’d take us back with her to work the last few hours on her shift. That Master System in the children’s recovery room was the most advanced video game system we had ever seen. That we had never seen past the first stage was a detail as easily ignored then as the game’s design flaws are now (with the benefit of rose-tinted glasses, of course).

Playing through the game again now, I think about everything that was just a little bit beyond me at the time: about my younger self, always falling just short of getting to that second level, all the while blissfully unconcerned with the hard work that brought me there.

Today, Astro Warrior can only really survive through nostalgia. For some, that might strip the game of all value; but, in the end, nostalgia isn’t about the object of a memory as much as the context that it returns us to. Astro Warrior might be objectively unremarkable, but to me it will always be a reminder of my mother–a symbol of her dedication to her children and her family. For that, Astro Warrior will always be a classic in my book.

Astro Warrior action shot

Why I’m abandoning strict Allman style in Puppet manifests

I pretty much always use Allman style in languages that have braces. I like the symmetry, and the visible separation of identifier from value.

Though Allman style has its roots in C, the only brace language I use these days is Puppet. (Python end-runs around this whole issue by omitting braces altogether, which I ultimately prefer.) Pedantic as I am, my choice of brace style has extended (as closely as I could) to writing Puppet manifests.

class motd

(
  $content = undef
)

{
  file
  { '/etc/motd':
    content => $content,
    owner   => '0',
    group   => '0',
    mode    => '0644',
  }
}

This isn’t what most people do, and it’s certainly not what the examples in the Puppet style guide do; but it’s also not in violation of any of the recommendations in the style guide.

I’ve been doing this for years, now; but today, I had one of those “aha” moments where I pleasantly realized that I’ve been doing it wrong.

Allman style works just fine for Puppet class definition; but Puppet resources provide their titles within the braces, rather than outside. This supports the compression of multiple resources into a single declaration.

file {
  '/tmp/a':
    content => 'a';
  '/tmp/b':
    content => 'b';
}

This syntax is explicitly discouraged in the style guide, but it’s part of the language’s legacy.

The problem with Allman style in this context is that is separates the resource title from the resource type. In most braced languages, the title of an element is written outside of the braces, after the type.

#! /bin/bash

function main
{
    # ...
}

In this example, it would be easy to grep a pile of Bash source files for scripts that declare a main function.

$ grep 'function main' *.sh

Not so with Allman style. I can grep for /etc/motd; but that would match against any reference to the file. Finding the declaration itself becomes a manual exercise with a contextual grep (grep --before-context 1).

All of this becomes much simpler, however, if resource declarations include the resource title (and the interstitial brace) on the same line as the resource type.

class motd

(
  $content = undef
)

{
  file { '/etc/motd':
    content => $content,
    owner   => '0',
    group   => '0',
    mode    => '0644',
  }
}

Even I have to admit that grep "file { '/etc/motd':" *.pp is much simpler.

This is immaterial for class declarations, since the class name is located before the brace.

class motd
{
  ...
}

I’d argue that Puppet should at least support a similar syntax for resources; one that puts the title directly after the type.

file '/etc/motd'
{
  ...
}

That could get a bit confusing, though, when using parameterized classes, as a parameterized class application syntax is somewhat close to regular class definition syntax.

# definition
class motd
{
  # ...
}

# declaration
class motd
{
  content => 'Hello, world!',
}

Spore | Gamespite Journal 11

At the 2005 Game Developers Conference luminary Will Wright of Maxis Software amazed attendees with his seminar, “The Future of Content.” He began simply, outlining for his audience the trials and travails of developing content for an increasingly high-definition medium. No one could have been surprised when he supported his arguments with experiences from his widely successful titles SimCity and The Sims; no one should have been surprised that his content strategies were being prototyped for a new game; but no one could have been prepared for Spore, or for the scope of the demonstration that promised an ever-growing, seemingly boundless galaxy filled with content birthed from players themselves.

From his perspective in the gaming industry, Wright had seen development budgets explode. Creating content to fill increasingly higher-fidelity worlds was getting more expensive every day; and though a developer might spend twice as much to create HD-ready models and textures, that expense carries a diminishing return in value for the player.

At the same time, Wright had noticed that some of his most engaging gaming experiences had little to do with anything the developer had defined beforehand. This had always been true of Maxis’s own games; but, with the advent of the Grand Theft Auto series and the era of sandbox gaming it had inspired, more and more players were creating their own stories in these virtual worlds. More importantly, these were the stories that players related to each other, often ignoring the story that had been written for them by the developer.

It seemed a perfect fit: on one side, an industry struggling to keep up with the cost of creation; on the other, an ever-growing mass of players eager to create. So Maxis created Spore: a set of intuitive content creation editors that would bring players and developers closer together, amplifying the player’s own creative abilities with procedurally generated models, textures, and animation.

Wright aimed beyond simply equipping players to create: Spore needed to inspire that creation as well. Inspiration comes from the context of the world we’re placed in, and the scope of what Wright had in mind was more than anyone present in the conference hall could have anticipated. With brief diversions into the philosophies of creation, play, and Care Bears, he unfolded Spore’s world. A single-celled organism grew to inhabit an ocean. He added legs to his creatures and they crawled onto dry land. Given tools, they formed tribes and built cities. Their starship carried them to other planets orbiting the sun and, then, to other stars, each with its own unique planets, cities, and creatures. Each iterative expansion of scope was met with another round of amazed applause.

This presentation was my first exposure to Spore, separated though I was from the event by a layer of Google Video. I was awestruck by the game’s depth: it seemed to be a world without end! As the player I would be able to create creatures, vehicles, tanks–entire planets! I could explore every drop of water at microscopic detail, or fly across an entire galaxy; and, thanks to Spore’s content distribution network, each new corner of that galaxy would be full of unique content created by other players.

Which is to say: I bought into the hype.

When Spore was released three years later, it seemed a shallow reflection of what had been on display at GDC. The creature editors were still intuitive and the galaxy was still near-infinite, but the magic of watching the game unfold was missing. I had imagined a world with depth to match the amazing breadth on display; but Spore lacked the essential details that make virtual worlds feel alive. The final game was, in fact, even shallower than what had been shown before. Exciting features like emergent abilities and behaviors, procedural verbs, sea creatures, and underwater civilizations were conspicuously missing. Creature evolution, originally rooted in iterative experimentation with the emergent properties of the game engine, was replaced with a linear tech tree of parts. Social interaction, once experimental and unpredictable, now amounted to little more than a repetitive game of “Simon Says” powered by static bonuses granted explicitly by parts purchased in the editor. Military conquest and religious proselytization were nearly indistinguishable, both playing out as a tank parked outside a city, shooting missiles (or is it music?) at an opposing population.

But what if my disappointment with Spore was simply the product of my own unrealistic expectations? The game I had imagined was the perfect fulfillment of every pipe dream that 2005 Will Wright had for it, but reality rarely matches up with such high expectations. Perhaps now, another three years divorced from the romanticism that had preceded the game’s launch, I would be able to appreciate Spore for what it is. After all: I don’t even remember my creature from my first session at launch. Resolved to approach the game on its own terms, I did my best to take ownership of my worlds and creatures this time around. As a result, I created the Qwertz: a race bent on conquest, dominance, and vague Klingon pretension.

Even as the simplest of cellular life the Qwertz survived by consuming and eradicating others. On land they preyed on those around them, quickly rising to dominance over Handwalkers, Skyles, Flirgle, and all manner of strange creatures.

The Qwertz were terrestrial conquerors, but the final frontier was cold, strange, and empty. Unable to anticipate the strength of other space-faring races, they fearfully allied with neighboring species: the Ixplix, snake-oil salesmen of the highest order, ready to sell of their goods and their lands to turn a profit; and the Leafel, quiet, continually distressed over this diseased native or that ecological collapse. For a time, the Qwertz lived in peace–in a different world, this peace might even have survived–but it was not to be. The Algernon Empire, ministers of the Spode, introduced its philosophy with the heated end of a laser cannon. The Qwertz had seen their fair share of religious fanatics at home and, finding security in the smallest foothold of familiarity, they met the Algernon threat with new-found ferocity.

Reassured by the fresh taste of victory, the Qwertz resumed their conquest. The Ixplix were the next to fall; then the Kirgle and the Ashonas, mutual opponents in the Spode. One by one the Qwertz hunted ever-greater opponents, leaving behind a trail of smoldering colonies and fallen empires as they made their trans-galactic pilgrimage to the core. There they finally met the Grox: the uncompromising, overpowering, infamous scourge of the galaxy.

This story that I told myself–of the Quertz, their conquest of the galaxy, and their eventual encounter with an oracle of their future selves–may have been birthed from my own imagination, but Spore lacks the depth to bring it to life: it exists almost entirely in my mind. My return to Spore left me sometimes frustrated, sometimes bored, and sometimes compelled to keep playing into the early morning; but I’m not sure that I ever found myself having fun. The content creation editors are as easy to use as they have always been, but they quickly start to feel like a chore. The player can meticulously design every building in every colony on every planet, but why bother when doing so has no impact on the game itself?

The pervasive absence of detail leaves Spore feeling soulless. I could play again–create another creature and make up another story–but the structurally uniform game world doesn’t provide any incentive to do so. Every planet has three plants (one small, one medium, and one large) and three creatures (two herbivores, one carnivore) per T-Score. Every sentient creature is shoehorned into the same handful of archetypes, and interactions with any creature within an archetype is identical to those with any other. I could terraform every planet in the galaxy to T3, place the maximum three colonies on each one, and purchase every item in the game; but the galaxy will always be mostly empty space homogeneously populated with pallet-swapped versions of the same shallow “characters” I saw when I first entered the space stage. They will always cry out with the same restricted vocabulary of mission templates: they’re either under attack, nearing ecological collapse, suffering a labor strike, conducting a planetary survey, or in search of some special artifact. One mission in particular sent me, to my surprise, to exterminate members of a rival religious group; but the mission framework, unable to distinguish even between religious homicide and disease control, instructed me to “exterminate the infected creatures.”

Yet, as disappointingly shallow as the narratives that emerge from the gameplay are, Spore does have at least one worthwhile story to tell. It’s a cautionary tale about the pitfalls of modern game development, starring the plucky hero Maxis and his fall from grace.

Spore was a dream of a new era in game development. It was an experimental response to the unsustainably ballooning budgets of the industry. In the best tradition of the original SimCity, Maxis sought to counter this trend while, at the same time, equipping the player to be creative, to experiment, and to share that experience with others. Wright wasn’t alone in his vision of the future, and his insights as shared by others are responsible for the burgeoning small and independent game development market that is thriving on digital distribution services like Steam, XBLA, PSN, and iOS. The Maxis of SimCity would fit right in with the likes of Studio Pixel, Amanita Design, and Frictional Games, creating focused conceptual experiments in gaming that inspire audiences and developers alike; but the Maxis of The Sims, toiling now in exile at the behest of the corporate behemoth that is Electronic Arts, is no longer afforded the freedom to commit to a singular vision (least of all a risky experiment) at the potential expense of wider market appeal. Spore’s pedigree begot a budget that crushed the life from its initial innovation, spreading it so thin as to be nearly inappreciable.

Spore’s ultimate failure to deliver on its potential might have robbed mainstream gamers of the experience of losing themselves in a world of their own creation, but the intent of its design lives on in a new generation of developers independent of the pressures of traditional game publishing. The communities that surround relatively niche titles like Minecraft and Dwarf Fortress are rooted in the same emergent storytelling that I had hoped to find in my procedurally generated galaxy. Spore might lack the depth to tell players a worthwhile story of their own creation, but perhaps it can tell developers large and small a story about a little studio called Maxis and what happened when innovation was exploited, rather than fostered.

Tracking user actions with the Linux Audit Subsystem

I was given a mandate to log “what the users are doing” on the Minerva cluster system at Mount Sinai. Actually, the original mandate was more prescriptive: implement an auditing ssh daemon on the login nodes.

So that’s what I started doing… or, trying to do. I grabbed the source for auditing ssh, which was, unfortunately, a big custom-patched tarball of openssh, hpn-ssh, and the auditing patches. There was a Red Hat specfile included, so I went to work building a set of packages from these sources.

Unfortunately, my packages, when installed, didn’t function. I say unfortunately, but it might have turned out to be a blessing in disguise. As I researched why my new auditing sshd wasn’t allowing any users to log in (explicitly, with a denied action) I kept coming up against a more general-purpose Linux audit system, built into the kernel.

I had seen bits of this system in use before. I had seen pam_loginuid in default pam stacks before, and anyone who has come up against selinux knows about /var/log/audit/audit.log; but I didn’t appreciate just how flexible the linux audit subsystem is, right down to, if we really want, the ability to log every tty keystroke. (That said, I think we really only need to log execs; but we’ll see.)

Introduction

The linux audit system is a kernel subsystem paired with a userspace daemon that, based on a set of rules stored at /etc/audit/audit.rules, maintains an audit log of events that take place in the kernel, either by instrumenting specific syscalls (e.g., open, execve) or by watching for access to specific inodes (e.g., to track changes to sensitive files.)

In particular, the Linux audit subsystem can be used in the implementation of a Controlled Access Protection Profile as defined by the NSA. Red Hat ships a ruleset, capp.rules, with the audit daemon to implement such a policy.

Goals

  • Track user access from login to logout as a single user.
  • Log all user actions.

User tracking

$ grep pam_loginuid /etc/pam.d/*
/etc/pam.d/crond:session    required   pam_loginuid.so
/etc/pam.d/login:session    required     pam_loginuid.so
/etc/pam.d/remote:session    required     pam_loginuid.so
/etc/pam.d/sshd:session    required     pam_loginuid.so
/etc/pam.d/ssh-keycat:session    required     pam_loginuid.so

Audit rules

-a exit,always -F arch=b32 -S execve
-a exit,always -F arch=b64 -S execve

Reporting script

$ sudo ausearch -r | audit-commands

audit-commands.py

https://www.centos.org/docs/5/html/5.1/Deployment_Guide/rhlcommon-section-0081.html