Posts about programming

backporting sudo's #includedir

sudo version 1.7.2 (possibly earlier) adds the ability to fragment the sudoers file into smaller chunks via an #includedir directive. This is a boon for our use of puppet, as it affords us the ability to configure sudo in multiple modules at the same time, rather than centralizing all of our privilege escalation information in one module.

class s_gpfs
{
    [...]

    file
    { '/etc/sudoers.d/nrpe-mmfs':
      content => "nrpe ALL = NOPASSWD: /usr/lpp/mmfs/bin/mmgetstate\n",
      owner   => 'root',
      group   => 'root',
      mode    => '0440',
    }

    [...]
}

Here, we allow the nrpe user (part of our automated monitoring infrastructure) to run the gpfs command mmgetstate as root.

Unfortunately, we also have to support systems who's sudo implementation predates this new feature. (cough SLES 10 cough) In order to provide this functionality forward–compatibly, I wrote a Python script that inlines the contents of files as indicated by an #includedir directive that would otherwise be ignored as a comment in older versions of sudo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#!/usr/bin/env python


import sys
import re
import glob
import os
import fileinput


include_directive = re.compile(r'^[ \t]*#includedir[ \t]+(.*)$')


def main ():
    for line in fileinput.input():
         match = include_directive.match(line)
        if match:
            directory = match.group(1)
            sys.stdout.write(inlined_content(directory))
        else:
            sys.stdout.write(line)


def inlined_content (directory):
    files = get_files(directory)
    return ''.join(read_all(files))


def get_files (directory):
    return [f for f in glob.glob(os.path.join(directory, '*'))
            if os.path.isfile(f)]


def read_all (files):
    for file_ in files:
        try:
            yield open(file_).read()
        except IOError:
            yield ''


if __name__ == '__main__':
    main()

Python Batteries: webbrowser

Today's reverse spelunking through the Python standard library reveals the webbrowser module. As a library, it allows a Python application to interact with the default web browser on the host OS, opening a url in a new browser window or tab. This functionality is exposed at the shell, as well:

$ python -m webbrowser
Usage: /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/webbrowser.py [-n | -t] url
    -n: open new window
    -t: open new tab

You can use the webbrowser module like this to open web pages from any shell or shell script.

$ python -m webbrowser -t http://www.civilfritz.net/

The module does what it can to do the right thing given your environment, and will open a browser on X11, OS X, Windows, and will even open a text–mode browser if no graphical browser is available.

I noticed (on OS X, anyway) that if you don't specify a protocol (e.g., http) you get an exception.

$ python -m webbrowser -n www.civilfritz.net
0:39: execution error: An error of type -2110 has occurred. (-2110)

The browser still opened successfully, though.

You can find more information on the webbrowser module at the Python website.

Python Batteries: zipfile

Did you know that many modules in the Python standard library have __main__ sections? They're included mainly as examples for using the standard library itself, but there's some useful functionality in there.

You can run these examples on the command-line with python -m $module.

Today, the zipfile module:

Usage:
    zipfile.py -l zipfile.zip        # Show listing of a zipfile
    zipfile.py -t zipfile.zip        # Test if a zipfile is valid
    zipfile.py -e zipfile.zip target # Extract zipfile into target dir
    zipfile.py -c zipfile.zip src ... # Create zipfile from sources

Using nothing but the standard library, you can list, test, extract, and create "standard" zipfiles, just like this:

numfar:~ janderson$ python -m zipfile -l ZZT.ZIP 
File Name                                             Modified             Size
TOWN.HI                                        1990-01-24 17:10:16         1590
TOWN.ZZT                                       1991-11-02 18:50:24        73959
CAVES.ZZT                                      1991-11-02 18:56:04        80376
DUNGEONS.ZZT                                   1991-11-02 18:56:58        71378
CITY.ZZT                                       1991-11-02 18:58:24        63943
DEMO.ZZT                                       1990-01-26 01:35:58        26465
TOUR.ZZT                                       1991-11-02 18:48:52        33941
ZZT.DAT                                        1992-01-10 16:59:24        35823
ZZT.CFG                                        1992-01-10 15:29:30           18
HELPME.EXE                                     1994-01-30 20:21:42         6288
ORDER.DOC                                      1997-09-26 15:56:28         5174
HELPME.DOC                                     1994-02-18 10:46:16         7884
PRINTME.BAT                                    1991-10-19 21:59:28          274
ZZT.EXE                                        1992-05-21 08:38:24        47764
LICENSE.DOC                                    1997-10-09 16:14:50        26112
license.txt                                    1997-10-09 16:10:30        14428
ORDER_UK.DOC                                   1997-09-26 16:43:46         4967

git-apply

I found myself wanting to revert changes made in a previous git commit... but only changes to a specific file (whereas the commit changed multiple files). As usual, git has just the combination of tools that I needed:

git diff $commit ${commit}^ $file | git apply -

Thank you, git.

Gah! I wanted to like modules

We use a program called Environment Modules to manage third-party software on Shaheen. It's basically softenv, if you're more familiar with that, but wasn't written by MCS.

Anyway...

I've been working on documentation and overall transparency to our users recently, and I found myself wanting to generate documentation from our modules for publication on our wiki.

It's a fairly typical shell script, and nothing unexpected.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#!/bin/bash

source /etc/profile.d/modules.sh

function main
{
    dest=$1
    mkdir -p $1
    modules=`module avail -t 2>&1 | grep -v :$`
    (
        echo "> automatically generated at" `date`
        echo
        echo "# Modules available"
        echo
        for module in $modules
        do
            echo >/dev/stderr $module
            echo "* \[[$module|$module]]"
            mkdir -p ${dest}/`dirname $module`
            mdwn=${dest}/${module}.mdwn
            module_to_mdwn $module >$mdwn 2>&1
        done
    ) > ${dest}.mdwn
}

function module_to_mdwn
{
    module whatis $1
    echo
    echo "# Module help"
    module help $1
}

main $@

The first annoyance comes from the fact that most output generated by modules is put on stderr. I'd argue philosophically that the right place for this output is stdout, but it's not too big of a deal: just a few 2>&1 redirects.

Or so I thought. Output (other than the logging I'm doing intentionally) appears at the terminal. Output that I intended to redirect to the markdown file.

I spend an hour or so trying various placements of parenthesis, expecting that this has something to do with shell functions (module_to_mdwn, or even the module command itself). Debugging further leads me to the end of the road:

/usr/bin/modulecmd bash help $module &>/dev/null

module is a shell function that calls modulecmd under the covers

That command produces seemingly impossible output at the terminal.

Time for strace:

[...]
write(2, "\t*** No Module Specific Help for"..., 43 *** No Module Specific Help for INTEL ***) = 43

It's writing to fd 2, which should be stderr redirected to /dev/null. What's happening to those file descriptors?

clusterbank migrates to civilfritz

Way back when I worked at Argonne I developed a little program called clusterbank that we used to manage resource allocations. Now that I'm at KAUST, we need the same thing again, so I've drudged the code up from the depths of one of my hard drives. (It was actually pretty nasty, and I'm lucky to have found as recent a copy as I did: the first copy that I tried to use had corruption that made "git fsck" fail.)

Anyway, I'm going to have to do a bit of coding to make it work at KAUST (surprise! it didn't just work out-of-the box) so I've migrated the most recent branch that I could find to civilfritz, its new canonical home.

I was told that any code developed on DOE time has to be open-sourced with a permissive license (e.g., BSD). I don't know what I should have actually done to make that happen at ANL, but I'm sure no one will complain if I just stick the BSD license on the code and call it good for now.