We use a program called Environment Modules to manage third-party software on Shaheen. It’s basically softenv, if you’re more familiar with that, but wasn’t written by MCS.

Anyway…

I’ve been working on documentation and overall transparency to our users recently, and I found myself wanting to generate documentation from our modules for publication on our wiki.

It’s a fairly typical shell script, and nothing unexpected.

#!/bin/bash

source /etc/profile.d/modules.sh

function main
{
    dest=$1
    mkdir -p $1
    modules=`module avail -t 2>&1 | grep -v :$`
    (
        echo "> automatically generated at" `date`
        echo
        echo "# Modules available"
        echo
        for module in $modules
        do
            echo >/dev/stderr $module
            echo "* \[[$module|$module]]"
            mkdir -p ${dest}/`dirname $module`
            mdwn=${dest}/${module}.mdwn
            module_to_mdwn $module >$mdwn 2>&1
        done
    ) > ${dest}.mdwn
}

function module_to_mdwn
{
    module whatis $1
    echo
    echo "# Module help"
    module help $1
}

main $@

The first annoyance comes from the fact that most output generated by modules is put on stderr. I’d argue philosophically that the right place for this output is stdout, but it’s not too big of a deal: just a few 2>&1 redirects.

Or so I thought. Output (other than the logging I’m doing intentionally) appears at the terminal. Output that I intended to redirect to the markdown file.

I spend an hour or so trying various placements of parenthesis, expecting that this has something to do with shell functions (module_to_mdwn, or even the module command itself). Debugging further leads me to the end of the road:

/usr/bin/modulecmd bash help $module &>/dev/null

module is a shell function that calls modulecmd under the covers. That command produces seemingly impossible output at the terminal.

Time for strace:

[...]
write(2, "\t*** No Module Specific Help for"..., 43 *** No Module Specific Help for INTEL ***) = 43

It’s writing to fd 2, which should be stderr redirected to /dev/null. What’s happening to those file descriptors?

Andi and I just finished dominating New Super Mario Bros. Wii. (I will curse level 9-7 for years to come.) We managed to get 100%, and I even purchased all of the hint videos at Peach’s castle for good measure.

The payoff was a bit disappointing, though: a quick message of congratulations is all we received after obsessively clearing every exit (including the warp canons). Oh well, I suppose.

In any case, that’s one more game off of my games I am playing list, even if it wasn’t the one I intended to finish next. (I’m stuck on the Nu Master. Hopefully someday soon I’ll work up the patience to beat him.)

Either that, or I’ll switch impatiently back and forth between the two DS Zelda games I have on tab, too.

I recently added ~/Library/Safari/Bookmarks.plist to my homedir git repository in the misguided hope that it would be a magically sensible way to keep my bookmarks sync’d between my various systems. This evening was the first real test:

Kay:~ janderson$ pgit pull
/Users/janderson
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From ssh://slice1.civilfritz.net/home/janderson/git/home
   9d1b3c1..5fdae45  master     -> origin/master
Updating 9d1b3c1..5fdae45
Fast-forward
 Library/Safari/Bookmarks.plist |  Bin 1951 -> 2498 bytes
 1 files changed, 0 insertions(+), 0 deletions(-)
/Users/janderson/org
Already up-to-date.

Not only did the day’s changes propagate to my local system, but Safari implemented the change immediately, in the current running process, in the current window, without intervention. I’m sure this has something to do with fsevents, but it’s just one more example of the extent of behavior you can get away with if you implement a system properly in the first place. (After all: it’s not as though Apple had this use case in mind.)

Of course, the true test will be how well the file format copes with an n-way merge… but I’ll cross that bridge when git blows up in my face.

I use a couple of git repositories in my home directory to track and sync my configuration and org files across multiple platforms. Originally I used a single repository for everything, tracked at ~/.git, but when I split the two concerns I knew that I would want a simple way to manage all of my local git repositories at once. Inspired by my use of existing pssh-style commands, I wrote a quick (not quite so) parallel pgit:

#!/bin/bash

for repo in ~ ~/org
do
    echo 1>&2 $repo
    cd $repo
    git $@
    cd - >/dev/null
done

I want to improve this at some point with a configurable repository list (likely a custom section in ~/.gitconfig) and actual parallel execution, but at least for day-to-day, this has already been satisfying to use.

Way back when I worked at Argonne I developed a little program called clusterbank that we used to manage resource allocations. Now that I’m at KAUST, we need the same thing again, so I’ve drudged the code up from the depths of one of my hard drives. (It was actually pretty nasty, and I’m lucky to have found as recent a copy as I did: the first copy that I tried to use had corruption that made “git fsck” fail.)

Anyway, I’m going to have to do a bit of coding to make it work at KAUST (surprise! it didn’t just work out-of-the box) so I’ve migrated the most recent branch that I could find to civilfritz, its new canonical home.

I was told that any code developed on DOE time has to be open-sourced with a permissive license (e.g., BSD). I don’t know what I should have actually done to make that happen at ANL, but I’m sure no one will complain if I just stick the BSD license on the code and call it good for now.

I just sent out the word that WatsonLinux is being decommissioned, and managed a relatively good sales pitch for Shaheen in the process. For those of you who wonder what I do…

Shaheen is powered by a 16-rack (65536-core) IBM Blue Gene/P system and a 96-(soon 128-)node IBM System x cluster. It currently ranks at #18 on the TOP500 list [1] of the world’s most powerful supercomputers, and remains the most powerful supercomputer in the middle east, capable of 222.82 teraflops peak (190.90 sustained).

Built with the environment in mind, Shaheen ranks #9 on the TOP Green500 list [2], providing 378.77 megaflops per watt.

If you are interested in using Shaheen in your research, contact our support desk at shaheen-help@kaust.edu.sa.

I’m obsessed with reason. I have to be able to explain things to myself; to understand why things are. That doesn’t mean that my reasons are rational, but I have to put it somewhere in the taxonomy of my mind.

When our house flooded, and the ceiling caved in, KAUST moved us to a house five doors down the block that was identical in design, except that it was the mirror image of our first house. To reconcile the disruption in my mind, I decided that the reversed house represented a turning point of the reversal of our experience at KAUST. From that point on, things would be easier… the reversal of the frustration we had experienced before.

We had a very frustrating time traveling back to Saudi Arabia after our trip to the states for thanksgiving. Our flight from Indianapolis to Chicago was delayed (picking us up) so our entire itinerary was disrupted. We were rerouted on a different airline network, so none of our new flights had any record of us. Andi’s ticket didn’t match her passport (both via nicknames and married names), causing us to almost miss our flight out of Heathrow after a nine-hour layover (three hours of which was spent arguing that Andi should be allowed to board the plane).

I spent a lot of time during our our last flight (Saudi Airlines from Heathrow to Jeddah) praying to God that our luggage would be at the airport when we arrived. We were returning to a strange country with no house to go to, and no idea where we would be staying. We had been through a lot of frustration in transit, and we were both frustrated and frightened. Andi was falling apart, and, as always, I decided it was my responsibility to make it right. To keep things together. If our luggage was at the airport, against all odds, I would have evidence that, through it all, God was watching out for us, and helping us through our times of trial.

We landed at a strange airport with no luggage, no idea where we were, and no driver to pick us up. I was crushed. I didn’t swear off my faith or trust in God, mind you; but my reasoning went something like this:

1. God does everything for a reason.

2. God had the power to guide our luggage to us at the airport.

3. Our luggage was not at the airport.

4. There was no reason why we should get our luggage later, rather than immediately.

5. God must not need us to get our luggage.

6. We would never see our luggage again.

I had it all figured out, and I had doomed myself.

A week later, though, and we had our luggage back. Why? Because I was driven back to the airport by a wonderful man named Arnold. The same man who took Andi and me to Jeddah for Nerph’s first visit to the vet. He told me all about his family back home, why he was working in Saudi Arabia, and how he hopes to be hired as a KAUST employee so he can bring his family with him.

Later that week, he came to my house and beat me at chess. Three times in a row.

My original plan was to get civilfritz set up as soon as I got my apartment in New York, and begin logging my experiences there. Though I suppose I have not yet violated that expectation, it seems that I won’t be in a permanent residence, with a regular ISP, for some time. Thus blogspot.

For those who haven’t heard, things at KAUST have not gone as planned. Though I had intended to live in New York for 12-18 months, I was greeted at IBM/Watson with the announcement that I would need to relocate to Saudi Arabia in June, leaving us with less than 30 days to prepare.

We’ve had a couple of difficult days (the worst of which was probably the day we transferred the contents of the U-Haul to a seemingly tiny storage space) but God has definitely been there with us. This was what we wanted, after all: to be challenged.

We’ve been through three separate hotels so far, including less than 12 hours in an Extended Stay America, but we finally have a wonderful room at the Marriott Westchester Residence Inn in White Plains. It’s a 10 minute walk to the train station, which has been wonderful.

We found a surprisingly great church in the area our first Sunday here: the First Community Church of the Nazarene. I seem to be destined to attend an ethnically-centered church: this time it’s an all-black congregation that is, apparently, mostly Jamaican (or otherwise Caribbean). The minister and his wife sought us out, though, and took us on a tour of Westchester County (including the Pepsi Headquarters), terminating at a seaside restaurant in Stamford, Connecticut.

Shaun came for a quick weekend visit (from New Mexico) to see us off before we leave the country. We spent a day in the city, including a lot of wandering about in the subway and a riverboat cruise around Liberty Island.

Work at IBM/KAUST has been a hectic cross-section of paperwork, IT, and meetings so far, including two days of meetings with the KAUST CIO, John Larson. It’s all gone respectably well so far, and I’m excited to get to Thuwal so I can start building things that matter long-term. Until then, though, it’s back to the 32-node Linux cluster, and more paperwork to get a visa for Saudi Arabia.

Despite the obvious lack of available information, the past few months have been eventful. We’re in our fifth hotel now, the Rosewood Corniche, having traveled from our home south of Chicago to White Plains, New York; San Diego, California; Paris, France (for all of two hours); and now Jeddah, Saudi Arabia.

We were originally supposed to relocate 15 June 2009. Rather, we were originally supposed to relocated in November of 2010, but some as-yet-unnamed individual (or groupthink) changed all that. We were handed a new relocation date (again, 15 June) and told to prepare.

For those of you who have never traveled to Saudi Arabia, let me digress for a moment to tell you a bit about the visa application process: it’s extensive. Very extensive. “Medical exam so thorough the doctors have never heard of the tests” extensive. A deceptively-single page covered in “positive/negative” checkboxes that certify that you are either worthy of entry or summarily fired. (I guess they usually do these before they hire you, but someone thought better of it since “we were going to be working in the US for over a year anyway.”) If the tests themselves–and the passport photos, and the background check, and the scholastic reports–were not complicated enough, we suffered mistakes. Little mistakes, but many. Forgotten tests. Incorrect tests. Mislabeled samples. Lost samples. Exploding samples. (I am quite glad that I was not present for that one.) Complicating matters even more, Andi was traveling the entire time. Round One in New York. Rounds Two and Three in Wisconsin. Rounds Four and Five in Florida. All within the span of two weeks. (At least the three times I had to go were all in the same state.) These complications and over-complications set our relocation date back by a month (or one month closer to our original date, however you care to see it).

For the curious, Andi and I are now both certified–to the satisfaction of the Saudi Arabian government–to be free of syphilis, malaria, HIV, hepatitis, cholera, parasites, mental retardation, mental disorder, paralysis, blindness, hearing disorder, speech disorder, and unborn children. Good to know.

After an epic three-way phone call with a testing clinic, more mad FedEx-overnighting than I’ve ever seen before, and a mysterious visit to the Saudi embassy in DC, Andi and I were each the proud owner of an entry visa for Saudi Arabia.

In the mean time, I had been working furiously to co-ordinate the sale of my 1999 Pontiac Grand Am to a good friend in need of a vehicle. It turns out it’s complicated to pay off a Wisconsin loan for an Illinois-titled car while living in New York and shipping to California… especially without a reliable mailing address. Still, just days after my entry visa arrived, the shippers appeared and carried away my car, leaving me to walk back to my hotel. As a born-and-bred midwesterner from the heartland of car culture, stranded (as it were) far from what my subconscious still considers home, that was a very odd feeling.

A eight-hour flight to Paris should be exciting. A little less so when you only get to be in Paris for two hours. Even less so when you never get to leave the airport… and when you’ve got another seven-hour flight after that… and when you’re flying coach.

When we finally arrived at the Jeddah airport, we were pleasantly surprised to find an entire KAUST greeting booth–with its now-familiar, large, colorful, and beckoning logo–manned and ready to receive us. We were ushered quickly through customs, and our bags were left unopened after a quick run through an x-ray, putting to rest months of fears about complications at the border. It took us thirty minutes to move from the plane to the car. (As Andi later remarked, “I thought they were driving us somewhere else for the actual customs check!”)

It might not surprise you to find out that Saudi Arabia is nothing like the US, so you’ll be very surprised to find out that it is exactly like the US… only hotter… and the men wear thobes and the women wear abayas… and everything is in Arabic… and businesses shut down five times a day for prayer… and there’s wild camels roaming the desert… ok, so maybe it’s not exactly like the US, but I was shocked by how familiar it can feel. One of the first things I saw (in the semi-conscious car ride from the airport to the hotel) was an iPhone billboard for the 3GS. Then a mall, in which I saw a Fudruckers. (That’s right, a Fudruckers.) Burger King… Applebees… TGI Fridays… There’s a Pizza Hut down the road from our hotel! (I’m starting to notice that I can’t think of much aside from the restaurants, so perhaps I’ve only underestimated the affect that familiar food can have.)

Oh, they have Wal-Mart, too. Only it’s not Wal-Mart, it’s HyperPanda, which, as I’m sure you are aware, is an infinitely better name than Wal-Mart. (Obviously, it follows that Saudi Arabia > the US.) Nothing makes the human experience seem more universal than watching a Saudi man stand awkwardly with his wife in the “feminine products” aisle.

Initial impressions at the KAUST campus were bad. My entire first day in Thuwal was spent in and around our datacenter… a ghastly sight. We entered the building through a hole in the wall, traversed the offices without floors, to reach the machine room: Shaheen splayed open, Blue Gene entrails spread callously across the floor. 85% humidity… heat… dust… and that’s with all the hardware turned off. This room was not fit to have a machine like Shaheen in it. (It still isn’t, really, though it’s closer every day.)

The next day was better. My project manager led me through more-complete areas of the campus: the vast tunnel that runs through the center of the academic square… the seafront… the residences… (Actually, those were depressing again, but we’re being positive now.) The more I saw, the broader my perspective became. Now I was one of the wide-eyed dreamers that provides no useful information to the comrades back home.

If I had one word to describe the campus it would be “behind.” With two words I would say, “behind schedule.” With four, though, It’s “behind schedule, but amazing.” Its proximity to the sea gives it both a fantastic view and a cooler climate than Jeddah. (Even a warm breeze can cool you down.) From a distance, I think the academic center is a bit sterile (though still grand) but in and around the buildings you explore passageways and walkways positioned to use the breeze while shielding from the sun. The center is an intellectually satisfying interweave of levels and walkways, all highlighted by palm-accented roads and water affects.

The residential districts (including the student accommodations) are similarly fantastic, if a bit repetitive. (It’s inevitable, I think, when you build this many homes all at once.) From a plaza at the center they are an amazing sight, and from there I began to get my first inkling of the full scale of what was being constructed.

That is the one problem, though: it’s all still being constructed. The site is large enough that you can get a perspective high enough that you can see just how amazing it really is; but up close there are few real roads, incomplete buildings, and an undeniable prevalence of unlandscaped desert. It’s a massive construction in the desert, with a few out-of-place grassy areas: not the miraculous oasis that it will one day be.

Andi and I each have an Iqama (ee-gah-mah) now: a sort of permanent residency permit, so they won’t kick us out of the country in thirty days. Andi has found work, too, in the university relations department! She’s doing exactly what she’s wanted to do, too: editing and writing, and even investigating and interviewing a bit. (I’ll let her talk about it, since she probably has much more interesting things to say about it than I do. I’m crazy-excited for her, though!) We’re still in the Rosewood Corniche, our stay in Jeddah having been extended until at least 8 September, due to the current state of the campus. We feel a bit stuck (again) but really, we can’t complain. We’re in this for the adventure, and we’re certainly having it. There’s virtually no information available (KAUST seems pathologically afraid of communication); but despite an unfortunate tendency toward freneticism (that I’m trying to calm) things get taken care of eventually. Just not on my schedule.

I have been particularly struck by the prevalence and visibility of worship here in Saudi Arabia. We thought that we knew what it would be like, (five daily prayers, fasting at Ramadan, mosques everywhere) but I did not anticipate quite the expression of devotion that we’ve seen. More than once we’ve witnessed a crowd of men exiting a coffee shop to pray together at the call, or pulled over at the side of the road to pray alone while other’s simply drive by. There’s a refreshing quantity of spiritual devotion that is both prevalent enough to earn specific notice but not so universal as to invalidate it via an excuse of cultural conformity.

That’s just my long winded way of saying that I admire the devotion that I’ve seen here, and have been thinking about it a lot in reference to my own devotion to the rituals and lifestyle of Christianity. Of course, the logical reaction was to observe Ramadan.

First, a quick primer on modern Ramadan observances in Jeddah. For a twenty-eight-day month you don’t consume any food or drink (including water) or indulge oneself in any way, really, from sunrise until sunset. You work a shortened, 6-hour work day to compensate. Each morning, you get up before sunrise to eat the Sahur meal, and each night, at sunset, you break the fast at the Iftar meal. The practice of denying yourself something of which your body continually reminds you is meant to develop patience and self-control while directing attention toward Allah. This all sounded pretty good (from some perspectives of good) and I have fasted before (though admittedly nothing to this extent) so I thought I’ve give it a try.

After two days, I felt as though a train was about to burst out of my forehead–on fire–right after having ripped up my throat on the way out. Every single person who saw me asked me why I looked so drained (as I hadn’t really publicized my attempt). I couldn’t even eat much at Iftar, feeling as though I might just vomit it all back up. Even after admitting defeat I did not fully recover until three days of eating and drinking normally.

I was doing it wrong.

That shortened work week? I didn’t take it. “I’ve never asked for special treatment during a fast before!” I thought. I hadn’t ever not had water during a fast, either. I guess that’s a bigger deal than I gave it credit for.

The Sahur? I didn’t really take it. I’d have fruit, bread, and water in the morning, but I didn’t feel comfortable enough to ask the hotel to serve me the real thing. As I found out later, I wasn’t even having a real Iftar, either. Apparently there are a few specific things that you are supposed to eat to replenish what your body has lost (like dates and whole milk), and I didn’t take the time to learn about any of it.

Most importantly, I didn’t actually use the experience to focus my attention on my faith, defeating the entire purpose of the exercise.

I’m not going to completely discount the possibility that I really did just get sick, but most likely I got an opportunistic infection after completely abusing myself for two days without sufficient replenishment, either physically, socially, or spiritually.

Lesson learned.