civilfritz.net (unsorted thoughts from Jonathon Anderson)

Splitting Warewulf Images Between PXE and NFS

Jonathon Anderson — Wed, 07 Dec 2022 23:30:00 GMT

This article was also published via the CIQ blog on 6 December 2022.

Warewulf 4 introduced compatibility with the OCI container ecosystem, which greatly streamlines the process of defining, importing, and maintaining compute node images compared to other systems--even compared to Warewulf 3! But one aspect of compute node images remains unchanged: they can quickly grow in size.

Warewulf (and the technique of PXE-booting a node image more broadly) expects that a compute node image will remain relatively small. Larger sets of software, like you might provide via an Environment Modules stack or, perhaps, via Spack, are typically deployed via a central NFS share, which is then mounted at runtime by the booted compute node. Even OpenHPC, with software packaged as operating system containers, supports this paradigm, with packages installed on the head node, landing in /opt, and then being shared from the head node to compute nodes.

However, there are still benefits to maintaining this software as part of a compute node image; but such a large image can quickly grow to tens of gigabytes, making network booting difficult.

In this article I'll demonstrate how a full software stack can be managed together with a given compute node image, but the resultant payload can be split in-place between PXE-served netbooting and an NFS-mounted file system.

NOTE

This procedure depends on support for /etc/warewulf/excludes, which was broken in Warewulf v4.3.0.

The root image

First, I start with the standard Rocky Linux 8 image as published by HPCng.

[root@wwctl1 ~]# wwctl container import docker://docker.io/warewulf/rocky:8 rocky-8-split

Installing some software

Using the OpenHPC project as a source, I install a set of typical scientific software. Most OpenHPC packages install software in /opt for distribution via NFS, which is what we're going to do: just a little bit differently than usual.

[root@wwctl1 ~]# wwctl container shell rocky-8-split
[rocky-8-split] Warewulf> dnf -y install 'dnf-command(config-manager)'
[rocky-8-split] Warewulf> dnf config-manager --set-enabled powertools
[rocky-8-split] Warewulf> dnf -y install epel-release http://repos.openhpc.community/OpenHPC/2/CentOS_8/x86_64/ohpc-release-2-1.el8.x86_64.rpm
[rocky-8-split] Warewulf> dnf -y install valgrind-ohpc {netcdf,pnetcdf,hypre,boost}-gnu9-mpich-ohpc

After installing the software our image is approaching 2GB. This isn't egregious (and the compressed image as sent over the network is even smaller), but gives us a point of comparison for what comes next.

[root@wwctl1 ~]# du -h /var/lib/warewulf/container/rocky-8-split.img{,.gz}
1.8G    /var/lib/warewulf/container/rocky-8-split.img
651M    /var/lib/warewulf/container/rocky-8-split.img.gz

Excluding the software from the final image

Warewulf consults /etc/warewulf/excludes within the image itself to define files that should not be included in the built image. For our example here, I exclude the full contents of /opt/, in anticipation that we'll be mounting it via NFS in stead.

[rocky-8-split] Warewulf> cat /etc/warewulf/excludes
/boot
/usr/share/GeoIP
/opt/*

Rebuilding the image with /opt/* excluded, the image is reduced in size, and further software installation would no longer increase the final size of the image delivered over PXE.

[root@wwctl1 ~]# du -h /var/lib/warewulf/container/rocky-8-split.img{,.gz}
1.1G    /var/lib/warewulf/container/rocky-8-split.img
483M    /var/lib/warewulf/container/rocky-8-split.img.gz

Exporting the software via NFS

With the software in /opt excluded from the image, we need to export it via NFS in stead. This is relatively easily done, though we must discover and hard-code paths to the container directory.

[root@wwctl1 ~]# readlink -f $(wwctl container show rocky-8-split)/opt
/var/lib/warewulf/chroots/rocky-8-split/rootfs/opt

Add an NFS export to /etc/warewulf/warewulf.conf, restart the Warewulf server, and configure NFS with wwctl. Note that I've specified mount: false for this export, as I want to control which nodes will mount it: presumably nodes that aren't using this image should not mount this image's software.

nfs:
  export paths:
  - path: /var/lib/warewulf/chroots/rocky-8-split/rootfs/opt
    export options: rw,sync,no_root_squash
    mount: false

[root@wwctl1 ~]# systemctl restart warewulfd
[root@wwctl1 ~]# wwctl configure nfs

Mounting the software on the compute node

We can mount this new NFS share just like any other, by listing it in fstab.

Warewulf typically configures fstab as part of the wwinit overlay. In order to mount this NFS share without setting mount: true for all nodes, I copy fstab.ww to a new overlay and add an additional entry.

[root@wwctl1 ~]# wwctl overlay list -a rocky-8-split
OVERLAY NAME                   FILES/DIRS
rocky-8-split                  /etc/
rocky-8-split                  /etc/fstab.ww

[root@wwctl1 ~]# wwctl overlay show rocky-8-split /etc/fstab.ww | tail -n1
{{ .Ipaddr }}:/var/lib/warewulf/chroots/rocky-8-split/rootfs/opt /opt nfs defaults 0 0

I can add the new overlay to our wwinit list, and the fstab in rocky-8-split will override the one in wwinit. (Note: --wwinit was specified as --system in Warewulf 4.3.0.)

[root@wwctl1 ~]# wwctl profile set --wwinit wwinit,rocky-8-split default
[root@wwctl1 ~]# wwctl profile set --container rocky-8-split default

From a compute node, we can see that /opt is mounted via NFS as expected.

[root@compute1 ~]# findmnt /opt
TARGET SOURCE                                                      FSTYPE OPTIONS
/opt   10.0.0.3:/var/lib/warewulf/chroots/rocky-8-split/rootfs/opt nfs4   rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.4,local_lock=none,addr=10.0.0.3

We can further confirm that /opt is empty on the local, PXE-deployed file system.

[root@compute1 ~]# mount -o bind / /mnt
[root@compute1 ~]# du -s /mnt/opt
0   /mnt/opt

Future work

As demonstrated here, we can already implement split PXE/NFS images using functionality already in Warewulf; but future Warewulf development may simplify this process further:

Container path variables in warewulf.conf

We could support referring to compute node images in warewulf.conf. For example, it would be nice to be able to replace

nfs:
  export paths:
  - path: /var/lib/warewulf/chroots/rocky-8-split/rootfs/opt
    export options: rw,sync,no_root_squash
    mount: false

with something like

nfs:
  export paths:
  - path: {{ containers['rocky-8-split'] }}/opt
    export options: rw,sync,no_root_squash
    mount: false

This way, our configuration would not have to hard-code the path to the container chroot.

Move NFS mount settings to nodes and profiles

Right now, NFS client settings are stored in warewulf.conf as mount options, mount, and implicitly via path; but if these settings were moved to nodes and profiles we could configure per-profile and per-node NFS client behavior without having to manually edit or override fstab.

Stateless provisioning of stateful nodes: examples with Warewulf 4

Jonathon Anderson — Fri, 21 Oct 2022 06:00:00 GMT

This article was also published via the CIQ blog on 30 November 2022.

When deploying Warewulf 4, we often encounter expectations that Warewulf should support stateful provisioning. Typically these expectations are born from experience with another system (such as Foreman, XCAT, or even Warewulf 3) that supported writing a provisioned operating system to the local disk of each compute node.

Warewulf 4 intentionally omits this kind of stateful provisioning from its feature set, following experiences from Warewulf 3: the code for stateful provisioning was complex, and required a disproportionate amount of maintenance compared to the number of sites using it.

For the most part, we think that arguments for stateful provisioning are better addressed within Warewulf 4's stateless provisioning process. I'd like to go over three such common use cases here, and show how each can be addressed to provision nodes with local state using Warewulf 4.

Local scratch

The first thing to understand is that stateless provisioning does not mean diskless nodes. For example, you may have a local disk that you want to provide as a scratch file system.

Warewulf compute nodes run a small wwclient agent that assists with the init process during boot and deploys the node's overlays during boot and runtime. wwclient reads its own initialization scripts from /warewulf/init.d/, so we can place startup scripts there to take actions during boot.

My test nodes here are KVM instances with a virtual disk at /dev/vda. This wwclient init script looks for a "local-scratch" file system and, if it does not exist, creates one on the local disk.

#!/bin/sh
#
# /warewulf/init.d/70-mkfs-local-scratch

PATH=/usr/sbin:/usr/bin

# KVM disks require a kernel module
modprobe virtio_blk

fs=$(findfs LABEL=local-scratch)
if [ $? == 0 ]
then
    echo "local-scratch filesystem already exists: ${fs}"
else
    target=/dev/vda
    echo "Creating local-scratch filesystem on ${target}"
    mkfs.ext4 -FL local-scratch "${target}"
fi

wwclient runs this script before it passes init on to systemd, so it is also processed before fstab. So we can mount the "local-scratch" file system just like any other disk in fstab.

LABEL=local-scratch /mnt/scratch ext4 defaults,X-mount.mkdir,nofail 0 0

The Warewulf 4 overlay system allows us to deploy customized files to nodes or groups of nodes (via profiles) at boot. For this example, I've placed my customized fstab and init script in a "local-scratch" overlay and included it as a system overlay, alongside the default wwinit overlay.

# wwctl overlay list -a local-scratch
OVERLAY NAME                   FILES/DIRS  
local-scratch                  /etc/        
local-scratch                  /etc/fstab.ww
local-scratch                  /warewulf/   
local-scratch                  /warewulf/init.d/
Local-scratch                  /warewulf/init.d/70-mkfs-local-scratch

# wwctl profile set --system wwinit,local-scratch default
# wwctl overlay build

Because local-scratch is listed after wwinit in the "system" overlay list (see above), its fstab overrides the definition in the wwinit overlay. 70-mkfs-local-scratch is placed alongside other init scripts, and is processed in lexical order.

A node booting with this overlay will create (if it does not exist) a "local-scratch" file system and mount it at "/mnt/scratch", potentially for use by compute jobs.

Disk partitioning

But perhaps you want to do something more complex. Perhaps you have a single disk, but you want to allocate part of it for scratch (as above) and part of it as swap space. Perhaps contrary to popular opinion, we actively encourage the use of swap space in an image-netboot environment like Warewulf 4: a swap partition that is at least as big as the image to be booted allows Linux to write idle portions of the image to disk, freeing up system memory for compute jobs.

So let's expand on the above pattern to actually partition a disk, rather than just format it.

#!/bin/sh
#
# /warewulf/init.d/70-parted

PATH=/usr/sbin:/usr/bin

# KVM disks require a kernel module
modprobe virtio_blk

local_swap=$(findfs LABEL=local-swap)
local_scratch=$(findfs LABEL=local-scratch)

if [ -n "${local_swap}" -a -n "${local_scratch}" ]
then
    echo "Found local-swap: ${local_swap}"
    echo "Found local-scratch: ${local_scratch}"
else
    disk=/dev/vda
    local_swap="${disk}1"
    local_scratch="${disk}2"

    echo "Writing partition table to ${disk}"
    parted --script --align=optimal ${disk} -- \
        mklabel gpt \
        mkpart primary linux-swap 0 2GB \
        mkpart primary ext4 2GB -1

    echo "Creating local-swap on ${local_swap}"
    mkswap --label=local-swap "${local_swap}"

    echo "Creating local-scratch on ${local_scratch}"
    mkfs.ext4 -FL local-scratch "${local_scratch}"
fi

This new init script looks for the expected "local-scratch" and "local-swap" and, if either of them is not found, uses parted to partition the disk and creates them. As before, this is done before fstab is processed, so we can configure these with fstab the standard way.

LABEL=local-swap swap swap defaults,nofail 0 0
LABEL=local-scratch /mnt/scratch ext4 defaults,X-mount.mkdir,nofail 0 0

This configuration went into a new parted overlay, allowing us to configure some nodes for "local-scratch" only, and some nodes for this partitioned layout.

# wwctl overlay list -a parted
OVERLAY NAME                   FILES/DIRS  
parted                         /etc/        
parted                         /etc/fstab.ww
parted                         /warewulf/   
parted                         /warewulf/init.d/
parted                         /warewulf/init.d/70-parted

# wwctl profile set --system wwinit,parted default
# wwctl overlay build

(Note: I installed parted in my system image to support this; but the same could also be done with sfdisk, which is included in the image by default.)

Persistent storage for logs

Another common use case we hear concerns the persistence of logs on the compute nodes. Particularly in a failure event, where a node must be rebooted, it can be useful to have retained logs on the compute host so that they can be investigated when the node is brought back up: in a default stateless deployment, these logs are lost on reboot.

We can extend from the previous two examples to deploy a "local-log" file system to retain these logs between reboots.

(Note: generally we advise not retaining logs on compute nodes: in stead, you should deploy something like Elasticsearch, Splunk, or even just a central rsyslog instance.)

#!/bin/sh
#
# /warewulf/init.d/70-parted

PATH=/usr/sbin:/usr/bin

# KVM disks require a kernel module
modprobe virtio_blk

local_swap=$(findfs LABEL=local-swap)
local_log=$(findfs LABEL=local-log)
local_scratch=$(findfs LABEL=local-scratch)

if [ -n "${local_swap}" -a -n "${local_log}" -a -n "${local_scratch}" ]
then
    echo "Found local-swap: ${local_swap}"
    echo "Found local-log: ${local_log}"
    echo "Found local-scratch: ${local_scratch}"
else
    disk=/dev/vda
    local_swap="${disk}1"
    local_log="${disk}2"
    local_scratch="${disk}3"

    echo "Writing partition table to ${disk}"
    parted --script --align=optimal ${disk} -- \
        mklabel gpt \
        mkpart primary linux-swap 0 2GB \
        mkpart primary ext4 2GB 4GB \
        mkpart primary ext4 4GB -1

    echo "Creating local-swap on ${local_swap}"
    mkswap --label=local-swap "${local_swap}"

    echo "Creating local-log on ${local_log}"
    mkfs.ext4 -FL local-log "${local_log}"

    echo "Populating local-log from image /var/log/"
    mkdir -p /mnt/log/ \
      && mount "${local_log}" /mnt/log \
      && rsync -a /var/log/ /mnt/log/ \
      && umount /mnt/log/ \
      && rmdir /mnt/log

    echo "Creating local-scratch on ${local_scratch}"
    mkfs.ext4 -FL local-scratch "${local_scratch}"
fi

For the most part, this follows the same pattern from the "parted" example above; but adds a step to initalize the new "local-log" file system from the directory structure in the image.

Finally, the new file system is added to fstab, after which logs will be persisted on the local disk.

LABEL=local-swap swap swap defaults,nofail 0 0
LABEL=local-scratch /mnt/scratch ext4 defaults,X-mount.mkdir,nofail 0 0
LABEL=local-log /var/log ext4 defaults,nofail 0 0

Some applications may write logs outside of /var/log; but, in these instances, it's probably easier to configure the application to write to /var/log than to try to capture all the places where logs might be written.

The future

There are a few more use cases that we sometimes hear brought up in the context of stateful node provisioning:

How can we use Ansible to configure compute nodes?
How can we configure custom kernels and kernel modules per node?
Isn't stateless provisioning slower than having the OS deployed on disk?

If you'd like to hear more about these or other potential corner-cases for stateless provisioning, get in touch! We'd love to hear from you, learn about the work you're doing, and address any of the challenges you're having.

Warewulf Deep Dive | CIQ Webinar

Jonathon Anderson — Thu, 07 Jul 2022 21:58:59 GMT

I'm really excited about Warewulf 4! I'm also really excited about OpenHPC, Apptainer, and containerizing HPC workloads, particularly MPI. Today I presented my most recent work in these areas in the CIQ webinar.

We're doing these webinars every week! I hope you'll join us Thursdays at 11:00 Pacific time, either via YouTube and LinkedIn.

Apptainer Signatures | CIQ Webinar

Jonathon Anderson — Thu, 19 May 2022 19:25:28 GMT

I had the pleasure of participating in my first CIQ webinar today! Check it out if you'd like to learn a bit about Apptainer's support for cryptographic signatures, using well-established PGP infrastructure and paradigms.

I hope you'll join us for our next session! We're live every Thursday at 11:00 Pacific time, streaming to YouTube and LinkedIn.

Ancient Astronauts | Scifi Camp with Andrew Sears

Jonathon Anderson — Mon, 25 Apr 2022 06:00:00 GMT

I had a lot of fun talking about Ancient Astronauts with my good friend Andrew Sears on his new podcast series Scifi Camp. We talk about Stargate, Mass Effect, and the fuzzy boundary with faith and religion. I hope you enjoy it as much as I did!

the person I love the least

Jonathon Anderson — Mon, 18 Nov 2019 04:28:09 GMT

I really only love God as much as I love the person I love the least.

~ Dorothy Day, journalist

God in your own image

Jonathon Anderson — Mon, 18 Nov 2019 04:06:35 GMT

You can safely assume you’ve created God in your own image when it turns out that God hates all the same people you do.

~ Anne Lamott quoting her “priest friend Tom” in Bird by Bird

an open letter to the TED team

Jonathon Anderson — Sun, 03 Nov 2019 22:57:14 GMT

Some time back I was talking with church leadership about possible opportunities for me to serve the church, including potential calling to leadership. At the time I insisted that theological differences between myself and the church disqualified me.

I still believe I have important theological differences with the church, even if only in my own questioning and theological formation; but I have felt a weight of consideration ever since, and I believe that the Spirit is provoking me to action. What that means I do not yet know, but I decided that, at the very least, I could formally put my name in for consideration by the church, and leave it to the church, rather than my own preemption, to disqualify me.

After applying I was asked to schedule a 30-minute conversation with the church's trustees, elders, and deacons (TED) team. As part of the conversation, I will be expected to share responses to the following questions:

Please share a brief statement (3-5 minutes) about your faith in Christ and how your personal relationship began and continues.

I have been raised in the church my entire life, my father a Christian Rock Band Jesus Hippie and my mother proselytized by him early in their relationship. I didn't appreciate the full impact of my upbringing until adulthood; but my father always had an earnest heart for God, and my mother a love for everyone around her. I grew up assuming these characteristics, and I am so thankful that they are encoded so deeply in who I am.

As a family we attended a Church of the Nazarene; but I also exclusively attended a small private school, kindergarten through highschool run by a fundamentalist Baptist church, followed by four years at a Nazarene univeristy. Church, has been embedded in virtual every aspect of my life for as long as I can remember; and more than that, a pervasively diverse church context means that I have always had to acknowledge and consider different, conflicting, and often opposing views of God and the scripture, even within Christianity.

This much pervasive access to church can make the experience of God somewhat mundane; so it wasn't until somewhere in my teens that I really felt God alive in my life. It's somewhat cliché, but I attended a weekend cursillo retreat run by local Methodist churches as part of The Upper Room ministries. I shouldn't have needed it--I had exactly the same kind of example at home already--but it was there that I first recognized the difference between assumptively "going to church" and living a life that is transformed by the Spirit and oriented toward God. I continued to work (and speak) at the retreat for years after, and my entire family attended successively afterward.

The next notable impact to my faith came as a result of my wife and I living for three and a half years in the Kingdom of Saudi Arabia. Though we were only there professionally, the facts of being a Christian in the nation of Isalm cannot be avoided--and why should they? There our understanding of God and Christ was deepended through comparison and contrast with the people around us. We met several strong Christians, largely among the students, and gathered to worship in our homes (eventually our home specifically) every week.

We have worshiped almost exclusively at Presbyterian churches since returning to the States, first at Redeemer Presbyterian in New York and now at First Pres in Boulder, an extension of our existing desire to challenge our own theological background and assumptions through comparison and contrast.

What is currently going on in your life spiritually? What is God teaching you? What growth are you experiencing?

Fankly, this is something going on in my life spiritually: I am trying to be ever more open to the leading of the Spirit, particularly where I habitually resist Him. For example: when we were first asked to open our home to our church in Saudi Arabia, I literally said, "OK; as long as I'm not expected to lead." Of course, it was scarcely a year later that I was preparing scripture readings, selecting songs for us to sing together, and distributing communion, after our previous leaders graduated from their respective academic programs and returned home.

I love theology, and I try to not let my esoteric interests get in the way of ministry and community. Most recently I have been reading the work of David Bentley Hart. I finished his defense of universal salvation "That All Shall Be Saved" and found it a profound challenge to what might be the last vestiges of my fundamentalist assumptions about hell; but I am currently re-reading the New Testament with his defense in mind to better discern my assumptions about scripture from my actual reading of it. After and during that I am also reading his defense of theism: "The Experience of God."

More personally, the Spirit is convicting me regarding how I respond to disagreement in my marriage. I say this here only because my comments so far have been largely academic, and I don't want to imply that the Spirit doesn't affect me more intimately as well; but, while I'm open to discussing such things, I tend to wait for them to be asked explicitly as well.

How did you come to be a part of First Pres? What experiences have you had as a part of our church family? What excites you about the future?

We came to Boulder largely chasing my wife's family (in Greeley) and the church family that we had had in Saudi: themselves largely from the Boulder area. When we surveyed the churches in the area we found mostly a certain type of closed conservatism, a certain type of loose liberalism, or a certain type of seeker congregation: none of these seemed to fulfill our expectations to be challenged toward growth in the faith. But we found First Pres, supported by our previous experiences at Redeemer, and encouraged by what we heard in sermon recordings online. We attended and, in our first services we encountered academic theology (led by Carl); passionate tradition (bagpipes); and earnest community (in coffee with Erik).

We showed up to Family Small Groups without prior arrangement; and, though there was no group for us to join on the spot, our chidren were cared for and we were told to enjoy the evening together.

At First Pres I feel at home, in a way that I have not felt since I left my childhood church.

How do you seek to discern God's will for you personally? How might you discern God's will in a group setting?

I seek God's will for me personally through prayer and study; but I cannot ignore the transforming work of the Spirit in my life as well. I can scarcely believe my life, and I am excited by the prospect of even deeper relationship with God.

In a group setting, if my self-assessment is accurate, I have a tendency and aptitude for listening to all perspectives and helping to bring parties to at least a common understanding. I consider what is said long after a group meeting, and often follow-up off-cycle to ask questions or assert possibilities. I pray, but more generally I believe that I feel the Spirit leading throughout the day, and I hope that I would be able to discern that leading in a more formal group as well.

Perhaps more technically, I argue. And I hope that that is not understood as argumentative; but I try what I believe to be Truth by presenting it for scruitiny. I am strong in my beliefs, but I am also quick to abandon my own misunderstandings. When I argue, I argue from scripture and (where it is a help and not a distraction) well-established shared belief.

I was also pointed to the Essential Tenents of ECO [1]; and it is here that I am afraid I will have the most trouble. As above, I don't intend to be argumentative; but I also do not want to conceal anything, so I will do my best to enumerate my concerns here.

I want to be clear: I do not begrudge ECO or First Pres these essential tenents. I recognize the importance of common doctrine, and I value the diversity of Christianity as expressed in the diversity among congregations: that diversity does not necessarily need to be expressed within each congregation. Still, First Pres is my home; and if I am to serve here, I have to be honest about what I believe as well.

Regarding "God’s Word: The Authority for Our Confession"

I have serious concerns regarding the common definition of the Word of God. Even today, Carl preached in his sermon that God's word is incarnate, proclaimed, and written; but the essential tenents omit acknowledgement of God's word as proclaimed altogether.

I further fear, and have for many years, that the veneration of the so-called "written Word of God" is a form of idolatry: the Bible serves as an image of God's Word, and its worship (in everything but name) is troubling to me. More striclty, I consider the scripture a testament to the Word of God, not the Word itself (as opposed to Christ, the Word Incarnate.)

I do acknowledge, as Paul taught Timothy, that "All Scripture is breathed out by God and profitable for teaching, for reproof, for correction, and for training in righteousness, that the man of God may be complete, equipped for every good work." But I'll still point out that Paul certainly was not talking about the gospels, the revelation, or much less the epistles (particularly that he was contemporaneously writing), but "the sacred writings" that Timothy had been acquanted with "from childhood." This is not to say that the Christian scriptures are not themselves "Breathed out by God," and I have faith in the work of the Spirit in preserving the Scripture through church tradition and study; but I believe the true nature of the scripture is more complex than is often exhorted in such essential tenents, and the true nature of the Word of God more complex still.

We confess that God alone is Lord of the conscience, but this freedom is for the purpose of allowing us to be subject always and primarily to God’s Word.

We are happy to confess ourselves captive to the Word of God

Perhaps there is some scriptural basis for this imagery that I am missing; but without it I am troubled by this imagery. Life in Christ is denotatively freedom; we are not captives, but heirs, ransomed from sin and death. I do not deny that "the Spirit will never prompt our conscience to conclusions that are at odds with the Scriptures that He has inspired"--perhaps this is an unnecessary complaint; but the heart with which we approach the Word matters to me, and I find it important to recognize that the Spirit changes our desires to be those of God; we are not captive to the Word, but freed by it.

Regarding "secondary authority"

[W]e affirm the secondary authority of the following ECO Confessional Standards as faithful expositions of the Word of God: Nicene Creed, Apostles’ Creed, Heidelberg Catechism, Westminster Confession, Westminster Shorter Catechism, Westminster Larger Catechism and the Theological Declaration of Barmen.

I don't know how I missed this before; but this greatly expands the scope of the so-called "essential" tenents. I have studied some of these; but certainly not all, and I would be loathe to assumptively confirm their authority in my theology, or my adherence to them, without further study (and given the differences raised by the primarily-stated essential tenents, I can only expect there would be further differences in a greater body of confessions).

regarding "Trinity and Incarnation: The Two Central Christian Mysteries"

I have strong, fundamental concerns regarding the doctrine of the Trinity.

But first let me be clear: I believe in God, non-contingent, transcendent, Father and creator of all. I believe that Jesus, the Christ, is the incarnate Word of God, one with the Father, in the Father and in whom the Father is. I believe in the Holy Spirit, the paraclete, the helper and advocate, who comes in the name of Christ and is sent by the Father, the Spirit of Christ and, thus, the Father.

But there is a great deal of distance between that and bold claims about God as having a fundamentally "trinitarian" nature. This is not quite idolatry; but it is seeking to define God by our experience of him, where he is more accurately transcendent. God has revealed himself freqntly through social terms, but it is eisegesis to read this as emphatically trinitarian. God did not direct Israel, for example, to worship "God the Creator; God the Fire of the Bush; and God the Pillar of Cloud"; but God. And if God did not direct worship to a plurality, but a unity God, then we should not break from that direction.

And what of the Word of God? Surely the Word of God is God, as John proclaimed. But if Christ is one part of a trinity God, then surely the Word is a person of God, existing before "the Word became flesh and dwelt among us." So perhaps the trinity is more accurately "the Father, the Word, and the Spirit"?

But I am particularly troubled by extra-Biblical habits I have observed recently of praying to individual "members of the trinity"; we are to pray to God, the Father, as Christ did and directed us to.

affirmed by all Christians everywhere

This is simply not true: there have been many Christians that have had different interpretations and understandings of the being of Christ. I may not agree with them, but to ignore them is distracting and disingenuous.

like us in having both a human soul and a human body

This anthropology isn't Biblical, so far as I can tell. Maybe it is technically true to say that Christ had a human soul; but this statement does not mean what a western mind will infer from it. God did not make a human body and then put a human soul into it; man became a living soul when God breathed into it. As such, to say that Christ is "like us in every way but sin" but then say that he has a "human soul" is both non-sensical and contradictory.

Regarding "God’s grace in Christ"

Our desires are no longer trustworthy guides to goodness, and what seems natural to us no longer corresponds to God’s design.

I hope that these tenents do not mean to indicate that we who are alive in the Spirit are unable to discern good. "We have received [...] the Spirit who is from God, that we might understand the things freely given us by God. And we impart this in words not taught by human wisdom but taught by the Spirit, interpreting spiritual truths to those who are spiritual." It is the promise of life in the Spirit that our hearts are turned towards the things of God; that our desires are made trustworthy, being those of the Spirit.

Jesus takes our place both in bearing the weight of condemnation against our sin on the cross

I require further study here; but I believe this to be incorrect theology. Christ did not "bear the weight of [presumably God's] condemnation against our sin"; in stead, his death paid the ransom to free us from our slavery to death.

Regarding "Election for salvation and service"

I am thankful that ECO does not, at least here, go so far as to proclaim limited atonement an essential tenent. (Perhaps it does implicitly by extension through one of the "secondary" authorities.) But I must say that the language of atonement does not appear, to me, to be concerned with eternal salvation or the church in general, but of specifically the work of the Spirit in Israel in the church age. "Israel failed to obtain what it was seeking. The elect obtained it, but the rest were hardened." But later "a partial hardening has come upon Israel, until the fullness of the Gentiles has come in. And in this way all Israel will be saved."

"just as you were at one time disobedient to God but now have received mercy because of their disobedience, so they too have now been disobedient in order that by the mercy shown to you they also may now receive mercy. For God has consigned all to disobedience, that he may have mercy on all."

Therefore I hold that, at the very least, the concept of election as expressed by Paul does not reflect eternal salvation, or its absence; but the work of God in the lives of some for the age towards an ultimately redemptive purpose for all.

One last thing: Paul explicitly doesn't use the word "elect" to refer to Gentiles; only Israel.

Regarding "Living in obedience to the Word of God"

I note here only to claim this commandment as expressed:

pursue truth, even when such pursuit is costly, and defend truth when it is challenged, recognizing that truth is in order to goodness and that its preservation matters;

on the defiance of expectations in Epic Mickey

Jonathon Anderson — Mon, 25 Mar 2019 06:00:00 GMT

I love it when a game defies my expectations sufficiently to make me uncomfortable. If a game can make me feel discomfort, there's something worth considering there--something that merits deeper understanding. There are things that a game can say about the player that couldn't be said in any other medium, and sometimes the message is all the more effective when I'm caught vulnerably by my own assumptions.

My first experience with this kind of discomfort came during my first playthrough of Mass Effect 2. Near the end of the game Shepard--the protagonist and player character--has accumulated a band of compatriots toward a final mission to stop the Reapers; but just before that final mission, the ship's crew is abducted by the Collectors.

Mass Effect is a role-playing game, and understanding genre tropes is an important aspect of interpreting a work and its impact. Many fantasy role-playing games have a similar plot point: the hero has completed his preparations. He is near the end of his journey. The stakes have never been higher, and the situation is urgent: Meteor is about to crash into Midgar; Gannon is about to destroy Hyrule; or, as is the case in Mass Effect 2, the Reapers are preparing to consume all life in the galaxy.

But role-playing games have another trope: the side quest. These are typically available throughout the game; but the moment before the final climactic mission is the last chance in most RPGs to finish up any side-quests that have been left undone. In Mass Effect 2, side quests take the form of "loyalty missions"--character-specific missions that provide additional backstory and inter-personal context for the members of your cohort. Completing these missions also improves an invisible but important loyalty stat which affects how team members respond to Shepard.

I'm a bit of a completionist, so I took this opportunity before the final mission to complete all of these loyalty missions. I did this all while I poked fun at the video game tropes on display: the big bad, poised and ready to attack; we, the player character, traipsing about the galaxy on unrelated menial missions. After all: Meteor won't crash into Midgar until the plot is ready for it; Gannon never will destroy Hyrule; and the Collectors will wait around until Shepard is good and ready to face them.

But that's not what happens. When I finally did embark on the final mission to stop the Collectors and rescue the crew, we found only Dr. Chakwas alive.

They're gone. All of them. I'm the only one left.

I watched them die. They were... processed--rendered down into some kind of raw genetic paste and pumped through these tubes.

What took you so long, Shepard? You could have saved them if you'd gotten here sooner!

Dr. Chakwas' words are true. While I was taking my time maximizing a gamified loyalty stat, the game was monitoring my activities after the abduction of the crew. Leave immediately, and you may save them all; but the longer you wait, the more of them die.

With this, the game defies trope, and punishes the player for approaching the work as a simple genre piece. In reality, Shephard would never meander about, but would prioritize the mission and the retrieval of the crew. But it's just a game, right?

But it is the fact that it is a game that enables this experience. A character in a book won't die because you waited a week to read the last chapter; but in my Mass Effect, we lost the entire crew: named characters with backstories and interactions that had developed throughout the game. And the consequences don't end there, either: Mass Effect is a three-part series, and the death of these characters carries on even into the next game.

Mass Effect 2 expects you to care about its characters; and, if you don't--if you just play it like a video game, expecting it to behave like other video games--it punishes you for it by taking those characters away.

But even then, I never would have expected to feel this same defiance of expectation from Epic Mickey.

Epic Mickey could hardly be more different from Mass Effect. It's a third-person platforming character action game with light adventure elements. It's a children's game, contrasted with media hysteria regarding Mass Effect's "mature" content. More immediately, Mass Effect is a good game; and I definitely wasn't enjoying Epic Mickey.

But I have kids, and those kids were excited about Mickey, so I was playing through it as a social activity with them. I really wasn't taking it seriously: jump on the platforms; paint the environment with the magic paintbrush; mash "A" when characters talk to you; make "progress."

Not too far into the game, I ran into a character called "Small Pete," a rendition of a classic Disney character, "Pete," who often serves as the antagonist of a Mickey Mouse story.

I spent years getting' along with gremlins. Only had to knock 'em around on occasion. Then, the ONE TIME I crash my boat into their village, they seem to think I'm some kinda villain.

Not that I give two hoots what they think, but it WAS an accident. And my ship's log will prove it.

Those little monsters won't let me near the wreck to get it, though. Hmm... I'll bet they'd let you.

I was immediately suspicious of Small Pete's story (assuming I paid it any mind at all, beyond just mashing "A"); but we got a quest objective and moved on.

I continued jumping between platforms, tagging the environment, and mashing "A," until we met Gremlin Shaky.

I smell treasure! You found it!

How's about you trade me that ship's log for a flashy new pin?

I still wasn't paying attention. Why would I? The platforming was mediocre. The characters were either flat or carbon-copies of each other. Each gremlin looks the same as all the others. So I interpreted this interaction with the same level of attention that I would pay to most collect-a-thon games:

"Oh, right. The ship's log. I guess I picked it up along the way. Pete wanted us to get that for him, right? What was that for, again? This must be the guy I'm supposed to give it to. And when I do I'll get a pin as a reward, eh? Ok, I guess it's a collectible, so I guess I'll do it."

Thank you very much. This will make excellent reading. Here's your pin.

But that wasn't all there was to it. Immediately after finishing the interaction I received a "quest failed" notification.

I kept playing, just accepting that I had failed the quest, and probably missed out on some minimal benefit. But something about the interaction bothered me. Small Pete seemed to be a character teetering on the edge of villany. He was willing to "knock 'em around on occasion"; but he seemed genuinely (if covertly) concerned with clearning his name. He wasn't a villain yet. He was a bully.

So you left my ship's log with those grubby gremlins, eh? Well, here's a little taste of what happens to those who cross me!

I had betrayed Small Pete. I hadn't done it out of malice. Worse: I had paid him no mind. He asked for help, and I ignored him. Eventually, I traded his name for a collectible pin I didn't even care about. In a literal sense, I had turned him into a villain: Small Pete had become a video-game boss, generating a combat encounter to punctuate the chapter.

I found myself considering what it would take to correct this mistaken path through the game's narrative. I had overwritten my save several times since I had given Pete's ship's log to Gremlin Shaky. I would have to start the game over from the beginning.

The very fact that I was considering it made me uncomfortable. I did not enjoy playing this game. But, for the sake of a fictional character as absurd as Small Pete, I was considering sacrificing some portion of my time in pursuit of his redemption.

I tell my kids that it's part of a parent's job to give them consequences that they can learn from and grow through, while protecting them from consequences that they can't recover from, if only for a time. In a small, but very real, way, Epic Mickey was that for me. I ignored a call for help. I was careless. A character was treated unjustly, and that injustice led him to embrace his own darker tendencies.

I never did go back and do right by Small Pete. In fact, I don't think I played the game again after that. I'm sure we were called down for dinner, and then distracted by another game I enjoyed playing more. But I still think about Small Pete, about the time I didn't pay enough attention, and about the consequences that might develop when I allow myself to become just a little bit more callous to the world around me.

Home (POA)

Jonathon Anderson — Mon, 17 Sep 2018 05:14:00 GMT

I just posted the entirety of the P.O.A. album, "Home."