Tech. Team weekly review for 31 July, 2017

This week at the University of Colorado Research Computing Tech. Team...

New HPC Storage Admin, Patricia Egeland

Were're bringing on a new team member, Patricia Egeland, as HPC Storage Administrator, starting Tuesday. Patricia will share general system administration duties with the other RC tech. team operational staff; but will carry primary responsibility for RC data storage systems, notably RC Core Storage and the PetaLibrary. She also plans to contribute to ongoing and upcoming software development efforts at RC, and we're looking forward to seeing more interest in that space both internally and in our user community.

Patricia worked most recently as a systems analyst for the Dark Energy Survey (DES) Science Portal, and previously worked as a server, system, and application administrator for the CERN Compact Muon Solenoid (CMS) experiment.

We couldn't be more pleased to have Patricia as a member of the RC tech. team! Please join me in welcoming her if you have an opportunity to work with her.

Final deployment of HPCF UPS

We'll have the final of our three UPS-related HPCF outages starting on Wednesday, and closing out in the afternoon on Friday. During this outage we'll be...

  • re-routing the power cabling between the UPS infrastructure and the in-row power-distribution infrastructure for future maintainability;

  • decommissioning the legacy UPS;

  • installing additional in-row power distribution infrastructure;

  • and bringing the new UPS into full production.

The new HPCF UPS will provide not only power conditioning (remediating a power quality issue that has led to several past Summit compute outages) but also at least 15 minutes of UPS-backed runtime in the event of a complete utility power outage (which should eventually provide us sufficient time to power the system off in a controlled manner).

PetaLibrary/2 RFP goes live

Research Computing successful PetaLibrary service is getting a refresh! Or, at least, that's the intent. We're publishing an RFP today for a new, unified infrastructure, which should extend the life of the PetaLibrary, simplify our service offerings, make the infrastructure more maintainable, and eventually allow us to add additional features and services.

Monday, 31 July 2017

RFP Posted online

Friday, 4 August 2017 (09:00)

Optional Pre-bid call

Friday, 11 August 2017

written questions are due

Tuesday, 5 September

RFP responses are due

XSEDE SSO hub authentication progress

RMACC Summit is intended, as its full name implies, to be an RMACC resource, not just a CU or CSU resource. We've planned from the beginning to support access to Summit through XSEDE credentials, but this has required additional (though already planned) service development at XSEDE. Those services are ready for beta testing now, and CU is on hand as an early adopter for their new "single sign-on hub for L3 service providers" service (XCI-36). We're working on deploying this now, and will hopefully be able to start bringing on early-adopters from the RMACC community soon.

Misc. other things

  • We're rebuilding the RC login environment. We've been through a few prototype efforts, but the current plan is to start by deploying a new tutorial login node, tlogin1, which will also be the first recipient of new XSEDE authentication services.

  • We're continuing to develop our internal "curc-bench" automated benchmarking utility for validating the performance of RC HPC resources over time (notably after we make changes). Development is primarily driven by Aaron Holt.

  • We had to rebuild Sneffels (originally "the viz cluster") after a security incident. That work is largely done, and service has been restored, but OIT is sill reviewing viz1 as part of our incident response process.

  • We're updating the Globus software for our data-transfer service, starting with dtn01. We're further taking this opportunity to re-build our DTN configuration in general, which should lead to better and more reliable data-transfer performance due to the correction of a number of networking irregularities on these servers. This work is being done primarily by Dan Milroy.

Tandem Gaming | Game Older

tandem bicycle

Andi and I play games together, but not really traditional multiplayer games. Join us for a long-delayed, rambling conversating about what games bring us together; and how it affects us, our friends, and our family.

Games mentioned

  • Myst (series)

  • Prince of Persia: Sands of Time

  • Legend of Zelda: Twilight Princess

  • Peasant's Quest

  • Donkey Kong Country

  • Donkey Kong Country Returns

  • Valkyria Chronicles

  • New Super Mario Bros.

  • Super Mario Galaxy (series)

  • Mass Effect (trilogy)

  • Metal Gear Solid (series)

  • Obduction

  • The Witness

  • Ibb and Obb

  • Dark Souls 2

  • Snipperclips

  • The Walking Dead

  • Heavy Rain

  • Indigo Prophecy

  • Half Life 2

  • Portal (series)

  • Fable 2

  • Oblivion

  • Final Fantasy

  • Final Fantasy X

  • Legend of Zelda: Oracle of Ages and Oracle of Seasons

  • World of Warcraft

  • Legend of Zelda: Minish Cap

Music used

  • Myst Theme by Robyn Miller, from Myst

  • Succeeded Wish (instrumental) from the Valkyria Chronicles "Steam" Launch Trailer

A prayer from MLK day

Chris Hill, a member of our church, shared this prayer in the context of the then-upcoming presidential inauguration and Martin Luther King, Jr. Day.

In a time of self-described conservatism vs liberalism, I found it remarkably neither, but only Christian.

Father we approach you today with many gratitudes, thoughts and requests. As a community we first empty our hands of those things that do not belong to us. We lay down our worldly possessions, those things that you have loaned us. We lay down our worldly successes and failures, which do not define us. And we lay down the pride that so easily devours us and those we live around. We bring before you our weakness, and thank you for it. We understand that without it, we would not see our desperation for you.

Behold us, Father, as we Behold you. See us. Understand us. Know our Human hearts. Together, today, we want to bring before you two events that we will undoubtedly carry with us this week. We bring these before you in faith that you are worth approaching and worth glorifying. We also bring these before you recognizing that only you are Good.

As Barak Obama and his family leave the presidency, we thank you for the ways you have worked during the 8 years he has served our country. Would you bless his family as they adjust to life outside of the white house. And as Donald Trump and his family transition into the presidency, we pray with hope and expectation that you would use them to strengthen the Kingdom of Heaven. Give us the strength to bear with one another in love and patience.

As we remember the life and work of Martin Luther King Jr. would you have mercy on us. Will you call us out of our complacency as middle to upper class white America to gaze across the scene and remember what’s painfully obvious and self-evident. That the person we see with a different skin color than our own is, in fact, a person. An image bearer. A jar of clay containing my blood, and your Holy Spirit. Today Father, we remember that much of what happened during the Civil Rights movement was guided and fueled by your Spirit. Thank you for gifting us a man who was able to give an ear to you and an ear to the people while persevering through an agony and persecution that few of us in this room understand. Thank you for the lives of our brothers and sisters who have fought to level our view of humanity. But if we are going to acknowledge what has been done Father, we will also acknowledge the work that still needs to be done. Would you equip those of us who are not white with hope, strength, and perseverance. Would you equip those of us who are white with ears to hear and eyes to see your children.

We believe that you, Holy Spirit, are the only one who can bring reconciliation between us and our neighbor. So we lean into you, ready to be led. Jesus, you are all and you are in all. We ask these things in your perfect name. Amen.

Best possible experience reading Silence

I read Shūsaku Endō's Silence as part of a book club with my pastor and a few other members of our church. The book was scheduled coming out of the holiday season so that if we didn't have time or motivation to read we could at least all watch the new film together and discuss the story.

I hadn't managed to finish the book by the time I reached the theatre. When I left Rodrigues (on the page) he was being brought before the authorities for interrogation and defending the purpose of the church in Japan.

When we passed that moment in the film, I appreciated the fresh perspective of watching the story play out on screen, but I realized that I had actually managed to remain unspoiled on the remaining plot. When Rodrigues was climatically confronted with the decision to trample on the fumi-e or allow others to suffer, I was overwhelmed by the cumulative anticipation of not one but two readings: I've never before experience so palpable a moment of, "I have no idea what is about to happen."

I've wrestled for years with the question of whether it would be sin to accept damnation--defined here as separation from God--for the sake of another's salvation. Self-sacrifice is good; but might such a sacrifice be construed an elevation of man over God?

Through Silence I've concluded that such a sacrifice is good, but that its consequence is inherent: damnation. In fact, in Christian theology, this is the sacrifice Christ made for us, and only Christ could both endure all of our damnation and still remain blameless.

And still, salvation through Christ is sufficient even for those who would deny him for the sake of others. It's obvious when you consider the apostle Peter, who famously denied association with Christ three times; but I hadn't before seen this portrayed so vividly, and the story of Peter is perhaps too familiar to be so impactful. It's easy to vilify Kichijiro when he repeatedly betrays the Kirishitans, and to become dismissive as Rodrigues when the acts of confession and atonement becomes rote and seemingly meaningless; but Rodrigues and Kichijiro both demonstrate what Peter did in the Passion: that Christ offers forgiveness and reconciliation even to those who betray him.

After more consideration, though, I fear that the Silence that has affected me so deeply exists only in my own heart and mind. The book, perhaps more than the film, might actually be more concerned with a technical definition of apostasy and Rodrigues' prideful self-image as a Christ figure than it is with deeper questions of the nature of salvation. He's a bit like Job, in a way: so assured of his blamelessness and rite of martyrdom that he can't see how he himself falls short of the perfection he aspires to.

But I still can't stop thinking about Silence, and I'm struck more than ever by the potential discontinuity between the story the author wrote and the story in my mind.

I can't imagine what Silence must mean to a Japanese Buddhist. From my western Christian perspective the story is familiar enough, and I implicitly understand the context and motivation of Rodrigues and his fellow Jesuits. But what I read was an English translation from original Japanese, ostensibly intended for a Japanese audience, and that presumably non-Christian. How could a Japanese person, with no experience with the church or Christ, possibly react to any of this?

The Witness | Game Older

The Witness

Long imagined but never before realized, I managed to get one of my non-gamer friends to actually play a game and share their experience with me! My father-in-law saw me playing The Witness over Christmas, and didn't want me to leave before I set him up to play it after I was gone. More than that, he totally bought in to engaging with it at a satisfying philosophical level. I loved having the conversation, and I hope you enjoy listening to it.

Music used

  • Escape Artist by Zoe Keating (from the trailer for The Witness)

The SSH agent

This is one part in a series on OpenSSH client configuration. Also read Elegant OpenSSH Configuration and Secure OpenSSH Defaults.

As part of another SSH client article we potentially generated a new ssh key for use in ssh public-key authentication.

$ ssh-keygen -t rsa -b 4096 # if you don't already have a key

SSH public-key authentication has intrinsic benefits; but many see it as a mechanism for non-interactive login: you don’t have to remember, or type, a password.

This behavior is dependent, however, on having a non-encrypted private key. This is a security risk, because the non-encrypted private key may be compromised, either by accidential mishandling of the file or by unauthorized intrusion into the client system. In almost all cases, ssh private keys should be encrypted with a passphrase.

$ ssh-keygen -t rsa -b 4096 -f test
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

If you already have a passphrase that is not encrypted, use the -p argument to ssh-keygen to set one.

$ ssh-keygen -p -f ~/.ssh/id_rsa

Now the private key is protected by a passphrase, which you’ll be prompted for each time you use it. This is better than a password, because the passphrase is not transmitted to the server; but we’ve lost the ability to authenticate without having to type anything.


OpenSSH provides a dedicated agent process for the sole purpose of handling decrypted ssh private keys in-memory. Most Unix and Linux desktop operating systems (including OS X) start and maintain a per-user SSH agent process automatically.

$ pgrep -lfu $USER ssh-agent
815 /usr/bin/ssh-agent -l

Using the ssh-add command, you can decrypt your ssh private key by inputing your passphrase once, adding the decrypted key to the running agent.

$ ssh-add ~/.ssh/id_rsa # the path to the private key may be omitted for default paths
Enter passphrase for /Users/user1234/.ssh/id_rsa:
Identity added: /Users/user1234/.ssh/id_rsa (/Users/user1234/.ssh/id_rsa)

The decrypted private key remains resident in the ssh-agent process.

$ ssh-add -L
ssh-rsa [redacted] /Users/user1234/.ssh/id_rsa

This is better than a non-encrypted on-disk private key for two reasons: first the decrypted private key exists only in memory, not on disk. This makes is more difficult to mishandle, including the fact that it cannot be recovered without re-inputing the passphrase once the workstation is powered off. Second, client applications (like OpenSSH itself) no longer require direct access to the private key, encrypted or otherwise, nor must you provide your (secret) key passphrase to client applications: the agent moderates all use of the key itself.

The default OpenSSH client will use the agent process identified by the SSH_AUTH_SOCK environment variable by default; but you generally don’t have to worry about it: your workstation environment should configure it for you.


At this point, there’s nothing more to do. With your ssh key added to the agent process, you’re back to not needing to type in a password (or passphrase), but without the risk of a non-encrypted private key stored permanently on disk.

Secure OpenSSH defaults

This is one part in a series on OpenSSH client configuration. Also read Elegant OpenSSH configuration and The SSH agent.

It’s good practice to harden our ssh client with some secure “defaults”. Starting your configuration file with the following directives will apply the directives to all (*) hosts.

(These are listed as multiple Host * stanzas, but they can be combined into a single stanza in your actual configuration file.)

If you prefer, follow along with an example of a complete ~/.ssh/config file.

Require secure algorithms

OpenSSH supports many encryption and authentication algorithms, but some of those algorithms are known to be weak to cryptographic attack. The Mozilla project publishes a list of recommended algorithms that exclude algorithms that are known to be insecure.

Host *

(More information on the the available encryption and authentication algorithms, and how a recommended set is derived, is available in this fantastic blog post, “Secure secure shell.”)

Hash your known_hosts file

Every time you connect to an SSH server, your client caches a copy of the remote server’s host key in a ~/.ssh/known_hosts file. If your ssh client is ever compromised, this list can expose the remote servers to attack using your compromised credentials. Be a good citizen and hash your known hosts file.

Host *
HashKnownHosts yes

(Hash any existing entries in your ~/.ssh/known_hosts file by running ssh-keygen -H. Don’t forget to remove the backup ~/.ssh/known_hosts.old.)

$ ssh-keygen -H
$ rm -i ~/.ssh/known_hosts.old

No roaming

Finally, disable the experimental “roaming” feature to mitigate exposure to a pair of potential vulnerabilities, CVE-2016-0777 and CVE-2016-0778.

Host *
UseRoaming no

Dealing with insecure servers

Some servers are old enough that they may not support the newer, more secure algorithms listed. In the RC environment, for example, the login and other Internet-accessible systems provide relatively modern ssh algorithms; but the host in the domain may not.

To support connection to older hosts while requiring newer algorithms by default, override these settings earlier in the configuration file.

# Internal RC hosts are running an old version of OpenSSH
Match host=*
MACs hmac-sha1,,hmac-ripemd160,,hmac-sha1-96