Ethan Hereth, High Performance Computing (HPC) Specialist.

I often wish I had an inspiring story about how I knew from a young age that I wanted to be an engineer, or that I’ve had a lifelong love of airplanes which led me into the field of CFD, or that I’ve been a computer geek since I was young — I always think those are the coolest success stories. Mine, however, isn’t quite like that; it is a more haphazard story.

I was born in North-West rural Arkansas, in the tourist town of Eureka Springs, fondly — or not-so-fondly, depending on your point of view — referred to as the San Francisco of Arkansas, for both its geography and its culture. I was home schooled from kindergarten all the way through high school. Pretty early on I developed an inclination for mathematics and excelled at it. I essentially taught myself towards the end as I soon surpassed my mother’s skill level such that she could no longer easily help me. This mathematical inclination is one of the main reasons I’ve ended up where I am today.

I played sports (basketball and baseball) while in high school and was a drummer in a rock band. After finishing high school, I moved to Seattle, WA with my band for a year and attempted to break into the languishing grunge rock scene (we were at least a decade too late!). This was a failed endeavor, but an experience that I’ll never forget. Afterward, I moved back to Arkansas and began my college career at UCA (I managed to score well enough on my ACT to be awarded a full scholarship there). After a semester or two, I declared myself an applied mathematics major. I graduated with my BS in 2007, and subsequently I accepted a position as a GTA at UCA while I pursued a master’s degree in applied mathematics. This is when I was introduced to the world of CFD. Dr. Clarence O. E. Burg was my advisor and had earned his Ph.D. in Computational Engineering from the SimCenter back when it was still at Mississippi State University. He was a contemporary of many of the current/past research faculty I work/worked with here at the SimCenter at UTC. Under his guidance, I began to learn the ropes. Probably one of the first meshes I ever made was for Dr. Burg using a version of an AFLR code he gave us on a CD. Two years later, I graduated with a master’s degree and a thesis involving 1D CFD titled “Numerical Simulation of Subaqueous Debris Flow in One Dimension.” Given that Dr. Burg knew most of the folks at the SimCenter, he helped a good friend of mine, Taylor Erwin, who was also working towards his M.S. under Dr. Burg’s tutelage, get a summer internship at the SimCenter. Taylor came back speaking very highly of his experience and the people at the SimCenter and intended to apply to their Ph.D. program. As I still had no idea what I was going to do with my life/career, I decided to apply as well. We were both accepted and began in the fall of 2009.

Then began intense full immersion into the world of CFD, and I do mean intense — sometimes it felt more like drinking out of a fire hose. The course work included several comprehensive 1D, 2D, and 3D CFD classes, at least two grid generation courses taught by Dr. Steve Karman, two very good parallel programming courses, amongst others. Everyone who survived the initial course work would have implemented an entire 3D solver from the ground up. I couldn’t have asked for a better program. After two years, however, I decided to take a full time job as a Research Associate at the SimCenter as a grid generation specialist (at the time I had three children — four now — and the stipend I was receiving was no longer making ends meet). At the time the SimCenter’s grid generator (often fondly referred to as ‘grid monkey’), my colleague and good friend, Bruce Hilbert, was also working towards his Ph.D. and was drowning in grid work and needed help. I had already been introduced to Pointwise during my first two years as a Ph.D. student, but this is when I began to really learn Pointwise. I enjoyed learning to become as proficient and efficient with the software as possible. I quickly developed a knack for unstructured meshing. While I’ve certainly done my share of structured meshing, it’s far from my specialty. At the 2017 AIAA SciTech conference in Grapevine TX, I saw Carolyn Dear Woeber present about her experiences meshing NASA’s High Lift Common Research Model (HL-CRM) geometry for AIAA’s third High Lift Prediction Workshop.  One of the meshes presented was a structured mesh and she showed an image demonstrating the topology of the mesh. Looking at that structured topology honestly made my head hurt. I admire people who can think like that as my brain doesn’t work that way! In order to become even more proficient, I began to train myself to use many of the accelerators Pointwise provides so I could keep my fingers on the keyboard as much as possible (as you might guess, I also exclusively use Vim). Finally, in order to become as efficient as possible, I learned to tap into the power provided by Pointwise’s Glyph scripting functionality, arguably my favorite feature of Pointwise. I even wrote a plugin for Pointwise, using their SDK, in order to export an in-house grid file format.

Five years later, in December 2016, I finally finished my Ph.D. One reason it took me so long was that I was interested in everything, grid generation, turbulence modeling, software development/optimization, etc. and I changed research topics/directions at least three times before finally choosing my final research area focusing on automatic octree-based grid generation software with an integrated CFD solution framework. See my dissertation titled “Automatic Parallel Octree Grid Generation Software with an Extensible Solver Framework and a Focus on Urban Simulation” here.

After graduating I accepted a position at the SimCenter as their High Performance Computing Specialist. Over the years, I have become a proficient Linux administrator and previously had the opportunity to fill in as the SimCenter’s HPC sysadmin for a little over a year when our long time sysadmin left for bigger and better opportunities. We have several clusters, the newest of which is a 33 node heterogeneous cluster containing the newest Nvidia P100 GPUs and 924 CPU cores. While I enjoy CFD and grid related research, I also thoroughly enjoy the hardware and software aspects of this business and I find the challenges of improving and maintaining HPC hardware/software very rewarding. I even get to use Pointwise now and again and relish the opportunities although I’m getting a little rusty!

• Location: Chattanooga, TN
• Current position: High Performance Computing (HPC) Specialist
• Current computer: Custom built desktop with twelve cores (two hex-core 2.4GHz Intel Xeon E5645s), 48 GB of RAM, and a Nvidia GeForce GTX 580 GPU
• One word that best describes how you work: Doggedly

What software or tools do you use every day?

I am probably as enthusiastic about Vim as Michael Mirsky is!  Without a doubt, I use Vim every day for many tasks, including writing email using the External Editor plugin for Thunderbird, keeping a personal knowledge base and journal using VimWiki, and editing/tweaking any number of Linux configuration files. I use Vundle to manage the myriad Vim plugins that I use daily. My favorite plugins include fugitive for tight integration of git (my favorite version control software) with Vim, easymotion for rapid text traversal, and supertab for enhanced tab completion, just to name a few. I use Google Chrome extensively to research any complexity that never fails to arise in my line of work, Thunderbird to keep up with my work email, Spotify for music to work with, Pointwise (well, almost every day), Bash (in vi mode of course!), among many other Linux utilities.

Lastly, concerning physical tools, I love my custom WASD mechanical keyboard with Cherry MX Blue switches. It seems a little silly I guess, but as much time as many of us spend on our computers, a high quality keyboard really does make a difference. I definitely notice the difference when I need to use a non-mechanical keyboard.

What does your workspace look like?

I spent a lot of time trying to decide whether or not I wanted to clean up and organize my office space before I took this picture and ultimately decided not to. My office always looks like this or worse. I always have pieces of paper lying around with random notes scribbled on them and spare computers hanging around to be imaged or moved. I’m just not super organized.

Note: This was Dr. Steve Karman’s original office at the SimCenter before he went to work at Pointwise. As I recall, it was much cleaner when he was in it!

What do you see are the biggest challenges facing CFD in the next 5 years?

I am going to parrot many things that others before me have already stated. I personally think that the geometry and grid generation challenges associated with the transition to exascale ranks very high on this list. The oft-cited CFD Vision 2030 Study makes this, among many other things, very clear. I know I’m preaching to the choir, so I’ll leave it at that.

Like others before me, I have the perception that the education of quality engineers in the field of CFD has been falling short recently. It feels to me like there are very few engineers in this arena that grok CFD like many of the founding fathers and engineers of CFD did before us. I feel that the latest generations of CFD engineers have become used to applications that do much of the work, often shielding them from having to really understand the underlying concepts.

One last thing that I’ll mention here relates to issues with higher order meshing, in particular, the visualization of said higher order meshes. I have recently had conversations with colleagues doing research in the area of higher order mesh generation, and one of the big things that they often complain about is the inability to easily visualize their meshes. I know that gmsh provides higher order finite element visualization functionality, but it tends to be difficult to use. I have heard rumors that Intelligent Light is working on improved support for visualization of higher order meshes/solutions in FieldView.

What are you currently working on?

Several things right now. High on my list currently is learning how to deploy an HPC cluster using OpenHPC, a modern, open source cluster management software. I’m learning about this by deploying a test cluster comprised of several old re-purposed Dell PowerEdge 1950 servers out of one of our oldest clusters. My hope is that I’ll learn what it takes to deploy a modern cluster using OpenHPC so that I can modernize some of our aging infrastructure’s operating systems and utilities. This test cluster will hopefully turn into a student resource for the UTC campus after I’m done learning with it.

Another task I’ve been spending a lot of time looking into recently is learning what it takes to deploy a big-data analytics cluster for the Computer Science researchers and Ph.D. students in our department/campus. This is quite different than what I’m used to here in my world of HPC and CFD, although there are some similarities including aspects of distributed computing. I’ve been learning about Apache Hadoop and trying to deploy a Cloudera cluster for big-data analytics using Cloudera Manager to deploy Cloudera’s CDH on yet another test cluster. I enjoy this work very much. There are many challenges along the way which help me to become a more rounded and skillful HPC administrator.

What would you say is your meshing specialty?

I’d say that unstructured meshing is what I do the most and the best and I also enjoy Glyph scripting for the automation-of-tedium. Lastly, I like using Pointwise’s keyboard shortcuts/accelerators as much as possible for increased efficiency (what can I say, recall, I’m an ardent user of vim and I like to keep my fingers on the keyboard). This skill is quickly lost however and I find myself losing my muscle memory as I use Pointwise less and less these days. In my opinion, this is the only area where Gridgen did it better than Pointwise. In my very limited experience with Gridgen, it was convenient to have pretty much every action a key-press away.

Any tips for our users?

Once again, I’m not going to say much original here as all of what I have to say has been said many times before me! I will mention four things:

Take advantage of the Layer Manager as it can help you organize and manage a large and complicated project by isolating pieces of geometry and/or mesh that make sense to group into separate layers and then have the ability to view and work on only those parts. This can be a life and time saver for any project — not just the exceedingly complicated ones. On a related note, the Groups feature of Pointwise can also be very useful.

Secondly, keep in touch with Pointwise and their support staff when you have a problem, are unsure how to use a feature, or would like to request an enhancement. I have had many stimulating and productive conversations with Pointwise’s support and invariably my problem is resolved or my confusion is clarified. I have even requested several enhancements that have made it into the production version of Pointwise.

Thirdly, become familiar with Glyph scripting; you don’t actually have to know how to write these scripts to benefit from them, there is a plethora of immensely valuable scripts on Pointwise’s GitHub Glyph Exchange that any user can take advantage of without having to know the first thing about Tcl (Tcl is the scripting language that Pointwise uses to create their Glyph scripting interface). I have also found a few gems by poking around on some of Pointwise’s developer’s personal GitHub profiles, e.g.  Travis Carrigan’s page and David Garlisch’s page. I’ll plug myself here too I suppose since I also have a few useful Glyph scripts on my GitHub page.

Lastly, a new feature was recently introduced to Pointwise in its V18.0 Release 1 which allows one to hide domains previously assigned to boundary conditions while assigning boundary conditions in the ‘Set Boundary Conditions’ dialog. This can be immensely useful, especially if you have a very large project with many domains composing the boundary conditions.

What project are you most proud of and why?

I am probably most proud of the work I did to obtain my Ph.D. This project took years and considerable effort — I’m honestly not sure I’d do it again faced with the choice — and I’m proud of what I finally accomplished.

What CFD solver and postprocessor do you use most often?

It’s been too long since I’ve actually used a CFD code and/or postprocessor. When I was doing the research for my dissertation I used my own CFD solver and automated grid generation code, unimaginatively named PAROS (PARallel Octree Solver), quite a bit. My postprocessing was done almost exclusively using ParaView.

Are you reading any interesting technical papers we should know about?

Not really. As I mentioned earlier, I have spent a lot of time looking into OpenHPC as a tool to help modernize our server room. I am also spending a significant amount of time researching Apache Hadoop and Cloudera’s CDH. I have been researching PBS Professional as a replacement for the much older cluster queueing system SGE that we’ve been using for ages here at the SimCenter. SGE does not support GPUs very well, and since our newest cluster contains GPUs, I think it’s time to look into modernizing the queueing system too.

Do you plan on attending any conferences or workshops this year?

I’m not sure yet. I will likely attend SuperComputing this year in Denver if my department will send me. I wanted to attend a training seminar hosted by TACC about Designing and Administering Large-scale Systems, but my attempt to register was too late and now I’m on a wait list.

What do you do outside the world of CFD?

Honestly, I’m quite lazy outside of my work world. I don’t have any hobbies that I spend a lot of time investing in. Although, now that I’m no longer a student, I have more free time than I’m used to and I need to find a way to spend it constructively. In an attempt to get back into shape after graduate school, I have tried to pick up running — which my knees and ankles aren’t sure they like. However, I completed my first half-marathon last month and I hope to compete in one of the next few Iron Man triathlons that happen here in Chattanooga annually. I have four wonderful children: Gabriel Alan, age almost-10, Jude Isaiah, age 8, Lael Sophia, age almost-7, and Emery Dawn, age 2. I try to spend as much time with them now before their fleeting youth has passed me by.

What is some of the best CFD advice you’ve ever received?

“There’s nothing more dangerous than an answer that looks about right.” – Dr. Timothy Swafford

If you had to pick a place to have dinner, where would you go?

The Terminal Brewhouse in downtown Chattanooga is my personal favorite. While their food is quite good in my opinion, it’s their locally brewed “West Coast IPA” that I love them for. I’m a big fan of IPA and their variant is my favorite.