[This article originally appeared in the May/June 2014 issue of The Connector where it quickly became the most widely read article during any issue’s first month of publication. It has been updated for this forum based on new thoughts and attendance at the panel discussion on  Transformative Aerospace System Analysis, Design and Certification: A Vision for CFD in 2030 at the AIAA Aviation 2014 Conference.]

Progress in computational fluid dynamics (CFD) is made every day, everywhere, by countless engineers and scientists. At dozens of conferences, symposia, user meetings and online forums (yes, even on social media) you can learn about the latest developments in the technology and its application. However, it is virtually impossible to keep up with it all due to the sheer volume. Because of that, it is often difficult to “see the forest for the trees.”

It is rare for someone to not only consider the future in considerable detail but also to publish the results. This is why the recent NASA-sponsored report CFD Vision 2030 Study: A Path to Revolutionary Computational Aerosciences (Reference 1) is a must-read for everyone involved in CFD, from the new student, to the applied practitioner, to the experienced researcher.

You might be tempted to believe that NASA’s goals for CFD do not apply to you; that they are too advanced or leading edge for the type of work you do. But because NASA has a strong legacy in groundbreaking CFD research and software development, what it does will influence everyone’s use of CFD.

The study’s purpose was to give NASA recommendations on how to renew its pre-eminence in computational science, allowing “NASA to play a leading role in the pursuit of revolutionary simulation-based engineering.”

Sexaflops vs. the Unglamorous

I was delighted to see Aviation Week’s technology editor, Graham Warwick, write about the study in the 19 May 2014 edition of the magazine under the title NASA Urged to Step Up Research Into CFD Capabilities. Via email Warwick told me that he planned to attend AIAA Aviation 2014 to learn more about the topic and revisit it in a future article.

When you read Warwick’s article you’ll see that he focuses on the high performance computing aspects of the problem – ensuring that CFD codes of tomorrow can efficiently utilize a computer with a million cores. (During the panel discussion at AIAA Aviation, one panelist asserted that most current CFD codes operate at only 3-5% of peak efficiency on HPC architectures so we apparently have a long hill to climb.)

But when virtually everyone in the CFD business moans about the fact that preprocessing consumes 75-80% of an engineer’s time, let’s not get distracted by what I call “sexaflops.”

“No, no – don’t look at those engineers hunched over workstations generating meshes. Look over here at this shiny supercomputer with all the blinking lights.” Reminds me of “Ah, I see you have the machine that goes ping.”

You see, I’ve been down this path before. In December 2007 I was fortunate enough to participate in a meeting at NASA headquarters hosted by the then Associate Administrator for the Aeronautics Research Mission Directorate. The topic was how NASA could spearhead an effort to regain engineering leadership in HPC. Lots of time and dollar signs were associated with processors (computers and computing) but very little with processing (especially preprocessing). Not to put too fine a point on it, but the attendees deemed meshing and related technologies “unglamorous.”

Other coverage of the study include:

• UW’s Mavriplis Advises NASA on Supercomputing Future
• NASA Flight Simulations May Soon Become More Realistic
• Seeking Reality in the Future of Aeronautical Simulations

CFD in the Year 2030

Let us jump right to the bottom line with the study’s vision of how we will be using CFD in the year 2030.

“A single engineer/scientist must be able to conceive, create, analyze, and interpret a large ensemble of related simulations in a time-critical period (e.g. 24 hours), without individually managing each simulation.”

All of us will be running codes that likely will be RANS-LES hybrids with quantified and manageable errors and uncertainties. Our geometry preprocessing and mesh generation for the thousands of cases in our test matrix will be performed automatically and invisibly, likely with mesh adaption. We will be using a massively parallel, heterogeneously architected computer and burning through 30 exaFLOPS of performance. Our computations will probably be multidisciplinary with results automatically reduced to engineering data and simultaneously compared with experimental and flight test data.

For those of you who might not necessarily think that goal is too exciting, let me remind you that I am old enough to remember when a single engineer (me) performed a single simulation (2D) over the course of non-critical period (a year) with 100 percent manual intervention. And that was considered progress. The study’s authors have truly established a fact-based and expertise-driven goal. More importantly, they have pointed us in the appropriate directions for our journey.

Recommendations: Throughput and Accuracy

At its broadest, the study’s recommendations focus on two things: increasing throughput and increasing accuracy.

I have interpreted the prioritized recommendations in the study to be the following:

1. Adapt current CFD software architectures and algorithms to allow for efficient performance on massively parallel computers utilizing millions of cores.
2. Figure out how to accurately model turbulence and transition.
3. Automate geometry preprocessing and mesh generation including adaption.

The secondary recommendations – again, in my opinion – are

1. Develop methods of automatically managing large datasets and large numbers of datasets up to and including solver-embedded post-processing leading to computational steering.
2. Create standards and frameworks that facilitate the sharing of data within a multi-disciplinary analysis and optimization (MDAO) framework.

The Long Road for Meshing

I will admit to not being surprised by the study’s findings that “mesh generation and adaptivity continue to be significant bottlenecks in the CFD workflow” and “most standard CFD analysis processes for the simulation of geometrically complex configurations are onerous.” This is neither omniscience nor apathy on my part but a practical realization of the state of the art combined with a knowledge of progress over the past 30 years plus a personal desire to own the unglamorous nature of the task.

In fact, the study admits that CFD advances in the past decade, during which algorithm development has stagnated, have come from “the use of larger meshes, more complex geometries, and more numerous runs,” all of which I believe are directly attributable to advances in meshing (i.e. meshing is more efficient, can handle CAD better, and can be done faster.)

However, the study is 100 percent accurate in stating that “ultimately the mesh generation process should be invisible [my emphasis] to the CFD user.”

What makes meshing highly visible today?

The Lack of Robust CAD Interoperability

CAD interoperability – or the lack thereof – means that geometry preprocessing for CFD is largely a manual process.

The CAD data interoperability issue has been discussed to death with no end in sight, either for the discussion or for resolving the issue. There is now an annual conference dedicated to the issue, the 3D Collaboration and Interoperability Congress (Reference 2). Also, at the recent Congress on the Future of Engineering Software (COFES, Reference 3), a discussion of cloud-based CAD came to a screeching halt when the moderator inquired whether there was any hope of resolving the interoperability issue and the resounding answer was “no.”

The crux of the matter was deemed political – CAD vendors have little incentive for letting data be portable because it makes it too easy to switch to a competing product. Even standards like IGES and STEP are ineffective because I have seen firsthand how some CAD systems write those files in ways that explicitly violate the standard, making exchange of those files problematic.

Politics aside, the technical issues for CAD interoperability are well known:

1. CAD files have either insufficient detail (tolerances that are larger than the required mesh sizes) or excessive detail (making extraction of the wetted surfaces for CFD burdensome).
2. File-based data exchange is complicated by lack of adherence to standards or lack of ability to read proprietary formats. It is my belief that the rise in popularity of faceted geometry for CFD is due largely to software developers’ lack of comfort with and lack of easy access to analytic geometry. However, faceted geometry presents its own challenges.
3. Geometry is represented differently by each CAD system or geometry kernel and this influences how its data is exported and translated to another kernel (i.e. it is a source of translation error). It also complicates direct access to the CAD by limiting a cross-CAD API to the least common denominator of functionality.

The situation is further complicated by the fact that CFD engineers usually don’t have access to the CAD system; they’re just given CAD files, often from multiple sources. Of course, this varies between organizations and the phase of the design (conceptual, detailed, and certification) in which CFD is being applied.

The Lack of HPC Support

The study states, “Despite the deficiencies in current CFD solver scalability, the situation for the surrounding infrastructure of pre- and post-processing software is even worse.” In other words, meshing software is much farther behind solvers in terms of effective use of HPC hardware.

From a certain perspective, this is not surprising. A CFD solver sees all problems as virtually the same: solving the RANS equations on a mesh with boundary and initial conditions that you iterate until convergence. A mesh generator sees each problem as intrinsically different: a blank screen, a unique CAD geometry, and any method will do when it comes to placing points where they’re needed.

Using a poor analogy, running a CFD solver is like highway driving in which you peg the speedometer, drive for several hours, and get the best mileage. Meshing is like city driving in which there’s a lot of stopping and starting, route changes, and your mileage is nowhere near as good.

Distributed computing for mesh generation is more about managing the large number of grid points/cells by spreading them across many machines than it is about speeding up the process.

That is not to say that meshing algorithms cannot be run in parallel; a lot of software does just that. I can think of several features inside Pointwise that could benefit. But comparing the compute needs of meshing to those of the solver is comparing apples to oranges.

The Lack of Automation

Ideally, meshers should be automated and easy for non-experts to use. The report states that meshing must be done manually largely because of “the inability of current mesh generation software to consistently produce valid high-quality meshes of the desired resolution about complex configurations on the first attempt.” (Note the adjectives “high-quality” and “complex.”)

Specialized, or so-called “vertical” applications, are the way to solve this problem. The lack of vertical applications also was identified at COFES as a roadblock to making the use of CAE software more widespread and effective, especially by non-expert users. (This is not just for CFD either; for example, how about a CAD system just for designing truck axles?) The study dismisses specialized solutions as only applicable to routine design work as though that were a limiting factor. Yet, the study also describes in detail the various design phases where CFD is applied, several of which are open to customization.

The general nature of current CFD and meshing software is the culprit. By its nature, a general purpose mesher has to

• Accept arbitrarily complex CAD geometry from any of a multitude of sources.
• Generate any mesh type (structured, unstructured, hybrid, Cartesian) suitable for a multitude of solvers.
• Meet the “high-quality” mesh threshold that varies from solver to solver.
• Ensure sufficient resolution where resolution is defined in terms of the solver and the desired flow quantities to be computed.

Of course, we will always need general purpose tools for “single-prototype designs and novel configurations,” but the day-to-day work of simulating design variations can best be automated by customized software, especially if the customized software can be built on top of a general purpose tool.

Mesh Generation in 2030

The targets established for mesh generation in the year 2030 by the study are hard to argue with.

• Streamlined CAD access and interfacing.
• Automated, adaptive techniques including anisotropy.
• A less burdensome, more invisible meshing process.
• Meshes with 1 trillion grid points.
• Efficient exploitation of massively parallel hardware.
• High order elements.

Panel Discussion at AIAA Aviation 2014

Getting together at AIAA Aviation to discuss CFD Vision 2030 was a great opportunity to get other people’s perspectives on the study. Panelists were:

• Robert D. Gregg III (Boeing) – moderator
• Wilson Felder (Stevens Institute of Tech.)
• Parviz Moin (Stanford)
• Stephen Morford (Pratt & Whitney)
• Cord-Christian Rossow (DLR)
• David Schuster (NASA Langley)
• Jeffrey P. Slotnick (Boeing) – one of the report’s authors

A video of the session is available for you to watch on Livestream.com: Transformative Aerospace System Analysis, Design, and Certification: A Vision for CFD in 2030.

A cool part of the panel discussion was real-time polling of the audience by text message. [Thank you to Jeff Slotnick for sharing the data with me and giving me permission to publish it.] The first question got right to the heart of the matter by asking the audience what they thought the main focus going forward should be (see below).

Hooray! Meshing and visualization came in second behind development of physics and algorithms. I’m OK with that because of the thrashing that turbulence modeling took in the study (and I’m assuming that “physics” refers to that and other physical models). Otherwise, a cynic would say these are the results you’d expect when polling a bunch of CFD solver writers.

In their opening remarks, each panelist noted what’s good and what’s bad about CFD.

The Value of CFD

• Tremendous performance gains have been made in the past decades, many of which are directly attributable to CFD.
• The insight CFD provides into aerodynamics has led to reduced testing (i.e. cost savings).
• CFD allows “what if” questions to be asked and answered.
• CFD allows investigation of extreme concepts (those with which we lack prior experience, those that would be difficult or impossible to test).
• CFD lets you explore cause and effect.

The Limitations of CFD

• CFD doesn’t always yield the correct answers and it’s often difficult to know when that happens.
• A priori grid generation must be eliminated.
• CFD simulations had reduced the number of tests but that benefit has plateaued lately.

Panel Q&A

Right off the bat, one thing was made clear. No one believes CFD can, will, or should replace wind tunnel or flight testing any time soon. The technologies are complimentary.

After that, the question and answer portion of the panel session focused almost entirely on the 800 pound gorilla in the room – money.

Who’s going to pay for this vision? And why? And how?

One panelist remarked that any proposal like this that called for additional federal funding is doomed to failure due to the current fiscal environment in Washington.

Despite that pessimistic viewpoint, most of those polled want government involvement and funding to move forward on the vision (see above). [It should be pointed out that the first polling question got about 3 times the number of responses than the others.]

Yet NASA funded the study so I’d be very interested in hearing an “official” response. One interesting aspect of NASA’s involvement was pointed out by one panelist. Even though 80% of NASA’s budget is for space exploration, the vast majority of the CFD vision is cast in terms of aeronautics. Maybe a focus shift is in order.

And while it may be true that NASA funded this study, the problems it identified are shared by a slew of acronym agencies – FAA, USAF, USN, DoD, DoE – and the industries with whom each are most closely related. And by including a European on the panel it was clear that someone knows this is an international problem. That fact was also echoed by an audience member. The whole world can benefit from working together to solve this problem.

After that “Kumbaya” moment was over, we were still left with the issue of how to begin making actual progress on this vision. And when it came to concrete first steps, the leading vote getter was establishing targets for CFD accuracy.

In terms of advocacy, some high-level suggestions were made. First, the vision needs to be recast in terms the stakeholders understand because certainly “turbulence modeling” doesn’t resonate with them. The idea of positioning the study as a means of protecting and strengthening the aerospace industry, the largest contributor to the nation’s exports, is something that might motivate stakeholders. And AIAA, our conference hosts, are well suited to carry this message to Washington for us.

And this is important because most poll respondents think the best way forward is to create some form of multi-agency government initiatives.

Other Highlights from the Study

Our turbulence modeling friends appear to be faced with the greatest challenge, according to the study. Turbulence modeling is said to be “the single most critical area in CFD simulation capability” because almost any type of separated flow is virtually unpredictable. We need to be able to confidently simulate the point of separation and the feedback from separated flow to the boundary layer.

The study advocates significant changes to NASA’s software development culture because it is currently stuck in a mode of making incremental improvements to existing tools and major re-architecting projects or new projects cannot get approval. Interestingly, the study also advocates an open-source model for non-critical components and/or as a mechanism for making the results of research available to the community.

The study recommends developing and implementing standards to address the impediments to realizing the MDAO capabilities envisioned by 2030. However, the study is 100 percent correct that “frameworks and standardization can lead to significant constraints and may not be the best solution in all cases.” In other words, a standard for interoperability often must sacrifice performance.

The most damning accusation in the report is that standards for verification and validation exist, but they are not widely used in the aerospace industry. Grid refinement “is seldom done in practice.”

Bonus Information

1. I learned what aleatory means: depending on chance, random.
2. The Cloud was not mentioned – not even once.
3. And as for democratization, “CFD is not sufficiently predictive and automated to be used in critical/relevant engineering decisions by the non-expert user.” I disagree. The study may be focused on typical NASA applications, but there are many examples of successful application of CFD by the non-expert. If that statement is true at this stage of the CFD game, we’ve all been abject failures.

Recommendations

The authors, sponsors, and contributors to the CFD Vision 2030 Study have succeeded in creating a thought-provoking vision of the future of CFD and also in developing an actionable plan for achieving those future goals.

To be clear, this is not prognostication and we shouldn’t quibble over precisely how achievable the goals are in the given timeframe. What is important is assessing where we are now relative to these goals and identifying the paths toward the goals’ achievement. Fortunately, NASA is uniquely suited to accomplishing and coordinating progress of these tasks.

My suggestions to you are to

1. Read the CFD Vision 2030 Study.
2. Follow verification and validation standards in your CFD work.
3. Participate in benchmark workshops such as the ones sponsored by AIAA (see References).

For further reading, you may want to take a look at the Simulation Driven Design study conducted by Lifecycle Insights that delves further into how simulation is currently applied.

But my number one recommendation is that you spend a day or two with us at the Pointwise User Group Meeting 2014 coming up on 29-30 October in Anaheim, California. Not only can you learn about what other people are doing with Pointwise you can see and hear our mesh generation development plans and judge for yourself how much more invisible we’re making meshing.

References

1. Jeffrey Slotnick, Abdollah Khodadoust, Juan Alonso, David Darmofal, William Gropp, Elizabeth Lurie, and Dimitri Mavriplis, CFD Vision 2030 Study: A Path to Revolutionary Computational Aerosciences, NASA/CR-2014-218178, March 2014, 51 pages. Download from the NASA Technical Report Server.
2. The 3D Collaboration and Interoperability Congress, http://www.3dcic.com/
3. The Congress on the Future of Engineering Software, http://cofes.com/
4. Michael Jefferies, Surface Mesh Extrusion on the GPU, SIAM Conference on Geometric and Physical Modeling, October 2011.
5. Pointwise Works to Make Aircraft Quieter , The Connector, Nov/Dec 2011.
6. AIAA Drag Prediction Workshop
7. AIAA High Lift Prediction Workshop
8. AIAA Sonic Boom Prediction Workshop
9. AIAA Propulsion Aerodynamics Workshop
10. Aeroelastic Prediction Workshop

P.S. I apologize for the length of this post. I took a long article and made it longer. As Mark Twain supposedly said, “I didn’t have time to write a short letter, so I wrote a long one instead.”