I can find inspiration for projects just about anywhere. While many ideas are fleeting and only grab enough attention for a day or two of work, some stick around and snowball into interesting projects. This is a look at one of those ideas turned projects, the idea that I could run a CFD calculation in the palm of my hand.
The idea for this project started to materialize back in October after receiving an Arduino Uno for another project I was working on. If you haven’t seen it, the Uno is small, about the size of a credit card. It got me thinking, what if I could run CFD calculations on a computer about the size of the Uno. A phone was the logical choice. But compiling something like OpenFOAM for an iPhone or an Android sounded too daunting to even consider. What about the Raspberry Pi?
The $35 Raspberry Pi, developed by the Raspberry Pi Foundation, first launched in the spring of 2012 with 100,000 boards shipping the first day! The goal was to bring an affordable computer into the classroom and teach children the mechanics of computing. But hobbyists had their eye on the intriguing little board that could pump out 1080p video and interface with a number of sensors and other devices. Now, two years after the launch, there are more than 2.5 million Raspberry Pis in the wild.
After doing the research I decided that I’d hold off on the Pi for a while. Then, in early February, a #SimulationFriday post caught my eye. It was a tweet by Momentum Analysis with a picture of the driven cavity OpenFOAM case being run on two Raspberry Pis!
An aside, #SimulationFriday is a great hashtag on Twitter where engineers post simulation photos and videos every Friday.
The guys at Momentum Analysis had used an OpenFOAM build provided by Rheologic. Finally, there was a build available for the Pi and I no longer had to worry about compiling it on my own. I could get a Pi and be up and running in no time!
A couple of days later, Momentum Analysis wrote a great how-to blog article about setting up what they refer to as a twoPi cluster. In the article they discuss how to setup the Pi, get the OpenFOAM binaries, and even described some of the tests they performed.
Ok. It was time to get a Pi of my own.
I should mention, not long after their post, I stumbled upon another article about a 40-node Raspberry Pi cluster here in the Dallas/Fort Worth area. The build quality is incredible and I encourage you to check out the video below.
I did a lot of research before I purchased a Raspberry Pi. What I learned was that while the board is around $35, the accessories can add up. That is unless you have chargers, cables, and keyboards at home. After looking around I knew what I needed and where I would purchase everything. I settled on ModMyPi. ModMyPi started out manufacturing cases for the Pi, but quickly thereafter began selling a range of accessories. And while they aren’t affiliated with the Raspberry Pi Foundation, they donate 5% of their profits back to the foundation.
Rather than purchasing one of their kits, I decided to stick with the bare essentials and just find what I didn’t already own. Below is a list of everything I purchased from ModMyPi if you’re interested in pursuing the same experiment. Note, prices have increased slightly since I bought these items. If you register and follow them on Twitter you’ll get a discount.
- Raspberry Pi – Model B
- 16GB Samsung Micro SD Card Pre-Loaded with NOOBS (Class 10 UHS-I)
- Cyntech Blackberry Raspberry Pi Case with SD Card Cover
- HDMI to HDMI Cable 2m V1.4 (Gold Plated)
- Raspberry Pi Heat Sink Kit
The Model B Pi comes with 512MB of RAM opposed to the Model A which only has 256MB. This is important for both compiling software and running CFD calculations. I elected to spend a little more on a high performance SD card with NOOBS preinstalled. The NOOBS (New Out of Box Software) package allows for easy installation of many supported Linux distributions, including the official Raspbian distro based on the popular Debian OS.
The heat sinks are interesting. I wanted the ability to overclock the 700Mhz processor to 1Ghz without significantly reducing the life of the Pi. While the heat sinks don’t dissipate that much heat, they do look pretty cool.
OpenFOAM for the Pi
Everything arrived last week, so I’ve only had a few days to scratch the surface. But, the first thing I did was load up the Momentum Analysis blog article so I could configure my Pi to run OpenFOAM. After applying all the necessary updates and playing with a few settings, I downloaded OpenFOAM and had the cavity case running in less than an hour. I was running OpenFOAM on a Raspberry Pi. I was able to run a CFD simulation on a device that fit in the palm of my hand. Momentum Analysis deserves a big thanks as their instructions were spot on. Thank you guys!
A couple of days later I decided to run a Pointwise generated mesh in OpenFOAM on the Raspberry Pi. I ran a steady-state RANS calculation for the NACA 0012 at an angle of attack to the flow. The solution converged in five minutes. I was ecstatic! Not only could I run CFD problems on the Raspberry Pi, but I could run our meshes on the Pi.
If you’re interested in this case, you can download it and try it yourself.
SU2 for the Pi
I had accomplished my goal. I ran a CFD calculation in the palm of my hand. But, I couldn’t shake the fact that I hadn’t really done anything new. The Momentum Analysis guys already did this…in parallel. What could I do differently? What about a different solver? Enter SU2.
A couple of years ago, a group from the Aerospace Design Lab in the Department of Aeronautics and Astronautics at Stanford University released an open source CFD solver. The Stanford University Unstructured software suite, or SU2, is a collection of C++ tools for solving partial differential equations, including CFD problems. The team released version 3.0 of the code in January of this year.
Over the last few weeks I’ve been exploring SU2 for some in-house CFD work, so I’m somewhat familiar with the code and can build it reliably. So, I decided to download the source on the Raspberry Pi and attempt to compile it. My new goal was to package up the SU2 binaries for the Raspberry Pi and run the NACA 0012 problem. Easier said than done.
I wasn’t looking to do anything fancy, so a basic single-threaded compile is all I needed. I followed the well written installation guide provided by the SU2 developers and configured the makefile using only the prefix argument followed by issuing the make command to compile the source code. About 30 minutes into the build the Pi threw up a cryptic error message. Shortly after that, all the applications crashed and the GUI was unresponsive. Unfortunately, the Pi is not equipped with a power switch, so turning off the device is just a matter of unplugging the power, about all I could do at this point.
I reconnected the power to turn the Raspberry Pi back on and was greeted with a kernel panic. Wonderful.
Unplug power. Reconnect power. Kernel panic.
Unplug power. Reconnect power. Nothing. Now the Pi would no longer boot.
I pulled the board from the case, tested the voltage using the two contacts on the board itself. No power. It turns out a fuse had blown and needed a few minutes to reset. Great. Long story short, this process repeated itself a half-dozen times followed by running the fsck utility.
The problem was either the SD card or the power supply. My guess was the board, ethernet cable, HDMI cable, SD card, and USB keyboard dongle were pulling more current than the ASUS Nexus 7 charger could handle. This in turn dropped the voltage below 4.75V and caused some instability. I have since replaced the power supply with a 5V 2.1A iPad 3 charger and have not encountered any stability issues.
Another potential issue was memory. I had been monitoring the memory usage during the build and noticed it creeping up to the 512MB limit. So, using the raspi-config utility I disabled the GUI and gave the GPU access to the minimum amount of memory. Next, I gave the Pi a static IP on my home network and enabled SSH access. The HDMI cable and USB dongle were no longer necessary. Not only did this solve the memory problem, but again I had fewer stability issues.
Every time I tried something new I reattempted the build, getting further each time. I’d estimate the entire build took around three hours to complete. Once it was finished I ran make install then archived and zipped up the result. I now had a Raspberry Pi Raspbian build of SU2!
The first thing I did was run through the detailed quick start tutorial to ensure that at the very least SU2_CFD was working. Sure enough, the steady, Euler calculation for the NACA 0012 converged in under three minutes.
The SU2 team graciously offered to host the binaries on their site for download. If you’d like SU2 for the Raspberry Pi, please visit http://su2.stanford.edu/download.html. You may notice there is also an SU2_EDU link for the Pi. It’s coming soon, but more on that later.
This was a great project. And while I’m happy with the work that Momentum Analysis and Rheologic have done, I’m glad I didn’t stop with the precompiled OpenFOAM build. It was great tearing through the electronics, decrypting error messages, and learning how to compile and run CFD software on a device that fits in the palm of my hand.
If you are interested in this project and are planning on purchasing a Raspberry Pi of your own, here are a few lessons I learned along the way.
1. Use a good power supply.
I can’t stress this enough. While the Raspberry Pi board doesn’t require that much power, it starts to add up once you plug things in. The Pi calls for a 5V 1A power supply. Most phone chargers will work, but in my experience some are flakey. I recommend going with something stable, preferably 5.25V to account for any slight voltage drops and 2A allowing for more peripherals to be plugged directly into the board.
2. Give the Raspberry Pi a static IP address and enable SSH.
If you plan to do any kind of development or CFD, ditch the GUI for the command line and SSH into the Pi. Telling my router to give the Pi a static IP wasn’t enough, I also had to edit the network interfaces file. Here’s a great video describing how to give your Pi a static IP address.
Also, you’ll need to enable SSH as it’s off by default. This can be done through raspi-config, the Raspberry Pi Software Configuration Tool seen below.
3. Expand the filesystem and adjust the CPU/GPU memory split.
If you use the NOOBS package to install Raspbian, the filesystem will automatically occupy the entire SD card. However, if you install the OS in another way, you’ll need to manually expand the filesystem to take advantage of all the memory available on the card. You can also do this using the raspi-config utility. You may also want to give the GPU the minimum amount of memory. This can be done using the same utility.
Lastly, overclock. The Raspberry Pi doesn’t generate much heat and doesn’t draw much power. Exploit the efficiency by overclocking. A stock Raspberry Pi runs at 700Mhz. I’ve overclocked the CPU to 1GHz giving it what I’ve been told is an extra 50% more horsepower. This is where the heat sinks come in handy.
wow. This is “underkill”. But Good job! 🙂
Thank you Daniel. I’m glad you liked it!
Up next: running CFD on a toaster 😛 Good writeup!
Personally, I’m hoping to try OpenFOAM on an Intel NUC.
I appreciate it. OpenFOAM on an Intel NUC sounds like a great project. Keep us posted!
Very cool Travis… I’ve been looking for an excuse to pick up a Pi. This could be it… unfortunately, I have very little free time. I, too, have been playing with SU2.
Thank you Ethan! CFD is a great excuse to pick up a Pi 😉
Hey, that’s awesome!! 🙂
Pingback: This Week in CFD | Another Fine Mesh
OK. This is really cool solution. Now I need a problem to solve with it 🙂
Thanks for the challenge, Travis! _This_ is underkill –>
Fantastic work! I’m downloading it tonight and I’ll give it a shot.
As always Travis persistence is the key in getting code to build. I had a heckuva time just getting gfortran to build on late version of ubuntu. SU2 is imo superior to openfoam although openfoam does come with mesh generation code but not to compare with pointwise. It’s good that pointwise now can output a native su2 file rather than going through cgns.
Hey Travis …
Nice and very useful writeup.
As an Computational guy … I am interested knowing about the performance of Openfoam simulations on the Pi … Can you comment on how fast it was compared to the parallel simulations (2 processors on i3 2.x GHz). I expect it to be slower … but I want to know how the speedup increases when adding more Pi. I am really interested in this aspect.
Thanks in advances
Thank you. I’m glad you enjoyed the article. Regarding the performance of OpenFOAM, it’s very, very slow. Adding more Pi’s might improve performance, but in terms of biggest bang for your buck, a workstation class processor is better at tackling these types of problems. However, as far as fun factor, running OpenFOAM on the Pi is hard to beat.
It is indeed, as Travis said, that running OpenFOAM on the Pi is a fun project rather than serious CFD simulation. There is nothing wrong with fun however 😉 So indeed, people did try to add additional Pi’s to test the parallel computing performance. One such test with 2 Pi’s in parallel was done by Momentum Analysis, a UK based CFD consultancy: http://goo.gl/iT9b9M
At the bottom of the article you find the actual comparisons. When running the tests, we advised Momentum Analysis to also vary the number of cells, since this is critical for the performance of parallel CFD simulations with OpenFOAM. Given enough cells (250k in this simple example) results in a very nice speedup of almost 2. Using less cells will create some overhead and the actual computational performance of the processors will be undercut by the speed with which data can be shoveled from one processor to the other.
Paralleling OpenFOAM calculations successfully depends on a number of factors and there is no one-size-fits-all solution.
While we (http://rheologic.at) use much faster hardware – when we are not toying around with Pi’s that is 😉 – we asked David Guill, creator of a 40-node Pi cluster (http://goo.gl/ngY3mZ) to give our OpenFOAM RheologicRemix (http://rheologic.at/?q=node/48) a spin. If you want to dig deeper into this topic, your best option is probably to contact David.
Hi Adi, Markus and Travis.
We (www.momentum-analysis.co.uk) ran up to four Pi’s in the parallel tests, and for the resolution of the 3D driven cavity we were looking at saw that the parallel scaling was linear with two Pi’s but then dropped off with three and four Pi’s. The point at which scaling drops off from linear (doubling the number of Pi’s results in halving of solve time) does depend on the number of cells in the model, but this doesn’t really tell us what is happening. Two of the factors (and there are many) to consider are the locations of the boundaries between the partitions in the parallelised mesh – if they are located in regions where there is rapid change of flow variables with iteration or with time, then this can result in greater inter process communication. Also, the matrix solution technique applied for the discretised equations in OpenFOAM affects the amount of inter process communication with some solvers being more comms heavy than others.
For us it’s reasons like these that make the use of Pi’s for CFD more than just fun – for relatively little money (by current computing hardware standards) a simple set-up can be used to investigate the parallel scaling sensitivity of domain decomposition and matrix solvers.
We haven’t haven’t explored this much further than our original tests, but its on our list. Sadly we lost one of our Pi’s to a separate electronics project 🙁
Pingback: This Week in CFD | Another Fine Mesh
Pingback: I’m Travis Carrigan and This Is How I Mesh | Another Fine Mesh
Hi Travis …
I am interested knowing about the performance of Openfoam simulations.I am interested in this area.
thanks for sharing
There is some info out there posted by Simon (see previous post) other than that, I’m not aware that somebody is testing OpenFOAM on alternative platforms.
We toyed around with some ideas, but things are on the back burner at the moment.
I guess you are aware of the basic trade-offs – parallel computing adds speed, but if the number of cells per core fall beyond a certain threshold (depending on the case) the overhead from data exchange slows things down and adding more cores will only get you “diminishing” return in terms of computing time.
Info from MomentumAnalysis:
Our binaries for various platforms:
Feel free to contact me, if you have questions.
Nice page explaining CFD on Raspberry Pi. I am new to using both OpenFOAM and Raspberry pi.
I am trying to install OpenFOAM on Raspberry Pi 2, Model B. I downloaded the OpenFOAM for Pi from RheologicRemix-2.2.1 for Raspberry Pi (Raspbian) ARMv6 (without Paraview) download link.
I am having an issue installing it into the Pi. Can anyone briefly explain the procedure?
I used tar-xf to extract the OpenFOAM and tried installing it. When I did ./Allwmake in ThirdParty folder. I got an error..
./Allwmake: 34: ./Allwmake: wmakeCheckPwd: not found
Error: Current directory is not $WM_THIRD_PARTY_DIR
The environment variables are inconsistent with the installation.
Check the OpenFOAM entries in your dot-files and source them.
It would be great if you could list out the important steps and also possible issue that might arise during installation.
Thanks in advance…
Thank you for the detailed note. Turns out you don’t actually have to compile OpenFOAM from Rheologic, they’ve already done that for you. You simply need to download it and unpack it into the /opt directory. I mentioned a how-to blog post by Momentum Analysis in the article where they describe the procedure for getting OpenFOAM up and running on the Pi. I suggest taking a look.
Thanks Travis for the reply. I did get it to run on single Pi. Now I intend to run OpenFOAM on a 2 pi cluster and intend to extend it further.
I got both the Pi’s ready with OpenFOAM installed and running. Now the issue is how to build a cluster out of them. From the twoPi – Initial Config & Test blog post by Momentum Analysis I changed the hostnames of the pi’s but couldn’t infer what to do next. How do I run my OpenFOAM simulation in a cluster? More importantly how to I make a cluster?
Could you direct me towards any detailed text available?
Thanks in advance..
You probably have some issue with your environment variables – see here: https://openfoam.org/download/source/setting-environment/
Pingback: Top Posts of 2017 on Another Fine Mesh | Another Fine Mesh
I could really do with an idiot’s guide to getting OpenFOAM running on a Pi 4. I really don’t know where to start. Any pointers? Thanks, Dave
Hi Dave. Thanks for your comment. As far as I can tell, Rheologic still offers a build for the Pi here, https://rheologic.at/en/download-RheologicRemix-en. Unfortunately, the original Momentum Analysis post is no longer available.
Pingback: This Week in CFD - Another Fine MeshAnother Fine Mesh