A Magnetically Insulated Line Oscillator (MILO) as modeled in ICEPIC showing (computational) electrons.
Bio available here.
ICEPIC is a 2/3d-3v (two or three position components, three velocity components) fully kinetic PIC (Particle-In-Cell) plasma simulation code designed to model complex plasma systems on one or more processors. ICEPIC is designed by the AFOSR to model high-energy plasma devices that can't accurately be modeled by bulk or fluid models ICEPIC uses a discrete number of particles (usually, far less than there would be in an actual plasma) coupled with Maxwell's equations for a field solve to resolve plasma behavior from first principles. A parallel matrix solve is used for electrostatics while Yee's lattice (a local algorithm) is used for fully electromagnetic systems. The charged particles are moved relativistically. Peter Mardahl of the Air Force Research Lab has used ICEPIC to model high-power microwave tubes, including magnetrons, Klystrons, gyrotrons, and Magnetically Insulated Line Oscillators (MILOs).
A Magnetically Insulated Line Oscillator (MILO) as modeled in ICEPIC showing (computational) electrons.
Domain decomposition for MILO shown above showing message-passing boundaries and arrows denoting message-passing.
ICEPIC is designed for the simulation of three dimensional electromagnetic and electrostatic kinetic plasma devices, such as magnetrons and gyrotrons. Kinetic simulations are necessary when the velocity distribution functions of the plasma species deviate from a Maxwellian. Many practical plasma devices are fully 3-D and kinetic, and require a code like ICEPIC to simulate. In the past, three-dimensional Particle-In-Cell (PIC) simulations were inconceivable for practical devices due to the high amount of computing time, but with modern massively parallel computers, such simulations are now within reach.
ICEPIC achieves its design objective. ICEPIC computational results were compared to experimental results for the University of Michigan magnetron. The extracted power and frequency compared well, as did many other qualitative characteristics of the system. Other systems have been simulated and the results are currently being compared to experimental data and theory.
ICEPIC targets highly distributed machines with tens to hundreds of CPUs (or more), though it is also possible to run on serial machines. ICEPIC is a C program which uses the Message Passing Interface (MPI) to communicate between processors. ICEPIC uses asynchronous messaging to reduce overall computational time between processors
The application has been run on brainerd (http://www.arl.hpc.mil/userservices/ibm.html) and huinalu (http://www.mhpcc.edu/doc/huinalu.html), among other computers. Though it is difficult to guess from the information at hand, I believe brainerd is ranked 282 on the Top 500 list for November 2003.
Flop rate data (for single and parallel systems) was not found for this application.
ICEPIC uses dynamic load balancing to achieve a high degree of parallelism. The grid is divided into different sections initially based on achieving a load balance between processors. As the simulation progresses, the boundaries are adjusted to keep the total work per processor per cycle roughly equivalent. Particle and cell data is moved between processors in efforts to minimize communication time.
Illustration of dynamic load balancing as applied to the MILO simulation.
In addition to load balancing, taking advantage of asynchronous communication can help take advantage of parallelism by performing tasks with local data while sending and receiving message to other nodes.
Several benchmarks were run and the efficiency (Speedup(p)/p) is noted. For particle dominated runs (large numbers of particles per cell), efficiency was 0.7 (256 processors on brainerd) or better. For cell dominated runs, efficiency was not as good. For the scaled problem (1 million cells per CPU), efficiency was a little over 0.4 for 256 processors on brainerd. For the fixed problem (1.2 million cells with 1000 particles), efficiency dropped to about 0.1 for 128 processors on huinalu. Results are shown below.
Particle dominated runs, scaled problem efficiency. 16 millions particles and 1,000 cells per CPU.
Particle dominated runs, fixed problem size efficiency. 16 million particles and 50,000 cells total. The efficiencies above 1 are probably due to the particle data fitting in cache.
Cell dominated runs, scaled problem efficiency, 1 million cells per CPU.
Cell dominated runs, fixed problem size efficiency. 1.2 million cells, with 1000 particles.
Not all of these details will be available for all applications. You ought to explain what you find noteworthy about the application or its implementation.
A few movies of ICEPIC in action are available in MPEG format:
ICEPIC is a noteworthy application for its dynamic load balancing between processors and good scalability to massively parallel machines, and also for its implementation of a generalized and extensible Particle-In-Cell code useful for many practical engineering problems in plasma physics.
ACKNOWLEDGEMENTS: Thank you, Peter Mardahl, for providing me with the necessary information about ICEPIC, and for your beautiful graphics and mpegs. All graphics and mpegs on this page are intellectual property of the US government (Air Force Research Lab) and may not be reproduced without permission.
REFERENCES: