NERSC Jacquard System Essentials
[
Connecting |
Storage |
Tools |
MPI |
Interactive Jobs |
Batch Jobs |
Repository
]
Be sure to visit the
NERSC site; they have quite an extensive
documentation
on using the Jacquard system. All of the information here can be found
in more detail there; I just list the most essential items on this page.
The frontend login node is jacquard.nersc.gov.
To log on, use a SSH program (such as
OpenSSH and PUTTY;
others are listed at here).
If you need to change your password, login to NIM.
Contact support
if you have problems.
There are two storage provided for you on the Jacquard system:
- $HOME. 5 GB of space, avalable from every node.
- $SCRATCH. 50 GB of scratch space, available from every node.
There is a nominal 7-day deletion policy, but files may be deleted
anytime after the job completes, so in your batch script you probably
want to copy back the relevant data to more permanent place, like
$HOME.
Neither of the above is backed up. Usage of your quota can be checked
by the program myquota. Do not use /tmp and
/var/tmp; they are slow and limited in space. Note that some
programs (such as the IBM compilers xlc and xlf)
use /tmp unless the you set the environmental variable TMPDIR.
You can set to $SCRATCH by something like
export TMPDIR=$SCRATCH
Various software on Jacquard is managed by
modules.
To load a particular module, you need to run module load module_name.
So for example, to load the GNU C C compiler gcc, you'll need to execute
module load gcc. To see all the available modules, use
module avail. To list the modules which have been loaded, use module list.
Some useful modules include
- gnu. This provides various GNU tools, including gzip,
autoconf, and automake. If you are used to GNU version
of various tools, this is a must!
- gcc. This provides the GNU
compiler collection. Version 3.3.3 is the current default, but
other versions are also available if necessary.
- scalapack. Provides the ScaLAPACK library which implements various
linear algebra routines for parallel machines.
- tau. Provides TAU (Tuning and Analysis Utilities),
tools for analyzing the performance of C, C++, Fortran and Java programs.
See the NERSC
TAU page.
- papi. Provides PAPI, a common interface to various hardware
performance counters. See
PAPI website, as well as
NERSC's PAPI page.
- totalview. Graphical debugger for C/C++/Fortran programs
which has support for parallel programs including MPI and OpenMP.
See NERSC's
totalview page.
More complete list (with links to documentation for various modules)
is available at the NERSC Seaborg software page.
To use the MPI library (in module mvapich) on Jacquard, prefix the compiler
name with "mpi". For example, instead of cc you would use
mpicc, instead of f77 use mpif77. No extra
include paths or library are needed.
See NERSC's MPI page for details.
Jacquard uses the PBS queue system for submitting all kinds of jobs. To run parallel interactive jobs (where you can see the output
on your terminal as it runs) you use the qsub command with the -I option.
To run on two nodes, using all processors on each node:
Use qsub -I -q interactive -A reponame -l nodes=2:ppn=2 to launch an interactive shell. You can then run your code from the shell.
To run on two nodes, using 1 processor on each node:
qsub -I -q interactive -A reponame -l nodes=2:ppn=1
Note that you will be charged for both the processors
for both examples above, regardless of whether you used all the processors
or not. Interactive jobs are limited to 16 nodes (32 processors) and
30 minutes. See NERSC's
interactive jobs page for more details.
Jacquard uses PBS queue system for batch jobs. Jobs are submitted
as a shell script with PBS directives embedded in the comments.
Following is a sample script
#PBS -l nodes=8:ppn=2,walltime=00:30:00
#PBS -N jobname
#PBS -o job.out
#PBS -e job.err
#PBS -A repo
#PBS -q debug
#PBS -V
./my_program
Lines starting with # are PBS directives. You can find the details about these directives
here.
Supercomputer time is a limited (and precious) resource.
Our class have been given 50,000 processor-hours in repository mp309.
Each person has been allocated some fraction of this with some slack.
If you use a node, all the processors on that node belongs to you,
and hence you will be charged for both the processors. Processor time
quota can be checked with getnim -U<username>.
See NESRC's
accounts
page for more information about how accounts are charged.
To prevent unnecessary charge, please
- Debug before submitting a large job: check serial program, try
a small number of processors (might as well use 2 processors,
since you will be charged for it).
- Set memory and time limits to your batch job: if a bug in your program
causes an infinite loop, the job will be stopped earlier.