NERSC Jacquard System Essentials

Updates

More information on the machine is in this paper (see the description of the Opteron machine). To obtain other information, use 'cat /proc/cpuinfo'.

Be sure to visit the NERSC site; they have quite an extensive documentation on using the Jacquard system. All of the information here can be found in more detail there; I just list the most essential items on this page.

Connecting to Jacquard

The frontend login node is jacquard.nersc.gov. To log on, use a SSH program (such as OpenSSH and PUTTY; others are listed at here). If you need to change your password, login to NIM. Contact support if you have problems.

Storage

There are two storage provided for you on the Jacquard system:

$HOME. 5 GB of space, avalable from every node.
$SCRATCH. 50 GB of scratch space, available from every node. There is a nominal 7-day deletion policy, but files may be deleted anytime after the job completes, so in your batch script you probably want to copy back the relevant data to more permanent place, like $HOME.

Neither of the above is backed up. Usage of your quota can be checked by the program myquota. Do not use /tmp and /var/tmp; they are slow and limited in space. Note that some programs (such as the IBM compilers xlc and xlf) use /tmp unless the you set the environmental variable TMPDIR. You can set to $SCRATCH by something like

export TMPDIR=$SCRATCH

Tools

Various software on Jacquard is managed by modules. To load a particular module, you need to run module load module_name. So for example, to load the GNU C C compiler gcc, you'll need to execute module load gcc. To see all the available modules, use module avail. To list the modules which have been loaded, use module list. Some useful modules include

gnu. This provides various GNU tools, including gzip, autoconf, and automake. If you are used to GNU version of various tools, this is a must!
gcc. This provides the GNU compiler collection. Version 3.3.3 is the current default, but other versions are also available if necessary.
scalapack. Provides the ScaLAPACK library which implements various linear algebra routines for parallel machines.
tau. Provides TAU (Tuning and Analysis Utilities), tools for analyzing the performance of C, C++, Fortran and Java programs. See the NERSC TAU page.
papi. Provides PAPI, a common interface to various hardware performance counters. See PAPI website, as well as NERSC's PAPI page.
totalview. Graphical debugger for C/C++/Fortran programs which has support for parallel programs including MPI and OpenMP. See NERSC's totalview page.

More complete list (with links to documentation for various modules) is available at the NERSC Seaborg software page.

MPI

To use the MPI library (in module mvapich) on Jacquard, prefix the compiler name with "mpi". For example, instead of cc you would use mpicc, instead of f77 use mpif77. No extra include paths or library are needed. See NERSC's MPI page for details.

Interactive Jobs

Jacquard uses the PBS queue system for submitting all kinds of jobs. To run parallel interactive jobs (where you can see the output on your terminal as it runs) you use the qsub command with the -I option.

To run on two nodes, using all processors on each node:

Use qsub -I -q interactive -A reponame -l nodes=2:ppn=2 to launch an interactive shell. You can then run your code from the shell.

To run on two nodes, using 1 processor on each node:

qsub -I -q interactive -A reponame -l nodes=2:ppn=1

Note that you will be charged for both the processors for both examples above, regardless of whether you used all the processors or not. Interactive jobs are limited to 16 nodes (32 processors) and 30 minutes. See NERSC's interactive jobs page for more details.

Batch Jobs

Jacquard uses PBS queue system for batch jobs. Jobs are submitted as a shell script with PBS directives embedded in the comments. Following is a sample script

#PBS -l nodes=8:ppn=2,walltime=00:30:00
#PBS -N jobname 
#PBS -o job.out
#PBS -e job.err
#PBS -A repo
#PBS -q debug 
#PBS -V

./my_program

Lines starting with # are PBS directives. You can find the details about these directives here.

Repository

Supercomputer time is a limited (and precious) resource. Our class have been given 50,000 processor-hours in repository mp309. Each person has been allocated some fraction of this with some slack. If you use a node, all the processors on that node belongs to you, and hence you will be charged for both the processors. Processor time quota can be checked with getnim -U<username>. See NESRC's accounts page for more information about how accounts are charged.

To prevent unnecessary charge, please

Debug before submitting a large job: check serial program, try a small number of processors (might as well use 2 processors, since you will be charged for it).
Set memory and time limits to your batch job: if a bug in your program causes an infinite loop, the job will be stopped earlier.

Last updated January 26, 2007