NERSC Seaborg System Essentials

Be sure to visit the NERSC site; they have quite an extensive documentation on using the Seaborg system. All of the information here can be found in more detail there; I just list the most essential items on this page.

Connecting to Seaborg

The frontend login node is seaborg.nersc.gov. To log on, use a SSH program (such as OpenSSH and PUTTY; others are listed at here). If you need to change your password, login to sadmin.nersc.gov. After a password change/reset, you may need to wait up to an hour for the change to propagate to all the nodes. Contact support if you have problems.

Storage

There are two storage provided for you on the Seaborg system:

$HOME. 15 GB of space, avalable from every node.
$SCRATCH. 250 GB of scratch space, available from every node. There is a nominal 7-day deletion policy, but files may be deleted anytime after the job completes, so in your batch script you probably want to copy back the relevant data to more permanent place, like $HOME.

Neither of the above is backed up. Usage of your quota can be checked by the program myquota. Do not use /tmp and /var/tmp; they are slow and limited in space. Note that some programs (such as the IBM compilers xlc and xlf) use /tmp unless the you set the environmental variable TMPDIR. You can set to $SCRATCH by something like

export TMPDIR=$SCRATCH

Tools

Various software on Seaborg is managed by modules. To use a particular module, you need to run module use module_name. So for example, to load the IBM C compiler xlc, you'll need to execute module use xlc. To see all the available modules, use module avail. Some useful modules include

GNU. This provides various GNU tools, including gzip, autoconf, and automake. If you are used to GNU version of various tools, this is a must!
gcc. This provides the GNU compiler collection. Version 3.4.1 is the current default, but other versions are also available if necessary.
xlc. Provides the IBM C/C++ compiler (xlc and xlC). NERSC's xlc page has more information.
xlf. Provides the IBM Fortran 77/90/95 compiler. NERSC's xlf page has more information.
lapack. Provides the LAPACK library which implements various linear algebra routines. Also available is the 64-bit version lapack_64.
atlas. Provides the auto-tuned BLAS library, which implements the basic vector and matrix operations.
vampir and vampirtrace. Graphical tools for performance analysis. See the NERSC VAMPIR page.
tau. Provides TAU (Tuning and Analysis Utilities), tools for analyzing the performance of C, C++, Fortran and Java programs. See the NERSC TAU page.
papi. Provides PAPI, a common interface to various hardware performance counters. See PAPI website, as well as NERSC's PAPI page.
totalview. Graphical debugger for C/C++/Fortran programs which has support for parallel programs including MPI and OpenMP. See NERSC's totalview page.

More complete list (with links to documentation for various modules) is available at the NERSC Seaborg software page.

MPI

To use the MPI library on Seaborg with IBM's compiler, prefix the compiler name with "mp". For example, instead of xlc you would use mpxlc, instead of xlf use mpxlf. No extra include paths or library are needed. See NERSC's MPI page for details.

Interactive Jobs

To run parallel interactive jobs (where you can see the output on your terminal as it runs) you use the poe command.

To run on two nodes, using all 16 processors on each node:

poe ./my_program -procs 32 -nodes 2

To run on two nodes, using only 4 processors on each node:

poe ./my_program -procs 8 -nodes 2

Note that you will be charged for all 32 processors for both examples above, regardless of whether you used all the processors or not. Interactive jobs are limited to 8 nodes (128 processors) and 30 minutes. See NERSC's interactive jobs page for more details.

Batch Jobs

Seaborg uses LoadLeveler queue system for batch jobs. Jobs are submitted as a shell script with LoadLeveler directives embedded in the comments. Following is a sample script

#@ job_name        = myjob 
#@ account_no      = mp309 
#@ output          = myjob.out 
#@ error           = myjob.err 
#@ job_type        = parallel 
#@ notification    = complete 
#@ network.MPI     = csss,not_shared,us 
#@ class           = regular 
#@ node	           = 1 
#@ tasks_per_node  = 16 
#@ wall_clock_limit = 01:00:00 
#@ queue 
./my_program

Lines starting with #@ are LoadLeveler directives. Explanation of some fields:

account_no: should be always set to mp309, the class repository.
notification: if set to complete, email will be sent upon completion of the job.
class: specify which queue the job should go to. Different queue has different limits, reponse time, and charge. Usually should be set to regular, but can be set to debug for debugging jobs, which has a higher priority (but runs on fewer nodes). See NERSC's queue class page for details.
node. Number of nodes you need. Each node has 16 processors, so there are total of node x 16 processors will be available. If you want to use less processors per node, adjust task_per_node below.
tasks_per_node. Number of processors to use per node. Should be set to 16 unless you want to use less than 16 processors total or want to underutilize a node for some reason.
wall_clock_limit: specifies that your job requires at most this many hours to complete. Always use this to guard against bugs. Also jobs with shorter wall_clock_limit tend to get scheduled earlier.

Some useful commands:

llsubmit my_script -- submit the shell script my_script to the batch system.
llqs -- lists full queue.
llqs -c debug -- lists the debug queue. Other class such as regular and interactive can be listed as well.
llqs -u username -- lists your jobs.
llcancel job_id -- cancels your job with the given job ID.

See NERSC's batch jobs page for more details.

Repository

Supercomputer time is a limited (and precious) resource. Our class have been given 20,000 processor-hours in repository mp309. Each person has been allocated some fraction of this with some slack. If you use a node, all the processors on that node belongs to you, and hence you will be charged for all 16 processors. Processor time quota can be checked with getnim -U<username>. See NESRC's accounts page for more information about how accounts are charged.

To prevent unnecessary charge, please

Debug before submitting a large job: check serial program, try a small number of processors (might as well use 16 processors, since you will be charged for it).
Set memory and time limits to your batch job: if a bug in your program causes an infinite loop, the job will be stopped earlier.

You should also consider using the CITRIS cluster (in particular the batch system). You will get a faster response time.

UPC

To use the UPC compiler load the module upc:

module add upc

upcc64 compiles 64-bit applications, while upcc compiles 32-bit applications. You can do something like

upcc -o myprogram foo.upc -lsomelibrary

To run a UPC-compiled program, you can do something like

upcrun -n 32 -c 16 /path/to/myprogram

which runs 32 processes on 2 nodes (with 16 processes per node).

More information can be found at UPC Homepage, in particular see the UPC Users' Guide, the upcc and upcrun man pages.

[ Main CS 267 | GSI Page ]

Last updated March 16, 2005