Getting started on Iridis

Note that the most upto-date information on using Iridis can be found on the CMG Wiki pages for Iridis (these are accessible to registed Iridis users only). These provide a more dynamic and comprehensive  source of information on how to get the best out of using Iridis.  The current page reproduces the main features of the Wiki page as a guide for prospective users, but registered users should refer to the Wiki for definitive information. 

Access to Iridis 3

Users with an account can connect to one of the Iridis Linux login nodes using ssh. The Internet addresses of these nodes are iridis3_a.soton.ac.uk , iridis3_b.soton.ac.uk & iridis3_c.soton.ac.uk. The three nodes are functionally equivalent, so it doesn't matter too much which one you use. For security reasons, access via telnet & rlogin is not allowed. See the iSolutions ssh page for details of how to transfer files to and from your local computer and how to display a GUI from Iridis to your local machine.

File Store and Disk Space

When you login to Iridis 3 you will normally start off in your Iridis home directory. Please note that your home file store on Iridis is entirely separate from any other iSolutions file store. For users who require extra file store or higher IO performance there are two alternative file systems that can be used. These are /work and /temp.

  • /work - This is the equivalent of the /scratch directories on Iridis 2. Besides offering more space than is available in /home, it will give better IO performance than /home.
  • /temp - This is a large temporary storage area of approx 60TB that is optimised for very high IO performance on large serial read/write operations. It is never backed up and we will be introducing policies that delete files over a certain age, so it should never be used for long term storage. Output files from jobs which are written to this file system should be copied to either /home or /work as soon as is convenient after a job has finished.

Applications with intensive I/O requirements should consider using /temp for best performance. Note that this area is not backed up at all so users should copy any files they wish to save back to their /work or /home directories after jobs have finished.

Backups

The /home directory is fully backed-up. Note that while /work is backed up periodically, we do not have the capacity to do this on the same daily basis as for /home. While /work directories are backed up periodically, the backups of /work are intended primarily for Disaster Recovery (in the event of a complete filesystem failure for instance). At present the assumption is that individual's files or directories in /work can be recreated by the user, if necessary, in a reasonable time by re-running jobs. Files in /temp are never backed up and may be automatically deleted after a few weeks.

User Quotas

Users are allocated a quota of 40GB in /home. They would normally be expected to use their /work directory if they needed more working space, or alternatively to use /temp if the files are needed only during the duration of a job or and can be discarded or copied elsewhere shortly after. The default quota's on user's /work and /temp directories are 80GB and 500GB respectively. An increase in the quota's on /work & /temp may be requested if a genuine need arises, but the quota on /home would not normally be increased.

Which file system should I use?

Many user's needs should be met within the standard /home file system, particularly when they first start to use the system. For users who do find that they need access to larger and or higher performance file store, the recommended usage is that valuable files such as source files, input data files and results summaries should be kept in your /home directory as this backed up regularly. /work should be used for working space for large files produced by jobs in progress, for jobs which require better IO performance or for job output that had not been analysed as yet. The /temp file system is intended for short-term storage for jobs that have very demanding IO requirements, particularly if very large temporary files are involved.

Editing and viewing files, working with directories

Users who are already familiar with Linux/Unix and the vi or emacs editors will need to read no further. If you do not have any previous experience of using command line editors in Linux you will probably want to use the emacs or gedit text editors to modify files. The command emacs my_file displays a new window in which you can view and modify the contents of the text file my_file in an intuitive manner, without having to know anything about the many more powerful capabilities of advanced emacs use. (Note that you must be able to display new windows back to your local system - see our page on displaying a GUI. If you want to use a simple command-line editor in your original login window then nano is a good choice.)

To list which files and subdirectories you have in a directory use the ls command. (Directories are the equivalent of windows folders.)

To view or search the contents of the text file some_file use the command less some_file. To learn how to move around in the file or search with less type the letter "h" for help while in less.

Use the mkdir command to create a new directory, eg. mkdir new_dir to create a directory called new_dir. To change the directory use the cd command, eg. cd new_dir. Two dots are used to represent the directory above, so cd .. will change to the directory above. The pwd command will show you where you are in the directory structure.

Further information can be found on our introduction to Unix page.

Software and Application Packages

See the Iridis Software page for a list of the main applications, compilers and libraries installed on the cluster. A more up-to-date and complete list can be found on the CMG wiki pages for Iridis. This page may also have links to more information on using a package.

The user environment for most of these packages is set up using "environment modules". Before you first use an application you will need to load the appropriate module, eg. before you first use matlab in a session:

module load matlab

Thereafter you can run matlab as many times as you like. If you get get a message that an application can't be found (eg. matlab: Command not found) it generally means that the module is not loaded (or perhaps you've mis-typed the name of the application!).

The modules command can be used to check which applications are available through modules, eg.

module avail

It can also be used to get a brief summary of how to use the application, eg.

module help cfx

Note that some modules will be loaded for you when you first login (eg. those needed for the batch system). If you are likely to be using an application every time then you can ask for the appropriate module to be loaded for you every time you login with the module initadd command, eg.

module initadd fluent

If you want to check which modules you have loaded at any time use:

module list

See the Environment Modules page for more information.

Compiling

The use of the Intel compilers, ifort and icc is recommended on Iridis 3. The PGI Portland Group compilers also provided as are the gnu compilers gcc and g77.

When compiling MPI codes it is recommended to use the appropriate wrapper for the compilers to ensure that the correct libraries and include files are selected. These are mpif77 or mpif90 for Fortran and mpicc and mpicxx for C and C++.

Parallel Environments

MPI is the most popular environment for running parallel jobs, particularly for multi-node jobs. OpenMPI is the recommended implementation and versions for both the Intel and gcc compilers are available by selecting the appropriate environment module, with Intel as the default. Intel MPI is also available as an alternative to OpenMPI.

Support for shared-memory parallelism using OpenMP is available within both Intel and the Portland Compilers, and as Iridis 3 has 8-cores per node may offer an attractive option for parallel execution of critical section of code.

Many applications such as Fluent, CFX, Ansys or Abaqus have their own provision for running in parallel over multiple processor-cores, more details are given in the software pages for each application.

Running Jobs

All long-running compute-intensive applications be should be run on the compute nodes (not the login nodes as this degrades performance for all users and makes you quite unpopular). There are over 1000 computational nodes which can be accessed via jobs submitted the PBS/Torque resource manager (though we will be offering other possibilities for submitting jobs at a latter stage). Most jobs will be run in batch mode, though interactive jobs are possible. The order in which jobs are run is determined by the Moab scheduler. More documentation on how to submit jobs is given in the links below.

Example Jobs for Applications

The directory /local/software/examples contains examples of batch jobs for some common applications. Each example contains a _README_ file, a sample job file and associated input files plus sample output. You can copy them to your own filestore, to experiment with, using the command copy_example.

If you just use the copy_example command on its own it will list which examples are available. You can then use the command again to copy the desired example. For instance:

 copy_example fluent

Will copy a fluent example run to your own filestore, normally to ~/fluent_example (where ~ is shorthand for your home directory).

Change to this directory with:

 cd ~/fluent_example


The README file should tell you more about the example.
To run this example as a batch job on a compute node, use the command:

 qsub run_fluent

The output files produced by the job should be similar to those in the subdirectory sample_output. To adapt the example to your own jobs you need to provide your own case file and substitute your own set of fluent text commands for the elbow.jou file (refer to sec 1.3 of the Fluent User Guide initially, and then look at the Fluent Text Command List - use flu_man to access these manuals in a browser).

User Support

Apart from the usual support mechanisms, we are using a set of Wiki pages and User Forums to cover both general topics and particular application areas. These provide a more dynamic and comprehensive  source of information on how to get the best out of using Iridis. They are hosted by the Computational Modeling Group. See How_to_log_in; How_to_use_Iridis_forums & How_to_edit_Wiki_pages for more on how to participate. Many thanks to the CMG for providing this facility.