Diffferences between Iridis 2 & 3

While the user environment on Iridis 3 will will be similar to that on Iridis 2, there will be some significant differences and a few details that might catch you out. This page is a first attempt at sumarising what we think the main diferences will be in advance of the system arriving. Some hints are provided on how you could prepare yourself for these differences in advance of the initial user service being available.

Operating System

The system will run Red Hat Enterprise Linux 5.3 (RHEL5.3) instead of  RHEL 4.0. Many users will not be affected by the RHEL version, but there are differences in the versions of the glibc libraries and other system software such  that users will probably find that codes compiled under RHEL 4.0 on Iridis 2 will not run under RHEL 5.3 on Iridis 3. One short-term workaround that should work in some cases would be to compile a version of your your code statically. You should then be able to run the static version on Iridis 3 from day 1, though it would be beneficial to recompile your code on the new system as soon as you have an opportunity as the statically compiled code will not be able to take advantage of the advanced architectural features of the Nehalem processors on the new system.  Note that this approach will not work for parallel  MPI codes as we will be providing a new infiniband interconnect which will have to use a new version of MPI.

Infiniband Network

A high-performance infiniband network will be available on all Iridis 3 nodes replacing the slower Myrinet network which was only available on  64 of the  Iridis 2 compute nodes. This network is targeted at users with MPI applications (including Fluent) which have demanding inter-node communications requirements. Those applications where communications latency is currently a performance bottleneck will particularly benefit, though the increase in bandwidth will also be beneficial. Non-parallel jobs may also benefit from the fast interconnect as this will also provide the connection to the GPFS filesystem giving much better I/O performance.

Quad-core nodes

Each node on Iridis 3 will have 8 computational cores as there are 2 processors with 4 computational cores in each processor. As most Iridis 2 nodes only had 2 processor-cores, it will become much more important on Iridis 3 to try to use all of the processors to get best use out of the system. There are several ways this can be achieved.

  • For users runing parallel MPI jobs it will be trivial to take advantage of the increased number of cores per node.
  • For users running lots of sequential jobs we will be experimenting with a change in the scheduling policy that will allow multiple jobs from the same user to run on the same node if there are processors free. If successful this will avoid the need to write scripts which run multiple sub-jobs within a single job.
  • Many applications have a parallel processing mode which enables jobs to use multiple processors. Users of Fluent, CFX and Ansys and Abaqus will be familar with this already, but it is worth noting that Matlab can also be used in parallel mode on upto 4 processors on a single node with the aid of the parallel computing toolbox though some code-rewriting will be necessary to gain any benefit.
  • When combined with the large memory individual nodes can be seen as highly capable shared-memory systems in their own right, so the use of  the OpenMP shared-memory programming model becomes attractive.

Application versions

We will be updating the versions of commercial applications such as Fluent, Ansys, Abaqus, Matlab etc and also the compiled versions of many open source or academic codes. As the older versions of software currently installed on Iridis may not be supported for much longer we will encourage users to switch to using newer versions of software packages at this stage and will only install older versions if there is a well-defined need.

Compiler versions

We will be moving to use of the Intel compiler suite and the associated Maths Kernel Libraries as the recommended compiler for Fortran based applications codes. Though the latest version of the PGI compilers will also be available. as will of course, the gcc compilers.

Software Environment

We will be using the Environment Modules package to provide an easy way for users to modify their working environment to work with the particular applications and software tools they are interested in. Modules are currently available as an optional extra on Iridis 2 which can be used to select mostly more recent versions of applications, but on Iridis they will become findamental to the way the environment is setup for most packages.

Default User Shell

On Iridis 2 all users currently have the tcsh shell set as their default shell when they login. This will be changed to the bash shell on Iridis 3. Many users will not notice much difference, unless they use the more advanced features of the shell. If you don't understand the difference between these shells already it probably means you don't need to worry!

Batch Job Environment and Scheduler

The batch job setup will be similar to that on  Iridis 2 but we will be using the Moab scheduler which is a commercially suported successor to the current Maui scheduler and should give us with the ability to provide a more sophisticated setup.

Diskless Nodes

The compute nodes on Iridis 3 will have no hard disk, so the O/S has to reside entirely in memory. This means that we will need to provide a cut-down O/S on most of the compute nodes to avoid using too much memory, which could be better used in running jobs. Hence we are unlikely to provide full X-windows support on most of the compute nodes, and you may find that other utilities are not available. If you find that there are crucial libraries or utilities missing let us know, and we'll check whether these can be added to the compute node build or moved onto the applications filesystem.  

The other consequence is that jobs cannot afford to run out of memory in RAM, as there is no substantial "swap" available on local disk (though we may configure a very small amount of swap space on remote disk). There is considerably more memory available on the Iridis 3 nodes (24GB v. 2GB on Iridis 2) so memory usage shouldn't be an immediate problem for most users but we will be issuing more guidance later.

Windows service

The current Iridis is entirely Linux based. Once we have the initial Linux setup running satisfactorily we will be rebuilding part of the system as a Windows cluster, with it's own login nodes.