Latest news for the Maths compute cluster

Introducing macomp19

Another node called macomp19 was added to the cluster on Wednesday December 16th and is now available for use. With four 16-core AMD Opteron processors providing a total of 64 processor cores and 512 GB of installed memory, this node will increase the ability to run large multi-processor/multi-threaded applications and those requiring a lot of contiguous memory.

Using the Linux du command on the cluster

Those of you who are using clustor2 for file storage may be mystified by the large discrepancy between file and folder sizes reported by the 'ls -l' and 'du' commands. This is because clustor2 uses file compression internally to improve read/write performance so that file and folder sizes as seen by the 'du' utility are the compressed sizes as stored in clustor2's ZFS disk pool, not the sizes seen by the operating system or yourself!

To support ZFS compression-based storage systems such as those used on clustor2, the latest version of du available for Ubuntu Linux now has the '--apparent-size' option which will report the actual file size, not the compressed size as seen on the clustor2 server. You can use this new option in conjunction with the existing du options such as 'h', 'm', 'c', etc. Here is an example, using a file that is 3.2 GB in size:

ls -l reports the file size is 3.2 GB as expected:

andy@macomp001:~$ ls -l 3_2GB_test_file.tar 
-rw-r--r-- 1 andy root 3343513600 Feb  2  2018 3_2GB_test_file.tar

but du shows it as less than half this size:

andy@macomp001:~$ du 3_2GB_test_file.tar 
1434360 3_2GB_test_file.tar

using the '--apparent-size' option to du now reports the size you would expect to see:

andy@macomp001:~$ du --apparent-size 3_2GB_test_file.tar 
3265150 3_2GB_test_file.tar

Using du to find sizes of files or folders on other servers attached to the compute cluster, for example silo2 or clustor, will show very similar sizes with or without the '--apparent-size' option since they do not use compression in their underlying storage systems.

major R upgrade for Maths compute cluster completed

Last autumn the core R installation on the cluster has been upgraded from version 3.4.3 to 3.4.4 which is a very minor upgrade but at the same time, the large additional R package collection - mostly from the CRAN repository - has also been rebuilt from current sources, many for the first time ever since the cluster was introduced.

Any questions?

As always, I'll be happy to answer any questions you may have.