Page tree
Skip to end of metadata
Go to start of metadata


(warning) External links on this page can only be accessed from outside the RE (warning)

Background

Genomics England is moving to a new HPC (Helix) in order to provide researchers with the best compute possible for their research. This cluster will replace our current HPC (Pegasus).

Information for Test Users

If you experience any issues or difficulties please raise a service desk ticket with the subject line "RE Testing Issue" (Service Desk). This is important so that we can catch and triage these issues as fast as possible.

We would also be interested in your feedback in how you find the new system. The feedback can also be submitted via service desk, using the feedback form (Service Desk).

How Access Will Change

Current Access Method

Currently access to pegasus is via ssh from the Research Environment desktop in the form:

depending on the user in question.

New Access Method for Gecip

Both the username and the hostname will change when we switch to Helix. Access will still be from the Research Environment desktop.

ssh [email protected]@phpgridzlogn001.int.corp.gel.ac
OR
ssh [email protected]@phpgridzlogn002.int.corp.gel.ac

New Access Method for Internal Users

Both the username and the hostname will change when we switch to Helix. Access will now be from P2 VPN, details available on confluence. (Go to /Platform engineering/environment/VPN/P2 Helix VPN access).

ssh [email protected]@phpgridzlogn003.int.corp.gel.ac

New Access Method for Discover Forum

Both the username and the hostname will change when we switch to Helix. Access will still be from the Research Environment desktop.

ssh [email protected]@phpgridzlogn004.int.corp.gel.ac

Main Usage Changes

Changes in the Module Framework

The way modules are organised in Helix has changed when compared to Pegasus. For example, loading bcftools, python and R:

bcftools
Pegasus:
module load bcftools/1.9

Helix:
module load bio/BCFtools/1.9-foss-2018b

Python
Pegasus:
module load python/3.6.5

Helix:
module load lang/Python/3.7.2-GCCcore-8.2.0

R
Pegasus
module load R/3.5.1

Helix:
module load lang/R/3.5.1-foss-2018b

Changes to Python Package and Environment Management

The conda package and environment management system is provided in keeping with current best practice for managing Python packages. This is provided alongside the EasyBuild-managed lmod modules environment following best practice for managing HPC software.

There is overlap in functionality which increases diversity and robustness, and supports debugging.

The conda environments are additionally built around Intel Distribution for Python, where possible, and performance benefits are expected. https://software.intel.com/en-us/distribution-for-python/benchmarks

To begin using these environments, make sure that on a fresh login, conda is not already in your path (using the command "which conda"). If you find conda is already in your path, you need to undo this with "conda init --reverse" and log in again.

Once the above has been checked, issue the command

. /resources/conda/miniconda3/etc/profile.d/conda.sh

taking care to include the dot and space at the front to source the file for bash (a similar file for csh in the same location). This method of accessing conda will be compatible with job scripts. "conda init" should be avoided unless you only use a single Python environment that can be safely loaded in ~/.bashrc.

The command "conda env list" will show the available environments. "idp" refers to the full Intel Distribution for Python, and "idpcore" refers to the core packages. https://software.intel.com/en-us/articles/complete-list-of-packages-for-the-intel-distribution-for-python

py2 and py3 refer to Python 2 and 3, respectively.

Environments with the prefix "test" are used to prepare the production environments. These test environments will additionally be used for testing and may change at any time so are unsuitable for production usage.

Issue "conda activate idppy3" to activate the desired environment, "idppy3" in this case. "conda env list" can be used to check the package versions available in your activated environment. "conda deactivate" will deactivate your environment after finishing your computation.

A suffix "revN" may appear on some environments, where N is potentially a multi-digit number, to denote revisions.

N.B. Contrary to standalone usage of conda, you should not issue the command "conda init" or else it will break integration with the module environment by making changes out of the control of the module command. If you accidentally call "conda init", reverse this with "conda init --reverse".

Changes to the Filesystem Layout

There have been changes to where folders and files are located, but symlinks have been created to maintain the existing paths. There should be no change in the user experience.

Changes to the Physical Compute Nodes

Pegasus has 152 compute nodes. Most of these nodes are 12-core, 368GB RAM, 1.5TB local SSD nodes, whilst a small number are 24-core, ~750GB RAM, 3TB local SSD nodes.

Helix has 60 compute nodes. Each node is 36-core, ~750GB RAM, NO LOCAL SSD nodes.

Helix also has 1 GPU (2x v100) nodes. These can be accessed using the reservation ID gpu1 via LSF ("brsvs" to show current reservations, users need to be added to a reservation to access).

Changes to the Operating Systems on the Nodes

All nodes on Pegasus are running Ubuntu 16.04.

All nodes on Helix are running CentOS 7.6.1810

This means that any programs that a user may have compiled on Pegasus will need to be recompiled for Helix.

Changes to the Queueing System and Schedulers

There are no changes to the existing LSF queues. The current configuration has been copied over from Pegasus to Helix.

There will be 50 nodes in production that will run LSF.

There will be 10 nodes in production that will run SLURM. There is a desire in Genomics England to switch over to SLURM by the end of December 2020. Therefore, a small SLURM cluster is provided for users to get familiar with the system. As of 2020-07-13, the SLURM nodes have been taken to expand capacity in the LSF cluster so there is essentially no SLURM capacity at the present time (two nodes were previously taken to join a dev/test LSF cluster).


Changes to Proxy Environment variables for external access

The following variables should be set for access to external sites to work

http_proxy=http://pfsense.int.corp.gel.ac:3128

ftp_proxy=http://pfsense.int.corp.gel.ac:3128

rsync_proxy=http://pfsense.int.corp.gel.ac:3128

https_proxy=http://pfsense.int.corp.gel.ac:3128

no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com,.gel.zone,.cluster

You can create a .Renviron file in your home directory to use CRAN and bioconductor. Please be aware that R and Bioconductor packages installed to a shared location outside of Helix (e.g. using an Ubuntu operating system) are highly unlikely to work as they may rely on libraries not found within Helix. R and Bioconductor packages should be installed within Helix for use within Helix.

As of 2020-07-13, the bash variables are set by default in /etc/profile.d/proxy.sh on login nodes.

CRAN mirrors are unlikely to be whitelisted for access via the proxy so you may wish to call

options(repos=c(CRAN="https://cran.r-project.org"))

Other useful commands include

Sys.getenv("R_LIBS_USER")

install.packages("package_name", dependencies=TRUE)

Container Support

Helix will support Singularity (https://sylabs.io/docs/) for containerised workflows. If your workflow is written in Docker, this is also ok, as Singularity is able to read Dockerfiles and convert them into Singularity images.

List of Currently Available Tools on Helix - Last updated  

The document below shows a list of tools available in Helix. We would encourage everyone to use the latest versions however we understand that you may require an older version which will need to be requested. If you require additional tools, please raise a service desk ticket with the subject line "RE Additional Tools required in Helix" (Service Desk). This is important so that we can catch and triage these as fast as possible.  

A non-exhaustive list of Python modules and packages in each conda environment is shown below. - Last updated  

N.B. pip/pypi is known to have broken pandas in the "ipy3pypi" and "ipy3pypirev1" environments. The error you will see is "ImportError: cannot import name 'maybe_downcast_numerical'". "py3pypi" and "ipy3nopypirev1" should be tried as alternatives.

FAQ


Frequently Asked Questions Area

Why do simple utilities such as "time ls" sometimes take a long time to return?

The primary cause of this has been found to be a combination of reasons, but luckily work arounds are available as the various contributory issues are fixed.

The main reason is that a long LD_LIBRARY_PATH results in libraries such as "libselinux.so.1" being searched for in a huge number of inappropriate places. These metadata requests are in direct competition with the full traffic of the cluster so varies with the typically high amount of storage traffic generated by the entire cluster. A long LD_LIBRARY_PATH typically arises because of a large number of modules loaded, as can be inspected with "module list". 

The best current work around is to load modules within job scripts to avoid the latency when working interactively. Where this is not possible, the LD_LIBRARY_PATH can be shortened with "module purge" after using the loaded software.

Currently, it is not recommended to load modules in your ".bashrc" as this causes a long delay when logging in and triggers the problems mentioned above.

Where are the Ensembl VEP caches?

The Ensembl VEP caches are currently present in directories named ".vep" under subdirectories of /resources/data/vep.caches/from.pegasus/ and /resources/data/vep.caches/helix/ This is subject to future change as we consult with the curators of public data sources.

Why are my job run-times/wall-times longer than expected?

The "Resource usage summary" of the LSF job output should be inspected to ensure the time spent on the CPU (the CPU time) is a large proportion of your run-time/wall-time. The "Max processes" and "Max threads" lines should be inspected to ensure you have requested an appropriate number of job slots.

In the event that the CPU time is much lower than run-time, this means the computation is not doing much on the processor and is instead spending most of its time doing something else. Diagnostic commands such as "uptime", "free -h", and "df -h /" can be added to your job script to ensure your compute node is in a reasonable state.

The immediate follow-on test is to identify the input data, and output data location and synthetically test these.

Assuming $HOME/input is an input file, and $HOME/output is the output file, you may run the following two commands in your job script before the computation. "$HOME/output" is an arbitrary filename for testing. Do not specify an important file or it will be overwritten.
    time dd if=$HOME/input of=/dev/null bs=$((1024*1024))
    time dd if=/dev/zero of=$HOME/output bs=$((1024*1024)) count=100000
Be careful not to mix up input file (if) with output file (of) or you may lose data. Work on copies of data if you are unsure.

The input file should be one of the files used in your computation later. If an input file is especially large (e.g. over 100 GB), you can modify

    time dd if=$HOME/input of=/dev/null bs=$((1024*1024))

to include a count in units of MiB (roughly megabytes/MB)

    time dd if=$HOME/input of=/dev/null bs=$((1024*1024)) count=100000

Why does Helix use the UTC timezone?

On large systems with international users and requiring debugging and management of data across timezones, it is typical for the system to present the UTC timezone to users to avoid discontinuities across timezones and clock changes.

Why do I see the error "cannot create temp file for here-document: No space left on device" and what is a suitable scratch location?

The error "cannot create temp file for here-document: No space left on device" is typically seen when a user has been using the local directory /tmp as scratch space. At the present time, /tmp resides on local SSD which is limited to at most 500 MB/s performance which is slow compared to the parallel file system (max of 10 GB/s to 12 GB/s per node), but more importantly cannot be made large enough to support production workloads. The compute node operating system relies on a local /tmp directory so this location cannot be a shared location to avoid introducing unwanted dependencies.

An appropriate subdirectory of /re_scratch/re_gecip (or other Shared Working Space, depending on the user community) should be used as scratch space, otherwise the filling of /tmp will cause problems for all users of the node and has also been found to render entire nodes inaccessible to both users and system administrators. Typically it is found that misuse of bcftools most often hits the problem of filling /tmp.

Why does loading SAIGE in R-3.6.2-foss-2019b generate the error "libhts.so.2: cannot open shared object file: No such file or directory"?

Genomics England requested an in-place manual upgrade (roughly July 2020) of SAIGE which has the below additional dependency.

module load lang/R/3.6.2-foss-2019b bio/HTSlib/1.9-GCC-8.3.0

Such an upgrade could cause silent problems for users and invalidate certain results if the user is unaware, which is why best practice is to avoid such in-place upgrades. Luckily, this additional dependency will flag the change for users so that they are aware and won't have their work invalidated by a silent change.


Cluster Guidelines

DO NOT ...

  • DO NOT run applications directly on the login/submission nodes . This nodes must be used as a portal to the cluster to submit jobs (bsub) and no more.
  • DO NOT target execution nodes to run the job. Leave the decision making to scheduler
  • DO NOT implicit module load (eg. module load python (error)); Always explicit load (eg. module load python/3.6.0 (tick))

  • DO NOT ssh onto any cluster nodes execution nodes. Just login to submission node and use bsub to submit to the cluster 

    Queue policies:

  • The interactive queue is the only queue that permits interactive work; So please be mindful of the number of Jobs submitted in this queue.

  • Use the Time-based queue as per the workflow. e..g 

    • Short Queue for jobs that run within 4 hours

    • Medium Queue for jobs that can run for 24 hours

    • Long queue for anything more than 24 hours.

  • Try and strip long jobs into chunks of smaller jobs.Best Practices:

  • Used the re_scratch directory for temporary & large data storage. Please be mindful of the data under scratch area. Scratch directories will be reviewed and any large file age 30 days or more will be subject to clean up clean up regularly
    /nas/weka.gel.zone/re_scratch/<domain>/<project code>

    • e.g For machine learning the scratch area would be,
      /nas/weka.gel.zone/re_scratch/re_gecip/machine_learning

  • Always be considerate of other users in the cluster. Be mindful of the number of jobs you submitted. When you are submitting large numbers of jobs or large job arrays, please consider this, and break the job submissions into small batches wherever possible
  • Always log job output errors in a file using -o flag to bsub “-o output_of_my_job.txt”. 
    e.g. bsub -o </path/to/outputfile>

  • Always request resources (cpu, memory, licenses(if applicable))  used by your application(s) with rusage option in bsub.
    e.g. bsub -R 'rusage[mem=4096]' ... ; LSF will reserved 4gig of memory for your job

  • Good to safeguard your jobs (runaway jobs) by setting appropriate run limit (-W), if possible a memory limit (-M) and a file limit (-F). 

          e.g. bsub –M 10240 <rest_of_the_job> ; LSF will automatically kill the job when the job exceeds memory usage of 10 GB.

         

  •  (If applicable) Please exit all interactive jobs before logging off for the day as this will free up additional capacity for batch jobs in the off hours.
  • Avoid poor resource requirements 
  • No labels