dask-hpcconfig: database of cluster configurations#

dask, distributed and deployment packages like dask-jobqueue or dask-kubernetes have a dazzling amount of settings allowing to fine-tune a cluster.

However, as useful as that is for experts, new users easily get confused by the steps necessary to create a cluster. For dask-jobqueue:

  1. copy this configuration file to that directory and remember the name

  2. install the appropriate deployment package

  3. instantiate the appropriate class using that name, e.g. dask_jobqueue.PBSCluster

dask-hpcconfig aims to make this much easier by providing a unified API and by shipping a pre-configured set of cluster definitions. This will change the list above to:

  1. install the appropriate deployment package

  2. call dask_hpcconfig.cluster() with the desired cluster configuration name

where dask_hpcconfig.cluster() will tell the name of the package to install if it is still missing.