Supercomputers are usually a Linux machines, that requires user to have additional skills in Linux. And while high-level programming for Linux and Windows is quite similar, all the infrastructure and tools are di erent. This paper presents a way to hide Linux interaction level from user by introducing the new library for .NET framework that handles SSH communication internally. Library has ability to run single-CPU/GPU processes as well as multicore/multi-GPU. Slurm task scheduler is supported for parallelized programs. Library also contains helper classes that allows user to retrieve source code from remote location (git and Subversion are supported) and compile program from sources (make is supported).
SLURM re
Supercomputer calculations are not exotic anymore. With growth of data, which should be processed during science research, there is a need to use supercomputers for programmers of any skill and background. And there is a problem. Supercomputers usually (if not always) works under Linux operation system, while most widespread operation system for desktop computers is Microsoft Windows. Thus, there is a big amount of programmers, who can develop software under OS Windows, but are not experienced Linux developers (or even users).
On the other hand, even Linux developers may experience di culties working with remote supercomputer system using Secure Shell (SSH, the most common way to access Linux computers via network). This procedure includes many routine operations such as transmission of program source code to remote computer, compilation of source code, transmission of data to process and, at last, obtaining the results of calculations.
There are many di erent approaches to make easier remote access to
supercomputers: REST API, Grid access or Web interface. This techniques are
described in [
So, there is a need in small tiny wrapper that should minimize count of required manual operations over SSH during supercomputer usage, but still allow developer to modify executed programs. In this paper author presents new library developed for .NET programming platform, which has these abilities. Library is written with full support of asynchronous features of .NET platform and C# language and allows easy convenient access to remote Linux system. 1.1
Library Structure
Three types of classes are inherited from the base class. First one is VCS (version control system) Client classes, which are intended to get source code for use program from remote repository. Currently three providers are implemented: Git, Subversion and CVS. Password-based authentication schemes are supported.
Second group is Builder classes. This is an abstraction for a set of tools that can translate source code into executable. Make and CMake are supported.
Classes in third group are needed to execute user programs. There are two ways of execution. First is direct run, that could be used if target computer has no ability to perform time-separation of tasks of di erent users or if executed program does not use any shared resource. Second way is execution using Slurm workload manager by sending program to the execution queue. 2
SSH communication is implemented in a fully asynchronous way by using async/await paradigm of .NET 4.0. All of the operations can be executed in parallel.
All SSH communication level is encapsulated in RemoteExecutor class. Usage example is provided below. var exec = new RemoteExecutor (" server . com " , " user " , " password "); var git = new GitClient ( exec ); var path = await git . Instantiate (" https :// github . com / my / program "); var compiled = await new MakeClient ( exec ). Make ( path ); if ( compiled )
Task . WaitAll ( exec . Run ("( cd program ; ./ program )")); else
Console . WriteLine (" Compilation error ");
This program rst logs into Secure Shell server with provided credentials. Then, git repository with needed program is cloned from external location. After that, a compilation is executed with make command. And if the compilation succeeds, resulting program is executed.
As it can be seen from the example above, in this case there is no parallel execution. All created tasks are awaited before the execution continues. This is related to sequential character of operations: make cannot be executed until all source code is available, and program cannot be run until it compiled.
However, for more independent tasks the parallelization can be obtained in the same simple way: var cmd1 = exec . Run (" for i in `seq 1 10 `; do echo $i ; sleep 1; done "); var cmd2 = exec . Run (" for i in `seq 1 5 `; do echo $i ; sleep 2; done "); var results = await Task . WhenAll ( cmd1 , cmd2 );
Each command is simple bash loop, which takes 10 seconds to be executed fully. Both loops will be run simultaneously and program will still nish in 10 seconds. Result of each exec.Run operation is an instance of CommandOutput structure that contains return value, standard output and standard error streams (stdout and stderr ). No mixing of output is performed even when commands are run in parallel because new SSH stream is created for any new command. 3
The main goal of creation of this library was to provide an easy way for seamless
execution of software, which had been written not just for Linux platform, but
for Linux-based supercomputers. This includes programs working via MPI
technological stack or using CUDA/OpenCL libraries for GPU computations. Such
supercomputer systems in most cases are designed to be used by many users
simultaneously and, thus, contain some software for program execution
scheduling. Library, which is described in this paper, contains support for one of most
widespread schedulers, Slurm [
Class for execution of parallel program with Slurm scheduler is called ParallelCodeExecutor. Execution itself is performed with Execute method, which signature is public async Task < CommandOutput > Execute ( int nodesNumber , int gpuNumber , string partitionName , string program , params ProgramArgument [] arguments )
Here nodesNumber is number of CPU nodes that should be used by this execution, gpuNumber is the number of GPU devices, requested on each node, partitionName is a name of cluster partition, where program should be executed, program and arguments are executed program and its arguments.
Each element of arguments can be either xed le name on remote system or local le, which will be uploaded automatically. ProgramArgument encapsulates this logic.
Return value is the CommandOutput structure with information about return code and stdout /stderr, the same for non-parallel executor. All streams redirection is handled internally and no user attention is needed to these details. 4
The presented software library covers full spectrum of tasks that exists in execution of custom supercomputing programs: source code obtaining, compilation and execution. Library helps developer to speed up the development process. Novice users are isolated from SSH layer, which may be not well-known for them. Experienced users can obtain development acceleration by automating routing operations.
Two main tasks can be solved using this library: 1. Reduce manual operations count during development process 2. Allow users to execute parallel programs in the same way as non-parallel.
All logic related to execution queue, job identi cation and task completeness is encapsulated inside library
As the result, it is convenient to use described technique for iterative processes. If program to be executed should be run in a loop, each iteration of which depends on results of the previous one, then this process includes many manual operation such as le copy or command execution. Usage of the library can signi cantly reduce overall time needed to obtain the nal result.
Library is distributed \as is" under LGPL 3.0 license. Source code can be obtained from https://github.com/atsidaev/parallexec. Since the structure can easily be extended by additional blocks, author welcomes everyone interested to submit patches and pull-requests.