=Paper=
{{Paper
|id=Vol-2267/558-562-paper-107
|storemode=property
|title=Event-driven automation and ChatOps on IHEP computing cluster
|pdfUrl=https://ceur-ws.org/Vol-2267/558-562-paper-107.pdf
|volume=Vol-2267
|authors=Anna Kotliar,Victor Kotliar
}}
==Event-driven automation and ChatOps on IHEP computing cluster==
https://ceur-ws.org/Vol-2267/558-562-paper-107.pdf
Proceedings of the VIII International Conference "Distributed Computing and Grid-technologies in Science and
Education" (GRID 2018), Dubna, Moscow region, Russia, September 10 - 14, 2018
EVENT-DRIVEN AUTOMATION AND CHATOPS ON IHEP
COMPUTING CLUSTER
A. Kotliar a, V. Kotliar b
Institute for High Energy Physics named by A.A. Logunov of National Research Center “Kurchatov
Institute”, Nauki Square 1, Protvino, Moscow region, Russia, 142281
E-mail: a Anna.Kotliar@ihep.ru, b Viktor.Kotliar@ihep.ru
Dealing with cluster systems you have multiple ordinary situations which can be solved using
automation tools. StackStorm is a quite good event-driven system which helps to manage typical
problems and to communicate with cluster via ChatOps extension. Just write a rule for such event and
it will be triggered and solved. In the presented work will be shown an example of a real event-driven
system on IHEP computing cluster which use Nagios, Check_MK, StackStorm, Mattermost for
routine work automation as a part of multicomponent cluster management system.
Keywords: Event-driven automation, ChatOps, StackStorm, cluster management system
© 2018 Anna Kotliar, Victor Kotliar
558
Proceedings of the VIII International Conference "Distributed Computing and Grid-technologies in Science and
Education" (GRID 2018), Dubna, Moscow region, Russia, September 10 - 14, 2018
1. Introduction
To solve computation problems in the distributed environment distributed systems are build.
Such systems have many elements from the distributed computing theory which allow to work
smoothly and to perform their operations. Usually all such elements are gathered together in the
computing clusters that have one common goal and represent one complex multicomponent system.
There are many ordinary situations which should be solved in such system and automation tools play
vital role here. StackStorm is quiet a good generic purpose event-driven system which is used on IHEP
cluster [1] and helps to manage typical problems as automation tool. When a failure happens it
troubleshoots it, fixes known problems or escalates them to system administrators if needed. It also
allows to use a brand new ChatOps technology for cluster management.
2. Event-driven system overview
StackStorm consists of four main logical parts like sensors, triggers, rules and actions [2].
Sensors are python plugins for integration that receives or watches for events. When an event from
external systems occurs and is processed by a sensor than an internal trigger emitted into the system.
From event-driven system’s view triggers are representation of external events. At IHEP it is already
used timers, webhooks and integrations triggers. It is easy to define a new trigger type just by writing a
sensor plugin. Rules map triggers to simple actions or to tasks as a set of actions (workflows) by
applying matching criteria and by mapping trigger payload to action inputs. Actions are StackStorm
outbound integrations. There are already implemented many generic actions like ssh connect, crontab,
sending e-mail, REST calls and several integrations with systems like OpenStack, Docker, Puppet
Actions are either python plugins, or any scripts, consumed into StackStorm system by adding a few
lines of metadata. The great features of the system is that actions can be invoked directly by user via
CLI or API, or used and called as a part of rules and workflows. To deploy its self-build content packs
are used. They simplify the management and developing of StackStorm pluggable content by grouping
integrations (triggers and actions) and automations (rules and workflows). IHEP StackStorm pack is
managed by GitLab system. To audit logs of action executions, manual or automated, they are
recorded and stored with full details of triggering context and execution results into a NoSQL DB.
This information also captured in audit logs for integrating with external logging and analytical tools
like: LogStash, Splunk, syslog. At IHEP mostly rules and actions are used for daily operations which
are deployed in one pack.
2.1. Events, triggers, rules and actions
To use event-driven functionality first of all we need to have one or many systems which will
generate events. At IHEP cluster Check_MK-Nagios monitoring framework is used. To bind this
system with Stacktorm two steps are needed to be done:
On Stackstorm server it needs to be installed Check_MK integration pack;
On Check_MK server it needs to be installed StackStorm plugin.
After installation steps in Check_MK system some relevant events need to be set.
Figure 1. Check_MK event for StackStorm
On the figure 1 is shown the event (notification rule in Check_MK terms) which is triggered
on CRITICAL or UKNOWN states for the cluster computing nodes. When such events occurred the
information about them is sent to the event-driven system to handle it. On the StackStorm side a
special trigger installed by the integration pack is used to react to the problem. Now to really react to
the event or make an action in StackStorm special rule was created for a specific problem. To bind
specific problem and action a special rule parameter “Criteria” is used. Such rule-action schema is
559
Proceedings of the VIII International Conference "Distributed Computing and Grid-technologies in Science and
Education" (GRID 2018), Dubna, Moscow region, Russia, September 10 - 14, 2018
used to bind many actions for one event depending on event parameters (one-to-many relationship see
figure 2).
Rule Action
Rule Action
Event Trigger
...
Rule Action
Firgure 2. One-to-many relationship for event-action in StackStorm
As an example a real production trigger-rule-action set is shown on figure 3. Check_MK event
handler checks against a rule NodesCheckMkCheckOrphans for three criterias: CRITICAL event, host
name started with “wn0” and specific service “check orphans” which is triggered by the event. After
all conditions are met an action “core.remote” is performing. In the example it just kills all orphaned
processes on the computing node.
Figure 3. Trigger, rule, action for Check_MK event
2.2. Complex actions
Most cluster management processes consist of multiple distinct interconnected steps that need
be executed in a particular order in a distributed cluster environment. A system administrator can
describe such process as a set of tasks and their transitions. After that, it is possible to upload such
description into the StackStorm mistral service, which will take care of state management, correct
execution order, parallelism, synchronization and high availability. In mistral terminology such set of
tasks and relations between them is called a workflow [3].
At IHEP computing cluster a kernel upgrade on the computing nodes must be applied
periodically. Such upgrade consists of many steps which involve many different components of the
cluster like batch system, monitoring system, working node itself. It was always a big headache for
system administrators because such upgrade should be done as effectively as possible with minimum
interruption to the service operations. To make such task more effective and simple it was developed a
StackStorm mistral workflow in the distributed cluster environment. From event-driven system a
workflow look like the action which consists of the set of other actions and it is possible to use such
560
Proceedings of the VIII International Conference "Distributed Computing and Grid-technologies in Science and
Education" (GRID 2018), Dubna, Moscow region, Russia, September 10 - 14, 2018
complex action everywhere where it is possible to use simple actions. To trigger kernel upgrade action
at IHEP a system console is used. One thing that should be taken in to account here is the time for
workflow to be completed. That could be from few hours to several days and all that time internal
StackStorm authentication token should be valid.
The procedure itself is very simple and consists of few steps:
1. A system administrator run kernel upgrade workflow through StackStorm API;
2. First step of the workflow is to put computing node offline in the batch system;
3. As soon as node will be free from computation jobs an upgrade should be applied and
node need to be rebooted;
4. After node reboot it’s status should be checked in the monitoring system;
5. As soon as all monitoring metrics for the node become normal the workflow put node
back to the production and inform the administrator.
Looks like all these steps could be done without any special system in bunch of scripts but
here is the real power of StackStorm appears. All steps in the workflow task are relatively
synchronized and fully logged. It is easy to find in case of the problem where and why the workflow
failed and the system administrator do not need to think how to save the states for the workflow
operations.
3. ChatOps technology
ChatOps is an operation paradigm where work that is already happening in the background
today is brought into a common chatroom. Such paradigm allows to fully open IT daily operations and
unify the communication about what should get done with action history of work being done [4].
Operation chat is the place where any worth notification could be done, any question and conversation
between sysadmins could be logged, it could speed up a new team member teaching, and it could be
used as a knowledge, tips and tricks base for the operation issues. As soon as there are all software
platform supported it is possible to use it from smart phones, Linux or Windows personal computers,
and even from any web-browser and be in touch for operation any time from any place in a convenient
way.
To implement ChatOps for IHEP IT operations Mattermost and Hubot software were chosen
(Figure 4). Mattermost is a messaging workspace which has a possibility to use many different
integrations. One of them is Hubot. Hubot is an open source bot written in CoffeeScript on Node.js
which can do operation things that are needed. It is possible to post image to chat, to translate a text, to
give a weather forecast and many others. In case of IHEP it is used to communicate with StackStorm
and post answers to the common chat from the server.
Figure 4. ChatOps system at IHEP
By placing StackStorm system for Hubot as a cluster manager software on IHEP cluster the
big set of features become available from a chat room. It is possible to run remotely a command, list
executions, list and install packs, re-run executions. To handle the big output for Stackstorm messages
in Hubot it was modified to send a big messages to the chat room split by frames. The last but not least
561
Proceedings of the VIII International Conference "Distributed Computing and Grid-technologies in Science and
Education" (GRID 2018), Dubna, Moscow region, Russia, September 10 - 14, 2018
piece in the system is GitLab service where cluster configuration is stored. After any modification of
the configuration notification messages reflects changes at the operation chat.
4. Conclusion
A new event-driven automation system for IHEP computing cluster was presented in the
described work. This system is based on open source StackStorm software. Some of the already used
StackStorm features are described. StackStorm allows to automate many actions for such complex
system as a cluster for distributed computing and it becomes a core part of the multicomponent cluster
management system at IHEP. With growing knowledge base for solving operation issues and
implementing them as actions or workflows it is possible to implement basic principles of autonomic
computing for self-management systems that will allow to increase effectiveness for the computing
cluster. To clue a new operation tool with system administrators team a ChatOps paradigm is used for
day-to-day operations where all background operations are brought into a common chatroom.
For further developing of the event-driven system all manual actions are need to be
programmed in a way that they could be used in StackStorm system. It is also planned to use this
system as a part of anomaly detection system on IHEP computing cluster for self-healing. Experience
in operating with event-driven systems could be used not only for IT operations but also for
experimental sets for physics in the Institute.
References
[1] Viktor Kotliar. IHEP cluster for Grid and distributed computing// CEUR Workshop Proceedings.
February 2017: Vol. 1787.- pp. 312-316
[2] StackStorm documentation [StackStorm overview]. Available at:
https://docs.stackstorm.com/overview.html. (accessed 24.09.2018)
[3] OpenStack documentation [Mistral overview]. Available at
https://docs.openstack.org/mistral/latest/overview.html (accessed 24.09.2018)
[4] StackStorm documentation [What is ChatOps]. Available at
https://docs.stackstorm.com/chatops/chatops.html (accessed 24.09.2018)
562