{"id":279,"date":"2015-02-23T17:23:06","date_gmt":"2015-02-23T17:23:06","guid":{"rendered":"https:\/\/laur.ie\/blog\/?p=279"},"modified":"2015-02-23T17:23:06","modified_gmt":"2015-02-23T17:23:06","slug":"systemd-using-execstop-to-depool-nodes-for-fun-and-profit","status":"publish","type":"post","link":"https:\/\/laur.ie\/blog\/2015\/02\/systemd-using-execstop-to-depool-nodes-for-fun-and-profit\/","title":{"rendered":"systemd: Using ExecStop to depool nodes for fun and profit"},"content":{"rendered":"

Preface: There is a tonne of drama about systemd on the internets; it won’t take you long to find it, if you’re curious. Despite that, I’m largely a fan and focusing on all the cool stuff I can finally do as an ops person without basically re-writing crappy bash scripts for a living (cough sys-v init)\u00a0<\/em><\/p>\n

Process Supervision<\/h2>\n

Without going into the basics about systemd too much (I quite enjoy this post<\/a> as an intro), you tell systemd to run your executable using the “ExecStart<\/code>” part of the config, and it will go and run that command and make sure it keeps running. Wonderful! In this case, we wanted to keep HHVM<\/a> running all the time, so we told systemd to do it, in 3 lines. Waaaay easier than sys-v init.<\/p>\n

ExecStop<\/h2>\n

By default when you tell systemd to stop a process, and you haven’t told it\u00a0how<\/strong> to stop the process, it’s just going to gracefully kill the process and any other processes it spawned.<\/p>\n

However, there is also the ExecStop<\/code> configuration option that will be executed\u00a0before<\/strong> systemd kills your processes, adding a new “deactivating” step to the process. It takes any executable name (or many) as an argument, so you can abuse this to do literally anything as cleanup before your processes get killed.<\/p>\n

Systemd will also continue to do it’s regular killing of processes if by the end of running your ExecStop<\/code> script the processes are not all dead.<\/p>\n

Load balancer health checks<\/h2>\n

We have a load balancer that uses a bunch of health checks to ensure that the node that it’s asking to do work can actually still do work before it sends it there.<\/p>\n

One of these is hitting an HTTP endpoint we set up, let’s call it “status.php<\/code>” which just contains the text “Status:OK”. This way, if the server dies, or PHP breaks, or Apache breaks, that node will be automatically depooled and we don’t serve garbage to the user. Yay!<\/p>\n

Example: automatic depooling using ExecStop<\/h2>\n

Armed with my new ExecStop<\/code> super power, I realised we were able to let the load balancer know this node was no longer available\u00a0before<\/strong> killing the process.<\/p>\n

I wrote a simple bash script that:<\/p>\n