Managing Ssh Known Hosts With-Serf

Serf is a very interesting service discovery mechanism. Its dynamic membership and tags capability make it very flexible. Can we use it to generate a centralized ssh_known_hosts file?

Installing and Configuring Serf

I like to use configuration management to manage servers. Here I use a Puppet module to install and configure Serf:

class { 'serf':
  config_hash   => {  
    'node_name'  => $::fqdn, 
    'tags'       => {
      'sshrsakey' => $::sshrsakey
    'discover'   => 'cluster',

This particular module uses a hash to translate directly into the config.json file on disk. Notice how I’m using the new tags feature, and adding a sshrsakey tag, populated by Puppet’s facts.

Querying The Cluster

Once the servers have Serf installed and configured, the cluster can be queried using the serf command line tool:

$ serf members    alive    sshrsakey=AAAA...    alive    sshrsakey=AAAA...

Using the Data

Lets use this data to write out our /etc/ssh/ssh\_known\_hosts file, emulating the the functionality of ssh-keyscan:

$ serf members -format=json | jq -r '.members | .[] | "\(.name) ssh-rsa \(.tags[])" ' | tee /etc/ssh/ssh_known_hosts ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTfPpmHhc+LoD05puxC... ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmzk+Chzrq73c5ytU9I...

So… you can see I’m using jq to manipulate the JSON ouput of the serf command. I’m not super proud of it, but it works.

Lets see if we can use a script instead? Serf provides and RPC protocol to interact with it programmatically:

#!/usr/bin/env ruby
require 'serf/client'
client = Serf::Client.connect address: '', port: 7373
members = client.members.value.body['Members']
puts members.collect { |x| x['Name'] + ' ssh-rsa ' +  x['Tags']['sshrsakey'] }

Of course, no error handling or anything. This script achieves the same result using the serf-client ruby gem.

There are libraries to connect to the Serf RPC directly for many languages, or you can do it yourself using the msgpack RPC library to communicate directly on the tcp socket.


This is just the beginning. Serf allows retrieving the status of members, but also can spawn programs (handlers) whenever members join or leave.

Additionally you can invoke custom events for your own uses, like code deploys.

If you can deal with an AP discovery and orchistration system, then Serf could be a foundation for building great things!