Managing Access to Ephemeral Infrastructure At Scale

Managing a static fleet of strongDM servers is dead simple. You create the server in the strongDM console, place the public key file on the box, and it’s done! This scales really well for small deployments, but as your fleet grows, the burden of manual tasks grows with it.

With the advent of automated scaling solutions for our cloud environment like AWS Auto Scaling Groups, we need a way for our strongDM inventory to change in real-time along with the underlying servers. The solution: automation automation automation! The devops mindset is key, we want to automate cloud infrastructure so it operates without manual intervention. We can write scripts that hook into instance boot and shutdown events that will automatically adjust our strongDM inventory accordingly.

The examples in this post are written for AWS, but all major cloud providers should provide a similar API for instance information, lifecycle hooks, and metadata-like tags.

Automation— the hooks

For server access, there are two lifecycle events that we care about: server boot and server shutdown. We are going to write scripts that hook into these events and execute strongDM CLI commands to perform the necessary actions.

We’ll need API keys to talk to strongDM and our cloud provider.  In this case: AWS.

strongDM and AWS API Authentication

strongDM provides admin tokens that facilitate access to sdm admin CLI calls. The admin token has the following permissions:

On the AWS side, servers were given a programmatic (non-console) user account with one IAM policy attached. This IAM policy can also be attached to an instance role instead of embedding credentials into the script!

The policy contains one statement: allow EC2:DescribeTags. This API call is required to build out the server’s name in StrongDM.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ec2:DescribeTags",
"Resource": "*"
}
]
}

Naming Convention

Server names will be dynamically built using the server’s AWS EC2 tags, with the following schema:

$APP-$ENV-$PROCESS-$INSTANCE_ID

A server with the following tags would result in the following name in the strongDM inventory:

SERVER                             STATUS            PORT      TYPE

testapp-stage-wweb-00deadbeef      connected         60456     ssh

Startup

We've built a custom base AMI based on the Amazon Linux operating system. Our provisioning workflow includes a per-boot CloudInit script that executes when the server is powered on, including restarts. This script utilizes the strongDM command line tool to register automatically identify the information about the server, register it with strongDM, and install the on-demand SSH keys.

In /var/lib/cloud/scripts/per-instance/00_register_with_strongdm.sh

INSTANCE_ID="$(curl --silent http://169.254.169.254/latest/meta-data/instance-id)"

INSTANCE_ID_TRIMMED="$(echo $INSTANCE_ID | cut -d '-' -f 2)"

LOCAL_IP="$(curl --silent http://169.254.169.254/latest/meta-data/local-ipv4)"

APP="$(aws ec2 describe-tags --filters Name=resource-id,Values=$INSTANCE_ID Name=key,Values=app --query 'Tags[0].Value' --output text)"

ENV="$(aws ec2 describe-tags --filters Name=resource-id,Values=$INSTANCE_ID Name=key,Values=env --query 'Tags[0].Value' --output text)"

PROCESS="-$(aws ec2 describe-tags --filters Name=resource-id,Values=$INSTANCE_ID Name=key,Values=process --query 'Tags[0].Value' --output text)-"

SDM_SERVER_NAME="$APP-$ENV$PROCESS$INSTANCE_ID_TRIMMED"

curl --silent -o sdm.zip -L https://app.strongdm.com/releases/cli/linux

unzip sdm.zip

mv sdm /usr/local/bin/sdm

rm sdm.zip

mkdir -p "/home/sshuser/.ssh/"

touch "/home/sshuser/.ssh/authorized_keys"

chmod 0700 "/home/sshuser/.ssh/"

chmod 0600 "/home/sshuser/.ssh/authorized_keys"

chown -R "sshuser:sshuser" "/home/sshuser/"

/usr/local/bin/sdm login

PUBLIC_KEY=$(/usr/local/bin/sdm admin servers add -p "$SDM_SERVER_NAME" "sshuser@$LOCAL_IP")

# Touch a "lockfile" so the server can be deregistered when it's shutdown

# (see shared/roles/strongdm_target/templates/remove_from_strongdm.sh.j2:10)

touch /var/lock/strongdm-registered