When your infrastructure depends on one person

Many companies have a person without whom operations cannot run. They know where everything runs, they know the passwords and they know what to do when something stops working. As long as they are available, everything works. The problem comes with a holiday, an illness or a resignation.

Usually this is nobody’s fault. It is the natural result of how a company and its systems grew over time. The good news is that you can get out of it in small steps.

How to tell if this applies to you

• The question “how does this work?” is always answered by the same person.
• Only one or two people know the passwords and keys.
• Documentation is missing or out of date.
• Changes to the environment are made manually and not recorded anywhere.
• Nobody is sure whether restoring from backups would actually work.

Step 1: Map what you have

List your servers, services, applications, their dependencies, access and backups. The goal is not perfect documentation right away, but an overview: what exists, what it is for and what is critical.

Step 2: Get access under control

• Store passwords and keys in a shared password manager or secrets vault.
• Use personal accounts instead of shared ones.
• Give everyone only the permissions they actually need.
• Have a procedure to revoke access quickly.

Step 3: Introduce Infrastructure as Code

Infrastructure as Code (IaC) and configuration management describe the environment in files that are versioned just like applications. Such a description is also documentation that does not go stale — because the environment is actually built from it. Changes are reviewed and traceable.

Step 4: Verify backups and recovery

A backup that has never been restored is only an assumption. Test recovery regularly and record the result, including how long it took.

Step 5: Share the responsibility

• Write down procedures for routine tasks and the most common incidents.
• Assign a stand-in and let them try the procedures.
• Send monitoring alerts to more than one person.
• Consider an external partner who takes over agreed activities and knows the environment.

Summary

The goal is not to get rid of your key person. The goal is to take some weight off them and make sure the company is not vulnerable when they are unavailable. Mapping, access control, IaC, verified backups and shared procedures get you there in small, manageable steps.