Securing Immutable Servers in a Serverless World

By Derek Weeks, VP and DevOps Advocate, Sonatype

Snowflakes are beautiful, unique creations. But, let’s keep them in nature. They don’t belong in our server infrastructure. Snowflake servers, where every configuration is just a little different, can introduce unnecessary security vulnerabilities and complications. While common in IT infrastructure, in the DevOps realm, they are gradually becoming ancient history.

At the recent All Day DevOps conference, Erlend Oftedal (@webtonull), with Blank and head of the OWASP Norway chapter, discussed the benefits of immutable infrastructure practices within serverless architectures. Erland walked us through the positive effects on security as well as offered insight on potential problems.

Erlend outlined the progress from physical servers to immutable servers, discussing the benefits of each step and the concerns that lead to the next phase, ending with immutable servers – that is, they can’t be changed. Once setup, they are set. Automated processes coupled with  immutability principles can eliminate snowflake servers.

Immutability isn’t a new concept. Some classic examples include: live Linux CDs, Internet cafes, schools, libraries, hotel business centers, etc. To set up immutable servers, Erlend outlines these steps:

1. Find a suitable base image
2. Build a specific image for the application (Dependencies, Application, Secrets)
3. Deploy image to server(s)
4. Repeat steps 2-3 for new versions.

Of course, this immediately raises questions.

What about data? He notes that, because data inherently changes, data has to be stored externally.

How do you deploy changes? Well, you don’t change the server. You take one out and put a new one in. What if that one fails? Rollback to the previous one.

How do you manage secrets? You can build secrets in, leverage cloud key management, or use third-party services.

How do we apply security patches? If truly immutable, you deploy a new server with the patch. However, you can have a semi-immutable server that allows for automatically installing patches but still doesn’t allow for logins or manual changes.

Speaking of security, what are the security advantages? For starters, if an attacker is in your server, they will be thrown out when you change servers, although they may come back if the vulnerability isn’t patched. It also allows for very specialized images by removing all unneeded packages.  With these specialized images, you can more easily  audit and monitor the system for unexpected file changes, logins, and connections.

Erlend discussed containers because they are a step closer to a serverless world. Containers isolate processes and all run on the same OS. Of course, you can make them immutable, but, whether they are or not, there are some key security best practices you’ll want to consider:

– Don’t run as root inside container
– Implement user namespaces so the root account within the container is separated from the root account outside the container
– Utilize the least privileges necessary for capabilities and resources
– Keep them up-to-date
– Automate scanning using tools such as Sonatype Nexus, Twistlock, Clair, or Aqua.
– Separate untrusted and trusted containers

Erlend also dove into serverless architecture. While there is debate over what exactly qualifies as serverless, there are 5 principles that Erlend highlighted:

– Use a compute service to execute code on demand
– Write single-purpose, stateless functions
– Design push-based, event-driven pipelines
– Create thicker, more powerful front ends
– Embrace third-party services

Migrating to immutable, serverless infrastructure is a topic to be discussed well beyond this blog post and even Erlend’s session. Whatever you do in the future, though, take his parting words to heart, “the cloud can solve many of your problems, but in the end you cannot transfer your responsibility for security.”

Erlend’s session was packed with information. If this piqued your interest, you should watch his complete session in his full All Day DevOps conference session (just 30 minutes). The other 56 presentations from the All Day DevOps Conference are also available online, free-of-charge here.