Securing Your Web App’s Staging Ground

A Guide to Non-Prod Access Control

When building a web app, it is a common practice to control who can access the non-prod environments for reasons such as intellectual property protection and other security concerns. In this blog, I will recap the approach we used in a recent client project, the lessons we learned, and some alternative strategies.

As Charlie Munger said,

The more hard lessons you can learn vicariously, instead of from your own terrible experiences, the better off you will be.

I hope by the end of this blog, you will be equipped with some pragmatic strategies for this seemingly straightforward problem.

Who should have access to non-prod?

The answer feels obvious, you might think of something like:

• Developers
• Testers
• Designers
• Product owner and other stakeholders

I had the same thought at the start of the project, but later I learned there is more to the list:

• Content creators (often freelancers) need access to build marketing and training materials before production launch
• The CI/CD pipeline requires access to run end-to-end tests against non-prod
• Users participate in the User Acceptance Test (UAT)

Some of these late discoveries really caused a few headaches, which I will explore later in this blog.

Web app architecture

Our web app is a typical single-page app (SPA) hosted on AWS. We followed a standard hosting architecture, similar to what is outlined in the AWS documentation. For general security, we utilise some AWS Managed Rules for the Web Application Firewall (WAF), but this does not affect non-production access control.

Access control with IP whitelisting rules for WAF

Besides using AWS Managed Rules for WAF, we can also define custom rules to only allow whitelisted IP addresses to access our web app. We chose this approach early in the project because of its simplicity. It was easy to gather a list of IP addresses and CIDR blocks for our non-prod users and define a WAF rule.

This approach worked well for most of the project. However, as we approached UAT, things quickly got out of hand:

• As more testers joined the UAT, we were bombarded with frequent requests to whitelist IP addresses and ranges. Some testers were not tech-savvy, and we even got 192.168.0.x IP addresses 😂.
• Our IP whitelist soon became a messy collection of duplicate and overlapping IP addresses and ranges. We used AI tools to deduplicate the list, which helped, but I then realised some of the duplicate IP ranges are from the same Zero Trust tools (such as CloudFlare WARP or Zscaler) used by different participating organisations.
• When we transitioned testing focus from desktop to mobile, we quickly discovered that whitelisting IP addresses doesn’t work for mobile users, as their IPs can change rapidly. Even on a desktop, a user’s IP can change when they switch Wi-Fi networks or use a mobile network.
• Very often, these types of requests came in with a sense of urgency, so our engineers had to drop what they were doing to handle them.

In short, the seemingly simple solution we started with became a high-touch, low-value process near the end of the project. How I wished we had the luxury of time to revamp our approach, but what are the alternatives?

Access control with corporate VPN

In an enterprise setting, it is common to put non-prod environments behind a corporate firewall. Users can access it only via a corporate VPN. In practice, besides the cost overhead, this approach has several limitations:

• Not all legitimate users of non-prod are internal employees with VPN access. This includes content creators, UAT participants (discussed in Who should have access to non-prod), and other external stakeholders.
• It is not straightforward for a CI/CD pipeline (discussed in Who should have access to non-prod)to use a corporate VPN.
• For solutions designed with internet-facing services, such as the Amazon CloudFront (described in Web app architecture), it can be difficult to put the entire solution behind a corporate firewall.

Access control with custom header rules for WAF

This is a similar approach to what we used, but instead, we define a WAF rule to check for a specific HTTP header key and a secret value. If the header is present and the value matches, the request is allowed to proceed; otherwise, the WAF blocks the request and returns a 403 Forbidden error.

With this method, engineers are no longer burdened with managing the IP whitelist. However, the non-prod users will need to configure their browsers to add the required HTTP header and value to all requests for the non-production environments. While browser extensions like ModHeader for Chrome can make this process easier, it’s still an extra step and can be a hurdle for less technical users.

Access control with Lambda@Edge

Switching gears, let’s explore Lambda@Edge, an AWS service that works with CloudFront. It intercepts viewer requests, allowing you to run custom logic before CloudFront returns a response. For our purposes, we can use Lambda@Edge to authenticate users and control access to non-production environments. There are several blogs explaining this solution using various authentication mechanisms:

• With Amazon Cognito, explain in this blog post — Authorization@Edge — How to Use Lambda@Edge and JSON Web Tokens to Enhance Web Application Security.
• With other IdP, explained in this AWS blog post — Securing CloudFront Distributions using OpenID Connect and AWS Secrets Manager.
• With an external authentication server, explained in this blog post — External Server Authorization with Lambda@Edge, or
• Even with a simple basic auth, discussed in this Hackernoon blog post — Serverless: password protecting a static website in an AWS S3 bucket.

Access control at scale

While the aforementioned solutions are well-established patterns and can serve different use cases, they come with inevitable engineering overhead. If an organisation has multiple web apps that all need controlled access to non-prod environments, the engineers would have to replicate the solution for each one, which multiplies the maintenance effort. When this repeated effort becomes prominent, we should consider leveraging a shared Infra-as-Code repository or Infra-as-Code components (such as CDK Level 3 constructs).

From an infrastructure perspective, we will still end up running multiple access control workloads, each doing similar things for each web app. If I have my software developer hat on, I can say, “Since this is a common problem, why don’t we build a solution that centrally manages access to multiple web apps?” But with the mantra “No code is the best code” in the back of my head, I might explore some Off-the-Shelf solutions like Pomerium.

Wrap up

I’ve put together a summary table to recap the solutions we’ve covered.

There is no clear winner, and my answer is “it depends”. Before you design your non-production access control system, take the time to thoroughly understand who your users are and what they need. And remember, a simple solution that works at the beginning may evolve into something difficult to manage. As engineers, we should always be ready to iterate on our solutions to better serve new business demands.