Posts

How I tracked down a ‘Phantom Error’ in a high performance Authorization layer that defied logic, race detectors, and logs.

During a new feature rollout of Skipper (An opensource ingress controller from Zalando), we hit a puzzling issue: <5% of requests to a specific route failed consistently in one pod, while the same configuration worked perfectly everywhere else. The culprit? A timing-dependent bug in how OPA’s plugin manager handles state notifications - one that Go’s race detector won’t catch. (If a more visual representation is appealing to you, refer the slide deck shared at https://pushpalanka.com/posts/stale-snapshot-case. It was prepared as an industry example for concurrency and operating system students.) ...

Background This learning comes from a project aimed at providing Externalized Authorization as a Service (AaaS), integrated directly into the platform. The solution leverages Open Policy Agent (OPA) as the Policy Decision Point (PDP), with policy enforcement handled by Skipper — an open-source ingress controller and reverse proxy. Skipper integrates with OPA to serve as the Policy Enforcement Point (PEP). For a detailed overview, refer to this Zalando Engineering blog. As an ingress controller, Skipper is designed to introduce minimal overhead to requests. Given the large number of deployed instances, any inefficiency in resource allocation can quickly scale into significant costs. Similarly, even a delay of just a few milliseconds per request becomes expensive when multiplied across thousands of requests flowing through Skipper. Now should we save the goat or the cabbages? ...

I have been studying SPIFFE (Secure Production Identity Framework For Everyone) [1] for some time and here I am drafting the flow as I have understood now, for the benefit of anyone else trying to understand the flow. Core Concepts Identity Registry — The SPIRE server(A SPIFFE implementation) has its own identity registry which keeps two coarse-grained attributes that decide how the SPIFFE IDs will be issued to a workload. A separate registration API is provided to manage these entries in the identity registry. Node Selector — This defines a machine (physical or virtual) where a workload can be running on. The exact type of selector to be used is decided based on the infrastructure provider (AWS, GCP, bare metal) that the workload is running on. E.g., AWS EC2 Instance ID, or the serial number of a physical machine. Node attestors act based on the infrastructure provider to honor their selectors. Workload Selector — This defines how to identify a process as representing a workload, after the node is identified. This can be described in terms of attributes of the process itself (e.g., Linux UID) or in terms of indirect attributes such as a Kubernetes namespace. The node agent is responsible for verifying that a particular process on a machine qualifies for its workload selector. Workload attestors act based on the process attributes to honor the process selectors. SPIRE Node Agent — A process that sits on the node, verifies the provenance of workloads running on the node, and provides those workloads with certificates via the Workload API, based on the selectors. SPIFFE/SPIRE Architecture Communication Flow ...