I was laid off by Atlassian
- Introduction and Context: The author reflects on his experiences after being affected by layoffs at Atlassian, where he worked for approximately 8 years. He shares details about the technical architecture he built, key achievements, and non-technical lessons learned to inspire others in similar situations.
- The Interview Process (8 Years Ago):
- Began with an online coding quiz on HackerRank, which he aced with full marks.
- The first technical round involved reading a Cloudflare white paper on custom domains for 10 minutes and then answering architectural questions regarding microservices and containers.
- The second technical round was a live troubleshooting simulation of a real Atlassian incident (an application issue causing a Denial of Service). He also faced questions about latency-based DNS routing.
- During the values interview, when asked what success would look like in 12 months, the interviewers outlined the need for an internal platform application to provide self-service load balancing for Atlassian dewelopers.
- Building the Open Service Broker (OSP):
- In his first few weeks, the author built an application adhering to the Open Service Broker API specification to automate infrastructure provisioning in a Kubernetes environment.
- Internal developers declared their infrastructure requirements using configuration files in version control, which build servers then uploaded to the broker.
- The system was originally built in Python using the Connexion library (routing based on OpenAPI documents), later migrated to pure Flask, and eventually transitioned to FastPI.
- The architecture utilized an asynchronous task queue model: FastAPI received requests, pushed task details to AWS SQS, and background workers handled tasks (like creating DNS records or CloudFront distributions) while writing status updates to DynamoDB.
- Transitioning to Envoy Proxy and Sovereign (Control Plane):
- Atlassian decided to replace expensive corporate enterprise load balancers with Envoy Proxy, an open-source, cloud-native proxy.
- The author built a custom Envoy management server/control plane named Sovereign (which was open-sourced on Bitbucket).
- Built with FastAPI, Sovereign pulled dynamic context data from the broker's database and AWS S3 buckets, injected it into templates for Envoy resources (clusters, routes, listeners), and dynamically pushed updated configurations to running proxies over the wire.
- Infrastructure as Code and Image Automation (AMI):
- The entire proxy infrastructure—comprising around 2,000 proxies across 13 AWS regions—was deployed using AWS CloudFormation templates defining VPCs, subnets, Network Load Balancers (NLBs), Security Groups, and Auto Scaling Groups.
- To create standardized images, the team used HashiCorp Packer combined with SaltStack (a declarative configuration management tool similar to Ansible or Chef).
- The resulting AMI had pre-installed and optimized components, including Envoy, network tuning configurations, security hardening layers, and observability agents for logging, tracing, and metrics.
- Mass Migration and Edge Centralization:
- Following the initial framework setup, the team spent roughly two years migrating major Atlassian core products (Jira, Confluence, Bitbucket, Statuspage) and thousands of microservices behind this centralized edge infrastructure.
- The platform locked down public exposures; microservices could no longer be accessed publicly by accident. Developers had to explicitly signal intent through the proxy configuration.
- Centralizing these features saved millions of dollars and massive development time, sparing thousands of developers from having to independently implement features like authentication, authz, or rate limiting on their own backends.
- Sidecar Architecture and Custom Rust Tools:
- While DDoS protection was offloaded to AWS CloudFront and Access Logs were captured natively via Envoy's HTTP Connection Manager filters, more complex features required a sidecar container model running locally on the proxy EC2 instances.
- The author personally designed and wrote a custom authentication sidecar container from scratch using Rust ("the Lord's language").
- Other specialized internal teams contributed separate sidecar containers for authorization and rate-limiting.
- Non-Technical Growth and Professional Lessons:
- Diplomacy and Conflict Resolution: Working with various managers and diverse personalities for nearly a decade forced the author to dramatically mature his skills in persuasion, mentoring, and navigating interpersonal friction.
- Code Churn and Long-Term Maintenance: The author notes that building software is easy, but maintaining its malleability over time is hard. Codebases develop highly predictable areas of continuous modification ("code churn"), which serve as code smells indicating growing complexity that must be actively refactored before coupling paralyzes development.
- Mentoring vs. Training: The author successfully mentored an intern to achieve the highest possible performance rating and a return offer. However, he reflects on mentoring as a highly challenging balancing act—knowing how to guide someone without giving away answers or letting them get overly frustrated—distinguishing it from his everyday engineering strength of breaking down complex system architectures into easily digestible mental models for peers.
