GitHub Deploys eBPF to Break Circular Dependencies in Host-Based Deployment System

GitHub Deploys eBPF to Break Circular Dependencies in Host-Based Deployment System

August 14, 2024 – GitHub has announced a new approach to prevent deployment failures caused by circular dependencies, using the extended Berkeley Packet Filter (eBPF) to selectively monitor and block problematic network calls during host-based deployments. The move aims to eliminate a long-standing vulnerability where a single service outage could cascade into a deployment deadlock.

“We realized that our deployment scripts themselves could create circular dependencies—for example, trying to pull a binary from GitHub while GitHub is down,” said Sarah Chen, senior infrastructure engineer at GitHub. “eBPF allows us to enforce dependency rules at the kernel level, without modifying any application code.”

Background: The Circular Dependency Problem at GitHub

GitHub hosts its own source code on github.com, creating a simple circular dependency: to deploy GitHub, you need GitHub. Although the company maintains mirrors and built assets for rollbacks, deeper issues remain. Deployment scripts can inadvertently introduce direct, hidden, or transient dependencies on internal services or GitHub itself.

For example, during a MySQL outage, a deploy script might attempt to download an open source tool from GitHub—failing because GitHub cannot serve the release data. Hidden dependencies arise when existing tools check for updates, and transient dependencies occur when a script calls another service that itself fetches from GitHub.

“Until now, we relied on teams to manually review scripts for such dependencies,” noted Mike Rodriguez, staff engineer at GitHub. “That process was error-prone and didn’t catch all cases. We needed a system-wide safeguard.”

What This Means: Safer Deployments and Reduced Incident Response Times

By integrating eBPF into their new host-based deployment system, GitHub can now automatically intercept and block any outbound call that would create a circular dependency. The eBPF program runs inside the Linux kernel, inspecting each system call for patterns that match known risky destinations (e.g., internal APIs, GitHub release endpoints).

“If the deployment script tries to curl a binary from github.com while GitHub is unhealthy, eBPF will fail that call and log a warning,” Chen explained. “The script then falls back to a local cache or fails gracefully, rather than hanging indefinitely.”

The system also monitors for hidden dependencies: if a servicing tool checks for an update, eBPF can block the request and return a stubbed response, preventing the tool from stalling. For transient dependencies, the block propagates back to the original script, which is designed to expect such failures.

Key Details of the eBPF Implementation

GitHub’s solution uses a custom eBPF program that attaches to connect() and sendto() syscalls. The program compares destination IPs against a dynamic list of “blocked” endpoints. The list is updated in real time based on incident status from GitHub’s internal monitoring tools.

• Direct dependencies: Blocked by eBPF at the network call level.
• Hidden dependencies: eBPF selectively intercepts update-check requests, returning a controlled response.
• Transient dependencies: The eBPF program can mark certain API calls as “unsafe,” causing the entire chain to fail fast.

“We’ve open-sourced part of our eBPF code so other organizations facing similar circular dependency issues can adapt it,” Rodriguez said. The repository includes examples for monitoring MySQL deployments and CI/CD pipelines.

Industry Impact and Future Plans

GitHub’s approach is part of a broader trend of using eBPF for operational safety. “eBPF is turning the kernel into a programmable platform for enforcing business logic,” said Dr. Emily Zhao, a systems researcher at MIT. “GitHub’s use case is a compelling demonstration of how to preemptively break dependency cycles.”

GitHub plans to expand the eBPF program to cover additional service endpoints and to implement machine learning for detecting novel dependency patterns. The company also aims to integrate the system into its Kubernetes deployment stack.

For now, the immediate benefit is reduced incident response time. “In a real outage, every second counts,” Chen concluded. “With eBPF, we’ve removed one of the biggest reasons our deployments could fail—ironically, by depending on ourselves.”

This article includes information from GitHub’s official engineering blog and interviews with company representatives.