Building Web Frameworks at Google Scale: Productivity, Performance, and Dependability

Google is renowned for the scale and reliability of its web applications. Billions of users depend on products like Search, Gmail, Drive, Maps, and AdWords every day, and Google must maintain a high bar for performance, security, and user experience across its vast portfolio.

But equally impressive is what happens behind the scenes to power these apps – Google‘s approach to building the fundamental tools and frameworks that its many products are built upon. In particular, examining how Google builds web frameworks like AngularDart provides valuable insights into the challenges and opportunities of large-scale software engineering.

By the Numbers: Google‘s Massive Monolithic Repository

The first key thing to understand is the sheer scale of Google‘s codebase. Virtually all of Google‘s code – a staggering 2 billion lines – resides in a single unified code repository. This includes not just end-user applications, but also the frameworks, libraries, and tools they‘re built with.

Consider these statistics:

• 25,000+ Google engineers work in this single repo across dozens of offices worldwide
• 16,000+ code changes are committed per day on average
• Google‘s codebase doubles in size every 18-24 months
• The AngularDart framework alone comprises over 430,000 lines of code
• 3,200+ commits were made to the AngularDart framework in 2020

Supporting development at this scale requires a significant investment in tooling and infrastructure. Google has built advanced proprietary systems for version control, code search, testing, debugging, and more to make working with its vast codebase tractable.

The Tradeoffs of a Monolithic Repo

Having a single repository for an entire organization is highly unorthodox – most companies split their code across many smaller repositories by project, team, or function. Google‘s monolithic approach has some significant benefits, including:

• Simplified dependency management – no need to deal with versioning and publishing of internal libraries
• Extensive code sharing and reuse across projects
• Unified versioning and release management
• Ability to make atomic commits across projects and libraries
• Ease of collaboration and knowledge sharing between teams

However, it‘s not without its downsides and unique challenges, especially for foundational libraries and frameworks. Since there is only ever a single version of each library in the monorepo, it means that every project is always on the latest version, whether they like it or not.

For an AngularDart developer, a new commit to the framework is immediately used by every AngularDart application at Google. There‘s no way for projects to stay on an older, more stable version while others adopt the latest changes. This is what Google engineers jokingly refer to as "living on the bleeding edge".

Preventing Breakages with Extreme Testing

So how does Google ensure that changes to mission-critical frameworks like AngularDart don‘t constantly break all the projects that depend on them? The answer lies in an extreme commitment to automated testing.

AngularDart has its own extensive test suite, with over 3,200 unit tests covering the framework. But that‘s just the tip of the iceberg. With every change to AngularDart, Google also runs every relevant test for every project across the company that uses the framework.

Currently, that means a single change to AngularDart triggers around 75,000 tests! This includes unit tests, integration tests, and end-to-end tests for hundreds of real-world Google applications built with the framework.

While it may seem excessive, this extensive real-world testing is crucial for catching regressions and incompatibilities before they hit production. It‘s a testament to the investment Google makes in the reliability and stability of its core frameworks.

Fixing What You Break

Even with such thorough testing, breaking changes are still sometimes unavoidable, especially for a framework as complex as AngularDart. APIs evolve, better patterns emerge, and dependencies update. How does Google handle breaking changes at such a large scale?

The key policy is simple: "You break it, you fix it". If your change to AngularDart breaks a test in some other project, it‘s your responsibility to go and fix that project, whether you‘re familiar with the codebase or not.

This ensures that the AngularDart team is always in tune with the real-world impact of their changes. They can‘t push breaking changes without considering and addressing the upgrade path for their many internal customers. It enforces a careful, incremental approach to evolution.

Here‘s an example of how a major breaking change to AngularDart might be handled:

1. The AngularDart team identifies the need for a breaking API change
2. They make the relevant changes to the framework code itself
3. Automated testing identifies the hundreds of places in Google‘s codebase that are broken by the change
4. The AngularDart team works through each broken project one-by-one, updating them to adapt to the new API
5. Extensive code reviews are conducted with each affected team to ensure the upgrades are safe and correct
6. Only once everything is fixed can the breaking change be committed and released

It‘s a lot of work, but it‘s necessary to avoid leaving behind a trail of broken projects. And it has the added benefit of keeping the framework grounded in the reality of how it‘s used and the upgrade paths it needs to support.

Tooling for Large-Scale Changes

Making sweeping changes across a codebase the size of Google‘s is no small feat. To support this, Google has invested heavily in developer tooling and language features, many of which are open-sourced for the broader Dart community to leverage.

For example, Dart‘s strong static typing and Google‘s analysis tools allow for reliable large-scale refactoring. Developers can confidently make API changes, rename symbols, and move code around, trusting that the tools will catch any breakages.

The Dart formatter tool, dart format, is also crucial for making large codebases manageable. By having a single, unambiguous formatting standard, it removes all debate and inconsistency around code style, making changes easier to author and review.

Google‘s build tool, Bazel, also plays a key role. Its hermetic, reproducible build system ensures that builds and tests are consistent across machines and accurately reflect the state of the codebase. Bazel‘s advanced caching and incremental rebuild capabilities are what make AngularDart‘s 75,000 test suite feasible to run on every change.

Benchmarks from the Real World

Stability is critical for Google‘s frameworks, but so is performance. And for a web framework like AngularDart, performance doesn‘t just mean raw computational speed – it means actual user-perceived performance in real applications.

AngularDart benefits from being benchmarked against real, production Google applications, not just synthetic toy examples. Teams across Google instrument their apps to measure key user experience metrics like:

• Time to interactive: how quickly does the app become responsive to user input?
• Frames per second: how smooth are animations and scrolling?
• Memory usage: how much RAM does the app consume on average and at peak?
• Bundle size: how much JavaScript must be downloaded and parsed at startup?

Every change to AngularDart is evaluated against these real-world benchmarks. A performance regression in any of these key metrics will block a new version from being released. This real-world benchmarking is a significant advantage of frameworks being built in-house at large companies like Google.

Balancing Pace and Stability

The heavy testing, real-world benchmarking, and fix-what-you-break policies required to build frameworks at Google‘s scale inevitably slow down the rate of change. While other frameworks might release new major versions every few months, AngularDart tends to evolve more gradually and carefully.

This is a necessary and intentional tradeoff. For a framework that powers some of Google‘s most important and revenue-critical applications, stability and reliability are non-negotiable. Breakages and performance regressions have an immediate bottom-line impact.

However, the AngularDart team still manages to deliver significant new capabilities and improvements in a steady stream of releases. In 2020 alone, they shipped two major new versions, AngularDart 6 and 7, with headline features like null safety, new templating syntax, and improved build performance.

The key is finding the right balance – evolving the framework with new features and best practices while still maintaining a high bar for dependability and backward compatibility. It‘s a challenging tightrope to walk, but it‘s necessary at Google‘s scale.

The Power of Dogfooding

Perhaps the single greatest benefit Google‘s frameworks enjoy is the fact that they are heavily "dogfooded" – used by Google itself for its most important applications. The Google teams building these frameworks also feel the pain of any shortcomings in their daily work.

This tight feedback loop keeps frameworks closely aligned with real-world needs. When Google developers are blocked by a missing AngularDart feature, a confusing API, or a performance problem, the framework team hears about it immediately and viscerally. There‘s no more powerful motivator for improvement than feeling your own pain.

Dogfooding also ensures deep organizational investment in the long term sustainability and improvement of a framework. Google has a strong vested interest in keeping AngularDart healthy and evolving for the long haul, as the cost of replacing it would be immense.

For outside adopters, this is one of the strongest signals of a framework‘s staying power. When a hugely successful company like Google depends on a tool for its most critical applications, you can have confidence that it will be maintained and invested in for the foreseeable future.

Conclusion: Lessons for Any Scale

Not every organization can or should emulate Google‘s exact approach to building software frameworks. The sheer scale and complexity involved is mind-boggling and far beyond what most teams need to cope with.

However, there are valuable principles and practices that any organization can learn from:

1. Deeply dogfood your own tools and frameworks to keep them grounded in real needs
2. Invest heavily in automated testing, including real-world integration testing
3. Benchmark performance against actual user-facing metrics, not just synthetic microbenchmarks
4. Build tooling and processes that make large-scale changes safe and routine
5. Maintain a high bar for stability and reliability, especially for critical paths
6. Balance the pace of change with the need for dependability and ease of upgrade

Most of all, Google‘s approach exemplifies the value of having a strong engineering culture and of building tools to empower and amplify that culture. By coupling clear policies like "you break it, you fix it" with powerful tooling for testing, refactoring, formatting, and more, Google creates an environment where large-scale development is not only possible but routine.

While the specifics may vary, that‘s a worthy goal for any engineering organization. Invest in your tools, your processes, and your culture, and you‘ll be able to build more ambitious, more impactful, and more reliable software, no matter your scale. The lessons of how Google builds its frameworks offer a roadmap for any team looking to step up their engineering game.