Accelerating AI Robot Development: Physical Intelligence’s Success with Aspect Workflows

Background

Jimmy Tanner, Senior Software Engineer at Physical Intelligence, first connected with Aspect in September 2024. His team had chosen Bazel for their infrastructure and they were seeking expert guidance and support. They were also exploring options for hosted Remote Build Execution (RBE). Their multi-language codebase included Python, TypeScript, and C/C++, and they were using GitHub Actions for CI along with Docker for containerization. 

Even with a few ex-Googlers on their team, Physical Intelligence recognized Bazel’s complexity and sought expert guidance from the outset to avoid costly technical debt. In addition to implementation support, they were also interested in experiencing the benefits of Aspect’s platform firsthand.

After evaluating Aspect alongside BuildBuddy and other solutions, they agreed to a 30-day trial.

Challenges

Physical Intelligence aimed to transition their robot development pipeline to Bazel to achieve faster, more efficient workflows. Their objectives were twofold:

1. Migrate their Continuous Integration (CI) pipeline to Bazel for significantly faster execution.
2. Shift their robot builds—primarily Python binaries packaged in Docker containers—to Bazel for smaller, more hermetic builds with improved cache hits and faster deployment.

Their starting point was a monorepo that was only partially integrated with Bazel. While the team had begun onboarding to Bazel and setting up its structure, their robot systems operated entirely outside of Bazel. In CI, Bazel was used experimentally with a remote cache, but the tests were optional and run for comparison, not yet fully replacing their Python pytest and GitHub Actions setup. CI times averaged 15 minutes, slowed by heavy load phases that diminished cache benefits. Robot builds were similarly inefficient, with build times on GitHub Actions’ larger runners averaging 10 minutes due to repeated downloading of external dependencies. A subset of low-level C++ hardware code was built with Bazel into binaries, which were then copied into Docker containers alongside Python code managed outside Bazel. Unifying these disparate processes under Bazel was a key goal.

The prolonged time from pull request (PR) to code deployment on robots hindered rapid iteration cycles critical for researchers. Additionally, GitHub Actions costs, approximately $12,000 per month, highlighted considerable inefficiencies due to the lack of a Bazel-based workflow. The team explored setting up their own runners but found Aspect Workflows to be an easier to use "out of the box" solution. Their goals were to reduce CI and build times, decrease workflow costs, and avoid trade-offs between speed and expense.

During the 30-day trial with Aspect, the focus was on fully migrating CI and robot builds to Bazel to achieve these performance gains. The team also aimed to move away from Docker to rules_oci to create container images more directly, particularly for complex CUDA-based images required for GPU-enabled robot components. While they could build basic images, they needed expert guidance for advanced configurations. Aspect believed that by onboarding the team to Bazel and leveraging Aspect Workflows, Physical Intelligence would achieve a strong return on investment, with faster builds and reduced costs effectively paying for the solution itself.

Results

By the end of the trial, Physical Intelligence successfully migrated their Continuous Integration (CI) pipeline and robot builds to Bazel, achieving significant performance improvements despite a substantial increase in build complexity. With Aspect’s expert guidance, the team transitioned their Docker containers to rules_oci, enabling faster and more direct creation of container images. When the collaboration began in November, Physical Intelligence was in a Bazel dark launch, with a repository containing 138 BUILD.bazel files. By the trial’s end, this number had grown to 364, reflecting a more than twofold increase in build complexity. Despite this, median build times for the main branch improved from 10–15 minutes to 5.5 minutes, and the entire pipeline median reached approximately 10 minutes. This represents a significant efficiency gain given the expanded build graph. Notably, prior to the collaboration, Physical Intelligence had no builds completing in under 2 minutes; in the last month of the trial, 21 such builds were recorded, showcasing Aspect’s impact on performance.

A key benefit, as highlighted by Jimmy Tanner, was the creation of a bespoke Docker build process that optimized every step of the build, push, and deployment workflow. This reduced the “time-to-robot” for production code from 90 minutes to 5 minutes, enabling Physical Intelligence to deploy productionized code to most robots. Previously, the 90-minute delay was intolerable, leading researchers to self-deploy code to robots manually using git. This improvement transformed developer iteration cycles, aligning with the team’s priority of maximizing speed.

Aspect Workflows reduced GitHub Actions costs by 7% despite heightened CI usage and a more than twofold increase in build files (from 138 to 364). This cost savings, achieved despite increased complexity, highlights substantial efficiency gains. Jimmy Tanner noted that the team currently values developer iteration speed most, though additional cost savings could be achieved through further build optimizations. He praised the collaboration: “Aspect Workflows has made Bazel an order of magnitude easier to adopt and significantly more valuable for our team.” The trial’s success fostered an ongoing partnership between Physical Intelligence and Aspect, with both teams satisfied with the outcomes and committed to continued collaboration.