Transform your data pipeline
Learn how companies around the world are using Pachyderm to automate complex pipelines at scale.
Request a DemoWoven Planet, a subsidiary of Toyota, is building the safest mobility in the world. With a focus on automated driving, Woven Planet invests in new technologies, software, and business models that deliver secure, connected, reliable, and sustainable mobility solutions for all.
Woven Planet uses Pachyderm as an orchestration system to support their Automated Mapping team that processes petabytes of mapping data to extract semantic information (i.e., what does a road feature mean to the vehicle) in a fast, iterative, and reproducible manner.
Woven Planet’s Automated Mapping team is building automotive-grade maps for use in the automated-driving vehicles of today and the autonomous-driving vehicles of tomorrow. Key to this effort is the use of aerial orthographic projection. While this kind of aerial- and satellite-derived data have long been used in the development of consumer-grade navigational maps, using these kinds of data to meet the rigorous requirements of automated driving—and to do so at continental scale—is an unprecedented challenge.
For automated driving applications, maps need a level of detail, accuracy and precision far beyond those of their consumer-grade counterparts. Developing this level of granularity requires processing voluminous amounts of data. In order to achieve this at scale, Woven Planet’s Automated Mapping team is pioneering new, cutting-edge machine learning solutions.
In selecting an orchestration system as part of this effort, the Automated Mapping team had two key priorities: First was ensuring their pipeline had maximum flexibility—the ability to scale to meet elastic workloads and to easily toggle between structured and unstructured datasets. And second was long-lived pipeline stability that would support continuous, region-based map updates—critical to maintaining map accuracy in a constantly evolving world.
Woven Planet’s Automated Mapping team had strong expertise in Kubernetes (K8s); and as Pachyderm employs the powerful K8s’s orchestration engine to rapidly scale up workers to deal with elastic workloads, it offered a solution that could be easily integrated into the Automated Mapping team’s workflow and pipeline.
Pachyderm is a seamless complement to our pipeline and processes. It already has built-in, ready-to-go systems for managing commit history, automatically tracking datum status, and it can reliably restart a process and keep the state of the overall pipeline consistent. And, of course, everyone has experienced an inability to roll back or snapshot data and Pachyderm just makes that easy and automatic. You don't have to think about it. It just does it right out of the box.
Also supporting the Automated Mapping team’s decision was Pachyderm’s ability to handle petabytes of machine learning workloads, which made it well-suited for the next-generation processes Woven Planet’s Automated Mapping team is pioneering.
Woven Planet's Automated Mapping team uses Pachyderm for a range of processes that augment their ingestion-to-training pipeline. With data sourced from an array of satellite providers and databases, the Automated Mapping team adopts a unique approach to image recognition that divides images into various geolocations and sections. Pachyderm pipelines complement this approach, offering hundreds of pipelines that allow for parallel processing of different road features. This allows the Automated Mapping team to extract semantic information (i.e., what does a road feature mean for the vehicle) from the data far more efficiently than via a linear approach.
In addition to parallel processes, there is also a sequential coordination, with a waterfall-like cascade of different threads continually transforming the work of the pipelines that came before it. As one pipeline completes its work, processing different sections of the imagery at the same time, the results can be collated from their distributed workers before passing onto the next stage for new transformations. Combined with Kubernetes' tremendous scalability at the worker level, the Automated Mapping team’s use of Pachyderm speeds up time-to-value, helping Woven Planet deliver maps that have continental-scale coverage and always-on accuracy.
Lastly, the Woven Planet engineers have enjoyed working with the Pachyderm engineering and support teams, giving credence to the truism that a powerful platform is great but it’s only as good as the support that comes with it.
The Pachyderm team has been so receptive and it's been great to have constant communication with the team, running new ideas by them. They've been infinitely helpful at anything and everything, whether that is bug fixing or ideation or anything we need.
Learn how companies around the world are using Pachyderm to automate complex pipelines at scale.
Request a Demo