Pachyderm has been acquired by Hewlett Packard Enterprise - Learn more
Introducing Pachyderm 2.8
Bhavani Rao
Share
We are excited to announce the general availability of Pachyderm 2.8! This release includes a multitude of exciting features and enhancements including Global Pipeline Configurations, a more convenient Console experience, and IntelliSense support for Pipelines. This release focuses on bringing our users a powerful, more intuitive, and streamlined experience across the whole Pachyderm platform.
Global Pipeline Configuration
Users are now able to efficiently bulk create and manage a massive scale of Pachyderm pipelines with the new Global Pipeline Configuration feature. This new feature introduces the concept of Cluster default configurations. When users set default parameters on the Cluster default configuration, pipelines inherit these parameters. Pipeline creation and updates are streamlined, allowing users to iterate faster and reduce risk.
Scale Pachyderm pipelines with new Global Pipeline Configurations
Set default parameters for pipelines on the cluster level
Pipelines can inherit these parameters, saving time and effort
Users can create a base pipeline configuration using Global Pipeline Configuration, allowing for better organization, maintenance, and consistency when creating and updating pipelines. This also unlocks rapid pipeline deployment as users can quickly create pipelines. Cluster default configuration reduces the complexity for new users, who can quickly onboard, and for existing users, who can better manage the growing number of pipelines.
Consistent, organized, and easily-maintained pipelines with Global Pipeline Configurations
Quickly deploy and iterate new Pipelines using Cluster default configuration
Simplify Pipeline creation for new users getting started and existing users managing Pipelines at scale
Find out more about Global cluster defaults in our docs.
How to leverage Global Pipeline Configuration to set Cluster default configuration
Do Even More in the Web Interface
Users can now do more than ever without having to leave Console in your web browser. Create and update pipelines in Console, using VS Code IntelliSense to help you write your code. Users can also configure cluster-level pipeline defaults to apply common settings to all your pipelines. Teams can see how projects are connected in the DAG viewer, and re-run jobs with a single click. Users can filter jobs and commits by global ID and see the details in the sidebar. And you can preview more file types in Console, including images, videos, audio, programming languages, markdown, HTML, XML, text, and CSV.
Create and update pipelines in Console, powered by VS Code IntelliSense
Configure cluster-level pipeline defaults using Global Pipeline Configuration
Connections between projects are now visible in the DAG viewer
A new action is available to re-run jobs with a single click
Global ID filters now show relevant job and commit info in the sidebar
More file types can be previewed in Console
15 image formats
24 programming languages
17 video file extensions
10 audio formats
Markdown, HTML, XML, text, and CSV all have improved support
IntelliSense Support for Pipelines
Pachyderm 2.8 introduces IntelliSense support for Pipeline specifications, a feature that will boost user’s productivity and accuracy. With IntelliSense, users receive real-time auto-completion and feedback as they draft pipeline specifications, reducing the risk of manual input errors. Users can also use IntelliSense with VSCode, which supports both JSON and YAML formats for your pipeline specifications. IntelliSense is based on a JSON Schema file, which means that you can customize it to work with your preferred editors, linters, and other tools if you are familiar with JSON schema support. Finally, we have integrated IntelliSense into Console, so users can benefit from the streamlined specification process within the Console code editing interface when you create or update a pipeline.
Real-time auto-completion and suggestions, aiding users in swiftly defining pipelines while minimizing errors
Pachyderm’s IntelliSense is optimized to work seamlessly with both JSON and YAML formats
Integration with your preferred editors, linters, and other tools, based on a JSON Schema file
For detailed installation and usage instructions. More details are provided in our docs.
Suggestions and feedback for configurations in the Console powered by IntelliSense
Pachyderm SDK Tutorials and Examples
With the previous release of Pachyderm 2.7.0, we enthusiastically announced our new official Python client, the Pachyderm SDK.
If you are as excited as we are about the new Pachyderm SDK, we encourage you to check out some of our tutorials and examples. Learn how to create your first project here or walk through examples including Breast Cancer Detection, Distributed Image Processing, or Spout Pipelines. Feedback is always encouraged and welcome!
Summary
Pachyderm 2.8 offers users the most performant and intuitive version of Pachyderm to date. With these new features and enhancements, customers will experience greater ease of use and flexibility when working across the platform, from Pipelines to the Console. For a comprehensive list of features, read the release notes.
Are you new to Pachyderm and searching for a data pipelining or MLOps solution? Take the next step and schedule a demo tailored to your environment.