Create scalable and reliable data pipelines easily using Pachyderm Key Features * Learn how to build an enterprise-level reproducible data science platform with Pachyderm * Deploy Pachyderm on cloud platforms such as AWS EKS, Google Kubernetes Engine, and Microsoft Azure Kubernetes Service * Integrate Pachyderm with other data science tools, such as Jupyter Notebook and Kubeflow Book Description Pachyderm is an open-source project that enables data scientists to run reproducible data pipelines and scale them to an enterprise level. This book will teach you how to implement Pachyderm to create collaborative data science workflows and reproduce your ML experiments at scale. You'll begin by understanding the importance of data reproducibility and comparing different data science platforms. Next, you'll explore how Pachyderm fits into the picture and its significance, followed by understanding instructions for installing Pachyderm locally on your computer or on a cloud platform of your choice. You'll then discover the architectural components and main pipeline principles and concepts of Pachyderm. The book will demonstrate how to create your first data pipeline using Pachyderm and its components. You'll further explore common operations with data in Pachyderm, such as uploading data to and from Pachyderm to create complex pipelines. Based on what you have learned, you will then develop an end-to-end ML workflow, before understanding the hyperparameter tuning technique and the different supported Pachyderm language clients along with examples. Finally, you'll integrate Pachyderm with JupyterHub and Kubeflow. By the end of this book, you'll be able to create repositories and pipelines, and implement Pachyderm in your infrastructure. What you will learn * Understand the importance of reproducible data science for enterprise * Explore the basics of Pachyderm, such as commits, branches, and more * Create a Pachyderm repository and upload data into it * Implement common pipeline operations in Pachyderm * Create a real-life example of hyperparameter tuning in Pachyderm * Combine Pachyderm with Pachyderm language clients in Python and Go Who This Book Is For This book is for data scientists and machine learning engineers who want to build scalable infrastructures for their data science projects. Basic knowledge of Python programming and Kubernetes will be beneficial. Table of Content The Problem of Data Reproducibility * Pachyderm Basics * Pachyderm Pipeline Specification * Installing Pachyderm Locally * Installing Pachyderm on a Cloud Platform * Creating Your First Pipeline * Pachyderm Operations * Creating an End-to-End Machine Learning Workflow * Distributed Hyperparameter Tuning with Pachyderm * Pachyderm Language Clients * Integrating with JupyterHub * Integrating with Kubeflow

Book Information
ISBN 9781801074483
Author Svetlana Karslioglu
Format Paperback
Page Count 348
Imprint Packt Publishing Limited
Publisher Packt Publishing Limited
Weight(grams) 75g