Overview

Prow is a Kubernetes-based Continuous Integration and Continuous Deployment (CI/CD) system. It provides automated testing, code review automation, and project management features for Kubernetes and other open-source projects. Prow was originally developed as part of the Kubernetes project and has since become a widely-used CI/CD platform.

Jobs can be triggered by various types of events and report their status to many different services. In addition to job execution, Prow provides GitHub automation in the form of policy enforcement, chat-ops via /foo style commands, and automatic PR merging.

See the GoDoc for library docs. Please note that these libraries are intended for use by prow only, and we do not make any attempt to preserve backwards compatibility.

For a brief overview of how Prow runs jobs take a look at “Life of a Prow Job”.

To see common Prow usage and interactions flow, see the pull request interactions sequence diagram.

What Problem Does It Solve?

Modern software development, especially in large open-source projects like Kubernetes, requires sophisticated CI/CD infrastructure to:

• Automate Testing: Run tests automatically on every pull request and commit
• Code Review Automation: Automate repetitive review tasks and enforce project policies
• Project Management: Automate issue management, labeling, and project workflows
• Scalability: Handle thousands of repositories and millions of test runs
• Integration: Integrate with GitHub, Gerrit, and other code hosting platforms
• Resource Management: Efficiently manage compute resources across multiple clusters

Prow solves these problems by providing a Kubernetes-native CI/CD platform that scales horizontally, integrates deeply with code hosting platforms, and offers extensive plugin-based automation.

Who Is It For?

Primary Users

1. Kubernetes SIG Testing: The team responsible for maintaining Kubernetes CI infrastructure
2. Open Source Project Maintainers: Teams managing large open-source projects requiring robust CI/CD
3. CI/CD Engineers: Engineers building and maintaining CI/CD pipelines
4. Developers: Contributors who interact with Prow through GitHub/Gerrit

Skill Level Requirements

• Basic Users: Need to understand YAML configuration and basic CI/CD concepts
• Advanced Users: Should be familiar with Kubernetes, Go programming, and CI/CD best practices
• Contributors: Need strong Go skills, Kubernetes API knowledge, and understanding of distributed systems

Key Features

1. Job Execution

Prow executes CI jobs as Kubernetes Pods:

• Presubmit Jobs: Run on pull requests before merge
• Postsubmit Jobs: Run after code is merged
• Periodic Jobs: Run on a schedule (cron-based)
• Batch Jobs: Run multiple jobs in parallel

2. Webhook Handling

Prow’s hook component processes GitHub/Gerrit webhooks:

• Validates webhook signatures
• Triggers appropriate jobs based on events
• Executes plugins based on webhook content
• Manages job lifecycle

3. Plugin System

Extensible plugin architecture for automation:

• Label Management: Automatically label PRs and issues
• Milestone Management: Track project milestones
• Approval Automation: /approve, /lgtm commands
• Test Automation: /test, /retest commands
• Issue Management: Auto-close, auto-assign, etc.
• Custom Plugins: Write your own plugins

4. Tide - Automated Merging

Tide automatically merges pull requests when:

• All required tests pass
• Code is approved by required reviewers
• Branch protection rules are satisfied
• No merge conflicts exist

5. Deck - Web UI

Deck provides a web interface for:

• Viewing job status and history
• Browsing test results and logs
• Managing Prow configuration
• Viewing plugin help
• Spyglass integration for artifact viewing

6. Spyglass - Artifact Viewer

Spyglass provides a unified interface for viewing:

• Build logs
• Test results (JUnit XML)
• Coverage reports
• Custom artifacts

7. Status Reporting (Crier)

Crier reports job status back to:

• GitHub (commit status, PR comments)
• Gerrit (code review comments)
• Pub/Sub (for external integrations)
• Slack (notifications)

8. Resource Management

• Sinker: Cleans up old ProwJobs and Pods
• Scheduler: Distributes jobs across clusters
• Plank: Manages job execution and Pod creation

9. Multi-Cluster Support

Prow can distribute jobs across multiple Kubernetes clusters:

• Cluster selection based on job requirements
• Load balancing across clusters
• Cluster-specific configurations

10. Integration Support

• GitHub: Full integration with GitHub API
• Gerrit: Support for Gerrit code review
• Slack: Notifications and chat-ops
• Pub/Sub: Event streaming
• GCS: Artifact storage

Additional Functions and Features

• Job execution for testing, batch processing, artifact publishing.
  • GitHub events are used to trigger post-PR-merge (postsubmit) jobs and on-PR-update (presubmit) jobs.
  • Support for multiple execution platforms and source code review sites.
• Pluggable GitHub bot automation that implements /foo style commands and enforces configured policies/processes.
• GitHub merge automation with batch testing logic.
• Front end for viewing jobs, merge queue status, dynamically generated help information, and more.
• Automatic deployment of source control based config.
• Automatic GitHub org/repo administration configured in source control.
• Designed for multi-org scale with dozens of repositories. (The Kubernetes Prow instance uses only 1 GitHub bot token!)
• High availability as benefit of running on Kubernetes. (replication, load balancing, rolling updates…)
• JSON structured logs.
• Prometheus metrics.

Documentation

Getting started

• With your own Prow deployment: “Deploying Prow”
• With developing for Prow: “Developing and Contributing to Prow”
• As a job author: ProwJobs

More details

• Components

• Plugins

• ProwJobs

• Building, Testing, and Updating

• General Configuration

• Pod Utilities

• Scaling Prow

• Tide

• Metrics

• “Life of a Prow Job”

• Getting more out of Prow

Tests

The stability of prow is heavily relying on unit tests and integration tests.

• Unit tests are co-located with prow source code
• Integration tests utilizes kind with hermetic integration tests. See instructions for adding new integration tests for more details

Useful Talks

KubeCon 2020 EU virtual

• Going Beyond CI/CD with Prow

KubeCon 2018 EU

• Automation and the Kubernetes Contributor Experience
• SIG Testing Deep Dive

KubeCon 2018 China

• SIG Testing Intro

KubeCon 2018 Seattle

• Behind you PR: K8s with K8s on K8s
• Using Prow for Testing Outside of K8s
• Jenkins X (featuring Tide)
• SIG Testing Intro
• SIG Testing Deep Dive

Misc

• Deploy Your Own Kubernetes Prow

Prow in the wild

Prow is used by the following organizations and projects:

• Kubernetes
  • This includes kubernetes, kubernetes-client, kubernetes-csi, and kubernetes-sigs.
• OpenShift
  • This includes openshift, openshift-s2i, operator-framework, and some repos in containers and heketi.
• Istio
• Knative
• Jetstack
• Kyma
• Metal³
• Caicloud
• Kubeflow
• Azure AKS Engine
• tensorflow/minigo
• Daisy (Google Compute Image Tools)
• KubeEdge (Kubernetes Native Edge Computing Framework)
• Volcano (Kubernetes Native Batch System)
• Loodse
• Feast
• Falco
• TiDB
• Amazon EKS Distro and Amazon EKS Anywhere
• KubeSphere
• OpenYurt
• KubeVirt
• AWS Controllers for Kubernetes
• Gardener

Jenkins X uses Prow as part of Serverless Jenkins.

Contact us

If you need to contact the maintainers of Prow you have a few options:

1. Open an issue in the kubernetes-sigs/prow repo.
2. Reach out to the #prow channel of the Kubernetes Slack.
3. Contact one of the code owners in OWNERS or in a more specifically scoped OWNERS file.

Bots home

@k8s-ci-robot lives here and is the face of the Kubernetes Prow instance. Here is a command list for interacting with @k8s-ci-robot and other Prow bots.