# PipedreamHQ/pipedream

> Source of Pipedream's 3,000+ app integrations (actions, triggers) plus the Connect SDKs that run them in Pipedream's cloud.

- Category: [API layer & connectors](https://llms-technical-reviews.com/connectors/)
- Repository: https://github.com/PipedreamHQ/pipedream (reviewed at commit `19a02750f24ccff09da19d27ac0b6bb6b8478bfc`, 2026-10-05)
- Stars: 11721 · Language: JavaScript · License: Pipedream Source Available License
- Canonical page: https://llms-technical-reviews.com/p/pipedream/

## Overview

This repository is the public half of Pipedream. It is not a self-hostable integration platform. It holds the *component registry source*: one directory per app under `components/` (about 3,300 app files, roughly 11,800 actions and 4,600 sources, not counting shared `common/` modules), and the small libraries those components import. It also holds the Pipedream Connect client SDKs that let your own product run those components for your end users. The workflow engine, the OAuth and credential vault, the webhook ingress, the trigger scheduler and the MCP server that exposes actions to agents all run in Pipedream's cloud and are not in this repository.

For an agent builder, the useful mental model is a large, human-reviewed catalogue of typed API wrappers. Each wrapper is a plain JavaScript object with `props`, an async `run({ $ })`, and, for actions, MCP-style `annotations`. Pipedream's cloud fills `this.$auth` with the connected account's credentials and runs the code. Connect's REST API (`/actions/run`, `/triggers/deploy`, `/proxy/...`) lets you invoke the same components on behalf of an `external_user_id`.

The code is published under the Pipedream Source Available License, not an OSI licence. It allows use and contribution, but it forbids offering a competing hosted service. The root `package.json` still says `MIT`, which is misleading.

## Architecture

```mermaid
flowchart LR
  APP["Your app backend"] -->|"BackendClient"| API["Connect REST API (cloud)"]
  FE["Your frontend"] -->|"BrowserClient iframe"| AUTH["Connect auth UI (cloud)"]
  AG["AI agent"] -->|"MCP (cloud)"| API
  API --> RT["Pipedream runtime (cloud)"]
  RT --> C["Component: app + action/source"]
  C --> PL["@pipedream/platform axios"]
  PL --> EXT["Third-party API"]
  GH["GitHub repo components/"] -->|"pd publish (CI)"| REG["Component registry (cloud)"]
  REG --> RT
```

| Component | Path | Role |
|---|---|---|
| App definitions | `components/<app>/<app>.app.mjs` | Shared auth access (`this.$auth`), `propDefinitions` with async option loaders, HTTP helper methods |
| Actions | `components/<app>/actions/*/*.mjs` | One operation each: `key`, `version`, `props`, `annotations`, `run({ $ })` |
| Sources | `components/<app>/sources/*/*.mjs` | Triggers: `$.interface.http` or `$.interface.timer`, `hooks`, `dedupe`, `$emit` |
| Platform library | `platform/lib/` | `@pipedream/platform`: axios wrapper, `ConfigurationError`, file streams, SQL props, `$.send` schemas |
| Types | `types/src/index.ts` | `@pipedream/types`: `defineApp` / `defineAction` / `defineSource` typings for TS components |
| Connect SDK | `packages/sdk/src/` | `BackendClient` (server), `BrowserClient` (frontend), shared REST methods |
| Connect React | `packages/connect-react/src/` | `ComponentForm` and controls that render a component's props for end users |
| CI and scripts | `.github/workflows/`, `scripts/` | Version checks, `pd publish` to the registry, app-file upload to Supabase, MCP annotation backfill |

## How a request flows

Take a product that lets its users create GitHub issues through Connect:

1. **Server credentials.** Your backend creates a `BackendClient` with a Connect project id, an environment (`development` or `production`, enforced) and OAuth client credentials ([server/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/index.ts#L213-L235)). `ensureValidOauthAccessToken` runs a client-credentials grant against `https://<apiHost>/v1/oauth/token`. It caches the token, refreshes it when less than one second remains, and makes up to three attempts ([server/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/index.ts#L281-L324)).
2. **Connect token.** `createConnectToken({ external_user_id })` POSTs to `/connect/<project>/tokens` ([server/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/index.ts#L347-L358)). Every Connect call goes through `makeConnectRequest`, which prefixes the project id ([shared/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/shared/index.ts#L1159-L1169)).
3. **Account connection.** In the browser, `connectAccount` opens a full-screen iframe on `pipedream.com/_static/connect.html?token=...&app=github`. It listens for `success`, `error` and `close` messages and returns an `authProvisionId` (`apn_...`) ([browser/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/browser/index.ts#L255-L292), [L313-L343](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/browser/index.ts#L313-L343)). The OAuth dance and token storage happen on Pipedream's side.
4. **Prop configuration.** For dynamic props, `configureComponent` asks the cloud to execute the prop's `options()` loader with the user's account and returns options and a `prevContext` for paging ([shared/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/shared/index.ts#L1386-L1411)). `connect-react` builds its forms on this call.
5. **Run.** `runAction({ externalUserId, actionId: "github-create-issue", configuredProps })` POSTs to `/actions/run` ([shared/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/shared/index.ts#L1505-L1527)).
6. **Component execution (cloud).** The runtime loads the published `github-create-issue` and injects the account into `this.github.$auth`. It then calls `run({ $ })`, which resolves the repo, calls `github.createIssue` and exports a `$summary` ([create-issue.mjs](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/actions/create-issue/create-issue.mjs#L53-L71)). The app file reads `this.$auth.oauth_access_token` and calls GitHub through Octokit or the platform axios wrapper ([github.app.mjs](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/github.app.mjs#L398-L438)).
7. **Errors.** On an HTTP error, the platform wrapper strips the request config, the request object and the stack trace from the AxiosError, so tokens do not leak into logs. It exports a `{status, statusText, headers, data}` summary to the step's `debug` key ([axios.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/platform/lib/axios.ts#L120-L160), [L181-L205](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/platform/lib/axios.ts#L181-L205)).

For anything without a pre-built action, `makeProxyRequest` base64url-encodes the target URL, re-prefixes headers as `x-pd-proxy-*`, and lets the cloud attach the user's credentials ([server/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/index.ts#L467-L490)).

## Key components

### The component contract

`@pipedream/types` is the clearest specification of what the runtime expects. An action has `key`, `version`, `type: "action"`, optional `ai: "optimized"`, `annotations` (`destructiveHint`, `idempotentHint`, `openWorldHint`, `readOnlyHint`), `props`, optional `additionalProps`, and `run` ([types/src/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/types/src/index.ts#L379-L401)). A source adds `hooks` (`deploy`, `activate`, `deactivate`) and a `dedupe` strategy of `last`, `greatest` or `unique`, and it emits with `$emit(event, { id, summary, ts })` ([types/src/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/types/src/index.ts#L275-L279), [L334-L351](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/types/src/index.ts#L334-L351)). Most components are plain `.mjs` with no type checking. The review guidelines and CI checks carry the weight.

### Sources and triggers

Webhook sources declare `http: "$.interface.http"` and `db: "$.service.db"`. In `activate`, they register `this.http.endpoint` with the provider and store the hook id in `db`. `deactivate` removes the hook ([common-webhook-orgs.mjs](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/sources/common/common-webhook-orgs.mjs#L54-L80)). Polling sources declare a `$.interface.timer` (15 minutes by default) and keep a cursor in `db` ([common-poling.mjs](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/crowdstrike_falcon/sources/common/common-poling.mjs#L1-L31)). Scheduling, webhook ingress and dedupe enforcement all live in the cloud. Through Connect, `deployTrigger` deploys a source for a user and forwards its events to your `webhookUrl` or a workflow ([shared/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/shared/index.ts#L1559-L1582)).

### `@pipedream/platform`

This is a small runtime library. It provides `axios($, config)` with debug exports and error sanitising. It handles OAuth 1.0a by POSTing the request to an external `oauthSignerUri`, which returns the `Authorization` header ([axios.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/platform/lib/axios.ts#L106-L117)). It also defines `ConfigurationError` for user-fixable failures, file-stream helpers, SQL prop helpers, and io-ts schemas for the `$.send` destinations (http, email, s3, sql, snowflake, sse, emit) ([index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/platform/lib/index.ts#L148-L167)).

### Agent surface

Agents reach components through Pipedream's hosted MCP server, which is not in this repository. What the repository contributes is metadata that makes the components usable as tools. The `annotations` block is the MCP hint set. `ai: "optimized"` marks actions rewritten for agents, with explicit descriptions and static, "MCP-friendly" prop variants such as `repoFullnameStatic` that avoid dropdown round trips. `scripts/tool-annotations/apply-annotations.js` backfilled annotations across the catalogue from CSV exports and bumped each patch version ([README.md](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/scripts/tool-annotations/README.md#L1-L26)).

## Extending it

- **New integration.** Add `components/<app>/` with an app file, actions and sources that follow the `.github/pipedream-*-guidelines.md` conventions, bump versions, and open a PR. On merge, CI publishes each changed action or source with `pd publish` and marks it as a registry component through a GraphQL mutation. App files are excluded from that path ([publish-components.yaml](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/workflows/publish-components.yaml#L53-L103)). App files are uploaded separately to a Supabase table.
- **Private components.** The same file format can be deployed to your own Pipedream workspace with the `pd` CLI, which is downloaded as a binary and is not in this repository.
- **Connect integration.** Use `@pipedream/sdk` for tokens, accounts, actions, triggers and the proxy. Use `@pipedream/connect-react` for embeddable prop forms. `packages/sdk/src/server/cli.ts` is a small `commander`-based Connect CLI driven by environment variables ([cli.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/cli.ts#L1-L28)).

## Running it

You cannot run it alone. Nothing in the repository executes a component outside Pipedream's cloud: there is no runtime, no `$auth` resolver, no scheduler and no HTTP ingress. Practically, you need a Pipedream account, and for Connect a project with OAuth client credentials. Then:

- `npm i @pipedream/sdk` on the server (Node 18+), and `createBackendClient({ projectId, environment, credentials })`. The default hosts are `api.pipedream.com` and `m.pipedream.net` for workflow invocation ([shared/index.ts](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/shared/index.ts#L1002-L1012)).
- `createFrontendClient({ tokenCallback, externalUserId })` in the browser for account connection.
- The repository itself is a pnpm workspace. Contributors lint and test components locally, and CI does the publishing.

## Strengths and caveats

- **Strength: breadth with review.** Thousands of apps with consistent structure, semver per component, CI version-floor checks, and agent-oriented descriptions and annotations on almost every action.
- **Strength: auth is solved for you.** Connect gives you hosted OAuth for each end user across the whole catalogue. Your code only handles a short-lived Connect token and `authProvisionId`s.
- **Strength: escape hatch.** The authenticated proxy covers endpoints that no action wraps, without your code ever touching the user's token.
- **Caveat: the platform is closed.** Execution, credential storage, retries, rate limiting, sandboxing and the MCP server are proprietary services. This repository cannot be self-hosted, and its licence forbids building a competing service from it.
- **Caveat: quality varies by component.** Retries, pagination and rate-limit handling are written per app. Some apps use `async-retry` or custom back-off, and many do none. The platform wrapper itself never retries.
- **Caveat: light typing.** Most components are untyped `.mjs`. Correctness rests on guidelines, review and CI lint rather than the type system.
- **Caveat: SDK drift.** The in-repo `@pipedream/sdk` is a hand-written 1.x client. Several methods have deprecated aliases (`actionRun`, `triggerDeploy`, `componentConfigure`), so check which API generation your docs refer to.

*Sources: code at 19a0275, deepwiki-open wiki (11 pages), OpenDeepWiki wiki (13 pages), verified Q&A.*

## How PipedreamHQ/pipedream answers the API layer & connectors questions

### How is third-party authentication implemented? (answered)

**OAuth2 flow.** App components read tokens from a runtime-injected `this.$auth` object, e.g. `this.$auth.oauth_access_token` (GitHub). The cloud platform executes the OAuth2 authorization-code grant; the open-source repo does not contain the redirect endpoint or token-exchange handler. Slack (v2) additionally accesses `oauth_uid`, `bot_token`, and a tenant-specific `base_url`. The `AppAuthType` enum in the SDK defines three categories: `OAuth`, `Keys`, `None`. OAuth1 requests are signed via the platform axios helper which POSTs to an external `oauthSignerUri` for the signature.

**Credential storage and encryption.** The credential vault is proprietary (pipedream.com cloud). The `push-registry-app-files-supabase` GitHub Action uploads only the `.app.mjs` files to Supabase; tokens and secrets never appear in the repo.

**Token refresh.** The server-side SDK manages OAuth client-credentials grants: `ensureValidOauthAccessToken()` maintains a cached access token with expiry checking and retry logic against `api.pipedream.com/v1/oauth/token`.

**Multi-tenant connected accounts (Pipedream Connect).** The SDK exposes `createConnectToken()` to mint time-bound tokens for end-users. The browser SDK provides `startConnect()` for the OAuth UI flow. `getAccountById()` retrieves connected accounts optionally including credentials. Apps are looked up by `name_slug` from the registry; each app record carries its `auth_type`.

> **Editor's note.** Correction: `ensureValidOauthAccessToken()` refreshes only the Connect SDK's own client-credentials token for api.pipedream.com; refreshing end users' third-party OAuth tokens happens in Pipedream's cloud and is not in this repository.

Citations: [components/github/github.app.mjs:402-404](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/github.app.mjs#L402-L404) · [components/slack_v2/slack_v2.app.mjs:266-277](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/slack_v2/slack_v2.app.mjs#L266-L277) · [packages/sdk/src/shared/index.ts:78-101](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/shared/index.ts#L78-L101) · [platform/lib/axios.ts:106-117](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/platform/lib/axios.ts#L106-L117) · [packages/sdk/src/server/index.ts:281-323](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/index.ts#L281-L323) · [packages/sdk/src/server/index.ts:347-358](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/index.ts#L347-L358)

### How is an integration / connector defined? (answered)

**Component format.** Every integration is a directory under `components/{app-slug}/` containing an app file (`{app}.app.mjs`), action files (`actions/*.mjs`), source files (`sources/*.mjs`), a `package.json`, and shared helpers in `common/`. The repo has 3,402 integration directories (one per app slug).

**App file.** The app file exports an object with `type: "app"`, `app: "slug"`, `propDefinitions` (reusable dropdown-loading props), and `methods` (shared HTTP helpers, pagination wrappers, webhook lifecycle). It is the single source of shared authentication and API-call logic for all actions and sources in that integration.

**Action file.** Each action exports `key` (globally unique, e.g. `github-create-issue`), `name`, `description`, `version` (semver), `type: "action"`, optional `annotations` (MCP hints), `ai: "optimized"`, `props`, and an async `run({ $ })` function. The `$` parameter provides runtime context (`$.export()`, `$.summary`, `$.service.db`).

**Versioning.** Components follow strict semver. A dedicated CI workflow (`component-registry-version-check.yaml`) verifies that every changed component's version exceeds the registry floor, failing closed if the version is insufficient. New components start at `0.0.1`.

**Publishing.** On push to master, the `publish-components.yaml` workflow uses a `pd publish` CLI to deploy changed `.mjs` action/source files to the Pipedream registry at `api.pipedream.com/v1/components/registry`. App files are separately pushed to a Supabase `registry_app_files` table. TypeScript components are compiled with esbuild before publishing. There are 12,073 action and 5,589 source `.mjs` files in the repository.


Citations: [components/github/actions/create-issue/create-issue.mjs:1-72](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/actions/create-issue/create-issue.mjs#L1-L72) · [.github/workflows/component-registry-version-check.yaml:1-241](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/workflows/component-registry-version-check.yaml#L1-L241) · [.github/workflows/publish-components.yaml:52-93](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/workflows/publish-components.yaml#L52-L93) · [scripts/upload-app-files-to-supabase.mjs:17-60](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/scripts/upload-app-files-to-supabase.mjs#L17-L60) · [.github/pipedream-component-guidelines.md:73-101](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/pipedream-component-guidelines.md#L73-L101)

### How are integrations exposed to LLM agents? (answered)

**MCP annotations in action components.** Every action can declare an `annotations` block containing `destructiveHint`, `openWorldHint`, and `readOnlyHint` booleans. These are MCP (Model Context Protocol) tool annotations that tell AI agents about the action's side effects (e.g. `destructiveHint: true` for delete operations, `readOnlyHint: true` for queries). Every GitHub action in the repository carries these annotations. The component guidelines mandate them.

**Annotation registry CSV files.** The repo ships two CSV files — one for Claude (`registry-actions-claude-2025-09-29.csv`, 6,709 entries) and one for ChatGPT (`registry-action-chatgpt-2025-09-25.csv`, 7,195 entries) — that map every published action's `key` to its `NAME`, `DESCRIPTION`, and all three MCP annotation values. These CSV files are consumed by `scripts/tool-annotations/apply-annotations.js`, which reads each entry, finds the corresponding action file, and injects/updates the `annotations` object.

**Pipedream Connect as the MCP gateway.** Actions are exposed as MCP tools via Pipedream Connect. The component guidelines state explicitly: "When exposed via MCP (Pipedream Connect), they appear as AI agent tools — the component's `description` and prop `description` fields become the agent's only documentation for how to call that tool correctly." The cloud-side Connect API serves the action registry as MCP-compatible tool definitions, with the annotations signalling capability constraints.

**Static prop variants (MCP optimization).** Several components define static, no-dropdown prop variants (e.g. `repoFullnameStatic`, `projectNumberStatic`) specifically so that AI-optimized actions resolve in a single tool call instead of requiring the multi-step configure-component/retrieve-options round-trip that dynamic dropdowns would need. This is documented in the prop descriptions as "MCP-friendly."


Citations: [components/github/actions/create-issue/create-issue.mjs:8-14](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/actions/create-issue/create-issue.mjs#L8-L14) · [scripts/tool-annotations/README.md:1-26](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/scripts/tool-annotations/README.md#L1-L26) · [scripts/tool-annotations/registry-actions-claude-2025-09-29.csv:1-5](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/scripts/tool-annotations/registry-actions-claude-2025-09-29.csv#L1-L5) · [.github/pipedream-component-guidelines.md:16-26](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/pipedream-component-guidelines.md#L16-L26) · [components/github/github.app.mjs:42-76](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/github.app.mjs#L42-L76)

### How is a tool call executed? (answered)

**Proxy vs direct.** Actions execute via the Pipedream cloud workflow engine — the open-source repository contains only the component definitions, not the runtime. Within a `run()` function, components call the external API using either the `@pipedream/platform` axios wrapper or a service-specific SDK (e.g. `@octokit/core` for GitHub, `@slack/web-api` for Slack, `@googleapis/sheets` for Google Sheets). The platform axios wrapper handles standard HTTP, OAuth1 signing, and error formatting; it injects auth from `this.$auth` but the cloud runtime resolves and populates that object before invoking the component.

**Sandboxing and rate limiting.** Sandboxing is handled by the cloud runtime (not in this repo). The `platform/lib/axios.ts` module converts Axios errors into structured summaries (`status`, `statusText`, `headers`, `data`) stripped of request config for safe exposure to users. Some components implement their own retry logic — e.g. Google Drive handles `RETRYABLE_STATUS_CODES` and `RATE_LIMIT_ERROR_REASONS` explicitly, and Slack uses `async-retry`.

**Pagination.** Pagination is handled per-component: the `propDefinitions` system supports cursor-based pagination via `prevContext` / `nextPageToken` patterns. Many action methods accept `page`/`per_page` parameters. The GitHub component uses `@octokit/plugin-paginate-rest` for REST API pagination and manual GraphQL cursor pagination.

**Error mapping.** The `ConfigurationError` from `@pipedream/platform` is used across components to surface configuration-level mistakes (e.g. missing permissions) to the user. Generic HTTP errors are returned as AxiosError summaries with context exported to the `debug` key.


Citations: [platform/lib/axios.ts:120-160](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/platform/lib/axios.ts#L120-L160) · [platform/lib/axios.ts:181-199](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/platform/lib/axios.ts#L181-L199) · [components/github/github.app.mjs:405-437](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/github.app.mjs#L405-L437) · [.github/pipedream-action-guidelines.md:39-49](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/pipedream-action-guidelines.md#L39-L49)

### How are data sync, webhooks and triggers implemented? (answered)

**Three delivery patterns.** Sources (triggers) use one of three patterns: polling, webhook, or hybrid. Each source component exports `type: "source"`, a `dedupe` strategy (standard: `"unique"`), and emits events via `this.$emit(data, { id, summary, ts })`.

**Polling (timer-based sync).** A polling source declares a `$.interface.timer` prop with a default `intervalSeconds` (commonly 900, the `DEFAULT_POLLING_SOURCE_TIMER_INTERVAL`). Persistent cursor state is kept in `$.service.db` — the common pattern encapsulates `_getLastTimestamp()` and `_setLastTimestamp()` in private methods. On each timer fire, the `run()` method fetches new items since the stored cursor and emits each via `$emit()`. The CrowdStrike Falcon common polling module illustrates this.

**Webhook ingestion.** A webhook source declares `$.interface.http` (which provides `this.http.endpoint`) and implements `activate()`/`deactivate()` hooks. `activate()` registers the webhook with the third-party service (e.g. `github.createOrgWebhook()`). `deactivate()` removes it. Incoming payloads are received by `run({ body, headers })` then passed to `processEvent()` which calls `this.$emit()`. The GitHub "New or Updated Issue" source shows the full hybrid pattern with sample events for both webhook and timer testing.

**Hybrid sources.** Many GitHub sources support both push (webhook) and timer (polling) via a shared base common module. The component detects which trigger mechanism fired at runtime.

**Deduplication.** The `dedupe: "unique"` strategy ensures events with duplicate `id` values are silently dropped, even across source restarts. The `id` must be a stable, globally-unique identifier for the real-world event. The component guidelines warn against using `Date.now()` or `Math.random()` as IDs. State in `$.service.db` survives across runs, providing incremental cursor persistence for polling sources.


Citations: [.github/pipedream-source-guidelines.md:8-64](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/pipedream-source-guidelines.md#L8-L64) · [.github/pipedream-source-guidelines.md:88-142](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/pipedream-source-guidelines.md#L88-L142) · [components/github/sources/common/common-webhook-orgs.mjs:54-80](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/sources/common/common-webhook-orgs.mjs#L54-L80) · [components/crowdstrike_falcon/sources/common/common-poling.mjs:1-31](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/crowdstrike_falcon/sources/common/common-poling.mjs#L1-L31) · [components/github/sources/new-or-updated-issue/new-or-updated-issue.mjs:1-50](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/components/github/sources/new-or-updated-issue/new-or-updated-issue.mjs#L1-L50)

### How is it self-hosted and what is open vs proprietary? (answered)

**Licence.** This repository is licensed under the **Pipedream Source Available License Version 1.0**, not GPL. The license permits use, modification, and distribution for non-commercial purposes but explicitly excludes "any commercial use of the software including, but not limited to, making available any software-as-a-service, platform-as-a-service, infrastructure-as-a-service or other online service that competes with the Software or any other Pipedream products or services." This is a classic source-available / business-source license that protects the hosted cloud business while allowing community use and contribution.

**What's in the repo (open / source-available).** The repository contains: all 3,402 integration component definitions (`.app.mjs` + actions + sources), the `@pipedream/platform` library (axios wrapper, runtime types, SQL helpers, file-stream utilities), the SDK packages (`packages/sdk/` — browser and server clients for Pipedream Connect, `packages/connect-react/` for UI components, `packages/prompts/` for prompt scaffolding), the TypeScript helpers and types packages, all CI/CD GitHub Actions workflows, and the build/annotation/upload scripts.

**What's proprietary / requires the hosted cloud.** The actual **workflow execution engine**, **credential vault** (OAuth token storage and encryption), **webhook HTTP endpoint** that receives incoming events, **agent/MCP runtime** that exposes actions as LLM tools, **admin dashboard UI** at pipedream.com, **user management** (teams, billing, usage tiers), and the **`pd publish` CLI** that connects to the registry are all proprietary cloud services. The `.github/publish-components.yaml` workflow reveals the registry at `api.pipedream.com/v1/components/registry` — this endpoint is not in the repo. The `pd` CLI referenced in the publish workflow is downloaded from `cli.pipedream.com`.

**Required services to self-host the components.** The components alone cannot be run standalone. They depend on the cloud runtime to resolve `this.$auth`, inject `$.service.db` and `$.interface.http`, call `$emit()`, run the workflow DAG, and schedule timer-based sources. The SDK documentation references `api.pipedream.com` as the default API host, and the Connect flow requires a project environment (`development` or `production`) tied to pipedream.com infrastructure.


Citations: [LICENSE:1-15](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/LICENSE#L1-L15) · [.github/workflows/publish-components.yaml:52-106](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/.github/workflows/publish-components.yaml#L52-L106) · [packages/sdk/src/server/index.ts:226-258](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/index.ts#L226-L258) · [packages/sdk/src/server/index.ts:29-42](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/server/index.ts#L29-L42) · [packages/sdk/src/shared/index.ts:39-53](https://github.com/PipedreamHQ/pipedream/blob/19a02750f24ccff09da19d27ac0b6bb6b8478bfc/packages/sdk/src/shared/index.ts#L39-L53)