PipedreamHQ/pipedream
Source of Pipedream's 3,000+ app integrations (actions, triggers) plus the Connect SDKs that run them in Pipedream's cloud.
Overview
This repository is the public half of Pipedream. It is not a self-hostable integration platform. It holds the component registry source: one directory per app under components/ (about 3,300 app files, roughly 11,800 actions and 4,600 sources, not counting shared common/ modules), and the small libraries those components import. It also holds the Pipedream Connect client SDKs that let your own product run those components for your end users. The workflow engine, the OAuth and credential vault, the webhook ingress, the trigger scheduler and the MCP server that exposes actions to agents all run in Pipedream’s cloud and are not in this repository.
For an agent builder, the useful mental model is a large, human-reviewed catalogue of typed API wrappers. Each wrapper is a plain JavaScript object with props, an async run({ $ }), and, for actions, MCP-style annotations. Pipedream’s cloud fills this.$auth with the connected account’s credentials and runs the code. Connect’s REST API (/actions/run, /triggers/deploy, /proxy/...) lets you invoke the same components on behalf of an external_user_id.
The code is published under the Pipedream Source Available License, not an OSI licence. It allows use and contribution, but it forbids offering a competing hosted service. The root package.json still says MIT, which is misleading.
Architecture
flowchart LR
APP["Your app backend"] -->|"BackendClient"| API["Connect REST API (cloud)"]
FE["Your frontend"] -->|"BrowserClient iframe"| AUTH["Connect auth UI (cloud)"]
AG["AI agent"] -->|"MCP (cloud)"| API
API --> RT["Pipedream runtime (cloud)"]
RT --> C["Component: app + action/source"]
C --> PL["@pipedream/platform axios"]
PL --> EXT["Third-party API"]
GH["GitHub repo components/"] -->|"pd publish (CI)"| REG["Component registry (cloud)"]
REG --> RT| Component | Path | Role |
|---|---|---|
| App definitions | components/<app>/<app>.app.mjs |
Shared auth access (this.$auth), propDefinitions with async option loaders, HTTP helper methods |
| Actions | components/<app>/actions/*/*.mjs |
One operation each: key, version, props, annotations, run({ $ }) |
| Sources | components/<app>/sources/*/*.mjs |
Triggers: $.interface.http or $.interface.timer, hooks, dedupe, $emit |
| Platform library | platform/lib/ |
@pipedream/platform: axios wrapper, ConfigurationError, file streams, SQL props, $.send schemas |
| Types | types/src/index.ts |
@pipedream/types: defineApp / defineAction / defineSource typings for TS components |
| Connect SDK | packages/sdk/src/ |
BackendClient (server), BrowserClient (frontend), shared REST methods |
| Connect React | packages/connect-react/src/ |
ComponentForm and controls that render a component’s props for end users |
| CI and scripts | .github/workflows/, scripts/ |
Version checks, pd publish to the registry, app-file upload to Supabase, MCP annotation backfill |
How a request flows
Take a product that lets its users create GitHub issues through Connect:
- Server credentials. Your backend creates a
BackendClientwith a Connect project id, an environment (developmentorproduction, enforced) and OAuth client credentials (server/index.ts).ensureValidOauthAccessTokenruns a client-credentials grant againsthttps://<apiHost>/v1/oauth/token. It caches the token, refreshes it when less than one second remains, and makes up to three attempts (server/index.ts). - Connect token.
createConnectToken({ external_user_id })POSTs to/connect/<project>/tokens(server/index.ts). Every Connect call goes throughmakeConnectRequest, which prefixes the project id (shared/index.ts). - Account connection. In the browser,
connectAccountopens a full-screen iframe onpipedream.com/_static/connect.html?token=...&app=github. It listens forsuccess,errorandclosemessages and returns anauthProvisionId(apn_...) (browser/index.ts, L313-L343). The OAuth dance and token storage happen on Pipedream’s side. - Prop configuration. For dynamic props,
configureComponentasks the cloud to execute the prop’soptions()loader with the user’s account and returns options and aprevContextfor paging (shared/index.ts).connect-reactbuilds its forms on this call. - Run.
runAction({ externalUserId, actionId: "github-create-issue", configuredProps })POSTs to/actions/run(shared/index.ts). - Component execution (cloud). The runtime loads the published
github-create-issueand injects the account intothis.github.$auth. It then callsrun({ $ }), which resolves the repo, callsgithub.createIssueand exports a$summary(create-issue.mjs). The app file readsthis.$auth.oauth_access_tokenand calls GitHub through Octokit or the platform axios wrapper (github.app.mjs). - Errors. On an HTTP error, the platform wrapper strips the request config, the request object and the stack trace from the AxiosError, so tokens do not leak into logs. It exports a
{status, statusText, headers, data}summary to the step’sdebugkey (axios.ts, L181-L205).
For anything without a pre-built action, makeProxyRequest base64url-encodes the target URL, re-prefixes headers as x-pd-proxy-*, and lets the cloud attach the user’s credentials (server/index.ts).
Key components
The component contract
@pipedream/types is the clearest specification of what the runtime expects. An action has key, version, type: "action", optional ai: "optimized", annotations (destructiveHint, idempotentHint, openWorldHint, readOnlyHint), props, optional additionalProps, and run (types/src/index.ts). A source adds hooks (deploy, activate, deactivate) and a dedupe strategy of last, greatest or unique, and it emits with $emit(event, { id, summary, ts }) (types/src/index.ts, L334-L351). Most components are plain .mjs with no type checking. The review guidelines and CI checks carry the weight.
Sources and triggers
Webhook sources declare http: "$.interface.http" and db: "$.service.db". In activate, they register this.http.endpoint with the provider and store the hook id in db. deactivate removes the hook (common-webhook-orgs.mjs). Polling sources declare a $.interface.timer (15 minutes by default) and keep a cursor in db (common-poling.mjs). Scheduling, webhook ingress and dedupe enforcement all live in the cloud. Through Connect, deployTrigger deploys a source for a user and forwards its events to your webhookUrl or a workflow (shared/index.ts).
@pipedream/platform
This is a small runtime library. It provides axios($, config) with debug exports and error sanitising. It handles OAuth 1.0a by POSTing the request to an external oauthSignerUri, which returns the Authorization header (axios.ts). It also defines ConfigurationError for user-fixable failures, file-stream helpers, SQL prop helpers, and io-ts schemas for the $.send destinations (http, email, s3, sql, snowflake, sse, emit) (index.ts).
Agent surface
Agents reach components through Pipedream’s hosted MCP server, which is not in this repository. What the repository contributes is metadata that makes the components usable as tools. The annotations block is the MCP hint set. ai: "optimized" marks actions rewritten for agents, with explicit descriptions and static, “MCP-friendly” prop variants such as repoFullnameStatic that avoid dropdown round trips. scripts/tool-annotations/apply-annotations.js backfilled annotations across the catalogue from CSV exports and bumped each patch version (README.md).
Extending it
- New integration. Add
components/<app>/with an app file, actions and sources that follow the.github/pipedream-*-guidelines.mdconventions, bump versions, and open a PR. On merge, CI publishes each changed action or source withpd publishand marks it as a registry component through a GraphQL mutation. App files are excluded from that path (publish-components.yaml). App files are uploaded separately to a Supabase table. - Private components. The same file format can be deployed to your own Pipedream workspace with the
pdCLI, which is downloaded as a binary and is not in this repository. - Connect integration. Use
@pipedream/sdkfor tokens, accounts, actions, triggers and the proxy. Use@pipedream/connect-reactfor embeddable prop forms.packages/sdk/src/server/cli.tsis a smallcommander-based Connect CLI driven by environment variables (cli.ts).
Running it
You cannot run it alone. Nothing in the repository executes a component outside Pipedream’s cloud: there is no runtime, no $auth resolver, no scheduler and no HTTP ingress. Practically, you need a Pipedream account, and for Connect a project with OAuth client credentials. Then:
npm i @pipedream/sdkon the server (Node 18+), andcreateBackendClient({ projectId, environment, credentials }). The default hosts areapi.pipedream.comandm.pipedream.netfor workflow invocation (shared/index.ts).createFrontendClient({ tokenCallback, externalUserId })in the browser for account connection.- The repository itself is a pnpm workspace. Contributors lint and test components locally, and CI does the publishing.
Strengths and caveats
- Strength: breadth with review. Thousands of apps with consistent structure, semver per component, CI version-floor checks, and agent-oriented descriptions and annotations on almost every action.
- Strength: auth is solved for you. Connect gives you hosted OAuth for each end user across the whole catalogue. Your code only handles a short-lived Connect token and
authProvisionIds. - Strength: escape hatch. The authenticated proxy covers endpoints that no action wraps, without your code ever touching the user’s token.
- Caveat: the platform is closed. Execution, credential storage, retries, rate limiting, sandboxing and the MCP server are proprietary services. This repository cannot be self-hosted, and its licence forbids building a competing service from it.
- Caveat: quality varies by component. Retries, pagination and rate-limit handling are written per app. Some apps use
async-retryor custom back-off, and many do none. The platform wrapper itself never retries. - Caveat: light typing. Most components are untyped
.mjs. Correctness rests on guidelines, review and CI lint rather than the type system. - Caveat: SDK drift. The in-repo
@pipedream/sdkis a hand-written 1.x client. Several methods have deprecated aliases (actionRun,triggerDeploy,componentConfigure), so check which API generation your docs refer to.
Sources: code at 19a0275, deepwiki-open wiki (11 pages), OpenDeepWiki wiki (13 pages), verified Q&A.
How it answers the API layer & connectors questions
Each answer was drafted by a code-reading agent at commit 19a0275. Its citations were checked mechanically. Compare with the other api layer & connectors →
How is third-party authentication implemented?
answeredOAuth2 flow. App components read tokens from a runtime-injected this.$auth object, e.g. this.$auth.oauth_access_token (GitHub). The cloud platform executes the OAuth2 authorization-code grant; the open-source repo does not contain the redirect endpoint or token-exchange handler. Slack (v2) additionally accesses oauth_uid, bot_token, and a tenant-specific base_url. The AppAuthType enum in the SDK defines three categories: OAuth, Keys, None. OAuth1 requests are signed via the platform axios helper which POSTs to an external oauthSignerUri for the signature.
Credential storage and encryption. The credential vault is proprietary (pipedream.com cloud). The push-registry-app-files-supabase GitHub Action uploads only the .app.mjs files to Supabase; tokens and secrets never appear in the repo.
Token refresh. The server-side SDK manages OAuth client-credentials grants: ensureValidOauthAccessToken() maintains a cached access token with expiry checking and retry logic against api.pipedream.com/v1/oauth/token.
Multi-tenant connected accounts (Pipedream Connect). The SDK exposes createConnectToken() to mint time-bound tokens for end-users. The browser SDK provides startConnect() for the OAuth UI flow. getAccountById() retrieves connected accounts optionally including credentials. Apps are looked up by name_slug from the registry; each app record carries its auth_type.
ensureValidOauthAccessToken() refreshes only the Connect SDK's own client-credentials token for api.pipedream.com; refreshing end users' third-party OAuth tokens happens in Pipedream's cloud and is not in this repository.How is an integration / connector defined?
answeredComponent format. Every integration is a directory under components/{app-slug}/ containing an app file ({app}.app.mjs), action files (actions/*.mjs), source files (sources/*.mjs), a package.json, and shared helpers in common/. The repo has 3,402 integration directories (one per app slug).
App file. The app file exports an object with type: "app", app: "slug", propDefinitions (reusable dropdown-loading props), and methods (shared HTTP helpers, pagination wrappers, webhook lifecycle). It is the single source of shared authentication and API-call logic for all actions and sources in that integration.
Action file. Each action exports key (globally unique, e.g. github-create-issue), name, description, version (semver), type: "action", optional annotations (MCP hints), ai: "optimized", props, and an async run({ $ }) function. The $ parameter provides runtime context ($.export(), $.summary, $.service.db).
Versioning. Components follow strict semver. A dedicated CI workflow (component-registry-version-check.yaml) verifies that every changed component's version exceeds the registry floor, failing closed if the version is insufficient. New components start at 0.0.1.
Publishing. On push to master, the publish-components.yaml workflow uses a pd publish CLI to deploy changed .mjs action/source files to the Pipedream registry at api.pipedream.com/v1/components/registry. App files are separately pushed to a Supabase registry_app_files table. TypeScript components are compiled with esbuild before publishing. There are 12,073 action and 5,589 source .mjs files in the repository.
How are integrations exposed to LLM agents?
answeredMCP annotations in action components. Every action can declare an annotations block containing destructiveHint, openWorldHint, and readOnlyHint booleans. These are MCP (Model Context Protocol) tool annotations that tell AI agents about the action's side effects (e.g. destructiveHint: true for delete operations, readOnlyHint: true for queries). Every GitHub action in the repository carries these annotations. The component guidelines mandate them.
Annotation registry CSV files. The repo ships two CSV files — one for Claude (registry-actions-claude-2025-09-29.csv, 6,709 entries) and one for ChatGPT (registry-action-chatgpt-2025-09-25.csv, 7,195 entries) — that map every published action's key to its NAME, DESCRIPTION, and all three MCP annotation values. These CSV files are consumed by scripts/tool-annotations/apply-annotations.js, which reads each entry, finds the corresponding action file, and injects/updates the annotations object.
Pipedream Connect as the MCP gateway. Actions are exposed as MCP tools via Pipedream Connect. The component guidelines state explicitly: "When exposed via MCP (Pipedream Connect), they appear as AI agent tools — the component's description and prop description fields become the agent's only documentation for how to call that tool correctly." The cloud-side Connect API serves the action registry as MCP-compatible tool definitions, with the annotations signalling capability constraints.
Static prop variants (MCP optimization). Several components define static, no-dropdown prop variants (e.g. repoFullnameStatic, projectNumberStatic) specifically so that AI-optimized actions resolve in a single tool call instead of requiring the multi-step configure-component/retrieve-options round-trip that dynamic dropdowns would need. This is documented in the prop descriptions as "MCP-friendly."
How is a tool call executed?
answeredProxy vs direct. Actions execute via the Pipedream cloud workflow engine — the open-source repository contains only the component definitions, not the runtime. Within a run() function, components call the external API using either the @pipedream/platform axios wrapper or a service-specific SDK (e.g. @octokit/core for GitHub, @slack/web-api for Slack, @googleapis/sheets for Google Sheets). The platform axios wrapper handles standard HTTP, OAuth1 signing, and error formatting; it injects auth from this.$auth but the cloud runtime resolves and populates that object before invoking the component.
Sandboxing and rate limiting. Sandboxing is handled by the cloud runtime (not in this repo). The platform/lib/axios.ts module converts Axios errors into structured summaries (status, statusText, headers, data) stripped of request config for safe exposure to users. Some components implement their own retry logic — e.g. Google Drive handles RETRYABLE_STATUS_CODES and RATE_LIMIT_ERROR_REASONS explicitly, and Slack uses async-retry.
Pagination. Pagination is handled per-component: the propDefinitions system supports cursor-based pagination via prevContext / nextPageToken patterns. Many action methods accept page/per_page parameters. The GitHub component uses @octokit/plugin-paginate-rest for REST API pagination and manual GraphQL cursor pagination.
Error mapping. The ConfigurationError from @pipedream/platform is used across components to surface configuration-level mistakes (e.g. missing permissions) to the user. Generic HTTP errors are returned as AxiosError summaries with context exported to the debug key.
How are data sync, webhooks and triggers implemented?
answeredThree delivery patterns. Sources (triggers) use one of three patterns: polling, webhook, or hybrid. Each source component exports type: "source", a dedupe strategy (standard: "unique"), and emits events via this.$emit(data, { id, summary, ts }).
Polling (timer-based sync). A polling source declares a $.interface.timer prop with a default intervalSeconds (commonly 900, the DEFAULT_POLLING_SOURCE_TIMER_INTERVAL). Persistent cursor state is kept in $.service.db — the common pattern encapsulates _getLastTimestamp() and _setLastTimestamp() in private methods. On each timer fire, the run() method fetches new items since the stored cursor and emits each via $emit(). The CrowdStrike Falcon common polling module illustrates this.
Webhook ingestion. A webhook source declares $.interface.http (which provides this.http.endpoint) and implements activate()/deactivate() hooks. activate() registers the webhook with the third-party service (e.g. github.createOrgWebhook()). deactivate() removes it. Incoming payloads are received by run({ body, headers }) then passed to processEvent() which calls this.$emit(). The GitHub "New or Updated Issue" source shows the full hybrid pattern with sample events for both webhook and timer testing.
Hybrid sources. Many GitHub sources support both push (webhook) and timer (polling) via a shared base common module. The component detects which trigger mechanism fired at runtime.
Deduplication. The dedupe: "unique" strategy ensures events with duplicate id values are silently dropped, even across source restarts. The id must be a stable, globally-unique identifier for the real-world event. The component guidelines warn against using Date.now() or Math.random() as IDs. State in $.service.db survives across runs, providing incremental cursor persistence for polling sources.
How is it self-hosted and what is open vs proprietary?
answeredLicence. This repository is licensed under the Pipedream Source Available License Version 1.0, not GPL. The license permits use, modification, and distribution for non-commercial purposes but explicitly excludes "any commercial use of the software including, but not limited to, making available any software-as-a-service, platform-as-a-service, infrastructure-as-a-service or other online service that competes with the Software or any other Pipedream products or services." This is a classic source-available / business-source license that protects the hosted cloud business while allowing community use and contribution.
What's in the repo (open / source-available). The repository contains: all 3,402 integration component definitions (.app.mjs + actions + sources), the @pipedream/platform library (axios wrapper, runtime types, SQL helpers, file-stream utilities), the SDK packages (packages/sdk/ — browser and server clients for Pipedream Connect, packages/connect-react/ for UI components, packages/prompts/ for prompt scaffolding), the TypeScript helpers and types packages, all CI/CD GitHub Actions workflows, and the build/annotation/upload scripts.
What's proprietary / requires the hosted cloud. The actual workflow execution engine, credential vault (OAuth token storage and encryption), webhook HTTP endpoint that receives incoming events, agent/MCP runtime that exposes actions as LLM tools, admin dashboard UI at pipedream.com, user management (teams, billing, usage tiers), and the pd publish CLI that connects to the registry are all proprietary cloud services. The .github/publish-components.yaml workflow reveals the registry at api.pipedream.com/v1/components/registry — this endpoint is not in the repo. The pd CLI referenced in the publish workflow is downloaded from cli.pipedream.com.
Required services to self-host the components. The components alone cannot be run standalone. They depend on the cloud runtime to resolve this.$auth, inject $.service.db and $.interface.http, call $emit(), run the workflow DAG, and schedule timer-based sources. The SDK documentation references api.pipedream.com as the default API host, and the Connect flow requires a project environment (development or production) tied to pipedream.com infrastructure.