Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ account (auth)
│ ├── connects → databases, storage
│ ├── runs on → a runtime
│ ├── invoked via → alias / HTTP / scheduler
├── durable functions ..... long-running, checkpointed logic → durable-functions.md
├── databases ............. managed Postgres → databases.md
│ └── schema changed by → migrations
├── storage ............... buckets → objects, policies → storage.md
Expand Down Expand Up @@ -81,6 +82,7 @@ Piped, CI, and `NO_COLOR` output remains plain. Machine output is unchanged.
| Account / auth | sign up, log in/out | `signup`, `login`, `logout` | [authentication.md](authentication.md) |
| Project | create, list, get, rename, delete, select, get anon keys | `projects …`, `use` | below |
| Functions | deploy, invoke, inspect, schedule, alias | `functions …` | [functions.md](functions.md) |
| Durable functions | deploy, start, inspect executions, schedule | `cloud durable …` | [durable-functions.md](durable-functions.md) |
| Databases | create, inspect, delete, migrate | `databases …`, `migrations …` | [databases.md](databases.md) |
| Storage | manage buckets, objects, policies | `storage …` | [storage.md](storage.md) |
| Variables | deploy, list, get, delete | `variables …` | [variables.md](variables.md) |
Expand Down
152 changes: 152 additions & 0 deletions docs/durable-functions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
---
title: "Durable functions"
description: "A durable function checkpoints its progress and resumes from the last completed step, so one execution can run for hours instead of the seconds a normal invocation allows."
---

## What it is

A durable function checkpoints its progress and resumes from the last completed
step, so one execution can run for hours instead of the seconds a normal
invocation allows. Use one for work that has to survive a restart: a multi-step
order pipeline, a long agent run, a nightly batch that calls out to a slow API.

Durable functions are cloud-only and JavaScript/TypeScript-only, and they are a
separate collection from standard functions:

- You **start** an execution instead of invoking a function. The start returns a
handle immediately; the result arrives later.
- Each **execution** is a resource with its own id, status, and result.
- The two collections never accept each other's names or ids. A function's kind
is fixed when it is created.

## How it relates

- Belongs to a **project**, and reads **variables**, **databases**, and
**storage** in that project like any function.
- Its sources live in `volcano/functions/` beside standard ones. Only
`volcano-config.yaml` says which are durable, so `volcano functions deploy
--all` skips them and `volcano cloud durable deploy --all` picks them up.
- Every region the project deploys to has to offer durable execution, or the
deploy is refused up front.
- Durable execution needs the durable authoring API, so a source whose runtime
does not offer one is refused before anything is uploaded.

```yaml
version: 1
project:
name: my-app
functions:
- name: hello
- name: order-pipeline
kind: durable
```

A durable entry takes `variable_scope` and `variables` like any other, and
`durable deploy` sends what the manifest declares. Declaring nothing leaves an
existing function's scope alone.

## CLI operations

| Operation | Command |
|---|---|
| Deploy all declared, or one | `volcano cloud durable deploy [--all \| -f <name\|path>]` |
| List | `volcano cloud durable list` |
| Get | `volcano cloud durable get <name>` |
| Delete | `volcano cloud durable delete <name>` |
| Start an execution | `volcano cloud durable start <name> [--input …] [--name …]` |
| List executions | `volcano cloud durable executions list <name> [--status …]` |
| Get one execution | `volcano cloud durable executions get <name> <execution-id>` |
| Stop an execution | `volcano cloud durable executions stop <name> <execution-id>` |
| Schedule executions | `volcano cloud durable schedulers create <name> --cron "0 * * * *" [--input …] [--regions …]` |
| List schedulers | `volcano cloud durable schedulers list <name>` |
| Pause or resume one | `volcano cloud durable schedulers disable\|enable <name> <scheduler-id>` |
| Delete one | `volcano cloud durable schedulers delete <name> <scheduler-id>` |

There is no top-level `volcano durable …`: the local development environment
does not run durable executions, and the local server refuses to create a
durable function rather than pretending to.

## Examples

```bash
# Deploy every function volcano-config.yaml declares durable
volcano cloud durable deploy --all

# Deploy one, and let anon keys start it
volcano cloud durable deploy -f order-pipeline --public

# Start an execution and follow it
volcano cloud durable start order-pipeline --input '{"order_id":4417}'
volcano cloud durable executions get order-pipeline 66666666-6666-4666-8666-666666666666

# Only the executions still running
volcano cloud durable executions list order-pipeline --status running

# End one where it is
volcano cloud durable executions stop order-pipeline 66666666-6666-4666-8666-666666666666
```

## Scheduling executions

A scheduler ticks on a cron expression and starts an execution instead of
invoking the function, so every tick produces an execution you can list, follow,
and stop like any other:

```bash
# Start one execution an hour, with the same input each time
volcano cloud durable schedulers create order-pipeline --cron "0 * * * *" --input '{"scope":"hourly"}'

volcano cloud durable schedulers list order-pipeline
volcano cloud durable executions list order-pipeline
```

Each tick names its execution after the run, so a tick Volcano has to retry
resolves to the execution it already started rather than beginning a second one.
Ticks draw on the same invocation allowance and concurrency cap a manual start
does; a tick that would exceed the cap fails that run rather than queueing.

`schedulers disable` stops the ticks and leaves the scheduler in place;
executions it already started keep running. `schedulers delete` removes the
scheduler and its run history, and also leaves running executions alone — stop
those with `executions stop`.

## Starting is asynchronous

`start` returns as soon as the execution is accepted, with the execution's id
and name. Read the execution to get its status and, once it has finished, its
result:

```bash
volcano cloud durable start order-pipeline --input order.json --name order-4417
volcano cloud durable executions get order-pipeline <execution-id>
```

`--input` takes inline JSON or the path to a JSON file, and is handed to the
function verbatim. Omit it to start with no input at all, which is not the same
as starting with `{}`.

`--name` is the execution's idempotency key. Starting again under a name that
already names an execution returns the existing one instead of beginning a
second, and is not charged again — so a retried start is safe. Omit it and
Volcano generates one.

## Visibility

`--public` lets a project's anon key start executions of one function;
`--private` takes that back. A public durable function is still not invocable
over HTTP the way a public standard function is — starting an execution is the
only thing the anon key can do. Polling and stopping always need a
project-scoped credential.

Omit both flags and a redeploy keeps the visibility the function already has. A
new durable function starts private.

## Stopping and deleting

`executions stop` ends one execution at its next checkpoint. Steps already
completed are not undone, and work already in flight is not interrupted
mid-attempt. Stopping one that has already finished reports the state it is in
rather than failing.

`durable delete` tears down the function and its execution history. It does not
wait for work in flight, so stop an execution you need ended first.
3 changes: 3 additions & 0 deletions docs/functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ is invoked over HTTP, by name/alias, or on a schedule.
- Its **visibility** (public/private) and **schedulers** can be declared in the
[declarative config](project-configuration.md) or managed with the CLI.
- Can be given an **alias** so you can invoke it by a friendly name.
- Work that has to run for hours belongs in a
[durable function](durable-functions.md) instead, which is a separate
collection with its own commands.

The CLI discovers functions in the `volcano/functions` directory, detects the
runtime from source file extensions, and uploads a packaged archive.
Expand Down
10 changes: 10 additions & 0 deletions docs/project-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ functions:
cron: "*/5 * * * *"
enabled: true
payload: { job: refresh }
- name: order-pipeline
kind: durable # asserted, not applied: a kind is fixed at creation
```

Cloud example — export the current cloud project's configuration to a file,
Expand All @@ -70,6 +72,14 @@ volcano cloud config deploy -f volcano-config.yaml # apply

Key semantics:

- `functions[].kind` declares which entries are
[durable functions](durable-functions.md). It is asserted rather than applied,
since a function's kind is fixed when it is created. It is also what tells the
deploy commands apart: `volcano functions deploy --all` skips these entries and
`volcano cloud durable deploy --all` deploys exactly them. Leave it out for a
standard function, and leave the invocation settings out of a durable one —
they describe synchronous HTTP invocation, which a durable function does not
have.
- Declared config sections are the source of truth. Variables, bucket policies,
OAuth providers, email templates, and function schedulers are fully synced
when declared: entries absent from the manifest are deleted. Omitted sections
Expand Down
Loading