Statement of work and steps

Every pipeline carries a Statement of Work: a document that says what the pipeline collects, what it leaves out, the parameters it exposes, what it produces, and the rules it follows while doing so. Jason writes it when the pipeline is built and rewrites it with every version. It is the first thing to read when a pipeline is new to you, and the place to ask for changes.

Open a pipeline, choose How it works, then Statement of Work in the rail.

1. Readthe statement of work
2. Select textany rule, field or step
3. Commentdescribe the change
4. Commitsend the batch to Jason
5. New versionreviewed, then active

The document

Cover

The top of the document carries the eyebrow Statement of work and Last updated on a date. Beside the objective sits a delivery card:

• Last successful delivery. A link to the most recent run that published, with <rows> rows · <pages> pages · <duration> · <ago> and a small sparkline of recent deliveries. Under it, a preview of the first rows of the result dataset and View collection run.
• No successful delivery has been recorded yet. Shown until the first publish.

Objective

One paragraph stating what the pipeline is for, written for someone who has never seen the sources. It is the sentence the reviewer checks candidate versions against.

Pipeline Scope

Two labelled paragraphs:

• In scope. What a row is, which sources and markets are covered, and the fields that are collected.
• Not included. What the pipeline deliberately does not do: pages it does not read, fields it does not keep, markets it ignores.

“Not included” is explicit on purpose. If you want something the scope excludes, that is a change request, not an assumption the pipeline will make. Comment on the line and commit it.

Parameters

A strip of chips, one per parameter the pipeline exposes: region = GB, max_products = 500, since = yesterday. Hover a chip for its description, type and whether it is required. The strip is absent when the pipeline has no parameters. Changing a default or a per-run value happens on the Inputs page or in the run dialog; see Inputs and parameters.

Produces

One card per output. Each card names the output and the dataset it publishes to, describes the rows, and lists every field as a typed chip (retailer · text, price · number, observed_at · datetime). Long field lists collapse behind View all.

Under the card, its guarantees: promises about the data that the rules below enforce, each citing the rule that backs it.

Every row has a price in the retailer’s currency, or price_missing_reason set (R4).

Click an output name to open a summary of that output: its dataset, row count, revision, when it last updated, and its declared columns with types and descriptions. Open dataset goes to the dataset page. Before the first run the summary reads No data yet. Run the pipeline to generate output.

How does it work?

The rules ledger. A numbered list of the processing behaviour you can rely on, each with a stable id (R1, R2, …) that stays the same across versions:

R3 — Promotions are recorded only when the retailer shows both a previous and a current price; badge-only “deals” are ignored.

Rules are extracted from the pipeline at every publish, so the ledger always describes the active version. A rule that changes keeps its id and notes the version it changed in. Rules are the most useful thing to comment on: a comment on R3 is a precise change request.

Input Data

The inputs the pipeline reads: a dataset from another pipeline, a value list such as URLs or product identifiers, or a file. Each is shown with a preview of its current values. Editing the values is done on the Inputs page.

Output data and Output data columns

The Output Data section lists each dataset and file the pipeline publishes, with a link to it. Output data columns is the full field table for each output: name, type, description. The step diagram’s outputs point at the same tables.

Schedule

A single line: Runs every day at 09:00 (Europe/London). or Runs only when triggered manually. Click it to open the schedule editor without leaving the document. The editor’s fields are described in Settings reference and Scheduling.

Sources and How it runs

Below the contract, Sources lists the sites that fed the latest output, and How it runs shows a compact version of the step diagram (The active definition, stage by stage.). Click an output node to open its summary, or Open full flow to go to the Steps page.

When there is no document yet

A pipeline whose document has not been written shows Generate document, which asks Jason to write it. Sections that would be empty are hidden; if nothing applies the page reads Nothing to show here yet.

Commenting and committing changes

The statement of work is where you ask for changes with precision. Rather than describing a rule in a chat, select the rule and say what should be different.

1. Select the text. Highlight any part of a rule, guarantee, field row or step paragraph. A floating Comment to change button appears.
2. Describe the change. The composer opens anchored to the selection, with the placeholder Describe the change you want…. Press Comment to save it. The selection is highlighted in the document and a card appears in the right margin.
3. Add more if you need to. Each comment is a separate card. Reply to your own comment with Reply…, or remove it with Dismiss. Nothing is sent yet.
4. Commit. A sticky bar at the top of the document counts your open comments: 3 pending changes. Press Commit (3). Every open comment is sent to Jason as one change request scoped to this pipeline, and the chat opens so you can follow the work.
5. Review the version. Jason makes the change, tests it, and sends it for independent review. When it passes, the version chip advances and the document is rewritten. See Versions and changelog.

On R3: also record promotions when the retailer shows a percentage-off badge with the current price, and store the percentage in a new promo_percent column.

Comments are visible to everyone in the workspace. Committed comments disappear from the margin once Jason has picked them up; the request lives on in the chat and in the next version’s changelog.

Comments attach to rules, guarantees, field rows and step descriptions. The cover, the schedule line and the parameters strip are not commentable: change those from Settings and Inputs, or ask Jason directly with Edit pipeline.

Viewing an older version

The document always shows the version selected in the version chip. When you pick an older one, an amber bar reads Viewing v1 · current is v2 with Restore v1 and Back to current. The Steps page switches to the same version. Comments are still possible on an older version, but they are committed against the current one.

Pipeline Steps

Pipeline Steps in the rail shows the pipeline as a diagram: inputs on the left, steps in the middle, outputs on the right. It is the same picture that animates on a live run, drawn here without run data so you can read the structure.

Blocks

Block	Shape	What it is
Input	Dashed edge into the first step	A dataset the pipeline reads, or a value list such as URLs. Static values show their current value under the name.
Step	Rounded box	One stage: fetch a set of pages, extract fields, process rows. A map step repeats its work for every input row; the diagram labels its input edge with the row count.
Route	Diamond	A decision that sends rows down one of several branches.
Output	Box with a table icon	A dataset or file the pipeline publishes. Click it to open the output summary.

The reading panel

Selecting a block fills the panel on the right. Depending on the block you see Stages and Steps (the position in the pipeline), Reads inputs and Read by (what flows in and out), Saves (the outputs a step writes), Parameters read in this stage (or This step reads no run parameters.), Primary key, and Last run facts such as the rows it produced. Buttons take you onwards: Open inputs to the Inputs page, Open output or Open dataset to the output.

Escape or the × in the panel clears the selection. Zoom in and Zoom out sit in the corner of the canvas.

A version without a diagram shows No diagram available for this version. and the rail entry is hidden.

A pipeline built from a brief like “track prices across three retailers” typically shows one input (the product list), three parallel fetch-and-extract steps (one per retailer), a matching step that joins them, a check step, and one output. Selecting the matching step tells you which key it joins on and which rules apply.

What’s next