Statement of work and steps

Every pipeline carries a Statement of Work: a document that says what the pipeline collects, what it leaves out, the parameters it exposes, what it produces, and the rules it follows while doing so. Jason writes it when the pipeline is built and rewrites it with every version. It is the first thing to read when a pipeline is new to you, and the place to ask for changes.

Open a pipeline, choose How it works, then Statement of Work in the rail.

1. Readthe statement of work
2. Select textany rule, field or step
3. Commentdescribe the change
4. Commitsend the batch to Jason
5. New versionreviewed, then active

The document

The document reads top to bottom: What it’s for, How it works, Sources, How it runs, Pipeline Scope, then Notes. Last updated on a date sits in the top right corner; click it to open the changelog.

What it’s for

One paragraph stating what the pipeline is for, written for someone who has never seen the sources. It is the sentence the reviewer checks candidate versions against.

Pipeline Scope

Two labelled paragraphs:

• In scope. What a row is, which sources and markets are covered, and the fields that are collected.
• Not included. What the pipeline deliberately does not do: pages it does not read, fields it does not keep, markets it ignores.

“Not included” is explicit on purpose. If you want something the scope excludes, that is a change request, not an assumption the pipeline will make. Comment on the line and commit it.

Parameters

A strip of chips, one per parameter the pipeline exposes: region = GB, max_products = 500, since = yesterday. Hover a chip for its description, type and whether it is required. The strip is absent when the pipeline has no parameters. Changing a default or a per-run value happens on the Inputs page or in the run dialog; see Inputs and parameters.

Produces

One card per output. Each card names the output and the dataset it publishes to, describes the rows, and lists every field as a typed chip (retailer · text, price · number, observed_at · datetime). Long field lists collapse behind View all.

Under the card, its guarantees: promises about the data that the rules below enforce, each citing the rule that backs it.

Every row has a price in the retailer’s currency, or price_missing_reason set (R4).

Click an output name to open a summary of that output: its dataset, row count, revision, when it last updated, and its declared columns with types and descriptions. Open dataset goes to the dataset page. Before the first run the summary reads No data yet. Run the pipeline to generate output.

How it works

The rules ledger. A numbered list of the processing behaviour you can rely on:

1. Promotions are recorded only when the retailer shows both a previous and a current price; badge-only “deals” are ignored.

Jason writes the rules with every version, so the ledger always describes the active version; a version cannot be published without them. Rules are the most useful thing to comment on: a comment on a rule is a precise change request.

Input Data

The inputs the pipeline reads: a dataset from another pipeline, a value list such as URLs or product identifiers, or a file. Each is shown with a preview of its current values. Editing the values is done on the Inputs page.

Output data and Output data columns

The Output Data section lists each dataset and file the pipeline publishes, with a link to it. Output data columns is the full field table for each output: name, type, description. The step diagram’s outputs point at the same tables.

Schedule

A single line: Runs every day at 09:00 (Europe/London). or Runs only when triggered manually. Click it to open the schedule editor without leaving the document. The editor’s fields are described in Settings reference and Scheduling.

Sources and How it runs

After the rules, Sources lists the sites that fed the latest output, and How it runs shows a compact version of the step diagram (The active definition, stage by stage.), opened on its first steps. Fit shows the whole diagram. Click an output node to open its summary, or Open full flow to go to the Pipeline page.

Notes

The last section is a plain text box for standing instructions and gotchas. Jason reads it whenever it works on this pipeline. Notes are saved on the pipeline rather than in a version and autosave as you type; see Settings reference.

When there is no document yet

A pipeline whose document has not been written shows Generate document, which asks Jason to write it. Sections that would be empty are hidden; if nothing applies the page reads Nothing to show here yet.

Commenting and committing changes

The statement of work is where you ask for changes with precision. Rather than describing a rule in a chat, select the rule and say what should be different.

1. Select the text. Highlight any part of a rule, guarantee, field row or step paragraph. A floating Comment to change button appears.
2. Describe the change. The composer opens anchored to the selection, with the placeholder Describe the change you want…. Press Comment to save it. The selection is highlighted in the document and a card appears in the right margin.
3. Add more if you need to. Each comment is a separate card. Reply to your own comment with Reply…, or remove it with Dismiss. Nothing is sent yet.
4. Commit. A sticky bar at the top of the document counts your open comments: 3 pending changes. Press Commit (3). Every open comment is sent to Jason as one change request scoped to this pipeline, and the chat opens so you can follow the work.
5. Review the version. Jason makes the change, tests it, and sends it for independent review. When it passes, the changelog gains a version and the document is rewritten. See Versions and changelog.

On rule 3: also record promotions when the retailer shows a percentage-off badge with the current price, and store the percentage in a new promo_percent column.

Comments are visible to everyone in the workspace. Committed comments disappear from the margin once Jason has picked them up; the request lives on in the chat and in the next version’s changelog.

Comments attach to rules, guarantees, field rows and step descriptions. The cover, the schedule line and the parameters strip are not commentable: change those from Settings and Inputs, or ask Jason directly with Edit pipeline.

Viewing an older version

The document always describes the current version. To read an older one, open Changelog, click the version’s row and press View v<n>; its statement of work and step diagram open in a window with Restore v<n>. See Versions and changelog.

Pipeline

Pipeline in the rail shows the pipeline as a diagram: inputs on the left, steps in the middle, outputs on the right. It is the same picture that animates on a live run, drawn here without run data so you can read the structure.

Blocks

Block	Shape	What it is
Input	Dashed edge into the first step	A dataset the pipeline reads, or a value list such as URLs. Static values show their current value under the name.
Step	Rounded box	One stage: fetch a set of pages, extract fields, process rows. A map step repeats its work for every input row; the diagram labels its input edge with the row count.
Route	Diamond	A decision that sends rows down one of several branches.
Output	Box with a table icon	A dataset or file the pipeline publishes. Click it to open the output summary.

The reading panel

Selecting a block fills the panel on the right. Depending on the block you see Stages and Steps (the position in the pipeline), Reads inputs and Read by (what flows in and out), Saves (the outputs a step writes), Parameters read in this stage (or This step reads no run parameters.), Primary key, and Last run facts such as the rows it produced. Buttons take you onwards: Open inputs to the Inputs page, Open output or Open dataset to the output.

Escape or the × in the panel clears the selection. Zoom in and Zoom out sit in the corner of the canvas.

A version without a diagram shows No diagram available for this version. and the rail entry is hidden.

A pipeline built from a brief like “track prices across three retailers” typically shows one input (the product list), three parallel fetch-and-extract steps (one per retailer), a matching step that joins them, a check step, and one output. Selecting the matching step tells you which key it joins on and which rules apply.

What’s next