Changing a pipeline

A pipeline is never edited in place. Every change you ask for becomes a new version that is built, verified on real data, reviewed independently and then activated. The previous version stays in the history. This page shows how to ask, what to expect back, and how to check the result.

Three ways to ask

From the pipeline page

Edit pipeline in the pipeline’s header opens a dialog with one box (What should change? Jason picks it up straight away.), Attach, and a row of suggestion pills based on the pipeline’s specification and last run. Press Send to Jason (or ⌘↵). A new chat scoped to this pipeline opens in the Jason panel and the request is sent immediately.

From a chat

In any chat, mention the pipeline with @ so there is no ambiguity about which one you mean, then describe the change. This is the right route when the change involves more than one pipeline or a dataset.

@inventory-monitoring should also read the “Only 3 left” badge into a low_stock flag. @product-identity-matching then needs to carry that flag through to the matched rows.

From the statement of work

On the pipeline’s Statement of Work, select any passage and a small floater offers to comment. Type the change in the box (Describe the change you want…) and add it. Comments collect in the right margin as anchored cards you can reply to or dismiss. When you are done, the amber bar at the top reads N pending changes with Commit (N); committing sends the whole batch to Jason as one request, so related changes land in one version.

Selected: “One row per product per retailer per day.” Comment: “Change to one row per product per retailer per observation; we want to run twice a day from next week.”

What Jason does

1. Reads the request together with the pipeline’s specification, steps, inputs, recent runs and the chat so far. If anything is ambiguous, Jason asks before changing anything (see questions).
2. Builds a candidate version. The transcript shows Start pipeline change and Edit pipeline, followed by the pages Jason fetches to check its assumptions.
3. Verifies it. Test candidate runs the candidate against real sources. The status row reads Testing candidate; the run appears under Latest runs as a Test run, and publishes nothing.
4. Sends it for review. An independent reviewer compares the candidate’s output with the statement of work. Its card shows Passed, Failed or Needs evidence, with View report.
5. Activates it. A passed candidate becomes the current version. The step diagram embeds in the chat as Pipeline — name · vN, and the reply ends with a Task completed card.

If the reviewer rejects the candidate, Jason revises it and tries again. If it cannot produce a version that passes, it says so and explains what it found instead of activating something unverified.

Candidate tests can take up to an hour on a large pipeline because they collect real pages. After 30 seconds the chat offers Notify me so you do not have to wait on the page.

What you get back

Item	Where
A new version	The changelog gains a Version 3 row marked Current version. Earlier versions remain listed as Version published.
A changelog entry	Changelog on the pipeline page records what changed and why, in Jason’s words.
A verification run	The Test run that validated the candidate, in Runs and on Latest runs. Its outputs show what the new version produces without touching the published dataset.
A reviewer report	The Pipeline reviewer card in the chat, with findings and Activated as pipeline version N.
An updated statement of work	The Statement of Work reflects the new scope, fields and rules.

The next scheduled or manual run uses the new version and publishes a new dataset version.

Good change requests

Be concrete about the outcome; leave the mechanics to Jason. Some shapes that work well:

A new field

Add a unit_price column: the price per 100g or per 100ml as the retailer displays it, as a number, blank when the site does not show one.

A source added or removed

Add Waitrose as a fourth retailer for the same product list. Drop Ocado; we no longer stock against them.

A schedule

Run twice a day, 06:00 and 18:00 Europe/London, instead of once. Keep the daily rollup pipeline on its current schedule.

A matching rule

When matching our SKUs to retailer products, require the pack size to match as well as the brand and name. A 6-pack and a 12-pack are different products.

A threshold or check

Hold any row where the price is more than 40% below the previous observation for the same product, and flag it in the run instead of publishing it.

A format

Store all prices in pence as whole numbers and add a currency column set to GBP.

Attach an example when you have one: a row showing the value you expect, or a screenshot of the badge you want captured.

What Jason asks instead of guessing

Some requests have more than one reasonable reading. Jason pauses with a question card rather than pick one silently:

You ask	Jason asks
“Add the discount”	Whether you want the percentage, the amount off, or the promotional text as shown.
“Include out-of-stock products”	Whether to publish them with a blank price, a status column, or hold them separately.
“Match by name”	How strict: exact, normalised (case and punctuation), or fuzzy with a threshold.
“Run more often”	The cadence and time zone.
“Use the new site”	Which fields, if any, should change when the new site shows different ones.
“Prices per unit instead of per pack”	Whether to replace the existing price column or add a second one, since dashboards depend on the first.

A short, specific answer gets the change moving again. Details of the question card are on Chatting with Jason.

A request that would remove data other things rely on — a column a dashboard reads, a dataset another pipeline consumes — gets a question, not a silent removal.

Checking the change

1. Open the changelog. How it works › Changelog lists every version with its changelog line, Current version on the active one, and when it was activated. See Versions and changelog.
2. Read the note. Each version’s changelog line says what changed. Compare it with what you asked for.
3. Check the verification run. In Runs, open the Test run from the change. Outputs shows the rows the new version produced; Steps shows what changed in the flow.
4. Look at the latest outputs. After the next live run, Latest outputs on the pipeline page and the dataset itself show the change in published data. A new column should be present and typed; a removed source should be absent from the row’s origin.
5. Watch the next scheduled run. If the change affects volume, an alert rule with the row drop condition tells you when a run returns far fewer rows than recent runs.

Undoing a change

Every version is kept, and restoring is itself a change:

1. Open the changelog, click the row of the version you want to go back to, and press View vN in the details window. The version opens in a window with its statement of work and step diagram.
2. Press Restore vN. That exact version, which was already reviewed, becomes the new current version straight away. Jason is not involved. The toast reads Restored vN as vM.

Restoring never rewrites history. If the current version is v5 and you restore v3, you get v6 that is v3, and v4 and v5 remain in the list. Your current input values and parameter defaults are kept.

A restore is a pipeline change, not a dataset change. Published dataset versions are not rolled back; the next run of the restored version publishes a new one. To read older rows, switch the dataset version instead. See Dataset versions.

Changes Jason makes on its own

Two kinds of version appear without a request from you:

• Auto-heal. When a live run fails, the pipeline is degraded and Jason diagnoses it, repairs the pipeline as a new candidate, verifies and activates it, then confirms the fix with a live run. The failed run is not re-run. The version’s changelog line says what changed, and the Changelog’s Autoheal row links the failed run and the repair conversation. See Verification and repair.
• Settings changes. Turning the schedule on, changing the fetch cache or the automatic repair setting from the pipeline’s Settings does not create a version; those settings live beside the pipeline. See Settings reference.

What’s next