Changing a pipeline

A pipeline is never edited in place. Every change you ask for becomes a new version that is built, verified on real data, reviewed independently and then activated. The previous version stays in the history. This page shows how to ask, what to expect back, and how to check the result.

Three ways to ask

From the pipeline page

Edit pipeline in the pipeline’s header opens a dialog with one box (What should change? Jason picks it up straight away.), Attach, and a row of suggestion pills based on the pipeline’s specification and last run. Press Send to Jason (or ⌘↵). A new chat scoped to this pipeline opens in the Jason panel and the request is sent immediately.

From a chat

In any chat, mention the pipeline with @ so there is no ambiguity about which one you mean, then describe the change. This is the right route when the change involves more than one pipeline or a dataset.

@inventory-monitoring should also read the “Only 3 left” badge into a low_stock flag. @product-identity-matching then needs to carry that flag through to the matched rows.

From the statement of work

On the pipeline’s Statement of Work, select any passage and a small floater offers to comment. Type the change in the box (Describe the change you want…) and add it. Comments collect in the right margin as anchored cards you can reply to or dismiss. When you are done, the amber bar at the top reads N pending changes with Commit (N); committing sends the whole batch to Jason as one request, so related changes land in one version.

Selected: “One row per product per retailer per day.” Comment: “Change to one row per product per retailer per observation; we want to run twice a day from next week.”

What Jason does

1. Reads the request together with the pipeline’s specification, steps, inputs, recent runs and the chat so far. If anything is ambiguous, Jason asks before changing anything (see questions).
2. Builds a candidate version. The transcript shows Start pipeline change and Edit pipeline, followed by the pages Jason fetches to check its assumptions.
3. Verifies it. Test candidate runs the candidate against real sources. The status row reads Testing candidate; the run appears under Latest runs as a Test run, and publishes nothing.
4. Sends it for review. An independent reviewer compares the candidate’s output with the statement of work. Its card shows Passed, Failed or Needs evidence, with View report.
5. Activates it. A passed candidate becomes the current version. The step diagram embeds in the chat as Pipeline — name · vN, and the reply ends with a Task completed card.

If the reviewer rejects the candidate, Jason revises it and tries again. If it cannot produce a version that passes, it says so and explains what it found instead of activating something unverified.

Candidate tests can take up to an hour on a large pipeline because they collect real pages. After 30 seconds the chat offers Notify me so you do not have to wait on the page.

What you get back

Item	Where
A new version	The version chip in the pipeline header increments, for example v2 → v3. Earlier versions remain in the chip’s menu.
A changelog entry	Changelog & Notes on the pipeline page records what changed and why, in Jason’s words.
A verification run	The Test run that validated the candidate, in Runs and on Latest runs. Its outputs show what the new version produces without touching the published dataset.
A reviewer report	The Pipeline reviewer card in the chat, with findings and Activated as pipeline version N.
An updated statement of work	The Statement of Work reflects the new scope, fields and rules.

The next scheduled or manual run uses the new version and publishes a new dataset version.

Good change requests

Be concrete about the outcome; leave the mechanics to Jason. Some shapes that work well:

A new field

Add a unit_price column: the price per 100g or per 100ml as the retailer displays it, as a number, blank when the site does not show one.

A source added or removed

Add Waitrose as a fourth retailer for the same product list. Drop Ocado; we no longer stock against them.

A schedule

Run twice a day, 06:00 and 18:00 Europe/London, instead of once. Keep the daily rollup pipeline on its current schedule.

A matching rule

When matching our SKUs to retailer products, require the pack size to match as well as the brand and name. A 6-pack and a 12-pack are different products.

A threshold or check

Hold any row where the price is more than 40% below the previous observation for the same product, and flag it in the run instead of publishing it.

A format

Store all prices in pence as whole numbers and add a currency column set to GBP.

Attach an example when you have one: a row showing the value you expect, or a screenshot of the badge you want captured.

What Jason asks instead of guessing

Some requests have more than one reasonable reading. Jason pauses with a question card rather than pick one silently:

You ask	Jason asks
“Add the discount”	Whether you want the percentage, the amount off, or the promotional text as shown.
“Include out-of-stock products”	Whether to publish them with a blank price, a status column, or hold them separately.
“Match by name”	How strict: exact, normalised (case and punctuation), or fuzzy with a threshold.
“Run more often”	The cadence and time zone.
“Use the new site”	Which fields, if any, should change when the new site shows different ones.
“Prices per unit instead of per pack”	Whether to replace the existing price column or add a second one, since dashboards depend on the first.

A short, specific answer gets the change moving again. Details of the question card are on Chatting with Jason.

A request that would remove data other things rely on — a column a dashboard reads, a dataset another pipeline consumes — gets a question, not a silent removal.

Checking the change

1. Open the version chip in the pipeline header. The menu lists every version with its source (build, edit, Auto-heal), the current marker and when it was activated. See Versions and changelog.
2. Read the changelog. Changelog & Notes describes each version. Compare the note with what you asked for.
3. Check the verification run. In Runs, open the Test run from the change. Outputs shows the rows the new version produced; Steps shows what changed in the flow.
4. Look at the latest outputs. After the next live run, Latest outputs on the pipeline page and the dataset itself show the change in published data. A new column should be present and typed; a removed source should be absent from the row’s origin.
5. Watch the next scheduled run. If the change affects volume, the row-count check compares it with previous runs before publishing.

Undoing a change

Every version is kept, and restoring is itself a change:

1. Open the version chip and click the version you want to go back to. The statement of work and steps switch to that version, and an amber bar reads Viewing vN · current is vM.
2. Press Restore vN. Jason receives a request to reproduce that version’s behaviour as a new current version, verifies it and has it reviewed like any other change.
3. Confirm in the chat. The toast reads Jason is restoring version vN; the panel opens on the restore chat.

Restoring never rewrites history. If the current version is v5 and you restore v3, you get v6 that behaves like v3, and v4 and v5 remain in the list. If Jason is already working on the pipeline, the restore waits until that turn finishes.

A restore is a pipeline change, not a dataset change. Published dataset versions are not rolled back; the next run of the restored version publishes a new one. To read older rows, switch the dataset version instead. See Dataset versions.

Changes Jason makes on its own

Two kinds of version appear without a request from you:

• Auto-heal. When a run fails persistently, Jason diagnoses it, repairs the pipeline as a new candidate, verifies and activates it, and resumes the run. The version’s source reads Auto-heal, and its changelog says what changed. See Verification and repair.
• Settings changes. Turning the schedule on, changing the fetch cache or the automatic repair setting from the pipeline’s Settings does not create a version; those settings live beside the pipeline. See Settings reference.

What’s next