Agents, workers and repairs

Three roles keep a dataset useful over time. Knowing which one is doing what makes the activity you see in the product easy to read.

Role	What it does	Where you see it
Jason	Understands the brief, explores sources, builds and repairs pipelines, answers questions about your data	Chats, activity on the pipeline page, and the new version it activates after a failure
Workers	Execute the active pipeline version on a schedule or on demand: fetch, extract, process, check, publish	Run detail (Summary, Steps, Activity, Integrations), the runs rail
Reviewer	Independently checks a candidate version against a sample before it can be activated	The Pipeline reviewer card in the chat, with its verdict and report, and the changelog on the version it approved

Jason: build and maintain

Jason is the agent you describe the task to. It reads your brief and attachments, opens the sources, decides how to collect from each, and assembles the pipeline. When something stops working, Jason investigates and produces a revised version.

Jason works in a chat. Each task is a turn: you send a message, Jason works through it, and replies with what it did and links to what it made. While a turn runs you can watch the activity, answer questions Jason asks, or send another message to steer it. See Chatting with Jason.

Add Target to the retailer list, keep the same fields, and let me know if any of the current products are not sold there.

Jason never runs the pipeline on your behalf silently. A build ends with a first run you can inspect; a change ends with a new version and a verification run you can inspect.

Workers: execute the steps

Workers carry out the pipeline’s steps. Independent sources are collected in parallel; a processing step waits for the collection it depends on. Workers are deterministic: the same version, inputs and pages give the same rows.

A restaurant-menu pipeline might find venue websites and delivery listings in parallel, extract menu items from each, then merge, de-duplicate and verify the records before publishing.

1. Fetchpages, in parallel
2. Extractfields from each page
3. Processnormalise, match, merge
4. Checktypes, nulls, duplicates, volume
5. Publisha new dataset version

Every worker action is recorded: which page was fetched, with what result, which step produced which rows. That record is what row provenance shows you.

Reviewer: an independent check

Before a candidate version can become active, a separate reviewer runs it against a sample and checks the output against the statement of work. A repair that produces the right shape of data but the wrong values does not pass. This applies to the first build, to every change you ask for, and to every automatic repair.

Datasets: the structured result

A dataset holds the fields the pipeline publishes. Useful fields usually include the source URL, collection time, units and currency alongside the main values; they let you interpret changes and compare records correctly. Every publish is a new version.

Verification checks missing values, types, duplicates and expectations specific to the task. A successful fetch alone does not establish that a value is right, which is why checks are part of the pipeline rather than an afterthought.

What automatic repair means

Suppose a retailer moves its price into a different part of the page. The collection step fails, or starts returning empty prices. Here is what happens, in the order you will see it in the product:

1. The run retries. Transient failures (a timeout, a rate limit) usually clear on retry. Up to three attempts are made before anything is escalated.
2. Jason diagnoses. It reads the failing step, refetches the page, and compares what the pipeline expected with what the page now shows.
3. Jason repairs. The fix is made as a new candidate version, with a changelog line describing what changed.
4. The reviewer verifies. The candidate is run on a sample and checked. If it passes, it is activated.
5. Jason confirms the fix. A live run of the new version publishes, the Changelog records Recovery confirmed, and the pipeline runs on its schedule again. The failed run stays failed; Resume on it finishes its remaining work on the new version if you want that.

A repair leaves a trail you can read afterwards, even though it happens without you:

What it does not mean

• Sources can go away. A page that is removed, moved behind a login you have not connected, or blocked cannot be repaired by changing the pipeline. Jason reports what it found and the last verified version stays available.
• Some fixes need a decision. If a site now shows prices per unit instead of per pack, the right answer depends on what you want. Jason asks rather than guessing.
• A repair must pass verification. If the candidate cannot be verified, it is not activated and you are told why.

Turn on alerts for run failures so you hear about escalations where your team already works. See Alert rules.

For ongoing operational support and a managed scope, see Enterprise.