How pipelines work
A pipeline turns a data brief into a repeatable sequence of collection, processing, verification and delivery steps, and repairs itself when a source changes.
- Describeyour brief
- BuildJason explores and builds
- Runworkers on a schedule
- Verifychecks before publishing
- Deliversheets, Slack, warehouse, MCP
- Repairwhen a source changes
1. Describe the result
Include the sources or the type of source, the fields you need, the market to cover, and the refresh schedule. A source list is useful when you already have one; otherwise describe what Jsonify should look for and Jason will propose sources.
Monitor restaurant menus across Manchester. Standardise item names and prices, keep the venue and section, and refresh the dataset weekly. Start with the 40 venues in the attached spreadsheet.
Attach files when you have them: a product catalogue, a list of URLs, an example of the output you want. Jason reads them as part of the brief.
2. Explore sources and build the pipeline
Jason opens the websites, apps or forms, works out how each exposes the data, and builds the pipeline: the steps that collect from each source, the processing that normalises fields and matches records, the checks that must pass, and the datasets to publish. Independent sources are collected in parallel; dependent steps run in sequence.
The result is a pipeline version with a statement of work describing what it does in plain language, and a step diagram showing how. You review the proposal before the build starts and the first run before the schedule begins.
3. Run it
A run executes the active version. Workers fetch pages, extract the fields, process the rows, and write the outputs. You can run a pipeline by hand from its page, let the schedule run it, or start it from your own system with a trigger.
Each run records what happened: pages fetched, rows collected, duration, the steps completed, activity as it happened, and what it sent where. See Run detail.
4. Verify before publishing
At the end of a run the outputs are checked against the expectations built into the pipeline: field types, required values and duplicates. A run that returns no rows fails rather than publishing an empty dataset, and an alert rule can warn you when a run returns far fewer rows than usual, because a quiet failure is worse than a loud one.
Rows that fail a check are held with a reason rather than published. Rows that pass become a new dataset version. Verification and repair has the details.
5. Deliver and refresh
A published version is checked by your alert rules, and the rules decide what goes where. An Every update rule sends the whole dataset to an output such as Google Sheets, an outgoing webhook, or a warehouse or bucket (Snowflake, BigQuery, Postgres, MySQL and S3). Other rules send a message, or only the rows that match. The version is also available over the serve API and to your own agent over MCP, and dashboards read it.
6. When a source changes
A changed page layout, a renamed field or a new anti-bot wall can make a step fail. Transient problems are retried. Persistent ones mark the pipeline as degraded and go to Jason, who reads the run, refetches the page, works out what changed, repairs the pipeline as a new candidate version, verifies it on a sample, and activates it. A live run of the new version then confirms the fix and the schedule carries on.
Automatic repair is a recovery process, not a guarantee. If a source disappears, blocks collection, or changes what it publishes, Jason tells you what it found and what it needs from you. The last verified dataset version stays available throughout.
Changing a pipeline later
Ask Jason in the pipeline’s chat. “Add the promotion end date”, “drop the two retailers we no longer stock”, “run at 5am instead of 6am”. Jason makes the change as a new version, and the version history records what changed and why. See Changing a pipeline.