Track the Hacker News front page

This tutorial builds a pipeline that collects the newest submissions on Hacker News: title, link, discussion link, author, points, comments and the time it was posted. It is deliberately small so you can see every part of the product once. The screenshots are from a real run.

You need: A Jsonify account
Rows used: 90 of your free 100
Time: About 10 minutes
Result: A dataset that grows on every run

What happens in this tutorial.

1. Describe the job

Open Jsonify and start a workspace. Go to factory.jsonify.com/setup. If you already have workspaces, choose New workspace from the workspace list.
Type the brief. Say what to collect, from where, and what a row is. You do not need to describe how the page works.

Collect the newest public Hacker News submissions from news.ycombinator.com/newest. For each post keep the title, the link it points to, the Hacker News discussion link, the author, points, comment count and when it was posted. One row per discussion link. Start with the latest 90 posts.

The setup dialog with the Hacker News brief typed into the goal field

The setup dialog. Attach files here if you have a URL list or an example of the output you want.

</figure>

Continue. Jason reads the page, proposes the fields and asks anything it is unsure about. For this brief it will ask nothing; press Build.

2. Watch the build

Jason opens the newest page, works out how posts are laid out and how the “More” link paginates, writes the pipeline, runs it against a sample, and sends it for review. You can follow along in the chat; each line of activity is something Jason did.

Jason's task-completed card at the end of the build with links to the pipeline, dataset and first run — The end of the build. The card links to what Jason made.

A build ends with a real first run, so the dataset you look at next is not a sample. It is the first version of your data.

3. Read the pipeline

Open the pipeline. The How it works tab shows the statement of work: what is in scope, what is not, the parameter Jason chose (posts = 90), and the output contract listing every field with its type.

Pipeline page for Latest Hacker News Posts showing the statement of work — The statement of work is the contract for this pipeline. Ask Jason to change it and a new version is produced.

Two things worth noticing:

“Not included” is explicit. The pipeline does not read article bodies or comment threads. If you want them, that is a change request, not an assumption.
The parameter is yours. posts controls how many submissions a run collects. Change it in Inputs or per run. See Inputs and parameters.

4. Look at the run

Open Runs. The first run completed in a few seconds, fetched a handful of pages and collected 90 rows across the steps.

Run summary showing status, version, duration, pages fetched and rows collected — The run summary. Outputs lists the dataset it published; Logs shows what the workers did.

5. Read the rows

Open the dataset from the run’s outputs or from the pipeline’s Latest outputs tab.

Dataset table of Hacker News posts with title, url, discussion_url, author, points, comments and posted_at columns — Ninety rows, one per discussion link. Columns carry formats, so URLs are links and numbers are numbers.

Click any row. Data shows the values; Provenance shows the run, step and fetch that produced them, with a screenshot of the page as it was collected.

Every value can be traced back to the page it came from.

See Row detail and provenance for what each tab shows.

6. Make it run on its own

The build left the schedule off. Ask Jason, or set it yourself in Settings:

Run this every hour and keep appending new posts. Skip posts we already have.

Jason sets the schedule and confirms that the pipeline already de-duplicates by discussion link, so re-runs only add new rows. From now on the dataset grows on its own, and each run is a new version you can compare.