Page through a URL backlink checker without gaps or loops
Rows per call, pages per query and a hard ceiling at 50,000. Here is a loop that respects all three and tells you when to stop.
Up to 1,000 rows come back per call. Each call is billed once, however many rows it carries. Deep pagination is capped at 50,000 rows, which is 50 calls at 1,000 rows each.
Why sorting comes before paging
Paging over an unsorted list is how duplicates appear. Always send sort and order together with page and per, so page 3 means the same thing on every request in the run.
Which column to sort by depends on the question. Biggest first is the usual choice for audits, because the cap then cuts off the long tail instead of the part you care about.
The data does not move during a run unless the monthly rebuild lands in the middle of it. Rare, but real. A nightly job that straddles the first of the month can return one page from the old index and the next from the new one, so record the start time in your log and compare.
A refdomains pull costs 2 credits per call, so 50 calls come to 100 credits, or one cent. Even the worst case is cheap; the real cost is a loop that does not terminate.
A paging loop that stops
TypeScript for a Node 20 worker. It stops on the first short page, on the cap, or on any ok that is not true.
const BASE = "https://seomasters.dev/api.php";
const CAP = 50_000;
const PER = 1000;
export async function pullAll(domain: string, key: string) {
const rows: unknown[] = [];
for (let page = 1; rows.length < CAP; page++) {
const u = new URL(BASE);
u.search = new URLSearchParams({
action: "refdomains", q: domain,
page: String(page), per: String(PER),
sort: "dr", order: "desc",
}).toString();
const res = await fetch(u, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401 || res.status === 402) throw new Error(`stop: HTTP ${res.status}`);
const body = await res.json();
if (!body.ok) throw new Error(JSON.stringify(body).slice(0, 300));
const chunk = findRows(body); // see note below
rows.push(...chunk);
if (chunk.length < PER) break; // short page means we are done
}
return rows;
}Troubleshooting a paging run
| What you see | Likely reason | What to do |
|---|---|---|
| Page 51 comes back empty or errors | You hit the 50,000 cap | Narrow with another action or a different sort, then merge by id |
| Same domain on two pages | Unsorted query, or the index rebuilt mid-run | Send sort and order; dedupe on the domain key |
| Fewer rows than the dashboard total | The total is for the whole graph, the pull is capped | Show "top 50,000" in the UI |
| Loop never ends | Condition checks rows.length only | Break on a short page, as above |
| Bill higher than expected | Retry on every non-200 | Retry only network faults, see errors |
Reading rows without guessing field names
The findRows helper above is deliberately left open. Make one real call, look at the JSON, and write the accessor against what you see rather than what a tutorial promised.
Then freeze it with a test. Save one response as a fixture file and assert on the row count and the first row's keys, so a future change fails in CI and not in a customer's chart.
For the cheaper cousin of this job, scoring many domains at once with no paging at all, read the bulk DR script.
Resuming after a crash
Persist the last finished page number after every call. Restart from that number plus one, not from the top, and you pay for 2 credits per page instead of re-buying the whole pull.
Crashes happen. A deploy restarts the worker, a laptop sleeps, a VPN drops, and the loop dies on page 31 of 50.
Keep an idempotent write. Upsert each row on its domain key, so a page fetched twice leaves one row, not two.
Log the sort and order alongside the page number. If someone changes sort between a crash and the restart, the saved page number no longer means the same slice. Reject the resume and start clean.
For lists above the ceiling, split the problem. Pull the top 50,000 by one sort, then ask anchors or toppages about the heavy hitters, which cost 2 credits too and answer different questions.