Web Scraping vs APIs: How to Choose, and When Neither Is Right
APIs are stable and constrained. Scraping is flexible and fragile. The right choice depends on the constraint you can least afford.
- Author
- HuiTu Technology
- Published
The choice between an official API and collecting from public pages is usually framed as a technical one. It is mostly a question of which constraint hurts you least: cost, coverage, stability or licence terms.
| Official API | Public page collection | |
|---|---|---|
| Stability | High; versioned and announced changes | Low; templates change without notice |
| Coverage | Only what the provider chooses to expose | Whatever is publicly visible |
| Cost at scale | Often the binding constraint | Engineering time rather than per-call fees |
| Licence clarity | Explicit, and often restrictive on storage | Requires judgement and care |
| Historical data | Usually limited retention | Only what you collected yourself |
| Maintenance | Low | Ongoing, and it never reaches zero |
Use the API when
- The data feeds a live product feature. Users notice breakage; a maintained API breaks far less often.
- Volumes are modest enough that pricing is not the deciding factor.
- You need fields that are not rendered on any public page.
- Provenance matters for compliance or audit reasons.
Collect from public pages when
- You need a complete census of a category across a geography, which most APIs will not let you enumerate.
- The data you need is displayed on a page but not exposed through any API.
- The work is a one-off research dataset, where API storage restrictions conflict with your purpose.
- There is no API at all, which remains the most common case.
The true cost comparison
API pricing is visible, so it feels expensive. Collection costs are mostly engineering time, so they feel free until the second year. A useful way to compare them is over a realistic horizon rather than at launch.
| Year 1 | Year 2 | Year 3 | |
|---|---|---|---|
| API | Per-call fees, low setup | Same fees, volume grows | Same again, plus any repricing |
| Collection | High build cost | Maintenance only | Maintenance only |
| Crossover | API usually cheaper | Depends on volume | Collection usually cheaper at scale |
The crossover point moves with volume and with how often the source changes. For a small feature, the API is almost always right. For a national census refreshed quarterly, it usually is not.
The hybrid that usually wins
In practice the best answer is often both. Use collection to build the complete universe of records once, because that is the thing APIs are worst at. Then use the API to keep a smaller, high-value subset current, because that is what APIs are best at. You pay for breadth once and for freshness continuously, rather than paying for both at the same rate.