When working with APIs to retrieve vast amounts of data—potentially thousands of items—there are several crucial aspects to consider, ensuring the process is efficient, flexible, and performant. Here’s a breakdown of the key factors to manage, along with a solution for PHP users.

Key considerations when retrieving large data via API

Let me share some key considerations for efficiently retrieving large datasets via API:

• Handling pagination: APIs typically deliver data in pages. To retrieve all the data, you need to manage pagination, performing multiple API calls while keeping track of the cursor or page number. Calculating the number of required API calls and managing this process is essential to ensure you get the complete dataset.
• Memory management: when fetching large datasets, loading everything into memory at once can overwhelm your system. It's crucial to avoid loading all results into memory at the same time. Instead, process data in chunks, ensuring your application remains responsive and doesn’t run into memory issues.
• Rate limiting & throttling: many APIs impose rate limits, such as restricting you to X requests per second or Y requests per minute. To stay within these limits, you must implement a flexible throttling mechanism that adapts to the API's specific restrictions.
• Parallel API requests: given the need to perform numerous API calls due to pagination, you want to retrieve data as quickly as possible. One strategy is to make multiple API calls in parallel, all while respecting the rate limits. This ensures that your requests are both fast and compliant with API constraints.
• Efficient data collection: despite making numerous paginated API requests, you need to combine the results into a single collection, handling them efficiently to avoid memory overload. This ensures smooth processing of data while keeping resource usage low.
• Optimized JSON parsing: many APIs return data in JSON format. When dealing with large responses, it's important to access and query specific sections of the JSON in a performant manner, ensuring that unnecessary data isn't loaded or processed.
• Efficient exception handling: APIs typically raise exceptions through HTTP status codes, indicating issues like timeouts, unauthorized access, or server errors. It’s important to handle these using the exception mechanism provided by your programming language. Beyond basic error handling, you should also map and raise exceptions in a way that aligns with your application's logic, making the error handling process clear and manageable. Implementing retries, logging, and mapping errors to meaningful exceptions ensures a smooth and reliable data retrieval process.

The "Lazy JSON Pages" PHP Solution

If you're working with PHP, you're in luck. The Lazy JSON Pages open source package offers a convenient, framework-agnostic API scraper that can load items from paginated JSON APIs into a Laravel lazy collection via asynchronous HTTP requests. This package simplifies pagination, throttling, parallel requests, and memory management, ensuring efficiency and performance.

You can find more information about the package, and more options to customize it in the readme of the official GitHub repository: Lazy JSON Pages.

I want to say thank you to Andrea Marco Sartori the author of the package.

Example: Retrieving Thousands of Stories from Storyblok

Here’s a concise example of retrieving thousands of stories from Storyblok using the Lazy JSON Pages package in PHP.
First, you can create a new directory, jump into the directory and start installing the package:

mkdir lazy-http
cd lazy-http
composer require cerbero/lazy-json-pages

Once the package is installed, you can start creating your script:

<?php

require "./vendor/autoload.php";

use Illuminate\Support\LazyCollection;  
$token = "your-storyblok-access-token";
$version = "draft"; // draft or published

$source = "https://api.storyblok.com/v2/cdn/stories?token=" . $token . "&version=" . $version;
$lazyCollection = LazyCollection::fromJsonPages($source)
    ->totalItems('total')
    ->async(requests: 3)
    ->throttle(requests: 10, perSeconds: 1)
    ->collect('stories.*');

foreach ($lazyCollection as $item) {
    echo $item["name"] . PHP_EOL;
}

Then you can replace your access token, and execute the script via the php command.

How it works

• Efficient pagination: the API results are paginated, and the lazy collection handles fetching all pages without needing to store everything in memory.
• Async API calls: the ->async(requests: 3) line triggers three API requests in parallel, improving performance.
• Throttling: the ->throttle(requests: 10, perSeconds: 1) line ensures that no more than 10 requests are made per second, adhering to rate limits.
• Memory efficiency: The use of lazy collections allows data to be processed item-by-item, reducing memory overhead, even with large datasets.

This approach offers a reliable, performant, and memory-efficient solution for retrieving large volumes of data from APIs in PHP.

References

• The Lazy JSON Pages package: https://github.com/cerbero90/lazy-json-pages
• The author of the open source package: https://github.com/cerbero90