Processing Large Datasets with Laravel Lazy Collections

When Laravel code reads a large file or database result, loading every record into memory before processing can exceed the available memory. A Laravel LazyCollection produces each value when code requests it. This can reduce the extra memory used to process a large input, as long as the data source and later code do not hold the full result in memory.

PHP generators make this possible. A generator is a function that yields values one at a time as a loop requests them. A lazy pipeline is a sequence of steps that filters or changes values; Laravel runs those steps as code iterates over the pipeline. For large database results, Laravel’s chunked lazy() method is a practical starting point. A cursor() streams model objects created from database rows from one query, but database-driver buffering can still use substantial memory. Laravel’s collection documentation and Eloquent retrieval documentation describe these trade-offs.

How a lazy collection processes values

A PHP generator pauses when it yields a value and resumes when the code using it requests the next one. The generator does not first build a complete array or provide the same random access as an array. Laravel uses generators so collection operations can work on values as they arrive. PHP’s generator documentation explains how generators provide data to loops without constructing the full array first.

For example, a lazy collection can read a file line by line and pass each line through a pipeline:

use Illuminate\Support\LazyCollection;

$lines = LazyCollection::make(function () {
    $handle = fopen('log.txt', 'r');

    try {
        while (($line = fgets($handle)) !== false) {
            yield $line;
        }
    } finally {
        fclose($handle);
    }
});

$lines
    ->filter(fn (string $line) => str_contains($line, 'ERROR'))
    ->each(function (string $line) {
        // Handle this matching line.
    });

Creating $lines and adding filter does not read the file. When each starts iterating, the generator reads one line, filter checks it, and each handles a matching line before the generator reads the next one. Laravel’s lazy collection examples use this same generator-based approach.

What changes compared with an eager collection

An eager collection loads all its items into memory before filtering or transforming them. For instance, User::all()->filter(...) loads the users first, then filters them. Laravel uses 10,000 models to illustrate this difference; that number is an example, not a benchmark.

A lazy collection lets values flow through operations during iteration. The table summarizes the practical difference:

Lazy processing changes when code produces and retains values, but it does not guarantee lower runtime or fixed memory use. If code stores every processed value in another array, it can use much of the same memory that eager loading would have used.

Choose between cursor(), lazy(), and lazyById()

Laravel’s cursor() runs one query and creates Eloquent model objects from database rows as iteration advances, holding one model at a time. However, Laravel warns that the database driver’s PDO layer can buffer raw query results. That buffering means a cursor does not guarantee low memory use for arbitrarily large results. Laravel also documents that cursors cannot eager-load relationships.

The lazy() method retrieves records in groups, then presents them as one LazyCollection. You can set the number of records in each group, as in this example:

Flight::where('destination', 'Zurich')
    ->lazy(500)
    ->each(function (Flight $flight) {
        // Process this flight.
    });

Laravel’s query builder API lists 1,000 as the default chunk size for lazy() in the documented version. That default is not a universal recommendation. The right size depends on the records and available resources, and the cited documentation does not establish an optimal size for every workload. Laravel recommends considering lazy() for very large Eloquent results and using it when the query needs related records loaded along with the models. The Eloquent documentation describes chunked retrieval, cursors, and relationship loading.

Use lazyById() when your code changes a column that determines which records match the query. It moves through records by comparing each ID with the ID of the last model in the previous group. For example, if a job updates a status that its query uses to select records, moving forward by ID avoids relying on that changing status to track progress.

When lazy iteration helps, and what can defeat it

Use lazy iteration when the input is too large to fit comfortably in memory and you can do the work one record at a time or in groups. Reading files, filtering database results, and handling matching records are good fits. For database work, choose lazy() when you need to retrieve records in groups or load related records with the models. Consider lazyById() when your code changes the condition it uses to select records.

An eager collection is still a reasonable choice when the result is small or your code needs all the items at once. You can also make an existing collection lazy to avoid creating another full-size result during later transformations. This does not free the original collection, so its items still occupy memory.

Operations that gather all values into a standard collection also stop the stream. Laravel’s collect() method, for example, turns a lazy collection into a regular collection. Code that saves every processed model or builds one large output string can also accumulate the data that lazy iteration was meant to avoid. Check what your code keeps in memory, not only whether the input uses LazyCollection.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com