Iterate over large company datasets using the Scroll API
masterFor datasets exceeding 10,000 companies, use client.companies.scroll instead of list. The Scroll API is more efficient for deep paging but has specific operational constraints.
Operational Rules:
- Concurrency: Only 1 scroll can be open per app at a time. Attempting to open a second will result in an error.
- Expiration: A scroll expires if it is not used for 1 minute. Subsequent calls with an expired scroll will fail.
- Completion: When the end of the dataset is reached, the
companiesarray will be empty and the scroll parameter will expire. - Error Handling: If you encounter an HTTP 500 error with the message
"Request failed due to an internal network error. Please restart the scroll operation."(often due to network timeouts on large datasets), you must restart the scroll operation from the beginning. You cannot resume from a specific point.
// Get the first page by sending an empty scroll_param or omitting it
const pageableResponse = await client.companies.scroll({
scroll_param: "scroll_param"
});
for await (const item of pageableResponse) {
console.log(item);
}