High-throughput ETL pipeline for web assets

Overview

An AI company needed to process and distribute a large volume of structured web asset data. At the required throughput, moving records from one service to another was only part of the problem. The pipeline also had to operate reliably and make it easy for engineers to inspect samples, investigate unexpected data, and catch problems early.

FerrisDev built software around the established extract, transform, load (ETL) pattern. It received data from upstream services, transformed it into the required structure, and delivered the results to storage and downstream systems.

Why Scale Changed the Problem

Small data-quality or processing errors become expensive when repeated across billions of assets. Engineers needed a way to probe the live data flow, inspect representative samples, and diagnose failures without waiting for an entire processing run to finish.

The pipeline therefore needed both sustained throughput and operational tooling. It had to communicate with other services, keep track of processing metadata, store the assets, and expose enough of its internal state for problems to be found while they were still limited in scope.

Pipeline Architecture

The system followed a conventional ETL pipeline:

1. Extract: services exchanged structured records over gRPC.
2. Transform: workers validated and converted records into the formats required by downstream consumers. RocksDB stored processing metadata close to the workers.
3. Load: assets were written to object storage, while structured results were distributed to the services that used them.

Operational Tooling

Alongside the main processing path, diagnostic tools let engineers probe the pipeline and sample data at different stages. This made malformed records, unexpected input, and processing failures visible before they affected a large part of the dataset.

Outcome

The production pipeline reliably processed billions of web assets and supplied structured data to the company's AI and intelligence products.

Where This Architecture Fits

This architecture is useful when a company needs to move a large proprietary dataset through several services and storage systems. It is particularly relevant when throughput matters, but engineers also need practical ways to inspect live data and diagnose failures before they spread across a large processing run.