How it works

Two things have to happen before the data lake receives data: configuration and execution. Configuration is done once, in the console. Execution happens at every snapshot, on its own.

The distinction matters because the pipeline completes successfully even without complete configuration. With no database marked for ingestion, the snapshot is exported, the Parquet files land in S3, and the copy job ignores them silently. There is no error anywhere — which is why the console overview lists what is still missing.

The path of a table

1. AWS takes the snapshot
   automatic (backup window) or manual

2. RDS exports it as Parquet
   exported/{cluster}/{export}/pagila/public.payment/1/*.parquet

3. Data Pump copies what you marked, under the name you gave
   catalog/{cluster}/pagila.pagamentos/*.parquet

4. Repartitions the configured tables
   catalog/{cluster}/pagila.pagamentos/ano=2024/mes=03/*.parquet

5. Glue catalogs it
   SELECT * FROM datalake.pagila_pagamentos WHERE ano = 2024

Steps 3 and 4 are where your decisions come in: which databases and tables (set up ingestion) and how to repartition (partitions).

The trigger

When a snapshot is ready, RDS publishes an event. A Lambda function receives it, checks whether that cluster is subscribed and, if so, starts the export. When the export finishes, the same path fires the processing pipeline.

The Lambda is the same in both moments — it tells them apart by the event type. Events that do not matter (creation started, copy, deletion) are logged and ignored.

The pipeline

Four jobs in sequence, all as Fargate tasks from the same image:

1. Clears the previous catalog — catalog/{cluster}/ is rebuilt at every run, so it always reflects the current state of the database.
2. Copies the files — only from the marked databases, applying table filters and aliases.
3. Repartitions — one task per configured table, in parallel.
4. Clears the raw export — optional.

At the end, the Glue crawler catalogs the result and a notification is published to an SNS topic, for anyone who wants to chain something afterwards.

Where the data lives in S3

exported/{cluster}/{export}/{banco}/{schema}.{tabela}/{parte}/*.parquet
   └── what RDS produces: everything the snapshot holds

catalog/{cluster}/{alias}.{tabela}/*.parquet
   └── what Glue catalogs: only what you marked

longterm/{cluster}/{alias}/{tabela}/
   └── history that survives a purge in the source database

The difference between catalog/ and longterm/ is what allows you to keep history after purging old data from Postgres: catalog/ is recreated at every run; longterm/ accumulates.

Tables already partitioned in Postgres

If a table is natively partitioned in Postgres, RDS exports the parent table and each child partition as separate tables. The parent already contains every row, so copying both would duplicate each row in the data lake.

Data Pump detects and ignores the children, using the export’s own metadata — not the name pattern. New partitions created in the database are recognized on their own; there is no list to maintain.