When something fails
The pipeline finished successfully and the data lake is empty
Section titled “The pipeline finished successfully and the data lake is empty”The most common symptom, and almost always configuration — not an error.
The pipeline does not fail for lack of configuration. With no database marked, the snapshot is exported, the Parquet files land in S3, and the copy job ignores them silently.
In order of likelihood:
| Cause | How to confirm |
|---|---|
| No database marked | The console overview shows the alert |
| A new database appeared in the cluster | Same, with the database name |
| Snapshot events turned off | Same — in this case not even the export runs |
| Table filter excluded everything | Under Bancos (Databases), see how many tables are marked |
The copy job log also records what was ignored:
IGNORADO: 56 arquivos do banco "pagila" -- ele nao esta marcadopara ingestao, entao nada dele chega ao datalake.A table disappeared from the data lake
Section titled “A table disappeared from the data lake”If the table existed and stopped appearing, check under Partições (Partitions) whether there is a configuration for it with replace the direct copy on and the partition disabled. In that combination, the copy skips the table and the partitioner does not run — nobody writes it.
Turning the partition back on, or turning off “replace the direct copy”, fixes it.
A run failed
Section titled “A run failed”Under Execuções (Runs), open the run to see the stages in order. The stage that failed carries the error message and the log.
The most common causes:
- Memory during partitioning — a table with many distinct values in the partition column. See how many levels to use.
- Cross-account permission — if step 3 of the cross-account installation was not done, the export fails when writing to the bucket.
- Timeout — each job has one hour. A very large table with sorting configured is the most likely suspect.
A table that fails does not bring the others down: the pipeline tolerates isolated failures in partitioning and the crawler catalogs what was written.
A strange column in the schema
Section titled “A strange column in the schema”If a partition_0 column shows up with values 1, 2, 3, the installation
is on a version earlier than the one that flattens the RDS parts directory.
Update the package and reprocess.
The same rows appear twice
Section titled “The same rows appear twice”Two possible causes:
Table partitioned in Postgres. RDS exports the parent table and each child
partition. Current versions detect and ignore the children; if you see tables
like payment_p2024_01 next to payment, update the package.
Partition without “replace the direct copy”. By default, a partitioned table appears twice on purpose — the whole copy and the partitioned one. Turn the option on if you want only the partitioned one.
Login does not work
Section titled “Login does not work”Cognito refuses callback URLs on http, except localhost. If the console serves
over http, login fails before it starts. Run setup-https.sh.
If the browser warns that the certificate is not trusted, that is the
self-signed certificate — it works, but to remove the warning use your own
certificate through CERTIFICATE_ARN.
I cannot reach the console
Section titled “I cannot reach the console”The load balancer is internal, on purpose: the console is not exposed to the internet. You reach it from inside the VPC, over VPN, or through a Session Manager tunnel.
Knowing which version produced a result
Section titled “Knowing which version produced a result”Each job records its own version when it starts, and the run stores that. Under Execuções (Runs), the version of each component appears in the details — useful when behavior changed between updates.