Requirements
Data Pump runs entirely in your AWS account. Before installing, three things must exist.
It is worth settling this before the handoff: the VPC, the bucket and the source accounts go into the template of your package, and changing them later requires a new package. See the handoff.
A VPC with private subnets
Section titled “A VPC with private subnets”The jobs run as Fargate tasks in private subnets, and the console sits behind an internal load balancer. The subnets need outbound access to the internet — through a NAT Gateway or VPC endpoints — to reach S3, DynamoDB, ECR and the AWS endpoints.
If you use VPC endpoints instead of NAT, you need: s3 (gateway), dynamodb
(gateway), and the interface ones for ecr.api, ecr.dkr, logs, sts and
secretsmanager.
An RDS cluster with snapshots
Section titled “An RDS cluster with snapshots”Data Pump reads snapshots — automatic or manual, from Aurora or RDS Postgres. If automatic backups are on (the default), there is already a snapshot to process.
The cluster can live in another AWS account. In that case, see cross-account database — the installation has one extra step.
Permission to create the stack
Section titled “Permission to create the stack”Whoever runs the installer must be able to create, through CloudFormation:
| Service | What for |
|---|---|
| S3 | The data lake bucket and a CloudFormation staging bucket |
| IAM | Roles for the tasks, the export and the crawler |
| KMS | The key that encrypts the snapshot export |
| ECS | Cluster, task definitions and the console service |
| Step Functions | The pipeline state machine |
| Lambda | The event and crawler functions |
| DynamoDB | Five configuration and state tables |
| Glue | The catalog database and the crawler |
| Cognito | The user pool that authenticates the console |
| SNS | The RDS event and completion topics |
| EC2/ELB | Security groups and the internal load balancer |
The installer requires CAPABILITY_NAMED_IAM, because the roles have fixed
names.
Data Pump creates nothing that gets expensive while idle. What weighs is proportional to volume:
- Fargate — the jobs run for a few minutes at every snapshot
- S3 — the data lake Parquet, plus the raw export if you do not clear it
- Console — one small task always up, plus the load balancer
- Athena — charged per data scanned, and that is what partitioning reduces
The snapshot itself you already pay for: AWS takes it either way.