Running Your Own Cumulus Aggregator

You only need to deploy your own Aggregator if you aren’t using the default Aggregator run by BCH.

Deployment Prerequisites

Requirements

Accessing via external URL

If you need your instance of the Aggregator to be accessible outside your organization, follow these steps prior to trying to deploy the Aggregator:

  • Register an appropriate domain via AWS Route 53
  • Update the Domain parameter in the template.certificate.yaml to match that domain
  • Deploy the certificate & hosted zone with 
    sam deploy --guided --stack-name=cumulus-dns -t template.hostedzone.yaml
  • In the Route 53 dashboard, under hosted zones in the sidenav, click on the zone named for the domain you registered in the table.
    • Find the row with the ‘Type’ column set to NS. In the ‘Value/Route traffic to’ column, copy the four DNS servers that are automatically assigned into a text editor.
  • In the Route 53 dashboard, clink on registered domains in the sidenav. Find the domain you registered, and update its DNS server info to match the four DNS servers you copied above. Publish those changes.
  • After the DNS info is published, the Certificate should finish deploying - this may take 5-10 minutes the first time you do it as these updates bounce around the internet. It’s amazing that this works! It’s truly indistinguishable from magic at this point. 🧙
    • Subsequent changes should not require manual DNS intervention unless you delete the cloudformation deployment.
  • Get the Aggregator Certificate ARN for the environment(s) you’re interested in and the Hosted Zone ID from the deployment output.
    • If you need these later, you can find them by looking in:
      • The Cloudformation dashboard, by selecting the ‘cumulus-dns’ template (or whatever you renamed it to), selecting the ‘Resources’ tab, and getting the physical ID.
      • The AWS Certificate Manager (ACM) dashboard, where they will be listed by domain name - click into the one you’re interested to find the ARN value.

Accessing via Amazon autogenerated id

If your Aggregator does not need public external access, remove/comment the Domain resource under SiteApiGateway inside of template.yaml

samconfig.toml setup

Once you’ve done this, we strongly recommend configuring your samconfig.toml to help you manage environments. To do this:

  • Run sam deploy --guided. This will ask you several questions, which you can accept the defaults/say yes to. What we’re looking for here is for SAM to identify the bucket it will use for deploying artifacts, which is usually a bucket it will manage for you separate from your other S3 infrastructure. When you are asked if you want to deploy, select no - we’ll want to edit the defaults next.
  • Open the samconfig.toml you just generated. We’ll want to take the default config and create one copy of it per environment, making sure they have unique stack names. In this config, we’ll want to provide the appropriate values for each of the parameters in the template, and name the stage for the environment. This is a pretty dense explanation! Here’s an example of what a configuration should look like for a dev and staging environment: ```toml [dev] [dev.deploy] [dev.deploy.parameters] stack_name = “cumulus-aggregator-dev” s3_bucket = “aws-sam-cli-managed-default-samclisourcebucket-[SAM generated bucket id]” s3_prefix = “cumulus-aggregator-dev” region = “us-east-1” confirm_changeset = true capabilities = “CAPABILITY_IAM” parameter_overrides = “RetentionTime=7 DeployStage="dev" AggregatorDomainName="dev.aggregator.yourdomain.org" AggregatorHostedZoneID="1234567890ABCDEFHIJ" AggregatorCertArn="arn:aws:acm:us-east-1:1234567890:certificate/12345678-90AB-CDEF-0123-4567890ABCD" DashboardApiDomainName="dev.api.yourdomain.org" DashboardApiCertArn="arn:aws:acm:us-east-1:1234567890:certificate/11112222-3333-CDEF-0123-FECDBA0987654"” image_repositories = []

[staging] [staging.deploy] [staging.deploy.parameters] stack_name = “cumulus-aggregator-staging” s3_bucket = “aws-sam-cli-managed-default-samclisourcebucket-[SAM generated bucket id]” s3_prefix = “cumulus-aggregator-staging” region = “us-east-1” confirm_changeset = true capabilities = “CAPABILITY_IAM” parameter_overrides = “RetentionTime=7 DeployStage="staging" AggregatorDomainName="staging.aggregator.yourdomain.org" AggregatorHostedZoneID="1234567890ABCDEFHIJ" AggregatorCertArn="aarn:aws:acm:us-east-1:1234567890:certificate/12345678-90AB-CDEF-0123-FECDBA0987654" DashboardApiDomainName="staging.api.yourdomain.org" DashboardApiCertArn="arn:aws:acm:us-east-1:1234567890:certificate/11112222-3333-CDEF-0123-FECDBA0987654"” image_repositories = []

Note that you'll use the same hosted zone for each stage, but a different certificate arn, one per environment. If you skipped external domain setup, just omit those entries


If for some reason you don't want to use the samconfig.toml, you could instead do these things, but it's much more error-prone, so please be careful:
- Add default parameters for `RetentionTime`, `AggregatorCertArn`, `AggregatorHostedZoneID`, `DashboardApiCertArn`, `DeployStage`, and if using, `AggregatorHostedDomainName` and `DashboardApiDomainName`
- Create environment variables and assign values to those parameters. Then, whenever you run `sam`, provide a parameter override in place of `--config-env`, with `--parameter-overrides AggregatorCertArn=$CUMULUS_AGG_CERT_ARN AggregatorHostedZoneID=$CUMULUS_AGG_ZONE_ID` (and deploy/domain if using.


## Deployment

Assuming you have done the above configuration, you can deploy the Aggregator with the following command:

```bash
sam build && sam deploy --config-env [stagename]

This will show you a list of changes to your environment and ask you to confirm. Responding to the prompt with ‘Y’ will kick off the deployment.

Local Development

If you are working locally and want to test a specific lambda function, first make sure that Docker is running.

You’ll want to use SAM to build an image, and then use a specific lambda function with an event file to mimic the real traffic. The following one-liner accomplishes this:

sam build && sam local invoke FetchUploadUrlFunction --event events/event-fetch-upload-url.json

Cloud Development

Assuming you have already generated a samconfig.toml, you can have the SAM CLI use this to hot deploy changes to AWS for live debugging/generating realistic event files. Only do this in dev! To configure this, you can copy your deploy config in samconfig.toml, and add one value to capabilities, CAPABILITY_NAMED_IAM. So that would look like this:

[dev]
[dev.deploy]
[dev.deploy.parameters]
#your existing config from earlier

[dev.sync]
[dev.sync.parameters]
stack_name = "cumulus-aggregator-dev"
s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-[SAM generated bucket id]"
s3_prefix = "cumulus-aggregator-dev"
region = "us-east-1"
confirm_changeset = true
capabilities = "CAPABILITY_IAM CAPABILITY_NAMED_IAM"
parameter_overrides = "RetentionTime=7 DeployStage=\"dev\" AggregatorDomainName=\"dev.aggregator.yourdomain.org\" AggregatorHostedZoneID=\"1234567890ABCDEFHIJ\" AggregatorCertArn=\"arn:aws:acm:us-east-1:1234567890:certificate/12345678-90AB-CDEF-0123-4567890ABCD\" DashboardApiDomainName=\"dev.api.yourdomain.org\" DashboardApiCertArn=\"arn:aws:acm:us-east-1:1234567890:certificate/11112222-3333-CDEF-0123-FECDBA0987654\""
image_repositories = []

To run the application in this mode, use the following command:

sam sync --config-env dev --watch

If you’d rather sync on demand, just omit the –watch parameter.

sam sync --config-env dev

After the build completes, you should be able to test with real events, either using the contents of the scripts/ directory, hitting the API gateway, or via another means of your choosing.

It might be more useful to live tail the logs, rather than waiting for the CloudFormation dashboard to update. To do this for a specific Lambda, for example, use the sam logs command:

sam logs -n PowersetMergeFunction --stack-name cumulus-aggregator-dev --tail

There is more advanced log filtering available - see the SAM Logs documentation

When you’re finished, you can clean up your deployment with the following command:

aws cloudformation delete-stack --stack-name cumulus-aggregator-dev

If your bucket has data in it, you need to delete the bucket contents before removing your deployment:

aws s3 rm s3://cumulus-aggregator-site-counts-dev --recursive && aws cloudformation delete-stack --stack-name cumulus-aggregator-dev