[Forecasted Pipeline] Persist forecasted grants in DynamoDB

[jeffsmohan]

Why is this issue important?

We want forecasted grants available in our dataset (alongside active grants) so we can integrate them into the product for our users.

Current State

The PersistGrantsGovXMLDB lambda only handles active grants today, not forecasted grants.

Expected State

We want the PersistGrantsGovXMLDB lambda to also take the XML snippets for forecasted grants (as produced by the change in #330) and persist those into our DynamoDB database alongside active grants.

Implementation Plan

Blocked on #3212 and #3213
Edit (authored by Marissa): This ticket isn't blocked by #3212 or #3213 if flags are used in similar fashion to IS_FORECASTED_GRANTS_ENABLED in this PR. Ideally, the same flag (IS_FORECASTED_GRANTS_ENABLED) would be used.

1. Update the type opportunity grantsgov.OpportunitySynopsisDetail_1_0 in handler.go with a new type that can handle both grant opportunities and forecasted grants, with a common interface for logging. (The rest of the DynamoDB update should work, as it's agnostic to the shape of the data.)
2. Create a second lambda_function in terraform/main.tf to invoke the lambda for forecasted grants as well:

  lambda_function {
    lambda_function_arn = module.PersistGrantsGovXMLDB.lambda_function_arn
    events              = ["s3:ObjectCreated:*"]
    filter_suffix       = "/grants.gov/v2.OpportunityForecastDetail_1_0.xml"
  }

3. Set the is_forecasted flag based on whether this is an opportunity or forecasted opportunity

Important: Both source XML record types (OpportunitySynopsisDetail_1_0 and OpportunityForecastDetail_1_0) provide a PostDate value, which has a different meaning depending on the source type:

• In OpportunitySynopsisDetail_1_0, the PostDate value represents the date when the finalized grant opportunity was published.
  • This value maps to the OpportunityMilestones.post_date field (at .opportunity_milestones.post_date of Grant objects) in our OpenAPI schema.
• In OpportunityForecastDetail_1_0, the PostDate value represents the date when the forecasted grant opportunity was published.
  • This value maps to the (new) OpportunityMilestones.forecast_creation_date field (at .opportunity_milestones.forecast_creation_date of Grant objects) in our OpenAPI schema.
• i.e. OpportunityForecastDetail_1_0 -> EstimatedSynopsisPostDate is basically the same (albeit forecasted/estimated) value as OpportunitySynopsisDetail_1_0 -> PostDate
  • Both map to the same OpportunityMilestones.post_date field.
  • When both values are available but differ, the value from OpportunitySynopsisDetail_1_0 takes precedence.

Therefore, we will need a way to disambiguate PostDate values between these two source record types in our DynamoDB records.

[jeffsmohan]

@TylerHendrickson I'd love a quick review of the implementation plan before moving this to ready, thanks!

[masimons]

@TylerHendrickson @jeffsmohan Can I crash that^ review if it hasn't happened yet? :)
editing your comment for fun lol