Skip to content

Commit

Permalink
Release 3.10.0 (2024-07-13)
Browse files Browse the repository at this point in the history 
### Features

* [IsolateBuyerAndSellerCodeExecution] Add wrapper for buyer's reportWin() udf
* [reporting] Enable reportResult execution for topLevelSeller
* Add chaffing feature flags in SFE/BFE.
* Add config flag enable_private_aggregation_generation to auction service's runtime_config
* Add distribution of IGs related metrics
* add EventMessage to log context
* Add request creation timestamp to ProtectedAuctionInput
* Add request/response to ExecuteInternal on async clients
* Add support for collecting metrics from the Roma callbacks.
* Add tee-container-log-redirect option in terraform
* Bash script for ASG and Cloud Map Custom HealthChecks
* Bid Currency Support for Top-Level Seller in Server-Orchestrated Multi-Seller Auctions
* configured public key urls are verified against an explicit allowlist
* Consented request replace enable_adtech_code_logging in Bidding Server
* Create inference model store for model management
* Create logging library for inference consented logs
* Debug Reporting for Bid Currency
* decrease aws/build_and_test duration by ~75%
* Feature Flag for TLS in Service Mesh
* Force the ML model reset with the probability of 0.1%
* Forward per request consented debugging config to roma callback
* Handle chaff and new request format on BFE
* Implement AWS Cloud Un-Map
* Implement new SFE <> BFE request format for chaffing
* Implement the probabilistic model reset for PyTorch
* Implement the probabilistic model reset for TensorFlow.
* Integrate with Trusted KV Server in Mesh
* Load Test Flag for AWS
* Log consented debugging information in inference sidecar
* log EventMessage in servers
* log udf log in EventMessage for non_prod debug_info
* Remove enableAdtechCodeLogging flag value
* Remove Envoy Access Logging
* Route consented inference requests to a consented model store
* send chaff requests from SFE
* Service Mesh in AWS
* Support CPU isolation in the inference sidecar
* Unpad KV server responses
* update code/cloud build to use tags
* Upgrade AWS Provider for Terraform from v3.xx to v4.xx
* Use gRPC for AWS Service Mesh Envoy HCs

### Bug Fixes

* [IsolateBuyerAndSellerCodeExecution] Add a new class for buyer's reporting code fetch and load
* [IsolateBuyerAndSellerCodeExecution] Add a new code wrapper with only scoreAd and reportResult
* [IsolateBuyerAndSellerCodeExecution] Add config flag to enable seller and buyer code isolation
* [IsolateBuyerAndSellerCodeExecution] Modify seller_udf_manager to fetch and load buyer udfs
* [IsolateBuyerAndSellerCodeExecution] Refactor the code fetch files.
* Add BidCurrency SUT to Pre-Submit
* add check back in default grpc client
* Add DebugInfo pointer for debugging log into RomaRequestContext
* Add generation_id to chaff requests
* Adds domain equality validations to buyer reporting UDF URL
* Align plaintext buyer request for basic SUT with encrypted version
* clean up log verbosity 3
* do not try to impersonate service accounts if TEST_MODE=true
* Eliminate Terraform Error Message about empty Authority Field
* Ensure instance id is set in logs on AWS when not using mesh.
* Execute Callback for empty HTTP request vector
* make num_chaff_requests not have a static lower bound
* Make terraform for aws delete cloud maps without error
* Redirect misleading log to /dev/null
* scorecard.yaml version updates
* shuffle request order (real and chaff) on SFE
* Update setup_2 and demo terraform configs to valid state
* Upgrades google terraform plugin to 5.31.0 to fix crash

Bug: N/A
GitOrigin-RevId: 0f8502ab785dc370b2e42ddb8cfadf9aa9c8db1f
Change-Id: I5aa982303ee547e1dab0c3906cfe8850d27fc5a9
  • Loading branch information
@akundla-google
akundla-google committed Jul 18, 2024
1 parent 332f7e1 commit e40a4fc
Show file tree
Hide file tree
Showing 118 changed files with 3,100 additions and 522 deletions.
3 changes: 3 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,6 @@ terraform.rc
docker-buildx-*.log

core

# Local Server built/temporary files for Protected Audience integration tests
tools/debug/pa-integration-tests/local-servers/*.pem
74 changes: 74 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,80 @@

All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.

## 3.10.0 (2024-07-13)


### Features

* [IsolateBuyerAndSellerCodeExecution] Add wrapper for buyer's reportWin() udf
* [reporting] Enable reportResult execution for topLevelSeller
* Add chaffing feature flags in SFE/BFE.
* Add config flag enable_private_aggregation_generation to auction service's runtime_config
* Add distribution of IGs related metrics
* add EventMessage to log context
* Add request creation timestamp to ProtectedAuctionInput
* Add request/response to ExecuteInternal on async clients
* Add support for collecting metrics from the Roma callbacks.
* Add tee-container-log-redirect option in terraform
* Bash script for ASG and Cloud Map Custom HealthChecks
* Bid Currency Support for Top-Level Seller in Server-Orchestrated Multi-Seller Auctions
* configured public key urls are verified against an explicit allowlist
* Consented request replace enable_adtech_code_logging in Bidding Server
* Create inference model store for model management
* Create logging library for inference consented logs
* Debug Reporting for Bid Currency
* decrease aws/build_and_test duration by ~75%
* Feature Flag for TLS in Service Mesh
* Force the ML model reset with the probability of 0.1%
* Forward per request consented debugging config to roma callback
* Handle chaff and new request format on BFE
* Implement AWS Cloud Un-Map
* Implement new SFE <> BFE request format for chaffing
* Implement the probabilistic model reset for PyTorch
* Implement the probabilistic model reset for TensorFlow.
* Integrate with Trusted KV Server in Mesh
* Load Test Flag for AWS
* Log consented debugging information in inference sidecar
* log EventMessage in servers
* log udf log in EventMessage for non_prod debug_info
* Remove enableAdtechCodeLogging flag value
* Remove Envoy Access Logging
* Route consented inference requests to a consented model store
* send chaff requests from SFE
* Service Mesh in AWS
* Support CPU isolation in the inference sidecar
* Unpad KV server responses
* update code/cloud build to use tags
* Upgrade AWS Provider for Terraform from v3.xx to v4.xx
* Use gRPC for AWS Service Mesh Envoy HCs


### Bug Fixes

* [IsolateBuyerAndSellerCodeExecution] Add a new class for buyer's reporting code fetch and load
* [IsolateBuyerAndSellerCodeExecution] Add a new code wrapper with only scoreAd and reportResult
* [IsolateBuyerAndSellerCodeExecution] Add config flag to enable seller and buyer code isolation
* [IsolateBuyerAndSellerCodeExecution] Modify seller_udf_manager to fetch and load buyer udfs
* [IsolateBuyerAndSellerCodeExecution] Refactor the code fetch files.
* Add BidCurrency SUT to Pre-Submit
* add check back in default grpc client
* Add DebugInfo pointer for debugging log into RomaRequestContext
* Add generation_id to chaff requests
* Adds domain equality validations to buyer reporting UDF URL
* Align plaintext buyer request for basic SUT with encrypted version
* clean up log verbosity 3
* do not try to impersonate service accounts if TEST_MODE=true
* Eliminate Terraform Error Message about empty Authority Field
* Ensure instance id is set in logs on AWS when not using mesh.
* Execute Callback for empty HTTP request vector
* make num_chaff_requests not have a static lower bound
* Make terraform for aws delete cloud maps without error
* Redirect misleading log to /dev/null
* scorecard.yaml version updates
* shuffle request order (real and chaff) on SFE
* Update setup_2 and demo terraform configs to valid state
* Upgrades google terraform plugin to 5.31.0 to fix crash

## 3.9.0 (2024-06-25)


Expand Down
8 changes: 4 additions & 4 deletions WORKSPACE
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,11 +85,11 @@ http_archive(

http_archive(
name = "service_value_key_fledge_privacysandbox",
# commit 89f678982d5cc1bcfde354980ed9380239f54b96 2024-03-22
sha256 = "dfa9da5e5b2b71aa706781e1edc0c6f4e752d4ebf30f82624f711e776553fbe2",
strip_prefix = "protected-auction-key-value-service-89f678982d5cc1bcfde354980ed9380239f54b96",
# commit d186bcf8d643888fac81784f9c91ffb2fb48791c 2024-04-19
sha256 = "53521403ffbe18a306417b8e7b41c2a13127ba0892ad921229861a451ad2045e",
strip_prefix = "protected-auction-key-value-service-d186bcf8d643888fac81784f9c91ffb2fb48791c",
urls = [
"https://github.com/privacysandbox/protected-auction-key-value-service/archive/89f678982d5cc1bcfde354980ed9380239f54b96.zip",
"https://github.com/privacysandbox/protected-auction-key-value-service/archive/d186bcf8d643888fac81784f9c91ffb2fb48791c.zip",
],
)

Expand Down
40 changes: 27 additions & 13 deletions production/deploy/aws/terraform/environment/demo/buyer/buyer.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@
locals {
region = "" # Example: ["us-central1", "us-west1"]
environment = "" # Must be <= 3 characters. Example: "abc"

# Set to true for service mesh, false for Load balancers
use_service_mesh = false
# Whether to use TLS-encrypted communication between service mesh envoy sidecars. Defaults to false, as comms take place within a VPC and the critical payload is HPKE-encrypted, and said encryption is terminated inside a TEE.
use_tls_with_mesh = false
}

provider "aws" {
Expand All @@ -27,7 +32,7 @@ module "buyer" {
source = "../../../modules/buyer"
environment = local.environment
region = local.region
enclave_debug_mode = false # Example: false
enclave_debug_mode = false # Example: false, set to true for extended logs
root_domain = "" # Example: "bidding1.com"
root_domain_zone_id = "" # Example: "Z1011487GET92S4MN4CM"
certificate_arn = "" # Example: "arn:aws:acm:us-west-1:57473821111:certificate/59ebdcbe-2475-4b70-9079-7a360f5c1111"
Expand All @@ -46,21 +51,30 @@ module "buyer" {
bidding_autoscaling_desired_capacity = 3 # Example: 3
bidding_autoscaling_max_size = 5 # Example: 5
bidding_autoscaling_min_size = 1 # Example: 1
country_for_cert_auth = "" # Example: "US"
business_org_for_cert_auth = "" # Example: "Privacy Sandbox"
state_for_cert_auth = "" # Example: "California"
org_unit_for_cert_auth = "" # Example: "Bidding and Auction Servers"
locality_for_cert_auth = "" # Example: "Mountain View"
kv_server_virtual_service_name = ""
use_service_mesh = local.use_service_mesh
use_tls_with_mesh = local.use_tls_with_mesh

runtime_flags = {
BIDDING_PORT = "50051" # Do not change unless you are modifying the default GCP architecture.
BUYER_FRONTEND_PORT = "50051" # Do not change unless you are modifying the default GCP architecture.
BFE_INGRESS_TLS = "false" # Do not change unless you are modifying the default GCP architecture.
BIDDING_EGRESS_TLS = "true" # Do not change unless you are modifying the default GCP architecture.
COLLECTOR_ENDPOINT = "127.0.0.1:4317" # Do not change unless you are modifying the default GCP architecture.
AD_RETRIEVAL_KV_SERVER_EGRESS_TLS = "true" # Do not change unless you are modifying the default GCP architecture.
KV_SERVER_EGRESS_TLS = "true" # Do not change unless you are modifying the default GCP architecture.
BIDDING_PORT = "50051" # Do not change unless you are modifying the default AWS architecture.
BUYER_FRONTEND_PORT = "50051" # Do not change unless you are modifying the default AWS architecture.
BFE_INGRESS_TLS = "false" # Do not change unless you are modifying the default AWS architecture.
BIDDING_EGRESS_TLS = "true" # Do not change unless you are modifying the default AWS architecture.
COLLECTOR_ENDPOINT = "127.0.0.1:4317" # Do not change unless you are modifying the default AWS architecture.
AD_RETRIEVAL_KV_SERVER_EGRESS_TLS = "true" # Do not change unless you are modifying the default AWS architecture.
KV_SERVER_EGRESS_TLS = "true" # Do not change unless you are modifying the default AWS architecture.

ENABLE_BIDDING_SERVICE_BENCHMARK = "" # Example: "false"
BIDDING_SERVER_ADDR = "" # Example: "dns:///bidding1.com:443"
BUYER_KV_SERVER_ADDR = "" # Example: "https://kvserver.com/trusted-signals"
TEE_AD_RETRIEVAL_KV_SERVER_ADDR = "" # Example: "xds:///ad-retrieval-host"
TEE_KV_SERVER_ADDR = "" # Example: "xds:///kv-service-host"
BIDDING_SERVER_ADDR = local.use_service_mesh ? "" /* Example for Mesh: "dns:///bidding-buyer1-prod-appmesh-virtual-service.bidding1.com:50051" */ : "" /* Example for internal Load Balancers: "dns:///bidding-buyer1-prod.bidding1.com:443" */
GRPC_ARG_DEFAULT_AUTHORITY = local.use_service_mesh ? "" /* Example for Mesh: "bidding-buyer1-${local.environment}-appmesh-virtual-service.bidding1.com" */ : "PLACEHOLDER" # "PLACEHOLDER" is a special value that will be ignored by B&A servers. Leave it unchanged if running with Load Balancers.
BUYER_KV_SERVER_ADDR = "" # Example: "https://kvserver.com/trusted-signals"
TEE_AD_RETRIEVAL_KV_SERVER_ADDR = "" # Example: "xds:///ad-retrieval-host"
TEE_KV_SERVER_ADDR = "" # Example: "xds:///kv-service-host"
AD_RETRIEVAL_TIMEOUT_MS = "60000"
GENERATE_BID_TIMEOUT_MS = "" # Example: "60000"
BIDDING_SIGNALS_LOAD_TIMEOUT_MS = "" # Example: "60000"
Expand Down Expand Up @@ -89,7 +103,7 @@ module "buyer" {
# "prepareDataForAdsRetrievalJsUrl": "",
# "prepareDataForAdsRetrievalWasmHelperUrl": "",
# }"
JS_NUM_WORKERS = "" # Example: "48" Must be <=vCPUs in bidding_enclave_cpu_count.
JS_NUM_WORKERS = "" # Example: "48" Must be <=vCPUs in bidding_enclave_cpu_count, and should be equal for best performance.
JS_WORKER_QUEUE_LEN = "" # Example: "100".
ROMA_TIMEOUT_MS = "" # Example: "10000"
# This flag should only be set if console.logs from the AdTech code(Ex:generateBid()) execution need to be exported as VLOG.
Expand Down
Loading

0 comments on commit e40a4fc

Please sign in to comment.