Release 4.0.0 (2024-09-09)

### ⚠ BREAKING CHANGES

* Move serialization of bidding signals per IG from Bidding to BFE

### Features

* [PAS threat mitigation] Add support for egressVector and temporaryUnlimittedEgressVector by replacing the $extraArgs
* [Private Aggregation] Add all the contributions from Auction Service to AuctionResult
* [Private Aggregation] Add integration test to verify parsing of PAAPI event
* [Private Aggregation] Add PrivateAggregateReportingResponse to AuctionResult proto
* [Private Aggregation] Convert 128 bit bucket into array of 64 bit integer in wrapper
* [Private Aggregation] Convert bucket offset object to SignalBucket
* [Private Aggregation] Convert value object in contribution to SignalValue
* Add a script to freeze a given TensorFlow model
* Add a script to list all TensorFlow ops
* Add a script to list all Torchscript ops
* Add AppendAdEventContributionsToPaggResponse with SignalValue only
* Add common utils to create test private aggregation contribution and response
* Add HandlePrivateAggregateReporting with support of SignalValue only
* Add HandlePrivateAggregationContributions, iterate over AdWithBids, filter contributions, post processing, test
* Add Inference Request count, duration and failure metric partitioned by model path for Tensorflow and Pytorch sidecars.
* Add metrics for inference model fetching and registration
* Add parser function for Private Aggregation's Signal Bucket Object
* Add parser function for Private Aggregation's Signal Value Object
* Add ToBaseValue Helper Function to convert corresponding base value strings to BaseValue Enum
* Add ToEventTypeString for converting EventType enum into its corresponding string in JSON object returned by ROMA
* Add utility function to calculate Signal Bucket's final value post auction ([3da9ab5]( )), closes [1#L192]( )
* Add utility function to parse and return BucketOffset from rapidjson document
* convert all remaining docker images into OCI
* Create declarative spec for generateBid
* Enable Protected App Signals by Default
* Freeze TorchScript models before serving
* Implement aggregated Error Reporting for Tensorflow and Pytorch sidecars.
* Implement ParseAndProcessContribution with SignalValue only
* Move serialization of bidding signals per IG from Bidding to BFE
* write event message to gcs for consented request
* write event message to s3 in aws for consented request

### Bug Fixes

* Add handling when rejection reason is specified as base value but not available and statuscode documentation for HandlePrivateAggregationValuePostAuction
* Changed base value string to be converted in ToBaseValue function and test cases
* Changed the input format of HandlePrivateAggregationReporting, added support for parsing PAgg bucket,  and add the logic for getting required BaseValues (winning-bid, highest-scoring-other-bid, rejection-reason).
* Enable threat mitigation with seller and buyer code isolation
* Populate temp rc file for inference sidecar
* Populate temp rc file for inference sidecar
* Populate temp rc file for inference sidecar
* Refactored conversion of private aggregation value post auction into returning absl::StatusOr<int> instead of implicitly editing PrivateAggregationValue that was passed in.
* Revert wrk2 test runner to an earlier version
* undo rules_oci migration to fix hash stability issues

Bug: N/A
GitOrigin-RevId: 9c81c00d35f3c5821bc98a289788ec2740ef2055
Change-Id: I274327201feade9fb9b5df1bd26d21cc2dff3533

.bazelrc

@@ -32,13 +32,14 @@ build:clang --cxxopt=-fsized-deallocation
 build:clang --host_cxxopt=-fsized-deallocation
 build:clang --cxxopt=-fnew-alignment=8
 build:clang --host_cxxopt=-fnew-alignment=8
+build:clang --per_file_copt=api/udf/.*@-fconstexpr-steps=1271242
 
 # Flag compiler warnings as errors.
 build:cpp_no_warn --copt=-Werror
 # Ignore warnings from Roma, zlib, differential privacy repo etc.
 build:cpp_no_warn --per_file_copt=.*external/.*@-Wno-error
 # Ignore deprecated declarations warnings from Roma, ProtectedAudience, Telemetry
-build:cpp_no_warn --per_file_copt=services/.*@-Wno-macro-redefined,-Wno-deprecated-declarations
+build:cpp_no_warn --per_file_copt=(services|api/udf)/.*@-Wno-macro-redefined,-Wno-deprecated-declarations
 build:cpp_no_warn --per_file_copt=tools/secure_invoke.*@-Wno-macro-redefined,-Wno-deprecated-declarations
 # Telemetry has some unchecked results after registering observers.
 build:cpp_no_warn --per_file_copt=services/.*_main.cc@-Wno-unused-result

CHANGELOG.md

@@ -2,6 +2,183 @@
 
 All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.
 
+## 4.0.0 (2024-09-09)
+
+
+### ⚠ BREAKING CHANGES
+
+* Move serialization of bidding signals per IG from Bidding to BFE
+
+### Features
+
+* [IsolateBuyerAndSellerCodeExecution] Add wrapper for buyer's reportWin() udf
+* [PAS threat mitigation] Add support for egressVector and temporaryUnlimittedEgressVector by replacing the $extraArgs
+* [Private Aggregation] Add all the contributions from Auction Service to AuctionResult
+* [Private Aggregation] Add integration test to verify parsing of PAAPI event
+* [Private Aggregation] Add PrivateAggregateReportingResponse to AuctionResult proto
+* [Private Aggregation] Convert 128 bit bucket into array of 64 bit integer in wrapper
+* [Private Aggregation] Convert bucket offset object to SignalBucket
+* [Private Aggregation] Convert value object in contribution to SignalValue
+* [PrivateAggregation] Add PrivateAggregateReportingResponse in Auction Service response
+* [PrivateAggregation] Append private aggregation wrapper to existing seller's wrapper
+* [reporting] Enable reportResult execution for topLevelSeller
+* Add `common.privateAggregation.createContribution` function for creating a contribution object.
+* Add `common.privateAggregation.ReservedEvent` object that holds reserved events string constants.
+* Add a script to freeze a given TensorFlow model
+* Add a script to list all TensorFlow ops
+* Add a script to list all Torchscript ops
+* Add AppendAdEventContributionsToPaggResponse with SignalValue only
+* Add auction_service.privateAggregation.contributeToHistogram/OnEvent
+* Add bucket protos (Bucket128Bit, BucketOffset, SignalBucket, PrivateAggregationBucket) and add bucket and value fields to PrivateAggregateContribution
+* Add chaffing feature flags in SFE/BFE.
+* Add checksum capability to model loading
+* Add common utils to create test private aggregation contribution and response
+* Add config flag enable_private_aggregate_reporting to bidding service's runtime_config
+* Add config flag enable_private_aggregation_generation to auction service's runtime_config
+* Add ConvertSignalValueToInt for calculating final value from contribution's base value, scale, and offset.
+* Add distribution of IGs related metrics
+* Add enable_private_aggregate_reporting for GetSellerWrappedCode and unit tests that uses GetSellerWrappedCode
+* Add event code and private aggregation objects for contributeToHistogramOnEvent, also added tests for event field
+* add EventMessage to log context
+* Add HandlePrivateAggregateReporting with support of SignalValue only
+* Add HandlePrivateAggregationContributions, iterate over AdWithBids, filter contributions, post processing, test
+* Add helper functions convertEventToInt and convertBaseValueToInt and tests for them for contributeToHistogramOnEvent for bidding service
+* Add Inference Metrics like count, duration, error and size to TF & Pytorch sidecars.
+* Add Inference Request count, duration and failure metric partitioned by model path for Tensorflow and Pytorch sidecars.
+* Add metrics for inference model fetching and registration
+* add option to switch DebugInfo in secure invoke encrypt
+* Add parser function for Private Aggregation's Signal Bucket Object
+* Add parser function for Private Aggregation's Signal Value Object
+* Add PrivateAggregateContribution proto with event field and enum EventType to the .proto file
+* Add proto messages for SignalValue, PrivateAggregationValue, and BaseValue enum.
+* Add request creation timestamp to ProtectedAuctionInput
+* Add request/response to ExecuteInternal on async clients
+* Add support for collecting metrics from the Roma callbacks.
+* Add tee-container-log-redirect option in terraform
+* Add ToBaseValue Helper Function to convert corresponding base value strings to BaseValue Enum
+* Add ToEventTypeString for converting EventType enum into its corresponding string in JSON object returned by ROMA
+* Add utility function to calculate Signal Bucket's final value post auction ([3da9ab5]( )), closes [1#L192]( )
+* Add utility function to parse and return BucketOffset from rapidjson document
+* Add wrapper and test files with method headers for contributeToHistogramOnEvent for bidding service
+* Added `isValidCustomEvent` validation function for common.privateAggregation
+* Added isValidValue validation function for common.privateAggregation
+* Added new files for JS private aggregation util for bidding service
+* Bash script for ASG and Cloud Map Custom HealthChecks
+* Bid Currency Support for Top-Level Seller in Server-Orchestrated Multi-Seller Auctions
+* Change js helper convertEventToInt to mapEventToEnum to return string corresponding to enum instead of int
+* configured public key urls are verified against an explicit allowlist
+* Consented request replace enable_adtech_code_logging in Bidding Server
+* contributeToHistogram with test for bidding service
+* contributeToHistogramOnEvent with tests, using common private aggregation helpers
+* convert all remaining docker images into OCI
+* Convert Private Aggregation wrapper functions from .js to C++ string
+* Create a periodic model fetcher library for inference
+* Create declarative spec for generateBid
+* Create inference model store for model management
+* Create logging library for inference consented logs
+* Debug Reporting for Bid Currency
+* Declare inference JS error schema
+* Declare schema for model fetching metadata file
+* decrease aws/build_and_test duration by ~75%
+* Enable blob fetcher to only fetch blobs with given prefixes
+* Enable dynamic model loading for inference
+* Enable Protected App Signals by Default
+* Enable Service Mesh on AWS by Default
+* export non-privacy log through otel with safe system context
+* Feature Flag for TLS in Service Mesh
+* Force chaffing to enabled for prod builds on BFE + update common repo dep
+* Force the ML model reset with the probability of 0.1%
+* Forward per request consented debugging config to roma callback
+* Freeze TorchScript models before serving
+* Handle chaff and new request format on BFE
+* Implement aggregated Error Reporting for Tensorflow and Pytorch sidecars.
+* Implement AWS Cloud Un-Map
+* Implement new SFE <> BFE request format for chaffing
+* Implement ParseAndProcessContribution with SignalValue only
+* Implement the probabilistic model reset for PyTorch
+* Implement the probabilistic model reset for TensorFlow.
+* Instrument inference JS error for the PyTorch sidecar
+* Integrate with Trusted KV Server in Mesh
+* Load models using model metadata config for the periodic model fetcher
+* Load Test Flag for AWS
+* Log consented debugging information in inference sidecar
+* log EventMessage in servers
+* log udf log in EventMessage for non_prod debug_info
+* Make chaffing flag configurable via Terraform
+* move ig metric to bfe
+* Move serialization of bidding signals per IG from Bidding to BFE
+* Propagate inference error back to JS caller
+* Remove enableAdtechCodeLogging flag value
+* Remove Envoy Access Logging
+* Route consented inference requests to a consented model store
+* send chaff requests from SFE
+* Service Mesh in AWS
+* Support CPU isolation in the inference sidecar
+* Unit test for loading parsing fake contributions in Roma for bidding service
+* Unpad KV server responses
+* update code/cloud build to use tags
+* Update Demo Terraform Configs with values for integrating TEE KV Servers into Mesh
+* Upgrade AWS Provider for Terraform from v3.xx to v4.xx
+* Use gRPC for AWS Service Mesh Envoy HCs
+* Write a fake generateBid() script which calls privateAggregation.contributeToHistogramOnEvent(event, <fake contribution>)
+* write event message to gcs for consented request
+* write event message to s3 in aws for consented request
+
+
+### Bug Fixes
+
+* [IsolateBuyerAndSellerCodeExecution] Add a new class for buyer's reporting code fetch and load
+* [IsolateBuyerAndSellerCodeExecution] Add a new code wrapper with only scoreAd and reportResult
+* [IsolateBuyerAndSellerCodeExecution] Add config flag to enable seller and buyer code isolation
+* [IsolateBuyerAndSellerCodeExecution] Modify seller_udf_manager to fetch and load buyer udfs
+* [IsolateBuyerAndSellerCodeExecution] Refactor the code fetch files.
+* Add BidCurrency SUT to Pre-Submit
+* add check back in default grpc client
+* Add createContribution inside contributeToHistogram/OnEvent and fixed typo with createContribution's value conditional statement.
+* Add DebugInfo pointer for debugging log into RomaRequestContext
+* Add generation_id to chaff requests
+* Add handling when rejection reason is specified as base value but not available and statuscode documentation for HandlePrivateAggregationValuePostAuction
+* Adds domain equality validations to buyer reporting UDF URL
+* Align plaintext buyer request for basic SUT with encrypted version
+* Allow up to 15 buyers when chaffing enabled
+* chaffing bugs ([027d5bb]( )), closes [/github.com/abseil/abseil-cpp/blob/master/absl/container/internal/raw_hash_set.h#L1572]( )
+* Changed base value string to be converted in ToBaseValue function and test cases
+* Changed raw string delimiter JSCODE to JS_CODE for consistency
+* Changed the input format of HandlePrivateAggregationReporting, added support for parsing PAgg bucket,  and add the logic for getting required BaseValues (winning-bid, highest-scoring-other-bid, rejection-reason).
+* clean up log verbosity 3
+* Correct output_filter typo
+* do not try to impersonate service accounts if TEST_MODE=true
+* Don't set chaff size for non-chaff requests
+* Eliminate Terraform Error Message about empty Authority Field
+* Enable threat mitigation with seller and buyer code isolation
+* Ensure instance id is set in logs on AWS when not using mesh.
+* Execute Callback for empty HTTP request vector
+* Fix bugs in reportWin URL validation in auction service
+* Fix release notes by adding a dedicated tag on main to generate the changelog
+* make num_chaff_requests not have a static lower bound
+* Make terraform for aws delete cloud maps without error
+* Populate temp rc file for inference sidecar
+* Populate temp rc file for inference sidecar
+* Populate temp rc file for inference sidecar
+* Redirect misleading log to /dev/null
+* Refactored conversion of private aggregation value post auction into returning absl::StatusOr<int> instead of implicitly editing PrivateAggregationValue that was passed in.
+* Revert wrk2 test runner to an earlier version
+* scorecard.yaml version updates
+* shuffle request order (real and chaff) on SFE
+* truncate curl failure info for fail count metric
+* undo rules_oci migration to fix hash stability issues
+* Update setup_2 and demo terraform configs to valid state
+* Update SUTs to include and test Experiment Group ID
+* Upgrades google terraform plugin to 5.31.0 to fix crash
+
+
+### Documentation
+
+* Add PAS input example to BFE
+* Document how to deployment B&A without inference
+* Update inference_sidecar README with model size limits
+* Updated expected format for INFERENCE_MODEL_BUCKET_PATHS in README
+
 ## 3.11.0 (2024-08-19)
 
 

WORKSPACE

@@ -1,15 +1,17 @@
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 
 ### register Python toolchain -- note this toolchain defines the path to a specific version of python
-load("//builders/bazel:deps.bzl", "python_deps")
+load("//builders/bazel:deps.bzl", "python_deps", "python_register_toolchains")
 
 http_archive(
     name = "io_bazel_rules_docker",
     sha256 = "b1e80761a8a8243d03ebca8845e9cc1ba6c82ce7c5179ce2b295cd36f7e394bf",
     urls = ["https://github.com/bazelbuild/rules_docker/releases/download/v0.25.0/rules_docker-v0.25.0.tar.gz"],
 )
 
-python_deps("//builders/bazel")
+python_deps()
+
+python_register_toolchains("//builders/bazel")
 
 # TODO: Remove bazel_clang_tidy once we sync to the common repo commit 9edb0c3 (4/3/2024) or later
 http_archive(
@@ -23,11 +25,11 @@ http_archive(
 
 http_archive(
     name = "google_privacysandbox_servers_common",
-    # 2024-08-06
-    sha256 = "776c90c2eb6961c4b834242db58ba187cf1e790c876c5e8ef4bdb912210b8266",
-    strip_prefix = "data-plane-shared-libraries-81262eb0b1dfad7b998eec6c6d38e902ed151482",
+    # 2024-08-26
+    sha256 = "dcd09e9241b9e2e85dfa9bd6b12768391f20e4587cba20589c781e4be1ba64a2",
+    strip_prefix = "data-plane-shared-libraries-da1550404faa919ccbbdaf9e91e6225934ad1620",
     urls = [
-        "https://github.com/privacysandbox/data-plane-shared-libraries/archive/81262eb0b1dfad7b998eec6c6d38e902ed151482.zip",
+        "https://github.com/privacysandbox/data-plane-shared-libraries/archive/da1550404faa919ccbbdaf9e91e6225934ad1620.zip",
     ],
 )
 

api/BUILD

@@ -23,6 +23,7 @@ proto_library(
     name = "bidding_auction_servers_proto",
     srcs = ["bidding_auction_servers.proto"],
     deps = [
+        "//api/udf:generate_bid_proto",
         "@com_google_googleapis//google/api:annotations_proto",
         "@com_google_protobuf//:struct_proto",
         "@google_privacysandbox_servers_common//src/logger:logger_proto",

api/bidding_auction_servers.proto

@@ -16,6 +16,7 @@ syntax = "proto3";
 
 package privacy_sandbox.bidding_auction_servers;
 
+import "api/udf/generate_bid.proto";
 import "google/api/annotations.proto";
 import "google/protobuf/struct.proto";
 import "src/logger/logger.proto";
@@ -196,6 +197,7 @@ message BuyerInput {
 }
 
 // Information about an Interest Group passed by the browser.
+// (-- LINT.IfChange(browser_signals_bidding) --)
 message BrowserSignals {
   // Number of times the group was joined in the last 30 days.
   int64 join_count = 1;
@@ -220,9 +222,12 @@ message BrowserSignals {
   // Only one of the recency or recency_ms is expected to present in the request.
   optional int64 recency_ms = 5;
 }
+// (-- LINT.ThenChange(/services/bidding_service/udf/api/generate_bid.proto:browser_signals_generate_bid) --)
 
 // Information passed by Android.
+// (-- LINT.IfChange(android_signals_bidding) --)
 message AndroidSignals {}
+// (-- LINT.ThenChange(/services/bidding_service/udf/api/generate_bid.proto:android_signals_generate_bid) --)
 
 // Specifies type of the ad. It can help differentiate between ads when
 // B&A is supporting multiple ad targeting use cases concurrently.
@@ -448,6 +453,17 @@ message AuctionResult {
     // Proptected App Signal: This refers to the buyer domain.
     string owner = 3;
 
+    // Owner of the IG to which the ad belongs to.
+    // Note: This is only populated for Protected Audience for
+    // Android apps.
+    string origin = 4;
+
+    // Name of the Interest Group (Custom Audience) to which the ghost
+    // winner belongs to.
+    // Note: This is only populated for Protected Audience for
+    // Android apps.
+    string ig_name = 5;
+
     // Private aggregation signals for the ghost winner.
     // Single seller auctions: This would correspond to a ghost winner
     // if available.
@@ -461,7 +477,7 @@ message AuctionResult {
 
       int32 value = 2;
     }
-    optional GhostWinnerPrivateAggregationSignals ghost_winner_private_aggregation_signals = 4;
+    optional GhostWinnerPrivateAggregationSignals ghost_winner_private_aggregation_signals = 6;
 
     // In case of multiseller auction, the associated data for the ghost winner
     // will be returned so that the ghost winning bid can be scored
@@ -496,9 +512,13 @@ message AuctionResult {
       // properties on the browser.
       string buyer_and_seller_reporting_id = 6;
     }
-    optional GhostWinnerForTopLevelAuction ghost_winner_for_top_level_auction = 5;
+    optional GhostWinnerForTopLevelAuction ghost_winner_for_top_level_auction = 7;
   }
   repeated KAnonGhostWinner k_anon_ghost_winners = 23;
+
+  // This field will be populated for all seller and buyer contributions in
+  // single seller auctions and server orchestrated multi seller auctions.
+  repeated PrivateAggregateReportingResponse top_level_contributions = 24;
 }
 
 message GetComponentAuctionCiphertextsRequest {
@@ -968,6 +988,7 @@ message PrivateAggregateReportingResponse {
 }
 
 // Bid for an ad candidate.
+// (-- LINT.IfChange(bid_bidding) --)
 message AdWithBid {
   // Metadata of the ad, this will be passed to Seller's scoring function.
   // Represents an opaque object that is eventually passed to seller Adtech
@@ -1016,6 +1037,7 @@ message AdWithBid {
   // Private aggregation object.
   repeated PrivateAggregateContribution private_aggregation_contributions = 12;
 }
+// (-- LINT.ThenChange(bid_generate_bid) --)
 
 // Bidding service operated by buyer.
 service Bidding {
@@ -1051,9 +1073,7 @@ message GenerateBidsRequest {
       string name = 1;
 
       // Used to fetch real time bidding signals from buyer's key/value server
-      // included in the request. The value of each key in this list will be
-      // passed from the bidding signals dictionary to the Interest Group's
-      // GenerateBid() function as the trustedBiddingSignals parameter.
+      // included in the request.
       repeated string trusted_bidding_signals_keys = 2;
 
       // Optional.
@@ -1086,6 +1106,11 @@ message GenerateBidsRequest {
         // information.
         BrowserSignals browser_signals = 7;
       }
+
+      // Real time bidding signals fetched from buyer's key/value service
+      // passed to the Interest Group's GenerateBid() function as the
+      // trustedBiddingSignals parameter.
+      string trusted_bidding_signals = 8;
     }
 
     // Interest Group is an input to bidding code.
@@ -1111,7 +1136,7 @@ message GenerateBidsRequest {
     string buyer_signals = 3;
 
     // Real Time signals fetched from buyer's Key/Value service.
-    string bidding_signals = 4;
+    string bidding_signals = 4 [deprecated = true];
 
     // A boolean value which indicates if event level debug reporting should be
     // enabled or disabled for this request.
@@ -1625,8 +1650,9 @@ message ScoreAdsResponse {
     // Protected Audience only).
     string interest_group_origin = 20;
 
-    // Private Aggregate contributions to be sent to the client
-    PrivateAggregateReportingResponse private_aggregate_reporting_response = 21;
+    // This field will be populated for all seller and buyer contributions in
+    // single seller auctions and server orchestrated multi seller auctions.
+    repeated PrivateAggregateReportingResponse top_level_contributions = 21;
   }
 
   // The response includes the top scored ad along with other related data.
@@ -1669,17 +1695,6 @@ message WinReportingUrls {
   ReportingUrls top_level_seller_reporting_urls = 3;
 }
 
-// Urls to support debug reporting, when auction is won and auction is lost.
-message DebugReportUrls {
-  // URL to be triggered if the interest group wins the auction.
-  // If undefined or malformed url it will be ignored.
-  string auction_debug_win_url = 1;
-
-  // URL to be triggered if the interest grou losses the auction.
-  // If undefined or malformed url it will be ignored.
-  string auction_debug_loss_url = 2;
-}
-
 // message to store the server request/ response
 message EventMessage {
   message KvSignal {