Hides user and credential field from search response

[owaiskazi19]

Description

• Hides user field from Search and Get APIs
• A painless script was added in Redacting credentials from Get Workflows and Search Workflows API #504 to filter out the credential field when present and adds a filter field instead of _source causing more client side work. Handled the credential field filtering in a better way.

Testing

Request

curl -XPOST --insecure -u 'admin:admin' --header 'Content-Type: application/json' "https://localhost:9200/_plugins/_flow_framework/workflow?provision=true" -d '{"name":"Deploy Claude Model","description":"Deploy a model using a connector to Claude","use_case":"PROVISION","version":{"template":"1.0.0","compatibility":["2.12.0","3.0.0"]},"workflows":{"provision":{"nodes":[{"id":"create_claude_connector","type":"create_connector","user_inputs":{"name":"Claude Instant Runtime Connector","version":"1","protocol":"aws_sigv4","description":"The connector to BedRock service for Claude model","actions":[{"headers":{"x-amz-content-sha256":"required","content-type":"application/json"},"method":"POST","request_body":"{ \"prompt\":\"${parameters.prompt}\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature},  \"anthropic_version\":\"${parameters.anthropic_version}\" }","action_type":"predict","url":"https://bedrock-runtime.us-west-2.amazonaws.com/model/anthropic.claude-instant-v1/invoke"}],"credential":{"access_key":"PUT_YOUR_ACCESS_KEY_HERE","secret_key":"PUT_YOUR_SECRET_KEY_HERE"},"parameters":{"endpoint":"bedrock-runtime.us-west-2.amazonaws.com","content_type":"application/json","auth":"Sig_V4","max_tokens_to_sample":"8000","service_name":"bedrock","temperature":"0.0001","response_filter":"$.completion","region":"us-west-2","anthropic_version":"bedrock-2023-05-31"}}}]}}}'

Response of Search API:

curl -XGET --insecure -u 'admin:admin' --header 'Content-Type: application/json' "https://localhost:9200/_plugins/_flow_framework/workflow/_search?pretty" -d '{"query":{"match_all":{}}}'
{
  "took" : 44,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".plugins-flow-framework-templates",
        "_id" : "YinyEo8B0L4I4ADcw5V9",
        "_version" : 2,
        "_seq_no" : 1,
        "_primary_term" : 1,
        "_score" : 1.0,
        "_source" : {
          "use_case" : "PROVISION",
          "created_time" : 1714009850537,
          "last_updated_time" : 1714009850537,
          "name" : "Deploy Claude Model",
          "description" : "Deploy a model using a connector to Claude",
          "workflows" : {
            "provision" : {
              "nodes" : [
                {
                  "user_inputs" : {
                    "protocol" : "aws_sigv4",
                    "name" : "Claude Instant Runtime Connector",
                    "description" : "The connector to BedRock service for Claude model",
                    "version" : "1",
                    "parameters" : {
                      "endpoint" : "bedrock-runtime.us-west-2.amazonaws.com",
                      "content_type" : "application/json",
                      "auth" : "Sig_V4",
                      "max_tokens_to_sample" : "8000",
                      "service_name" : "bedrock",
                      "temperature" : "0.0001",
                      "response_filter" : "$.completion",
                      "region" : "us-west-2",
                      "anthropic_version" : "bedrock-2023-05-31"
                    },
                    "actions" : [
                      {
                        "headers" : {
                          "x-amz-content-sha256" : "required",
                          "content-type" : "application/json"
                        },
                        "method" : "POST",
                        "request_body" : "{ \"prompt\":\"${parameters.prompt}\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature},  \"anthropic_version\":\"${parameters.anthropic_version}\" }",
                        "action_type" : "predict",
                        "url" : "https://bedrock-runtime.us-west-2.amazonaws.com/model/anthropic.claude-instant-v1/invoke"
                      }
                    ]
                  },
                  "id" : "create_claude_connector",
                  "type" : "create_connector",
                  "previous_node_inputs" : { }
                }
              ],
              "edges" : [ ],
              "user_params" : { }
            }
          },
          "version" : {
            "template" : "1.0.0",
            "compatibility" : [
              "2.12.0",
              "3.0.0"
            ]
          },
          "last_provisioned_time" : 1714009850887
        }
      }
    ]
  }
}

Response of GET API:

curl -XGET --insecure -u 'admin:admin' --header 'Content-Type: application/json' "https://localhost:9200/_plugins/_flow_framework/workflow/D0sjE48BLSYJV6YQCQ_K?pretty" 
{
  "name" : "test-1",
  "description" : "test-2",
  "use_case" : "SEMANTIC_SEARCH",
  "version" : {
    "template" : "1.0.0",
    "compatibility" : [
      "2.12.0",
      "3.0.0"
    ]
  },
  "workflows" : {
    "provision" : {
      "user_params" : { },
      "nodes" : [
        {
          "id" : "create_ingest_pipeline",
          "type" : "create_ingest_pipeline",
          "previous_node_inputs" : { },
          "user_inputs" : {
            "configurations" : "{\"description\":\"test-pipeline\",\"processors\":[{\"text_embedding\":{\"field_map\":{\"passage_text\":\"passage_embedding\"},\"model_id\":\"jyKh544B161srMZWwY93\"}}]}",
            "pipeline_id" : "nlp-pipeline"
          }
        }
      ],
      "edges" : [ ]
    }
  },
  "user" : {
    "name" : "admin",
    "backend_roles" : [
      "admin"
    ],
    "roles" : [
      "own_index",
      "all_access"
    ],
    "custom_attribute_names" : [ ],
    "user_requested_tenant" : null
  },
  "created_time" : 1714013014339,
  "last_updated_time" : 1714013014339,
  "last_provisioned_time" : 1714013014615
}

Issues Resolved

Resolves #546

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.80%. Comparing base (bae3aa2) to head (6cc5851).

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #680      +/-   ##
============================================
+ Coverage     72.27%   72.80%   +0.53%     
- Complexity      687      694       +7     
============================================
  Files            82       82              
  Lines          3585     3589       +4     
  Branches        295      296       +1     
============================================
+ Hits           2591     2613      +22     
+ Misses          863      845      -18     
  Partials        131      131              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbwiddis]

Approved, but it doesn't look like tests are covering the getSourceContext static method in either case. Are these covered in integ tests? Also it looks like the TODO has been handled.

[owaiskazi19]

@dbwiddis you might want to review again since I added more changes. Thanks

[owaiskazi19]

Approved, but it doesn't look like tests are covering the getSourceContext static method in either case. Are these covered in integ tests? Also it looks like the TODO has been handled.

Added tests for RestHandlerUtils and removed the TODO.

[dbwiddis]

(Re)approved with a question.