07_evals_L2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "59711e58-259d-4c9c-994d-137b9ca0c5da",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import json\n",
    "traindf = pd.read_csv('sample_data/honeycomb_train.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc8d62d8-41ed-49bb-b8f7-70890b2fe84d",
   "metadata": {},
   "source": [
    "# Optimize Model\n",
    "> Make the model better"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "252b617b-a56d-4394-900a-2ec520c24679",
   "metadata": {},
   "source": [
    "## Initial Feedback\n",
    "\n",
    "Phillip at Honeycomb did a vibe check and had the following feedback."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f593c7e-3818-4ea3-86ca-18ccb07a60a0",
   "metadata": {},
   "source": [
    "> latency for highest cart totals --> https://replicate.com/hamelsmu/honeycomb?prediction=qs6ybdlbwtpyvvb5rbbrfdhttu\n",
    "\n",
    "This was a good response.\n",
    "\n",
    "> request latency for highest cart totals --> https://replicate.com/hamelsmu/honeycomb?prediction=qs6ybdlbwtpyvvb5rbbrfdhttu\n",
    "\n",
    "Same thing, also good.\n",
    "\n",
    "> slow requests --> https://replicate.com/hamelsmu/honeycomb?prediction=vopc2qtbvjkks4bvplos4gkal4\n",
    "\n",
    "Poor response. A heatmap of duration is good, but it didn’t filter by parent_id does-not-exist, which is how you filter to requests.\n",
    "\n",
    "> slowest endpoints --> https://replicate.com/hamelsmu/honeycomb?prediction=5qa3rzdb35tqatrc4ltzntez4a\n",
    "\n",
    "Poor response. Same as slow requests. I was expecting a heatmap(duration_ms) and a max(duration_ms), grouped by http.route or http.target, and an ordering by max(duration_ms)\n",
    "\n",
    "> which endpoint is erroring --> https://replicate.com/hamelsmu/honeycomb?prediction=yezz223b5ez6sr6uhm77iss2gu\n",
    "\n",
    "Poor-ish response. HEATMAP is good to include, but this should also have a COUNT. Needs to filter by http.status_code > 400 or something like that. There’s no error column so it’s fine that it didn’t try to use that.\n",
    "\n",
    "> slowest traces --> https://replicate.com/hamelsmu/honeycomb?prediction=arqhqj3b7wevo4wpl7ykfblhhm\n",
    "\n",
    "Poor response. Should by a MAX(duration_ms) where parent_id does-not-exist, grouped by trace.trace_id , ordered by the max.\n",
    "\n",
    "> slow spans --> https://replicate.com/hamelsmu/honeycomb?prediction=qny73zlb4t7embjg4vfjynl7ey\n",
    "\n",
    "Weirdly didn’t produce a query. But I would expect a MAX(duration_ms) grouped by trace.span_id, ordered by the max"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a0adf8c-a333-4f0c-9199-ed8046c348f7",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "\n",
    "- I ran the same queries through GPT 3.5 (what Honeycomb uses in production) and found that OpenAI produces the same \"bad queries\".\n",
    "- The training data that Phillip gave me also has the same bad queries!\n",
    "\n",
    "### What do we do now?\n",
    "\n",
    "So what do we do, are we out of luck?  If we have bad data, and the baseline model is just as bad, what can we do?  It's not like we can encode Phillip himself into an AI?  I actually think we can!  This is my plan:\n",
    "\n",
    "> The idea is that we need Philip in a box. I’ve been experimenting with GPT-4 and I think it’s possible to encode you with some prompting and examples. I propose we manually look at 25 examples at a time in each round I try to make Philip in a box better and track more closely to the real Philip. I bet you I can get close. what we need for these 25 examples is detailed feedback like you gave above here in Slack then I can iterate once I’m done I can use Philip in a box to filter for only really good data but also to generate better data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85531dd9-d0aa-4017-ac36-b1e025a002fb",
   "metadata": {},
   "source": [
    "## Example Critiques\n",
    "\n",
    "- few shot examples of good critiques.\n",
    "- these examples were iterated on through an iterative process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "df5b0193-6ea8-418b-807a-31490b4316e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "CRITIC_PROMPT=\"\"\"## Background\n",
    "\n",
    "Honeycomb is an observability platform that allows you to write queries to inspect trace data.\n",
    "The specification of the Honeycomb query language is as follows:\n",
    "\n",
    "QUERY SPEC:\n",
    "All top-level keys are optional.\n",
    "\n",
    "```json\n",
    "\"calculations\":[\n",
    "    // ops: COUNT, CONCURRENCY, COUNT_DISTINCT, HEATMAP, SUM, AVG, MAX, MIN, P001, P01, P05, P10, P25, P50, P75, P90, P95, P99, P999, RATE_AVG, RATE_SUM, RATE_MAX\n",
    "    {\"op\": \"COUNT\"},// COUNT and CONCURRENCY are just op\n",
    "    {\"op\": \"HEATMAP\", \"column\": \"name\"}\n",
    "],\n",
    "\"filters\":[\n",
    "    // ops: =, !=, >, >=, <, <=, starts-with, does-not-start-with, exists, does-not-exist, contains, does-not-contain, in, not-in\n",
    "    {\"column\": \"name\", \"op\": \"exists\"}, // exists and does-not-exist ops only have column\n",
    "    {\"column\": \"name\", \"op\": \"=\", \"value\": \"something\"}\n",
    "],\n",
    "\"filter_combination\": \"AND\", // AND or OR\n",
    "\"breakdowns\":[\n",
    "    // columns in COLUMNS\n",
    "    \"column1\",\"column2\"\n",
    "],\n",
    "\"orders\":[\n",
    "    // HEATMAP not allowed\n",
    "    // Must come from breakdowns or calculations\n",
    "    {\"op\": \"op_in_calculation\", \"column\": \"column_in_calculation\", \"order\": \"ascending\"},\n",
    "    {\"op\": \"COUNT\", \"order\": \"descending\"}, // COUNT and CONCURRENCY have no column\n",
    "    {\"column\": \"column1\", \"order\": \"descending\"},\n",
    "],\n",
    "\"havings\":[\n",
    "    // HEATMAP not allowed\n",
    "    {\"calculate_op\": \"op_in_calculation\", \"column\": \"name\", \"op\": \"OPNAME\", \"value\": 100},\n",
    "    {\"calculate_op\": \"COUNT\", \"op\": \">\", \"value\": 10}, // COUNT and CONCURRENCY have no column\n",
    "],\n",
    "\"time_range\": 7200, // Relative time range in seconds.\n",
    "\"start_time\": 1234567890, // UNIX timestamp\n",
    "\"end_time\": 1234567890, // UNIX timestamp\n",
    "```\n",
    "\n",
    "QUERY SPEC TIPS:\n",
    "\n",
    "COUNT counts events/spans. COUNT_DISTINCT counts unique values of columns.\n",
    "HEATMAP shows value distributions.\n",
    "trace.parent_id does-not-exist shows root span of a trace. Use that to count requests and traces.\n",
    "name is a span or span event.\n",
    "parent_name is the name of a span that created a span event.\n",
    "\n",
    "When the user input is asking about a time range (such as \"yesterday\" or \"since last week\"),\n",
    "always use the time_range, start_time, and end_time fields. time_range\n",
    "is relative and can be combined with either start_time or end_time but not both.\n",
    "\n",
    "---\n",
    "\n",
    "## Instructions\n",
    "\n",
    "You are an EXPERT query evaluator that has advanced capabilities to judge if a query good or not.  You understand the nuances of the Honeycomb query language, including what is likely to be most useful from an analytics perspective.\n",
    "You are given the following three inputs: (1) NLQ, (2) A list of candidate columns (COLUMNS) that are allowed to be in the query, and (3) The query (QUERY). Your job is to evaluate and critique the QUERY relative to the provided NLQ and COLUMNS. \n",
    "\n",
    "The critiques must be provided in the same json format as provided in the examples below:\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: show me slowest trace\n",
    "\n",
    "COLUMNS: ['trace.trace_id', 'trace.span_id', 'trace.parent_id', 'duration_ms', 'name', 'faas.instance', 'faas.id', 'filter', 'telemetry.instrumentation_library', 'library.name', 'faas.name', 'span.kind', 'type', 'http.wrote_bytes', 'http.url', 'service.name', 'http.flavor', 'span.num_links', 'span.num_events', 'net.host.name', 'library.version', 'http.scheme', 'net.peer.name', 'http.method', 'meta.signal_type', 'cloud.region', 'cloud.provider', 'faas.version', 'http.read_bytes', 'http.user_agent', 'cloud.account.id', 'organization_id', 'cloud.platform', 'net.sock.peer.addr', 'page_size', 'net.sock.peer.port', 'page_token', 'status_code', 'http.client_ip', 'http.status_code', 'http.route']\n",
    "\n",
    "QUERY: {\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"MAX\"}],\"filters\":[{\"column\":\"trace.parent_id\",\"op\":\"does-not-exist\",\"join_column\":\"\"}],\"orders\":[{\"column\":\"duration_ms\",\"op\":\"MAX\",\"order\":\"descending\"}],\"limit\":1,\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"The response is nearly correct, as it is looking for the slowest trace by using MAX(duration_ms) and ordering by duration_ms in descending order, which is appropriate for finding the 'slowest' trace. Additionally, filtering with trace.parent_id does-not-exist correctly identifies root spans. However, the query should be grouping by trace.trace_id to actually show the slowest trace. Without that grouping, the query only shows the MAX(duration_ms) measurement over time, irrespective of which trace is responsible for that measurement.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: slow requests\n",
    "\n",
    "COLUMNS: ['duration_ms', 'deprecatedSource', 'error', 'severity', 'flags', 'rpc.method', 'deprecatedCount', 'ip', 'rpc.service', 'interface', 'metadata', 'reason', 'container.cpu.time', 'deprecatedLastTimestamp', 'name', 'regarding', 'namespace', 'message.type', 'watch-type', 'severity_text', 'apiVersion', 'kind', 'service.name', 'rpc.system', 'container.uptime', 'type', 'uid', 'span.kind', 'db.name', 'note', 'body', 'resourceVersion', 'library.version', 'db.statement', 'status_code', 'event.domain', 'managedFields', 'deprecatedFirstTimestamp', 'container.memory.page_faults', 'event.name', 'container.memory.available', 'k8s.container.cpu_request', 'container.id', 'library.name', 'container.memory.rss', 'message.uncompressed_size', 'rpc.grpc.status_code', 'net.peer.port', 'k8s.container.memory_request', 'container.memory.working_set', 'trace.parent_id', 'http.route', 'service_name', '']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"name\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"},{\"column\":\"duration_ms\",\"op\":\"MAX\"}],\"filters\":[{\"column\":\"trace.parent_id\",\"op\":\"does-not-exist\",\"join_column\":\"\"}],\"orders\":[{\"column\":\"duration_ms\",\"op\":\"MAX\",\"order\":\"descending\"}],\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"This response is adequate. The inclusion of a HEATMAP and a MAX calculation on the duration_ms column is appropriate for identifying slow requests. Correctly including a filter on 'trace.parent_id' with 'does-not-exist' ensures that only root spans (requests) are considered. Ordering by the max duration_ms in descending order is sensible to quickly highlight slower requests. The use of 'name' in a breakdown is good, because it groups results into span names. This can help you quickly narrow down behavior to a particular part of a system. It might be more insightful to break down by 'http.route', because 'http.route' will also break down more broadly by API call. Having both 'http.route' and 'name' included can give both a broader and narrower breadkdown of the data, and both views can be helpful at different times.. Overall, the query aligns well with the NLQ but could benefit from a more specific breakdown column.\", \"outcome\": \"good\"}\n",
    "\n",
    "---\n",
    "NLQ: records of clientid 5fcedb77-267c-44e3-8ea4-4d21786b2618\n",
    "\n",
    "COLUMNS: ['clientIds', 'sys-client-id', 'principal.clientId', 'client.id', 'clientName', 'metadata.clientId', 'payload.clientId', 'clientExternalId', 'informationReportingEvent.clientId', 'principalClientId', 'metadata.clientTxId', 'targetClientId', 'e2eId', 'client', 'transactionIds', 'clientId', 'client.external_id', 'metadata.clientName', 'sys-service-provider-id', 'transactionId', 'paymentRequestIds.identifying', 'serviceProviderId', 'service_provider.id', 'externalId', 'traceId', 'event.id', 'accounts-deleted', 'jobExecutionId', 'accounts-updated', 'syncContext.clientDto.id', '$metadata.requestId', 'subscriptionId', 'paymentExternalId', 'activeIntegrationIds', 'fileId', 'tracerId', '$metadata.extendedRequestId', 'moneyTransferId', 'paymentRequestIds', 'vendor.id', 'triggeredNumberOfClients', 'payload.eventId', 'existedTransactionIds', 'completedTransactionIds', 'metadata.paymentRequestExternalId', 'syncContext.clientDto.externalId', 'createdTransactionIds', 'transactionsDeletionServiceProviderId', 'idempotency_key', 'RequestId', 'status', 'error', 'duration_ms', 'http.route', 'name', 'http_path', 'trace.parent_id', 'http.status_code', 'meta.req.query', 'service_name', 'request_time']\n",
    "\n",
    "QUERY: {\"filters\":[{\"column\":\"client.id\",\"op\":\"=\",\"value\":\"5fcedb77-267c-44e3-8ea4-4d21786b2618\"}]}\n",
    "\n",
    "{\"critique\": \"Poor response. While the filter by client.id is correct to find records for clientid '5fcedb77-267c-44e3-8ea4-4d21786b2618', the query lacks breakdowns, calculations, or order to make the results more insightful. It would be more beneficial to include calculations such as COUNT to understand the volume of records.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: Find the traces for the slowest requests in the past 3 hours where http.host = internal-pcmws-alkqa-routing-black-1164727359.us-east-1.elb.amazonaws.com\n",
    "\n",
    "COLUMNS: ['trace.trace_id', 'http.host', 'exception.stacktrace', 'pcm.drawtraffic', 'SLI_geo-95-percentile-latency', 'pcm.server.worker-load-balancer', 'pcm.responsetime', 'aws.requestId', 'http.query-length', 'http.response-uri', 'trace.span_id', 'net.host.name', 'http-500-error-percent', 'http.url', 'net.host.port', 'pcm.server.worker-app-pool', 'http.request-headers', 'duration_ms', 'pcm.restful', 'http.response-version', 'trace.parent_id', 'pcm.initialmaprequest', 'http.response-headers', 'http.method', 'aws.region', 'service.instance.id', 'http.response-method', 'wcf.channel.path', 'pcm.server', 'pcm.userip', 'http.version', 'pcm.server-name', 'pcm.server-color', 'aws.service', 'pcm.stopcount', 'rpc.method', 'pcm.splcs', 'peer.service', 'wcf.channel.scheme', 'service.name', 'http.response_content_length', 'pcm.foserver', 'SLI_99.9-percent-without-500-error', 'pcm.httpcode', 'pcm.errortext', 'http.status-description', 'pcm.login', 'pcm.apikey', 'rpc.service', 'pcm.date', 'db.statement', 'http.route', 'parent_name', 'name', 'http.status_code', 'exception.message', 'error']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"trace.trace_id\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"MAX\"}],\"filters\":[{\"column\":\"http.host\",\"op\":\"=\",\"value\":\"internal-pcmws-alkqa-routing-black-1164727359.us-east-1.elb.amazonaws.com\",\"join_column\":\"\"}],\"filter_combination\":\"AND\",\"orders\":[{\"column\":\"duration_ms\",\"op\":\"MAX\",\"order\":\"descending\"}],\"time_range\":10800}\n",
    "\n",
    "{\"critique\": \"The query correctly uses time_range to indicate the past 3 hours and filters by the specific http.host value. However, it should also include a filter where trace.parent_id does-not-exist to identify the root spans or initial requests, which represent the start of a trace. Without this, the query might include intermediate spans that are not the start of the trace. The breakdown by trace.trace_id is correct, and ordering by the max duration is appropriate to find the slowest requests. It could be improved by including a calculation of COUNT to gauge the number of slow traces for each trace id, making it thorough for understanding general patterns or anomalies.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: show me traces where ip is 10.0.2.90\n",
    "\n",
    "COLUMNS: ['net.host.ip', 'trace.trace_id', 'net.peer.ip', 'net.transport', 'net.host.name', 'http.client_ip', 'http.host', 'host.name', 'trace.span_id', 'net.host.port', 'k8s.pod.ip', 'sli_route_latency_1500', 'ip', 'net.peer.port', 'trace.parent_id', 'sli_route_latency', 'http.server_name', 'service.name', 'service.instance.id', 'rpc.user_agent', 'rpc.service', 'http.route', 'db.name', 'http.url', 'area', 'requestID', 'service.version', 'http.user_agent', 'http.target', 'host.arch', 'orderid', 'process.pid', 'rpc.system', 'os.type', 'rpc.method', 'otlp.exporter.exported', 'runtime.jvm.gc.time', 'http.scheme', 'thread.name', 'name', 'runtime.jvm.memory.pool', 'message.type', 'runtime.jvm.memory.area', 'thread.id', 'enduser.id', 'error', 'userid', 'http.flavor', 'pool', 'telemetry.instrumentation_library', 'http.status_code', 'service_name', 'parent_name', 'duration_ms', 'db.statement', 'exception.message']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"trace.trace_id\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"net.host.ip\",\"op\":\"=\",\"value\":\"10.0.2.90\"}]}\n",
    "\n",
    "{\"critique\": \"The query satisfies the NLQ. It correctly filters for traces with an IP address of 10.0.2.90 and counts the occurrences of those traces, grouped by trace.trace_id, which is what the NLQ requests. The response is good as it meets the requirement of showing traces from a specific IP address without additional complexities.\", \"outcome\": \"good\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: count of moderation.flags.create api calls\n",
    "\n",
    "COLUMNS: ['response_flags', 'job_counts_MULTI_INDEX_CREATE', 'http_count', 'found_count', 'is_creation', 'is_create', 'job_counts_MULTI_INDEX_BATCH_CREATE', 'event_count', 'job_counts_MULTI_INDEX_CREATE_BE', 'job_counts_SEND_SLACK_UNFURLS_UPDATE', 'queued_count', 'keys_count', 'agenda_req_count', 'flags', 'job_counts_SEND_SLACK_DOC_THREAD_UPDATE', 'update_count', 'log_count', 'old_activity_badge_count', 'total_message_count', 'job_counts_SEND_SLACK_USER_MENTIONS_UPDATE', 'job_counts_UNIQUE_INDEX_CREATE', 'request_attempt_count', 'job_counts_ADD_TO_SLACK_DOCS_COMPANY', 'args.thread_counts_by_channel', 'job_counts_FINALIZE_EDIT_HISTORY', 'mc_queries_count', 'hitCount', 'api:warning', 'job_counts_MULTI_INDEX_BATCH_CREATE_BE', 'file_count', 'job_counts_UPDATE_THREAD_UNREAD_COUNTS_EF4', 'api:method', 'executing_count', 'job_counts_SEND_SLACK_FILES_UPDATE', 'failed_count', 'mpdm_count', 'count', 'grpc_req_count', 'job_counts_UPDATE_SLACK_REFERENCES_IN_THREAD', 'mc_queries_multi_count', 'channel_ids_count', 'job_counts_UPDATE_THREAD_PHRASES_ON_EDIT', 'total_participant_count', 'job_counts_MULTI_INDEX_DELETE', 'job_counts_UPDATE_THREAD_UNREAD_COUNTS_BE', 'kafkagate_count', 'authorized_apps_count', 'redirect_count', 'apps_count', 'bulk_generate_calls', 'duration_ms', 'slath', 'name', 'execution_context', 'trace_parent_id', 'db.statement', 'namespace', 'kind', 'error', 'message', 'reason', 'http.status_code', 'service_name', 'quip:call_path']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"slath\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"name\",\"op\":\"=\",\"value\":\"moderation.flags.create\",\"join_column\":\"\"}],\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"The query groups by 'slath' which is extraneous information - there is nothing in NLQ that indicates 'slath' is related to NLQ. Additionally, the filter clause should check for the 'contains' op instead of '='. While it is likely good to check for 'name' as that represents an operation name, which means it will likely have data NLQ is referring to, there is no guarantee that the actual name of the operation is exactly what was input in NLQ. Furthermore, the result should include 'name' in the breakdowns so that the results show the actual values of 'name'.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: What kind of events are contributing towards my rate limits\n",
    "\n",
    "COLUMNS: ['up', 'k8s.pod.memory.major_page_faults', 'k8s.pod.memory_limit_utilization', 'k8s.pod.cpu_limit_utilization', 'k8s.node.memory.page_faults', 'k8s.container.cpu_limit', 'k8s.node.memory.major_page_faults', 'k8s.pod.memory.page_faults', 'k8s.container.memory_limit', 'k8s.node.filesystem.usage', 'otelcol_processor_batch_batch_send_size.avg', 'k8s.pod.memory.rss', 'k8s.pod.filesystem.usage', 'k8s.node.network.io.transmit', 'prometheus.io/port', 'scrape_duration_seconds', 'eks.amazonaws.com/compute-type', 'otelcol_processor_batch_timeout_trigger_send', 'otelcol_receiver_refused_metric_points', 'k8s.node.memory.rss', 'otelcol_processor_batch_batch_send_size.count', 'eks.amazonaws.com/component', 'otelcol_processor_dropped_metric_points', 'k8s.pod.network.io.transmit', 'otelcol_processor_refused_metric_points', 'job.runner.gitlab.com/url', 'scrape_samples_scraped', 'prometheus.io/scrape', 'k8s.pod.memory.usage', 'k8s.node.cpu.time', 'otelcol_processor_batch_batch_send_size.p50', 'k8s.node.network.io.receive', 'k8s.node.filesystem.capacity', 'k8s.node.memory.usage', 'otelcol_processor_dropped_log_records', 'k8s.container.restarts', 'k8s-app', 'otelcol_exporter_queue_capacity', 'k8s.pod.memory_request_utilization', 'k8s.pod.network.errors.transmit', 'otelcol_processor_batch_batch_send_size.max', 'otelcol_receiver_accepted_metric_points', 'otelcol_exporter_sent_metric_points', 'k8s.pod.filesystem.capacity', 'project.runner.gitlab.com/id', 'processor', 'otelcol_scraper_errored_metric_points', 'otelcol_processor_batch_batch_send_size.p05', 'interface', 'otelcol_process_memory_rss', 'net.host.name', 'name', 'trace.parent_id', 'duration_ms', 'http.route', 'service_name']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"name\"],\"calculations\":[{\"column\":\"otelcol_processor_refused_metric_points\",\"op\":\"SUM\"},{\"column\":\"otelcol_processor_dropped_metric_points\",\"op\":\"SUM\"},{\"column\":\"otelcol_processor_dropped_log_records\",\"op\":\"SUM\"}],\"filters\":[{\"column\":\"otelcol_processor_refused_metric_points\",\"op\":\">\",\"value\":0},{\"column\":\"otelcol_processor_dropped_metric_points\",\"op\":\">\",\"value\":0},{\"column\":\"otelcol_processor_dropped_log_records\",\"op\":\">\",\"value\":0}]}\n",
    "\n",
    "{\"critique\": \"Good response. The query is grouping by 'name' to identify different event types and summing up refused, dropped metric points and dropped log records, which are potential indicators of hitting rate limits. Furthermore, it properly filters each of those to be greater than zero to only include relevant events. This helps to understand what events are contributing towards rate limits being exceeded.\", \"outcome\": \"good\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: count window-hash where window-hash exists per hour\n",
    "\n",
    "COLUMNS: ['window-hash', 'window_hash_get', 'prop-window_hash', 'fingerprint_window_stack_traces_hash', 'prop-window_hash_get', 'fingerprint_window_overloaded_hash', 'fingerprint_window_document_keys_hash', 'fingerprint_window_keys_hash', 'fingerprint_window_client_tools_hash', 'fingerprint_window_messages_hash', 'headers-hash', 'fingerprint_time_zone_anomaly_hash', 'best_hash_get', 'w_count', 'legacy-hash', 'fingerprint_window_stack_traces', 'fingerprint_time_zone_hash', 'siteverify_nojs-hdict_count', 'fingerprint_time_zone_region_hash', 'fingerprint_window_missing_features_hash', 'fingerprint_timing_hash', 'fetch-hash', 'client_timestamp_check', 'canvas-hash', 'fingerprint_unique_keys_stable_hash_hash', 'fingerprint_window_document_keys', 'fingerprint_window_keys', 'task_request_count', 'fingerprint_web_worker_stack_size_hash', 'fingerprint_window_overloaded', 'fingerprint_window_version', 'fingerprint_time_zone_anomaly', 'token_rate_1m_tarpit', 'r_count', 'art_up__asn_window', 'fingerprint_unique_keys_stable_hash', 'sk_wgl_canv_hash', 'fingerprint_dom_rect_decimal_hash', 'fingerprint_window_messages', 'webrtc-hash', 'legacy_hash_get', 'fingerprint_window_sus_chrome_key', 'md_hash', 'webgl_hash_get', 'fingerprint_unique_keys_hash', 'checkcaptcha-elapsed-diff', 'fingerprint_analytics_hash', 'fingerprint_label_performance_start_bucket', 'art__asn_window', 'fingerprint_unique_keys_meta_hash_hash', 'getcaptcha-host', 'parent_name', 'exception.message', 'service_name', 'base_url', 'app.exception_string', 'request.path', 'response.status_code', 'duration_ms', 'db.statement', 'score_reason', 'sk', 's3_upload_uri', 'trace.parent_id', 'name', 'art_ms', 'shard_id']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"window-hash\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"window-hash\",\"op\":\"exists\",\"join_column\":\"\"}],\"filter_combination\":\"AND\",\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}],\"time_range\":3600}\n",
    "\n",
    "{\"critique\": \"The response is mostly good, but misinterprets the time range and granularity settings. The QUERY correctly applies a COUNT calculation on events where 'window-hash' exists, which matches the NLQ requirement to count 'window-hash'. It breaks down by the 'window-hash' column for granularity and orders the results in descending order of the count, which is useful for identifying the most frequent 'window-hash' values. However, when NLQ says \"per hour\", it is likely seeking to show results with a granularity of 1 hour. As per query rules, this would require the 'time_range' to be at least '36000', which is 10 times the granularity value in seconds.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: latency distribution by status code\n",
    "\n",
    "COLUMNS: ['status_code', 'severity_code', 'status_message', 'rpc.grpc.status_code', 'duration_ms', 'container.uptime', 'rpc.method', 'message.type', 'rpc.service', 'rpc.system', 'severity_text', 'apiVersion', 'opencensus.resourcetype', 'k8s.node.network.io.transmit', 'meta.signal_type', 'ip', 'container.memory.major_page_faults', 'container.memory.page_faults', 'error', 'container.cpu.utilization', 'library.version', 'net.peer.port', 'container.memory.available', 'deprecatedLastTimestamp', 'container.memory.working_set', 'severity', 'k8s.pod.network.io.transmit', 'container.cpu.time', 'k8s.node.uptime', 'k8s.pod.uptime', 'container.filesystem.usage', 'container.filesystem.available', 'trace.trace_id', 'interface', 'deprecatedFirstTimestamp', 'deprecatedCount', 'span.kind', 'resourceVersion', 'k8s.node.network.errors.transmit', 'metadata', 'event.domain', 'trace.span_id', 'container.memory.usage', 'Kubernetes']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"status_code\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"},{\"column\":\"duration_ms\",\"op\":\"P95\"}],\"filters\":[{\"column\":\"status_code\",\"op\":\"exists\",\"join_column\":\"\"}],\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"Almost a good response. Having a HEATMAP(duration_ms) is excellent for visualizing the distribution of response times, and P95 gives a useful statistical measure of latency under typical conditions. However, filtering and breaking down by 'status_code' is unfortunately incorrect, as this does not refer to any system status code, but rather a span status. Instead, 'rpc.grpc.status_code' should be used instead. If there were an 'http.status_code' column, or an 'http.response.status_code' column, that could be a good grouping too.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: events by name\n",
    "\n",
    "COLUMNS: ['app.event_name', 'app.event_id', 'name', 'span.num_events', 'db.name', 'library.name', 'app.subscriber_name', 'parent_name', 'process.runtime.name', 'service.name', 'app.collection', 'job.enqueued_at', 'net.peer.name', 'db.system', 'http.route', 'error', 'job.created_at', 'exception.stacktrace', 'http.target', 'telemetry.sdk.name', 'http.host', 'messaging.system', 'db.statement', 'span.kind', 'type', 'http.method', 'service.version', 'messaging.destination', 'exception.message', 'exception.type', 'app.queued_times', 'process.pid', 'service.ownership', 'process.command', 'messaging.operation', 'duration_ms', 'library.version', 'meta.refinery.reason', 'messaging.protocol', 'telemetry.sdk.language', 'status_message', 'job.latency_ms', 'meta.annotation_type', 'messaging.rabbitmq.routing_key', 'deployment.environment', 'http.scheme', 'trace.link.trace_id', 'process.runtime.description', 'telemetry.instrumentation_library', 'trace.trace_id', 'trace.parent_id', 'http.status_code']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"app.event_name\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"app.event_name\",\"op\":\"=\",\"value\":\"Event drop\",\"join_column\":\"\"}],\"filter_combination\":\"AND\",\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"This is a bad response. The NLQ asks for events by name, which implies a need to count occurrences of events broken down by their name. The query correctly uses COUNT to aggregate events and breaks it down by app.event_name. However, it also includes a filter for a specific app.event_name value, which is unrelated to what NLQ asks for.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: /api/external_storage/gdrive/notify\n",
    "\n",
    "COLUMNS: ['document.api_status.dropbox_download_url', 'http.payment_endpoints_no_errors', 'http.document_upload_successful', 'http.document_upload_success_rate', 'http.is_document_upload', 'http.auth_endpoints_no_errors', 'status_message', 'document_processing_event.extension', 'document.s3_dir', 'http.space_endpoints_successful', 'http.grpc_requests_no_errors', 'document_processing_event.upload_source', 'http.is_payment', 'rpc.method', 'graphql.document', 'flipper.file_requests_in_dashboard', 'link.downloadable', 'rpc.service', 'messaging.operation', 'http.payment_success_rate', 'document.upload_error_rate', 'messaging.destination', 'link.re_signable', 'message', 'http.is_auth_endpoint', 'flipper.space_analytics', 'http.url', 'flipper.space_analytics_backfill', 'http.space_endpoints_success_rate', 'http.presentation_view_endpoints_successful', 'net.transport', 'messaging.system', 'link.type', 'document_processing_event.delayed_job_queue', 'request.id', 'http.grpc_requests_success_rate', 'http.elaine_web_request', 'http.link_successful_requests_rate', 'http.esign_successful_requests', 'link.watermark_enabled', 'service.component', 'user.login_success', 'document_processing_event.client', 'flipper.space_audit_log', 'status_code', 'document.document_group_id', 'http.link_successful_requests', 'http.target', 'link.recipient_kind', 'document.upload_unknown_error_rate', 'duration_ms', 'service.name', 'http.route', 'db.statement', 'error', 'trace.parent_id', 'http.status_code', 'name', 'user.signup_referer_path', 'exception.message', 'parent_name']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"name\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"}],\"filters\":[{\"column\":\"http.route\",\"op\":\"=\",\"value\":\"/api/external_storage/gdrive/notify\",\"join_column\":\"\"}],\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"The query correctly filters for the specific route '/api/external_storage/gdrive/notify'. Although it just shows a heatmap, because NLQ did not ask for anything specific, the heatmap is the best generat calculation to use. It may also be helpful to add a calculation for MAX or P95 to identify the slowest events explicitly. The breakdown by 'name' is generic and might not provide meaningful insights for this specific route unless 'name' is related to some aspect of the functionality at the route, but it is not a bad breakdown. It may also be helpful to filter to 'trace.parent_id' 'does-not-exist' so that it shows only requests to the endpoint, but since NLQ did not ask for requests, this isn't strictly necessary.\", \"outcome\": \"good\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: find the traces with the most spans\n",
    "\n",
    "COLUMNS: ['trace.span_id', 'trace.trace_id', 'trace_flags', 'trace_id', 'span_id', 'exception.stacktrace', 'trace.parent_id', 'span.num_links', 'span.num_events', 'span.kind', 'flags', 'nodejs_gc_duration_seconds.max', 'nodejs_gc_duration_seconds.avg', 'nodejs_gc_duration_seconds.p01', 'container.memory.usage.max', 'scrape_samples_post_metric_relabeling', 'container.duration', 'extensions', 'nodejs_gc_duration_seconds.sum', 'container.memory.utilized', 'nodejs_gc_duration_seconds.p001', 'nodejs_gc_duration_seconds.p99', 'scrape_samples_scraped', 'path', 'nodejs_gc_duration_seconds.p05', 'source', 'nodejs_eventloop_lag_mean_seconds', 'nodejs_eventloop_lag_max_seconds', 'nodejs_gc_duration_seconds.count', 'nodejs_gc_duration_seconds.min', 'container.cpu.onlines', 'up', 'partial_last', 'container.cpu.utilized', 'graphql.source', 'ecs.task.memory.usage.max', 'nodejs_gc_duration_seconds.p95', 'ecs.task.cpu.onlines', 'container']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"trace.trace_id\"],\"calculations\":[{\"op\":\"COUNT\"}],\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}]}\n",
    "\n",
    "{\"critique\": \"Bad response. The COUNT operation is incorrectly used because it does not count the number of spans per trace, but the total events within a window. Instead, the calculations should use a COUNT_DISTINCT(trace.span_id). Breaking down by 'trace.trace_id' is good, however, because that lets the user see each trace with a span count. Finally, the query should have an ordering of COUNT_DISTINCT(trace.span_id) 'desc' so that results are ordered by most spans.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: api calls that are failing\n",
    "\n",
    "COLUMNS: ['cronofy.oauth_client_notification_subscriptions.failed', 'apirequestjournal.failed_journal_reads', 'cronofy.notifier_client.api_call_status', 'auth.oauthinator.go_to.api_call_count', 'api.event.conferencing.error', 'cronofy.notifier_client.api_call_path', 'cronofy.notifier_client.api_call_ms', 'api.event.conferencing.sync.error', 'auth.oauthinator.zoom.api_call_count', 'cronofy.service_account.relinks.failed', 'auth.failure', 'cronofy.api.rate_limits.oauth_client_account_exceeded', 'cronofy.notifier_client.api_call_method', 'auth.oauthinator.ms_teams.api_call_count', 'cronofy.api.rate_limits.oauth_client_exceeded', 'cronofy.embedded_auth_token.errors', 'apirequestjournal.failed_dir_enumerations', 'indeed.put.error', 'apirequestjournal.read_dir.total_calls', 'integrations.indeed.api_response.status', 'api.free_busy.caching.hit', 'shipper.ingress.failed_open', 'error.service_unavailable_error?', 'auth.oauthinator.go_to.refresh_failure_count', 'auth.provider_error_code', 'error', 'auth.graph_access_token_attempt', 'apirequestjournal.results', 'eightbyeight.put.error', 'integrations.slack.api_response.status', 'sli.apirequestjournal.dashboard.ok', 'integrations.hubspot.api_response.status', 'integrations.api.operation', 'enterprise_connect.request_context.failure_key', 'api.params.google_event_ids', 'eightbyeight.persistent.get.error', 'auth.error_code', 'auth.graph_me_attempt', 'auth.op', 'eightbyeight.persistent_cache.get.error', 'cronofy.calendar_api', 'auth.oauthinator.zoom.refresh_failure_count', 'shared.put.error', 'filehousekeeper.delete.failures', 'api.event.conferencing.dial_in', 'api.free_busy.total_pages', 'integrations.zendesk.api_response.status', 'apirequestjournal.index_job.total_indexing_errors', 'ms_graph.failed_delta_link_fallback', 'api.free_busy.page_events', 'name', 'shipper.ingress.full_path', 'cronofy.account_profile', 'trace.parent_id', 'http.status', 'scheduler.build_availability_query.type', 'active_job.class', 'enterprise_connect.request_context.error', 'cronofy.oauth_client', 'duration_ms', 'sync.state', 'cronofy.environment.name', 'sync.sync_type', 'http.path', 'service_name']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"http.path],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"http.status\",\"op\":\"exists\",\"join_column\":\"\"},{\"column\":\"http.status\",\"op\":\"!=\",\"value\":\"200\",\"join_column\":\"\"}],\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}],\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"This query is too specific. The columns 'http.path' and 'http.status' are more generic columns, meaning that they capture information about more endpoints than the columns selected ('cronofy.notifier_client.api_call_status' and 'cronofy.notifier_client.api_call_path'). By using those more specific columns, the query makes an assumption that NLQ is referring to these specific columns, when there is no evidence that NLQ is doing that. Although the structure of this response is good, it should use the more generic columns 'http.status' and 'http.path' instead.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: ListWalletMultichainTokenHoldings\n",
    "\n",
    "COLUMNS: ['service_dc_2', 'service_dc_1', 'service_dc_3', 'service_dc_5', 'service_dc_4', 'success_rate_v3', 'error', 'success_rate_demo', 'duration_ms', 'name', 'dc_simple_refinery_reason', 'client-uuid', 'meta.stressed', 'go.tb', 'ip', 'span.kind', 'db.statement.fingerprint', 'db.statement', 'span.num_links', 'parent_name', 'dc_log10_duration', 'http.url', 'db.rowcount', 'meta.refinery.reason', 'span.num_events', 'host.name', 'dc_db_system_or_type', 'log10_duration_ms', 'service.name', 'type', 'meta.signal_type', 'error.object', 'trace.span_id', 'status_code', 'meta.annotation_type', 'trace.trace_id', 'meta.refinery.sample_key', 'http.method', 'db.canonical_name', 'dc_is_root_span', 'grpc.source', 'trace.parent_id', 'duration_per_rowcount', 'meta.refinery.original_sample_rate', 'go.tb.exists', 'opencensus.exporterversion', 'dc_is_success', 'meta.refinery.send_reason', 'http.status_code', 'dc_ensure_nonroot_server_span', 'http.route']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"name\"],\"filters\":[{\"column\":\"name\",\"op\":\"=\",\"value\":\"ListWalletMultichainTokenHoldings\"}],\"calculations\":[{\"op\":\"COUNT\"}],\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}],\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"The query correctly filters on the 'name' column to isolate the 'ListWalletMultichainTokenHoldings' service. Using COUNT as a calculation is appropriate to get the number of occurrences, and ordering by count descending will properly rank the results. It could be improved by adding a calculation that includes the 'duration_ms' column such as AVG or MAX to assess the latency, as the NLQ suggests a potential interest in performance metrics for this particular service.\", \"outcome\": \"good\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: overall distribution of latencies\n",
    "\n",
    "COLUMNS: ['duration_ms', 'container.uptime', 'container.cpu.time', 'event.domain', 'container.cpu.utilization', 'ip', 'regarding', 'container.memory.page_faults', 'severity', 'metadata', 'interface', 'container.memory.major_page_faults', 'span.kind', 'namespace', 'k8s.node.cpu.time', 'error', 'container.filesystem.usage', 'uid', 'container.memory.working_set', 'span.num_events', 'container.memory.usage', 'flags', 'k8s.node.uptime', 'event.name', 'k8s.pod.cpu.time', 'k8s.pod.uptime', 'container.filesystem.capacity', 'name', 'container.memory.available', 'k8s.node.network.io.transmit', 'severity_text', 'k8s.node.cpu.utilization', 'body', 'reason', 'note', 'kind', 'net.peer.port', 'k8s.pod.cpu.utilization', 'type', 'Kubernetes']\n",
    "\n",
    "QUERY: {\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"},{\"column\":\"duration_ms\",\"op\":\"P95\"}],\"breakdowns\":[{\"namespace\"},\"orders\":[{\"op\":\"P95\",\"column:\"duration_ms\",\"order\":\"descending\"}]]\n",
    "\n",
    "{\"critique\": \"This is a bad query. While it includes relevant calculations, there are far too many, which is an overwhelming response. Only a HEATMAP and a common aggregation like P95 should be used. Additionally, an ideal query would break down by a column like 'namespace' even though NLQ did not ask for it, because that can yield more interesting results.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: error\n",
    "\n",
    "COLUMNS: ['error', 'env', 'exception.message', 'type', 'name', 'http.error_message', 'exception.type', 'http.error_name', 'exception.stacktrace', 'status_code', 'status_message', 'span.kind', 'net.transport', 'library.name', 'tls.authorized', 'http.target', 'service.name', 'http.url', 'tls.protocol', 'duration_ms', 'process.owner', 'library.version', 'process.command', 'parent_name', 'process.pid', 'http.flavor', 'deployment.environment', 'http.method', 'next.bubble', 'http.host', 'next.route', 'net.host.ip', 'http.scheme', 'telemetry.sdk.language', 'deployment.name', 'trace.parent_id', 'tls.alpnProtocol', 'trace.span_id', 'http.response_content_length_uncompressed', 'net.host.name', 'trace.trace_id', 'net.host.port', 'http.status_code', 'http.response_content_length', 'process.command_args', 'net.peer.name', 'http.user_agent', 'telemetry.instrumentation_library', 'net.peer.port', 'process.runtime.description', 'http.route']\n",
    "\n",
    "QUERY: {\"breakdowns\":[\"name\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"error\",\"op\":\"=\",\"value\":true,\"join_column\":\"\"}],\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}],\"time_range\":7200}\n",
    "\n",
    "{\"critique\": \"Bad response. The query correctly filters by errors and uses COUNT, but it breaks down by 'error' instead of a more useful column like 'name', which is the name of an operation. Although NLQ is vague, it's best to be generally helpful and show more information than less.\", \"outcome\": \"bad\"}\n",
    "\n",
    "---\n",
    "\n",
    "NLQ: I would like to pull a number from a string\n",
    "\n",
    "COLUMNS: ['github.pull_requests.0.number', 'github.run_number', 'github.job.step.number', 'github.pull_requests.0.id', 'library.name', 'github.pull_requests.0.head.sha', 'github.pull_requests.0.url', 'name', 'github.pull_requests.0.base.sha', 'github.pull_requests.0.head.repo.id', 'github.pull_requests.0.head.repo.name', 'github.pull_requests.0.base.ref', 'github.job.name', 'service.name', 'github.pull_requests.0.head.ref', 'github.pull_requests.0.head.repo.url', 'github.job.id', 'github.job.step.name', 'telemetry.sdk.language', 'github.pull_requests.0.base.repo.name', 'github.job.conclusion', 'github.conclusion', 'span.kind', 'service.version', 'github.pull_requests.0.base.repo.url', 'error', 'github.pull_requests.0.base.repo.id', 'github.job.step.conclusion', 'status_code', 'span.num_events', 'service.instance.id', 'github.job.runner_name', 'github.job.run_id', 'github.workflow', 'github.html_url', 'github.run_attempt', 'github.event', 'github.job.runner_group_id', 'github.job.run_attempt', 'telemetry.sdk.name', 'github.workflow_id', 'github.head_commit.id', 'service.namespace', 'telemetry.sdk.version', 'github.author_name', 'github.head_commit.message', 'github.head_sha', 'trace.span_id', 'github.base_sha', 'github.workflow_url', 'trace.parent_id', 'duration_ms', 'http.route']\n",
    "\n",
    "QUERY: \n",
    "\n",
    "{\"critique\": \"This query is good because it does a best effort at aligning with a vague NLQ. By checking if 'name' contains 'string', the query is about as close to an interpretation of NLQ as-is possible without further clarification. A COUNT of events broken down by 'name' where 'name' contains 'string' is the best way to interpret a vague NLQ like this.\", \"outcome\": \"good\"}\n",
    "\n",
    "---\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ddc9cbf-6ce2-46c8-bb45-fde3f949d9a2",
   "metadata": {},
   "source": [
    "## Critic Prompt\n",
    "\n",
    "Our goal is build a query critic that can be used to identify good and bad queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d86261ab-4bd2-4ce6-8225-9ca6da83b4bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def critic_prompt(nlq, cols, query):\n",
    "    \"Construct a cirtic prompt.\"\n",
    "    return CRITIC_PROMPT + f\"\"\"\n",
    "For the below NLQ, QUERY and COLUMNS provide a critique as JSON in the format {{\"critique\": \"...\", \"outcome\": \"good\"|\"bad\"}} as shown above.\n",
    "\n",
    "NLQ: {nlq}\n",
    "\n",
    "COLUMNS: {cols}\n",
    "\n",
    "QUERY: {query}\n",
    "\"\"\"    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9ab6096-f463-4d86-acdb-067d80418c96",
   "metadata": {},
   "source": [
    "Let's see an example of a fully-assembled critic prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2f3c1f02-8b6f-4ff2-990a-3b005b3f8872",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## Background\n",
      "\n",
      "Honeycomb is an observability platform that allows you to write queries to inspect trace data.\n",
      "The specification of the Honeycomb query language is as follows:\n",
      "\n",
      "QUERY SPEC:\n",
      "All top-level keys are optional.\n",
      "\n",
      "```json\n",
      "\"calculations\":[\n",
      "    // ops: COUNT, CONCURRENCY, COUNT_DISTINCT, HEATMAP, SUM, AVG, MAX, MIN, P001, P01, P05, P10, P25, P50, P75, P90, P95, P99, P999, RATE_AVG, RATE_SUM, RATE_MAX\n",
      "    {\"op\": \"COUNT\"},// COUNT and CONCURRENCY are just op\n",
      "    {\"op\": \"HEATMAP\", \"column\": \"name\"}\n",
      "],\n",
      "\"filters\":[\n",
      "    // ops: =, !=, >, >=, <, <=, starts-with, does-not-start-with, exists, does-not-exist, contains, does-not-contain, in, not-in\n",
      "    {\"column\": \"name\", \"op\": \"exists\"}, // exists and does-not-exist ops only have column\n",
      "    {\"column\": \"name\", \"op\": \"=\", \"value\": \"something\"}\n",
      "],\n",
      "\"filter_combination\": \"AND\", // AND or OR\n",
      "\"breakdowns\":[\n",
      "    // columns in COLUMNS\n",
      "    \"column1\",\"column2\"\n",
      "],\n",
      "\"orders\":[\n",
      "    // HEATMAP not allowed\n",
      "    // Must come from breakdowns or calculations\n",
      "    {\"op\": \"op_in_calculation\", \"column\": \"column_in_calculation\", \"order\": \"ascending\"},\n",
      "    {\"op\": \"COUNT\", \"order\": \"descending\"}, // COUNT and CONCURRENCY have no column\n",
      "    {\"column\": \"column1\", \"order\": \"descending\"},\n",
      "],\n",
      "\"havings\":[\n",
      "    // HEATMAP not allowed\n",
      "    {\"calculate_op\": \"op_in_calculation\", \"column\": \"name\", \"op\": \"OPNAME\", \"value\": 100},\n",
      "    {\"calculate_op\": \"COUNT\", \"op\": \">\", \"value\": 10}, // COUNT and CONCURRENCY have no column\n",
      "],\n",
      "\"time_range\": 7200, // Relative time range in seconds.\n",
      "\"start_time\": 1234567890, // UNIX timestamp\n",
      "\"end_time\": 1234567890, // UNIX timestamp\n",
      "```\n",
      "\n",
      "QUERY SPEC TIPS:\n",
      "\n",
      "COUNT counts events/spans. COUNT_DISTINCT counts unique values of columns.\n",
      "HEATMAP shows value distributions.\n",
      "trace.parent_id does-not-exist shows root span of a trace. Use that to count requests and traces.\n",
      "name is a span or span event.\n",
      "parent_name is the name of a span that created a span event.\n",
      "\n",
      "When the user input is asking about a time range (such as \"yesterday\" or \"since last week\"),\n",
      "always use the time_range, start_time, and end_time fields. time_range\n",
      "is relative and can be combined with either start_time or end_time but not both.\n",
      "\n",
      "---\n",
      "\n",
      "## Instructions\n",
      "\n",
      "You are an EXPERT query evaluator that has advanced capabilities to judge if a query good or not.  You understand the nuances of the Honeycomb query language, including what is likely to be most useful from an analytics perspective.\n",
      "You are given the following three inputs: (1) NLQ, (2) A list of candidate columns (COLUMNS) that are allowed to be in the query, and (3) The query (QUERY). Your job is to evaluate and critique the QUERY relative to the provided NLQ and COLUMNS. \n",
      "\n",
      "The critiques must be provided in the same json format as provided in the examples below:\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: show me slowest trace\n",
      "\n",
      "COLUMNS: ['trace.trace_id', 'trace.span_id', 'trace.parent_id', 'duration_ms', 'name', 'faas.instance', 'faas.id', 'filter', 'telemetry.instrumentation_library', 'library.name', 'faas.name', 'span.kind', 'type', 'http.wrote_bytes', 'http.url', 'service.name', 'http.flavor', 'span.num_links', 'span.num_events', 'net.host.name', 'library.version', 'http.scheme', 'net.peer.name', 'http.method', 'meta.signal_type', 'cloud.region', 'cloud.provider', 'faas.version', 'http.read_bytes', 'http.user_agent', 'cloud.account.id', 'organization_id', 'cloud.platform', 'net.sock.peer.addr', 'page_size', 'net.sock.peer.port', 'page_token', 'status_code', 'http.client_ip', 'http.status_code', 'http.route']\n",
      "\n",
      "QUERY: {\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"MAX\"}],\"filters\":[{\"column\":\"trace.parent_id\",\"op\":\"does-not-exist\",\"join_column\":\"\"}],\"orders\":[{\"column\":\"duration_ms\",\"op\":\"MAX\",\"order\":\"descending\"}],\"limit\":1,\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"The response is nearly correct, as it is looking for the slowest trace by using MAX(duration_ms) and ordering by duration_ms in descending order, which is appropriate for finding the 'slowest' trace. Additionally, filtering with trace.parent_id does-not-exist correctly identifies root spans. However, the query should be grouping by trace.trace_id to actually show the slowest trace. Without that grouping, the query only shows the MAX(duration_ms) measurement over time, irrespective of which trace is responsible for that measurement.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: slow requests\n",
      "\n",
      "COLUMNS: ['duration_ms', 'deprecatedSource', 'error', 'severity', 'flags', 'rpc.method', 'deprecatedCount', 'ip', 'rpc.service', 'interface', 'metadata', 'reason', 'container.cpu.time', 'deprecatedLastTimestamp', 'name', 'regarding', 'namespace', 'message.type', 'watch-type', 'severity_text', 'apiVersion', 'kind', 'service.name', 'rpc.system', 'container.uptime', 'type', 'uid', 'span.kind', 'db.name', 'note', 'body', 'resourceVersion', 'library.version', 'db.statement', 'status_code', 'event.domain', 'managedFields', 'deprecatedFirstTimestamp', 'container.memory.page_faults', 'event.name', 'container.memory.available', 'k8s.container.cpu_request', 'container.id', 'library.name', 'container.memory.rss', 'message.uncompressed_size', 'rpc.grpc.status_code', 'net.peer.port', 'k8s.container.memory_request', 'container.memory.working_set', 'trace.parent_id', 'http.route', 'service_name', '']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"name\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"},{\"column\":\"duration_ms\",\"op\":\"MAX\"}],\"filters\":[{\"column\":\"trace.parent_id\",\"op\":\"does-not-exist\",\"join_column\":\"\"}],\"orders\":[{\"column\":\"duration_ms\",\"op\":\"MAX\",\"order\":\"descending\"}],\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"This response is adequate. The inclusion of a HEATMAP and a MAX calculation on the duration_ms column is appropriate for identifying slow requests. Correctly including a filter on 'trace.parent_id' with 'does-not-exist' ensures that only root spans (requests) are considered. Ordering by the max duration_ms in descending order is sensible to quickly highlight slower requests. The use of 'name' in a breakdown is good, because it groups results into span names. This can help you quickly narrow down behavior to a particular part of a system. It might be more insightful to break down by 'http.route', because 'http.route' will also break down more broadly by API call. Having both 'http.route' and 'name' included can give both a broader and narrower breadkdown of the data, and both views can be helpful at different times.. Overall, the query aligns well with the NLQ but could benefit from a more specific breakdown column.\", \"outcome\": \"good\"}\n",
      "\n",
      "---\n",
      "NLQ: records of clientid 5fcedb77-267c-44e3-8ea4-4d21786b2618\n",
      "\n",
      "COLUMNS: ['clientIds', 'sys-client-id', 'principal.clientId', 'client.id', 'clientName', 'metadata.clientId', 'payload.clientId', 'clientExternalId', 'informationReportingEvent.clientId', 'principalClientId', 'metadata.clientTxId', 'targetClientId', 'e2eId', 'client', 'transactionIds', 'clientId', 'client.external_id', 'metadata.clientName', 'sys-service-provider-id', 'transactionId', 'paymentRequestIds.identifying', 'serviceProviderId', 'service_provider.id', 'externalId', 'traceId', 'event.id', 'accounts-deleted', 'jobExecutionId', 'accounts-updated', 'syncContext.clientDto.id', '$metadata.requestId', 'subscriptionId', 'paymentExternalId', 'activeIntegrationIds', 'fileId', 'tracerId', '$metadata.extendedRequestId', 'moneyTransferId', 'paymentRequestIds', 'vendor.id', 'triggeredNumberOfClients', 'payload.eventId', 'existedTransactionIds', 'completedTransactionIds', 'metadata.paymentRequestExternalId', 'syncContext.clientDto.externalId', 'createdTransactionIds', 'transactionsDeletionServiceProviderId', 'idempotency_key', 'RequestId', 'status', 'error', 'duration_ms', 'http.route', 'name', 'http_path', 'trace.parent_id', 'http.status_code', 'meta.req.query', 'service_name', 'request_time']\n",
      "\n",
      "QUERY: {\"filters\":[{\"column\":\"client.id\",\"op\":\"=\",\"value\":\"5fcedb77-267c-44e3-8ea4-4d21786b2618\"}]}\n",
      "\n",
      "{\"critique\": \"Poor response. While the filter by client.id is correct to find records for clientid '5fcedb77-267c-44e3-8ea4-4d21786b2618', the query lacks breakdowns, calculations, or order to make the results more insightful. It would be more beneficial to include calculations such as COUNT to understand the volume of records.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: Find the traces for the slowest requests in the past 3 hours where http.host = internal-pcmws-alkqa-routing-black-1164727359.us-east-1.elb.amazonaws.com\n",
      "\n",
      "COLUMNS: ['trace.trace_id', 'http.host', 'exception.stacktrace', 'pcm.drawtraffic', 'SLI_geo-95-percentile-latency', 'pcm.server.worker-load-balancer', 'pcm.responsetime', 'aws.requestId', 'http.query-length', 'http.response-uri', 'trace.span_id', 'net.host.name', 'http-500-error-percent', 'http.url', 'net.host.port', 'pcm.server.worker-app-pool', 'http.request-headers', 'duration_ms', 'pcm.restful', 'http.response-version', 'trace.parent_id', 'pcm.initialmaprequest', 'http.response-headers', 'http.method', 'aws.region', 'service.instance.id', 'http.response-method', 'wcf.channel.path', 'pcm.server', 'pcm.userip', 'http.version', 'pcm.server-name', 'pcm.server-color', 'aws.service', 'pcm.stopcount', 'rpc.method', 'pcm.splcs', 'peer.service', 'wcf.channel.scheme', 'service.name', 'http.response_content_length', 'pcm.foserver', 'SLI_99.9-percent-without-500-error', 'pcm.httpcode', 'pcm.errortext', 'http.status-description', 'pcm.login', 'pcm.apikey', 'rpc.service', 'pcm.date', 'db.statement', 'http.route', 'parent_name', 'name', 'http.status_code', 'exception.message', 'error']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"trace.trace_id\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"MAX\"}],\"filters\":[{\"column\":\"http.host\",\"op\":\"=\",\"value\":\"internal-pcmws-alkqa-routing-black-1164727359.us-east-1.elb.amazonaws.com\",\"join_column\":\"\"}],\"filter_combination\":\"AND\",\"orders\":[{\"column\":\"duration_ms\",\"op\":\"MAX\",\"order\":\"descending\"}],\"time_range\":10800}\n",
      "\n",
      "{\"critique\": \"The query correctly uses time_range to indicate the past 3 hours and filters by the specific http.host value. However, it should also include a filter where trace.parent_id does-not-exist to identify the root spans or initial requests, which represent the start of a trace. Without this, the query might include intermediate spans that are not the start of the trace. The breakdown by trace.trace_id is correct, and ordering by the max duration is appropriate to find the slowest requests. It could be improved by including a calculation of COUNT to gauge the number of slow traces for each trace id, making it thorough for understanding general patterns or anomalies.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: show me traces where ip is 10.0.2.90\n",
      "\n",
      "COLUMNS: ['net.host.ip', 'trace.trace_id', 'net.peer.ip', 'net.transport', 'net.host.name', 'http.client_ip', 'http.host', 'host.name', 'trace.span_id', 'net.host.port', 'k8s.pod.ip', 'sli_route_latency_1500', 'ip', 'net.peer.port', 'trace.parent_id', 'sli_route_latency', 'http.server_name', 'service.name', 'service.instance.id', 'rpc.user_agent', 'rpc.service', 'http.route', 'db.name', 'http.url', 'area', 'requestID', 'service.version', 'http.user_agent', 'http.target', 'host.arch', 'orderid', 'process.pid', 'rpc.system', 'os.type', 'rpc.method', 'otlp.exporter.exported', 'runtime.jvm.gc.time', 'http.scheme', 'thread.name', 'name', 'runtime.jvm.memory.pool', 'message.type', 'runtime.jvm.memory.area', 'thread.id', 'enduser.id', 'error', 'userid', 'http.flavor', 'pool', 'telemetry.instrumentation_library', 'http.status_code', 'service_name', 'parent_name', 'duration_ms', 'db.statement', 'exception.message']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"trace.trace_id\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"net.host.ip\",\"op\":\"=\",\"value\":\"10.0.2.90\"}]}\n",
      "\n",
      "{\"critique\": \"The query satisfies the NLQ. It correctly filters for traces with an IP address of 10.0.2.90 and counts the occurrences of those traces, grouped by trace.trace_id, which is what the NLQ requests. The response is good as it meets the requirement of showing traces from a specific IP address without additional complexities.\", \"outcome\": \"good\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: count of moderation.flags.create api calls\n",
      "\n",
      "COLUMNS: ['response_flags', 'job_counts_MULTI_INDEX_CREATE', 'http_count', 'found_count', 'is_creation', 'is_create', 'job_counts_MULTI_INDEX_BATCH_CREATE', 'event_count', 'job_counts_MULTI_INDEX_CREATE_BE', 'job_counts_SEND_SLACK_UNFURLS_UPDATE', 'queued_count', 'keys_count', 'agenda_req_count', 'flags', 'job_counts_SEND_SLACK_DOC_THREAD_UPDATE', 'update_count', 'log_count', 'old_activity_badge_count', 'total_message_count', 'job_counts_SEND_SLACK_USER_MENTIONS_UPDATE', 'job_counts_UNIQUE_INDEX_CREATE', 'request_attempt_count', 'job_counts_ADD_TO_SLACK_DOCS_COMPANY', 'args.thread_counts_by_channel', 'job_counts_FINALIZE_EDIT_HISTORY', 'mc_queries_count', 'hitCount', 'api:warning', 'job_counts_MULTI_INDEX_BATCH_CREATE_BE', 'file_count', 'job_counts_UPDATE_THREAD_UNREAD_COUNTS_EF4', 'api:method', 'executing_count', 'job_counts_SEND_SLACK_FILES_UPDATE', 'failed_count', 'mpdm_count', 'count', 'grpc_req_count', 'job_counts_UPDATE_SLACK_REFERENCES_IN_THREAD', 'mc_queries_multi_count', 'channel_ids_count', 'job_counts_UPDATE_THREAD_PHRASES_ON_EDIT', 'total_participant_count', 'job_counts_MULTI_INDEX_DELETE', 'job_counts_UPDATE_THREAD_UNREAD_COUNTS_BE', 'kafkagate_count', 'authorized_apps_count', 'redirect_count', 'apps_count', 'bulk_generate_calls', 'duration_ms', 'slath', 'name', 'execution_context', 'trace_parent_id', 'db.statement', 'namespace', 'kind', 'error', 'message', 'reason', 'http.status_code', 'service_name', 'quip:call_path']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"slath\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"name\",\"op\":\"=\",\"value\":\"moderation.flags.create\",\"join_column\":\"\"}],\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"The query groups by 'slath' which is extraneous information - there is nothing in NLQ that indicates 'slath' is related to NLQ. Additionally, the filter clause should check for the 'contains' op instead of '='. While it is likely good to check for 'name' as that represents an operation name, which means it will likely have data NLQ is referring to, there is no guarantee that the actual name of the operation is exactly what was input in NLQ. Furthermore, the result should include 'name' in the breakdowns so that the results show the actual values of 'name'.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: What kind of events are contributing towards my rate limits\n",
      "\n",
      "COLUMNS: ['up', 'k8s.pod.memory.major_page_faults', 'k8s.pod.memory_limit_utilization', 'k8s.pod.cpu_limit_utilization', 'k8s.node.memory.page_faults', 'k8s.container.cpu_limit', 'k8s.node.memory.major_page_faults', 'k8s.pod.memory.page_faults', 'k8s.container.memory_limit', 'k8s.node.filesystem.usage', 'otelcol_processor_batch_batch_send_size.avg', 'k8s.pod.memory.rss', 'k8s.pod.filesystem.usage', 'k8s.node.network.io.transmit', 'prometheus.io/port', 'scrape_duration_seconds', 'eks.amazonaws.com/compute-type', 'otelcol_processor_batch_timeout_trigger_send', 'otelcol_receiver_refused_metric_points', 'k8s.node.memory.rss', 'otelcol_processor_batch_batch_send_size.count', 'eks.amazonaws.com/component', 'otelcol_processor_dropped_metric_points', 'k8s.pod.network.io.transmit', 'otelcol_processor_refused_metric_points', 'job.runner.gitlab.com/url', 'scrape_samples_scraped', 'prometheus.io/scrape', 'k8s.pod.memory.usage', 'k8s.node.cpu.time', 'otelcol_processor_batch_batch_send_size.p50', 'k8s.node.network.io.receive', 'k8s.node.filesystem.capacity', 'k8s.node.memory.usage', 'otelcol_processor_dropped_log_records', 'k8s.container.restarts', 'k8s-app', 'otelcol_exporter_queue_capacity', 'k8s.pod.memory_request_utilization', 'k8s.pod.network.errors.transmit', 'otelcol_processor_batch_batch_send_size.max', 'otelcol_receiver_accepted_metric_points', 'otelcol_exporter_sent_metric_points', 'k8s.pod.filesystem.capacity', 'project.runner.gitlab.com/id', 'processor', 'otelcol_scraper_errored_metric_points', 'otelcol_processor_batch_batch_send_size.p05', 'interface', 'otelcol_process_memory_rss', 'net.host.name', 'name', 'trace.parent_id', 'duration_ms', 'http.route', 'service_name']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"name\"],\"calculations\":[{\"column\":\"otelcol_processor_refused_metric_points\",\"op\":\"SUM\"},{\"column\":\"otelcol_processor_dropped_metric_points\",\"op\":\"SUM\"},{\"column\":\"otelcol_processor_dropped_log_records\",\"op\":\"SUM\"}],\"filters\":[{\"column\":\"otelcol_processor_refused_metric_points\",\"op\":\">\",\"value\":0},{\"column\":\"otelcol_processor_dropped_metric_points\",\"op\":\">\",\"value\":0},{\"column\":\"otelcol_processor_dropped_log_records\",\"op\":\">\",\"value\":0}]}\n",
      "\n",
      "{\"critique\": \"Good response. The query is grouping by 'name' to identify different event types and summing up refused, dropped metric points and dropped log records, which are potential indicators of hitting rate limits. Furthermore, it properly filters each of those to be greater than zero to only include relevant events. This helps to understand what events are contributing towards rate limits being exceeded.\", \"outcome\": \"good\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: count window-hash where window-hash exists per hour\n",
      "\n",
      "COLUMNS: ['window-hash', 'window_hash_get', 'prop-window_hash', 'fingerprint_window_stack_traces_hash', 'prop-window_hash_get', 'fingerprint_window_overloaded_hash', 'fingerprint_window_document_keys_hash', 'fingerprint_window_keys_hash', 'fingerprint_window_client_tools_hash', 'fingerprint_window_messages_hash', 'headers-hash', 'fingerprint_time_zone_anomaly_hash', 'best_hash_get', 'w_count', 'legacy-hash', 'fingerprint_window_stack_traces', 'fingerprint_time_zone_hash', 'siteverify_nojs-hdict_count', 'fingerprint_time_zone_region_hash', 'fingerprint_window_missing_features_hash', 'fingerprint_timing_hash', 'fetch-hash', 'client_timestamp_check', 'canvas-hash', 'fingerprint_unique_keys_stable_hash_hash', 'fingerprint_window_document_keys', 'fingerprint_window_keys', 'task_request_count', 'fingerprint_web_worker_stack_size_hash', 'fingerprint_window_overloaded', 'fingerprint_window_version', 'fingerprint_time_zone_anomaly', 'token_rate_1m_tarpit', 'r_count', 'art_up__asn_window', 'fingerprint_unique_keys_stable_hash', 'sk_wgl_canv_hash', 'fingerprint_dom_rect_decimal_hash', 'fingerprint_window_messages', 'webrtc-hash', 'legacy_hash_get', 'fingerprint_window_sus_chrome_key', 'md_hash', 'webgl_hash_get', 'fingerprint_unique_keys_hash', 'checkcaptcha-elapsed-diff', 'fingerprint_analytics_hash', 'fingerprint_label_performance_start_bucket', 'art__asn_window', 'fingerprint_unique_keys_meta_hash_hash', 'getcaptcha-host', 'parent_name', 'exception.message', 'service_name', 'base_url', 'app.exception_string', 'request.path', 'response.status_code', 'duration_ms', 'db.statement', 'score_reason', 'sk', 's3_upload_uri', 'trace.parent_id', 'name', 'art_ms', 'shard_id']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"window-hash\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"window-hash\",\"op\":\"exists\",\"join_column\":\"\"}],\"filter_combination\":\"AND\",\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}],\"time_range\":3600}\n",
      "\n",
      "{\"critique\": \"The response is mostly good, but misinterprets the time range and granularity settings. The QUERY correctly applies a COUNT calculation on events where 'window-hash' exists, which matches the NLQ requirement to count 'window-hash'. It breaks down by the 'window-hash' column for granularity and orders the results in descending order of the count, which is useful for identifying the most frequent 'window-hash' values. However, when NLQ says \"per hour\", it is likely seeking to show results with a granularity of 1 hour. As per query rules, this would require the 'time_range' to be at least '36000', which is 10 times the granularity value in seconds.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: latency distribution by status code\n",
      "\n",
      "COLUMNS: ['status_code', 'severity_code', 'status_message', 'rpc.grpc.status_code', 'duration_ms', 'container.uptime', 'rpc.method', 'message.type', 'rpc.service', 'rpc.system', 'severity_text', 'apiVersion', 'opencensus.resourcetype', 'k8s.node.network.io.transmit', 'meta.signal_type', 'ip', 'container.memory.major_page_faults', 'container.memory.page_faults', 'error', 'container.cpu.utilization', 'library.version', 'net.peer.port', 'container.memory.available', 'deprecatedLastTimestamp', 'container.memory.working_set', 'severity', 'k8s.pod.network.io.transmit', 'container.cpu.time', 'k8s.node.uptime', 'k8s.pod.uptime', 'container.filesystem.usage', 'container.filesystem.available', 'trace.trace_id', 'interface', 'deprecatedFirstTimestamp', 'deprecatedCount', 'span.kind', 'resourceVersion', 'k8s.node.network.errors.transmit', 'metadata', 'event.domain', 'trace.span_id', 'container.memory.usage', 'Kubernetes']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"status_code\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"},{\"column\":\"duration_ms\",\"op\":\"P95\"}],\"filters\":[{\"column\":\"status_code\",\"op\":\"exists\",\"join_column\":\"\"}],\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"Almost a good response. Having a HEATMAP(duration_ms) is excellent for visualizing the distribution of response times, and P95 gives a useful statistical measure of latency under typical conditions. However, filtering and breaking down by 'status_code' is unfortunately incorrect, as this does not refer to any system status code, but rather a span status. Instead, 'rpc.grpc.status_code' should be used instead. If there were an 'http.status_code' column, or an 'http.response.status_code' column, that could be a good grouping too.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: events by name\n",
      "\n",
      "COLUMNS: ['app.event_name', 'app.event_id', 'name', 'span.num_events', 'db.name', 'library.name', 'app.subscriber_name', 'parent_name', 'process.runtime.name', 'service.name', 'app.collection', 'job.enqueued_at', 'net.peer.name', 'db.system', 'http.route', 'error', 'job.created_at', 'exception.stacktrace', 'http.target', 'telemetry.sdk.name', 'http.host', 'messaging.system', 'db.statement', 'span.kind', 'type', 'http.method', 'service.version', 'messaging.destination', 'exception.message', 'exception.type', 'app.queued_times', 'process.pid', 'service.ownership', 'process.command', 'messaging.operation', 'duration_ms', 'library.version', 'meta.refinery.reason', 'messaging.protocol', 'telemetry.sdk.language', 'status_message', 'job.latency_ms', 'meta.annotation_type', 'messaging.rabbitmq.routing_key', 'deployment.environment', 'http.scheme', 'trace.link.trace_id', 'process.runtime.description', 'telemetry.instrumentation_library', 'trace.trace_id', 'trace.parent_id', 'http.status_code']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"app.event_name\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"app.event_name\",\"op\":\"=\",\"value\":\"Event drop\",\"join_column\":\"\"}],\"filter_combination\":\"AND\",\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"This is a bad response. The NLQ asks for events by name, which implies a need to count occurrences of events broken down by their name. The query correctly uses COUNT to aggregate events and breaks it down by app.event_name. However, it also includes a filter for a specific app.event_name value, which is unrelated to what NLQ asks for.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: /api/external_storage/gdrive/notify\n",
      "\n",
      "COLUMNS: ['document.api_status.dropbox_download_url', 'http.payment_endpoints_no_errors', 'http.document_upload_successful', 'http.document_upload_success_rate', 'http.is_document_upload', 'http.auth_endpoints_no_errors', 'status_message', 'document_processing_event.extension', 'document.s3_dir', 'http.space_endpoints_successful', 'http.grpc_requests_no_errors', 'document_processing_event.upload_source', 'http.is_payment', 'rpc.method', 'graphql.document', 'flipper.file_requests_in_dashboard', 'link.downloadable', 'rpc.service', 'messaging.operation', 'http.payment_success_rate', 'document.upload_error_rate', 'messaging.destination', 'link.re_signable', 'message', 'http.is_auth_endpoint', 'flipper.space_analytics', 'http.url', 'flipper.space_analytics_backfill', 'http.space_endpoints_success_rate', 'http.presentation_view_endpoints_successful', 'net.transport', 'messaging.system', 'link.type', 'document_processing_event.delayed_job_queue', 'request.id', 'http.grpc_requests_success_rate', 'http.elaine_web_request', 'http.link_successful_requests_rate', 'http.esign_successful_requests', 'link.watermark_enabled', 'service.component', 'user.login_success', 'document_processing_event.client', 'flipper.space_audit_log', 'status_code', 'document.document_group_id', 'http.link_successful_requests', 'http.target', 'link.recipient_kind', 'document.upload_unknown_error_rate', 'duration_ms', 'service.name', 'http.route', 'db.statement', 'error', 'trace.parent_id', 'http.status_code', 'name', 'user.signup_referer_path', 'exception.message', 'parent_name']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"name\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"}],\"filters\":[{\"column\":\"http.route\",\"op\":\"=\",\"value\":\"/api/external_storage/gdrive/notify\",\"join_column\":\"\"}],\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"The query correctly filters for the specific route '/api/external_storage/gdrive/notify'. Although it just shows a heatmap, because NLQ did not ask for anything specific, the heatmap is the best generat calculation to use. It may also be helpful to add a calculation for MAX or P95 to identify the slowest events explicitly. The breakdown by 'name' is generic and might not provide meaningful insights for this specific route unless 'name' is related to some aspect of the functionality at the route, but it is not a bad breakdown. It may also be helpful to filter to 'trace.parent_id' 'does-not-exist' so that it shows only requests to the endpoint, but since NLQ did not ask for requests, this isn't strictly necessary.\", \"outcome\": \"good\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: find the traces with the most spans\n",
      "\n",
      "COLUMNS: ['trace.span_id', 'trace.trace_id', 'trace_flags', 'trace_id', 'span_id', 'exception.stacktrace', 'trace.parent_id', 'span.num_links', 'span.num_events', 'span.kind', 'flags', 'nodejs_gc_duration_seconds.max', 'nodejs_gc_duration_seconds.avg', 'nodejs_gc_duration_seconds.p01', 'container.memory.usage.max', 'scrape_samples_post_metric_relabeling', 'container.duration', 'extensions', 'nodejs_gc_duration_seconds.sum', 'container.memory.utilized', 'nodejs_gc_duration_seconds.p001', 'nodejs_gc_duration_seconds.p99', 'scrape_samples_scraped', 'path', 'nodejs_gc_duration_seconds.p05', 'source', 'nodejs_eventloop_lag_mean_seconds', 'nodejs_eventloop_lag_max_seconds', 'nodejs_gc_duration_seconds.count', 'nodejs_gc_duration_seconds.min', 'container.cpu.onlines', 'up', 'partial_last', 'container.cpu.utilized', 'graphql.source', 'ecs.task.memory.usage.max', 'nodejs_gc_duration_seconds.p95', 'ecs.task.cpu.onlines', 'container']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"trace.trace_id\"],\"calculations\":[{\"op\":\"COUNT\"}],\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}]}\n",
      "\n",
      "{\"critique\": \"Bad response. The COUNT operation is incorrectly used because it does not count the number of spans per trace, but the total events within a window. Instead, the calculations should use a COUNT_DISTINCT(trace.span_id). Breaking down by 'trace.trace_id' is good, however, because that lets the user see each trace with a span count. Finally, the query should have an ordering of COUNT_DISTINCT(trace.span_id) 'desc' so that results are ordered by most spans.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: api calls that are failing\n",
      "\n",
      "COLUMNS: ['cronofy.oauth_client_notification_subscriptions.failed', 'apirequestjournal.failed_journal_reads', 'cronofy.notifier_client.api_call_status', 'auth.oauthinator.go_to.api_call_count', 'api.event.conferencing.error', 'cronofy.notifier_client.api_call_path', 'cronofy.notifier_client.api_call_ms', 'api.event.conferencing.sync.error', 'auth.oauthinator.zoom.api_call_count', 'cronofy.service_account.relinks.failed', 'auth.failure', 'cronofy.api.rate_limits.oauth_client_account_exceeded', 'cronofy.notifier_client.api_call_method', 'auth.oauthinator.ms_teams.api_call_count', 'cronofy.api.rate_limits.oauth_client_exceeded', 'cronofy.embedded_auth_token.errors', 'apirequestjournal.failed_dir_enumerations', 'indeed.put.error', 'apirequestjournal.read_dir.total_calls', 'integrations.indeed.api_response.status', 'api.free_busy.caching.hit', 'shipper.ingress.failed_open', 'error.service_unavailable_error?', 'auth.oauthinator.go_to.refresh_failure_count', 'auth.provider_error_code', 'error', 'auth.graph_access_token_attempt', 'apirequestjournal.results', 'eightbyeight.put.error', 'integrations.slack.api_response.status', 'sli.apirequestjournal.dashboard.ok', 'integrations.hubspot.api_response.status', 'integrations.api.operation', 'enterprise_connect.request_context.failure_key', 'api.params.google_event_ids', 'eightbyeight.persistent.get.error', 'auth.error_code', 'auth.graph_me_attempt', 'auth.op', 'eightbyeight.persistent_cache.get.error', 'cronofy.calendar_api', 'auth.oauthinator.zoom.refresh_failure_count', 'shared.put.error', 'filehousekeeper.delete.failures', 'api.event.conferencing.dial_in', 'api.free_busy.total_pages', 'integrations.zendesk.api_response.status', 'apirequestjournal.index_job.total_indexing_errors', 'ms_graph.failed_delta_link_fallback', 'api.free_busy.page_events', 'name', 'shipper.ingress.full_path', 'cronofy.account_profile', 'trace.parent_id', 'http.status', 'scheduler.build_availability_query.type', 'active_job.class', 'enterprise_connect.request_context.error', 'cronofy.oauth_client', 'duration_ms', 'sync.state', 'cronofy.environment.name', 'sync.sync_type', 'http.path', 'service_name']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"http.path],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"http.status\",\"op\":\"exists\",\"join_column\":\"\"},{\"column\":\"http.status\",\"op\":\"!=\",\"value\":\"200\",\"join_column\":\"\"}],\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}],\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"This query is too specific. The columns 'http.path' and 'http.status' are more generic columns, meaning that they capture information about more endpoints than the columns selected ('cronofy.notifier_client.api_call_status' and 'cronofy.notifier_client.api_call_path'). By using those more specific columns, the query makes an assumption that NLQ is referring to these specific columns, when there is no evidence that NLQ is doing that. Although the structure of this response is good, it should use the more generic columns 'http.status' and 'http.path' instead.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: ListWalletMultichainTokenHoldings\n",
      "\n",
      "COLUMNS: ['service_dc_2', 'service_dc_1', 'service_dc_3', 'service_dc_5', 'service_dc_4', 'success_rate_v3', 'error', 'success_rate_demo', 'duration_ms', 'name', 'dc_simple_refinery_reason', 'client-uuid', 'meta.stressed', 'go.tb', 'ip', 'span.kind', 'db.statement.fingerprint', 'db.statement', 'span.num_links', 'parent_name', 'dc_log10_duration', 'http.url', 'db.rowcount', 'meta.refinery.reason', 'span.num_events', 'host.name', 'dc_db_system_or_type', 'log10_duration_ms', 'service.name', 'type', 'meta.signal_type', 'error.object', 'trace.span_id', 'status_code', 'meta.annotation_type', 'trace.trace_id', 'meta.refinery.sample_key', 'http.method', 'db.canonical_name', 'dc_is_root_span', 'grpc.source', 'trace.parent_id', 'duration_per_rowcount', 'meta.refinery.original_sample_rate', 'go.tb.exists', 'opencensus.exporterversion', 'dc_is_success', 'meta.refinery.send_reason', 'http.status_code', 'dc_ensure_nonroot_server_span', 'http.route']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"name\"],\"filters\":[{\"column\":\"name\",\"op\":\"=\",\"value\":\"ListWalletMultichainTokenHoldings\"}],\"calculations\":[{\"op\":\"COUNT\"}],\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}],\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"The query correctly filters on the 'name' column to isolate the 'ListWalletMultichainTokenHoldings' service. Using COUNT as a calculation is appropriate to get the number of occurrences, and ordering by count descending will properly rank the results. It could be improved by adding a calculation that includes the 'duration_ms' column such as AVG or MAX to assess the latency, as the NLQ suggests a potential interest in performance metrics for this particular service.\", \"outcome\": \"good\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: overall distribution of latencies\n",
      "\n",
      "COLUMNS: ['duration_ms', 'container.uptime', 'container.cpu.time', 'event.domain', 'container.cpu.utilization', 'ip', 'regarding', 'container.memory.page_faults', 'severity', 'metadata', 'interface', 'container.memory.major_page_faults', 'span.kind', 'namespace', 'k8s.node.cpu.time', 'error', 'container.filesystem.usage', 'uid', 'container.memory.working_set', 'span.num_events', 'container.memory.usage', 'flags', 'k8s.node.uptime', 'event.name', 'k8s.pod.cpu.time', 'k8s.pod.uptime', 'container.filesystem.capacity', 'name', 'container.memory.available', 'k8s.node.network.io.transmit', 'severity_text', 'k8s.node.cpu.utilization', 'body', 'reason', 'note', 'kind', 'net.peer.port', 'k8s.pod.cpu.utilization', 'type', 'Kubernetes']\n",
      "\n",
      "QUERY: {\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"},{\"column\":\"duration_ms\",\"op\":\"P95\"}],\"breakdowns\":[{\"namespace\"},\"orders\":[{\"op\":\"P95\",\"column:\"duration_ms\",\"order\":\"descending\"}]]\n",
      "\n",
      "{\"critique\": \"This is a bad query. While it includes relevant calculations, there are far too many, which is an overwhelming response. Only a HEATMAP and a common aggregation like P95 should be used. Additionally, an ideal query would break down by a column like 'namespace' even though NLQ did not ask for it, because that can yield more interesting results.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: error\n",
      "\n",
      "COLUMNS: ['error', 'env', 'exception.message', 'type', 'name', 'http.error_message', 'exception.type', 'http.error_name', 'exception.stacktrace', 'status_code', 'status_message', 'span.kind', 'net.transport', 'library.name', 'tls.authorized', 'http.target', 'service.name', 'http.url', 'tls.protocol', 'duration_ms', 'process.owner', 'library.version', 'process.command', 'parent_name', 'process.pid', 'http.flavor', 'deployment.environment', 'http.method', 'next.bubble', 'http.host', 'next.route', 'net.host.ip', 'http.scheme', 'telemetry.sdk.language', 'deployment.name', 'trace.parent_id', 'tls.alpnProtocol', 'trace.span_id', 'http.response_content_length_uncompressed', 'net.host.name', 'trace.trace_id', 'net.host.port', 'http.status_code', 'http.response_content_length', 'process.command_args', 'net.peer.name', 'http.user_agent', 'telemetry.instrumentation_library', 'net.peer.port', 'process.runtime.description', 'http.route']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"name\"],\"calculations\":[{\"op\":\"COUNT\"}],\"filters\":[{\"column\":\"error\",\"op\":\"=\",\"value\":true,\"join_column\":\"\"}],\"orders\":[{\"op\":\"COUNT\",\"order\":\"descending\"}],\"time_range\":7200}\n",
      "\n",
      "{\"critique\": \"Bad response. The query correctly filters by errors and uses COUNT, but it breaks down by 'error' instead of a more useful column like 'name', which is the name of an operation. Although NLQ is vague, it's best to be generally helpful and show more information than less.\", \"outcome\": \"bad\"}\n",
      "\n",
      "---\n",
      "\n",
      "NLQ: I would like to pull a number from a string\n",
      "\n",
      "COLUMNS: ['github.pull_requests.0.number', 'github.run_number', 'github.job.step.number', 'github.pull_requests.0.id', 'library.name', 'github.pull_requests.0.head.sha', 'github.pull_requests.0.url', 'name', 'github.pull_requests.0.base.sha', 'github.pull_requests.0.head.repo.id', 'github.pull_requests.0.head.repo.name', 'github.pull_requests.0.base.ref', 'github.job.name', 'service.name', 'github.pull_requests.0.head.ref', 'github.pull_requests.0.head.repo.url', 'github.job.id', 'github.job.step.name', 'telemetry.sdk.language', 'github.pull_requests.0.base.repo.name', 'github.job.conclusion', 'github.conclusion', 'span.kind', 'service.version', 'github.pull_requests.0.base.repo.url', 'error', 'github.pull_requests.0.base.repo.id', 'github.job.step.conclusion', 'status_code', 'span.num_events', 'service.instance.id', 'github.job.runner_name', 'github.job.run_id', 'github.workflow', 'github.html_url', 'github.run_attempt', 'github.event', 'github.job.runner_group_id', 'github.job.run_attempt', 'telemetry.sdk.name', 'github.workflow_id', 'github.head_commit.id', 'service.namespace', 'telemetry.sdk.version', 'github.author_name', 'github.head_commit.message', 'github.head_sha', 'trace.span_id', 'github.base_sha', 'github.workflow_url', 'trace.parent_id', 'duration_ms', 'http.route']\n",
      "\n",
      "QUERY: \n",
      "\n",
      "{\"critique\": \"This query is good because it does a best effort at aligning with a vague NLQ. By checking if 'name' contains 'string', the query is about as close to an interpretation of NLQ as-is possible without further clarification. A COUNT of events broken down by 'name' where 'name' contains 'string' is the best way to interpret a vague NLQ like this.\", \"outcome\": \"good\"}\n",
      "\n",
      "---\n",
      "\n",
      "For the below NLQ, QUERY and COLUMNS provide a critique as JSON in the format {\"critique\": \"...\", \"outcome\": \"good\"|\"bad\"} as shown above.\n",
      "\n",
      "NLQ: latency distribution by status code\n",
      "\n",
      "COLUMNS: ['status_code', 'http.response.status_code', 'duration_ms', 'code.lineno', 'network.protocol.version', 'code.namespace', 'telemetry.sdk.language', 'meta.signal_type', 'server.address', 'library.version', 'idle_ns', 'span.type', 'busy_ns', 'trace.trace_id', 'span.kind', 'trace.span_id', 'os.type', 'http.route', 'code.filepath', 'http.request.method', 'url.scheme', 'service.name', 'telemetry.sdk.version', 'thread.id', 'span.num_events', 'user_agent.original', 'type', 'url.path', 'library.name', 'span.num_links', 'trace.parent_id', 'telemetry.sdk.name', 'name', 'thread.name', 'service_name']\n",
      "\n",
      "QUERY: {\"breakdowns\":[\"status_code\"],\"calculations\":[{\"column\":\"duration_ms\",\"op\":\"HEATMAP\"},{\"column\":\"duration_ms\",\"op\":\"P95\"}],\"filters\":[{\"column\":\"status_code\",\"op\":\"exists\",\"join_column\":\"\"}],\"time_range\":7200}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "_sample = traindf.iloc[0]\n",
    "print(critic_prompt(_sample.user_input, \n",
    "                    cols=_sample.candidate_cols,\n",
    "                    query=_sample.response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df242283-d748-46dc-b6c7-9f6233de01cb",
   "metadata": {},
   "source": [
    "## Human Labeling\n",
    "\n",
    "Phillip and I did a bunch of labeling and [analysis in this (private) spreadsheet](https://docs.google.com/spreadsheets/d/1zPl1Nr9JayEcFEG1L7LA3ppPEbCP0IQyaWK972cWUDM/edit?usp=sharing).  This is what we found:\n",
    "\n",
    "- Through a little bit of human labeling, we were able to progressively improve the prompt by adding Phillip's critique examples.\n",
    "- We quickly hit a plateau with few-shot prompting, but were able to get an AI critique that is ok at removing \"bad\" examples.\n",
    "\n",
    "_Note: I cannot share the spreadhseet_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a487b2d-113a-4a5a-9ca5-23fe6e6793ca",
   "metadata": {},
   "source": [
    "## More Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "970976ef-9595-40db-93cf-a3bd728ca163",
   "metadata": {},
   "source": [
    "This is level 2 evaluation as described in [this post](https://hamel.dev/blog/posts/evals/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdaa13ab-61b8-41f0-b119-e51062ebc4e1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}