ESS updates

docs/AI-for-security/ai-for-security.asciidoc

@@ -1,5 +1,5 @@
 [[ai-for-security]]
-= AI for security
+= AI for Security
 
 :frontmatter-description: Learn to use AI capabilities in {elastic-sec}.
 :frontmatter-tags-products: [security]

docs/AI-for-security/ai-security-assistant.asciidoc

@@ -24,7 +24,7 @@ WARNING: The Elastic AI Assistant is designed to enhance your analysis with smar
 
 * To set up AI Assistant, you need the **Actions and Connectors : All** {kibana-ref}/kibana-privileges.html[privilege].
 
-* You need an account with a third-party generative AI provider, which AI Assistant uses to generate responses. Supported providers are OpenAI, Azure OpenAI Service, and Amazon Bedrock.
+* You need an account with a third-party generative AI provider, which AI Assistant uses to generate responses. Supported providers are OpenAI, Azure OpenAI Service, Google Vertex, and Amazon Bedrock. Local open-source models are also supported.
 --
 
 [discrete]
@@ -68,27 +68,25 @@ You can also chat with AI Assistant from several particular pages in {elastic-se
 * <<data-quality-dash, Data Quality dashboard>>: Select the *Incompatible fields* tab, then click *Chat*. (This is only available for fields marked red, indicating they're incompatible).
 * <<timelines-ui, Timeline>>: Select the *Security Assistant* tab.
 
-NOTE: Each user's chat history and custom quick prompts are automatically saved, so you can leave {elastic-sec} and return to pick up a conversation later.
+NOTE: Each user's chat history (up to the 99 most recent conversations) and custom quick prompts are automatically saved, so you can leave {elastic-sec} and return to a conversation later. Chat history appears to the left of the AI Assistant chat window, and on the Conversations tab of the AI Assistant settings menu. To access the settings menu, use the global search field to search for "AI Assistant for Security".
 
 [discrete]
 [[interact-with-assistant]]
 == Interact with AI Assistant
 
 Use these features to adjust and act on your conversations with AI Assistant:
 
-* Select a _system prompt_ at the beginning of a conversation to establish how detailed and technical you want AI Assistant's answers to be.
-+
-[role="screenshot"]
-image::images/system-prompt.gif[The system prompt drop-down menu,90%]
-+
-System prompts provide context to the model, informing its response. To create a custom system prompt, open the system prompts dropdown menu and click *+ Add new system prompt...*.
-
+* (Optional) Select a _system prompt_ at the beginning of a conversation by using the **Select Prompt** menu. System prompts provide context to the model, informing its response. To create a system prompt, open the system prompts dropdown menu and click *+ Add new system prompt...*.
 * Select a _quick prompt_ at the bottom of the chat window to get help writing a prompt for a specific purpose, such as summarizing an alert or converting a query from a legacy SIEM to {elastic-sec}.
 +
 [role="screenshot"]
 image::images/quick-prompts.png[Quick prompts highlighted below a conversation,90%]
 +
-Quick prompt availability varies based on context — for example, the **Alert summarization** quick prompt appears when you open AI Assistant while viewing an alert. To customize existing quick prompts and create new ones, click *Add Quick prompt*.
+* System Prompts and Quick Prompts can also be configured from the corresponding tabs in the Security AI settings menu.
++
+image::images/assistant-settings-system-prompts.png[The Security AI settings menu's System Prompts tab,90%]
++
+* Quick prompt availability varies based on context — for example, the **Alert summarization** quick prompt appears when you open AI Assistant while viewing an alert. To customize existing quick prompts and create new ones, click *Add Quick prompt*.
 
 * In an active conversation, you can use the inline actions that appear on messages to incorporate AI Assistant's responses into your workflows:
 
@@ -104,22 +102,16 @@ TIP: AI Assistant can remember particular information you tell it to remember. F
 [discrete]
 [[configure-ai-assistant]]
 == Configure AI Assistant
-The *Settings* menu (image:images/icon-settings.png[Settings icon,17,17]) allows you to configure default conversations, quick prompts, system prompts, and data anonymization.
-
-[role="screenshot"]
-image::images/assistant-settings-menu.png[AI Assistant's settings menu, open to the Conversations tab]
+The *Security AI settings* menu allows you to configure AI Assistant. To access it, use the global search field to search for "AI Assistant for Security".
 
-The *Settings* menu has the following tabs:
+It has the following tabs:
 
-* **Conversations:** When you open AI Assistant from certain pages, such as Timeline or Alerts, it defaults to the relevant conversation type. Choose the default system prompt for each conversation type, the connector, and model (if applicable). The **Streaming** setting controls whether AI Assistant's responses appear word-by-word (streamed), or as a complete block of text. Streaming is currently only available for OpenAI models.
-* **Quick Prompts:** Modify existing quick prompts or create new ones. To create a new quick prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the quick prompt's text. 
+* **Connectors:** Manage all LLM connectors.
+* **Conversations:** When you open AI Assistant from certain pages, such as Timeline or Alerts, it defaults to the relevant conversation type. Choose the default system prompt for each conversation type, the default connector, and the default model (if applicable). The **Streaming** setting controls whether AI Assistant's responses appear word-by-word (streamed), or as a complete block of text. Streaming is currently only available for OpenAI models.
 * **System Prompts:** Edit existing system prompts or create new ones. To create a new system prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the system prompt's text. Under *Contexts*, select where the system prompt should appear.
-+
-NOTE: To delete a custom prompt, open the *Name* drop-down menu, hover over the prompt you want to delete, and click the *X* that appears. You cannot delete the default prompts.
-
+* **Quick Prompts:** Modify existing quick prompts or create new ones. To create a new quick prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the quick prompt's text. 
 * **Anonymization:** Select fields to include as plaintext, to obfuscate, and to not send when you provide events to AI Assistant as context. <<ai-assistant-anonymization, Learn more>>.
-
-* **Knowledge base:** Provide additional context to AI Assistant so it can answer questions about {esql} and alerts in your environment. <<ai-assistant-knowledge-base, Learn more>>.
+* **Knowledge base:** Provide additional context to AI Assistant. <<ai-assistant-knowledge-base, Learn more>>.
 
 [discrete]
 [[ai-assistant-anonymization]]
@@ -131,7 +123,7 @@ NOTE: To delete a custom prompt, open the *Name* drop-down menu, hover over the
 To modify Anonymization settings, you need the **Elastic AI Assistant: All** privilege, with **Customize sub-feature privileges** enabled.
 --
 
-The **Anonymization** tab of the AI Assistant settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated.
+The **Anonymization** tab of the Security AI settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated.
 
 [role="screenshot"]
 image::images/assistant-anonymization-menu.png[AI Assistant's settings menu, open to the Anonymization tab]
@@ -143,49 +135,17 @@ The *Show anonymized* toggle controls whether you see the obfuscated or plaintex
 When you include a particular event as context, such as an alert from the Alerts page, you can adjust anonymization behavior for the specific event. Be sure the anonymization behavior meets your specifications before sending a message with the event attached.
 
 [discrete]
-[[ai-assistant-knowledge-base]]
+[[ai-assistant-page-knowledge-base]]
 === Knowledge base
-beta::[]
 
-The **Knowledge base** tab of the AI Assistant settings menu allows you to enable AI Assistant to answer questions about the Elastic Search Query Language ({esql}), and about alerts in your environment. To use knowledge base, you must <<ml-requirements, enable machine learning>>.
-
-[discrete]
-[[rag-for-esql]]
-==== Knowledge base for {esql}
-
-NOTE: {esql} is enabled by default in {kib}. It can be
-disabled using the `enableESQL` setting from the
-{kibana-ref}/advanced-options.html[Advanced Settings]. This will hide the {esql} user interface from various applications. However, users will be able to access existing {esql} artifacts like saved searches and visualizations.
-
-IMPORTANT: {esql} queries generated by AI Assistant might require additional validation. To ensure they're correct, refer to the {ref}/esql-language.html[{esql} documentation]. 
-
-When this feature is enabled, AI Assistant can help you write an {esql} query for a particular use case, or answer general questions about {esql} syntax and usage. To enable AI Assistant to answer questions about {esql}:
-
-. Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled.
-. Click *Save*. The knowledge base is now active. A quick prompt for {esql} queries becomes available, which provides a good starting point for your {esql} conversations and questions. 
-
-NOTE: AI Assistant's knowledge base gets additional context from {ml-docs}/ml-nlp-elser.html#download-deploy-elser[Elastic Learned Sparse EncodeR (ELSER)].
-
-[discrete]
-[[rag-for-alerts]]
-==== Knowledge base for alerts
-When this feature is enabled, AI Assistant will receive multiple alerts as context for each of your prompts. It will receive alerts from the last 24 hours that have a status of `open` or `acknowledged`, ordered first by risk score, then by recency. Building block alerts are excluded. This enables it to answer questions about multiple alerts in your environment, rather than just the individual alerts you choose to include as context. 
-
-To enable RAG for alerts:
-
-. Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled.
-. Use the slider to select the number of alerts to send to AI Assistant. Click **Save**.
-+
-[role="screenshot"]
-image::images/knowledge-base-settings.png["AI Assistant's settings menu open to the Knowledge Base tab",75%]
-
-NOTE: Including a large number of alerts may cause your request to exceed the maximum token length of your third-party generative AI provider. If this happens, try selecting a lower number of alerts to send.
+The **Knowledge base** tab of the Security AI settings menu allows you to enable AI Assistant to remember specified information, and use it as context to improve response quality. To learn more, refer to <<ai-assistant-knowledge-base>>.
 
 [discrete]
 [[ai-assistant-queries]]
+[[rag-for-esql]]
 ### Get the most from your queries
 
-Elastic AI Assistant helps you take full advantage of the {elastic-sec} platform to improve your security operations. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. 
+Elastic AI Assistant helps you take full advantage of the {elastic-sec} platform to improve your security operations, such as by helping you write an {esql} query for a particular use case, or answering general questions about how to use the platform. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. 
 
 To maximize its usefulness, consider using more detailed prompts or asking for additional information. For instance, after asking for an {esql} query example, you could ask a follow-up question like, “Could you give me some other examples?” You can also ask for clarification or further exposition, for example "Please provide comments explaining the query you just gave."
 

docs/AI-for-security/knowledge-base.asciidoc

@@ -0,0 +1,98 @@
+[[ai-assistant-knowledge-base]]
+= AI Assistant Knowledge Base
+
+AI Assistant's knowledge base feature enables it to recall specific documents and other specified information, and to use it as context when responding to your queries. This page describes how to enable and add information to knowledge base.
+
+NOTE: When you upgrade from {elastic-sec} version 8.15 to a newer version, information previously stored by AI Assistant will be lost.
+
+.Requirements
+[sidebar]
+--
+
+* To use knowledge base, you need the `Elastic AI Assistant: All` privilege. To edit global knowledge base entries (information that will affect the AI Assistant experience for other users in the {kib} space), you need the `Allow Changes to Global Entries` privilege. 
+* To use knowledge base, you must <<ml-requirements, enable machine learning>> with a minimum ML node size of 4GB.
+
+--
+
+[discrete]
+[[enable-knowledge-base]]
+== Enable knowledge base
+
+There are two ways to enable knowledge base.
+
+NOTE: You must individually enable knowledge base for each {kib} space where you want to use it.
+
+[discrete]
+=== Option 1 — enable knowledge base from an AI Assistant conversation
+
+Open a conversation with AI Assistant, select a large language model, then click **Setup Knowledge Base**. If the button doesn't appear, knowledge base is already enabled.
+
+image::images/knowledge-base-assistant-setup-button[An AI Assistant conversation showing the Setup Knowledge Base button]
+
+Knowledge base setup may take several minutes. It will continue in the background if you close the conversation. After setup is complete, you can access knowledge base settings from the conversation settings menu.
+
+image::images/knowledge-base-assistant-menu-dropdown.png[AI Assistant's dropdown menu with the knowledge base option highlighted]
+
+[discrete]
+=== Option 2 — enable knowledge base from the Security AI settings menu
+
+. Go to **Stack Management → AI Assistants → Elastic AI Assistant for Security → Knowledge Base**.
+. Click **Setup Knowledge Base**. If the button doesn't appear, knowledge base is already enabled.
+
+image::images/knowledge-base-assistant-settings-kb-tab.png[AI Assistant's settings menu open to the knowledge base tab]
+
+[discrete]
+[[rag-for-alerts]]
+== Knowledge base for alerts
+When knowledge base is enabled, AI Assistant receives alerts from your environment as context for each of your prompts. It will receive alerts from the last 24 hours that have a status of `open` or `acknowledged`, ordered first by risk score, then by recency. Building block alerts are excluded. This enables it to answer questions about multiple alerts in your environment, rather than just the individual alerts you choose to include as context. 
+
+To enable knowledge base for alerts:
+
+. Make sure that knowledge base is <<enable-knowledge-base, enabled>>.
+. Use the slider on the Security AI settings menu's Knowledge Base tab to select the number of alerts to send to AI Assistant. Click **Save**.
+
+NOTE: Including a large number of alerts may cause your request to exceed the maximum token length of your third-party generative AI provider. If this happens, try selecting a lower number of alerts to send.
+
+[discrete]
+[[knowledge-base-add-knowledge]]
+== Add knowledge 
+
+To view all knowledge base entries, go to the Security AI settings menu's Knowledge Base tab. You can add either individual documents, or entire indices containing multiple documents. Each entry in the knowledge base (a document or index) has a **Sharing** setting of either `private` or `global`. Private entries do not affect other users in the {kib} space, whereas global entries affect all users. Each entry can also be `Required knowledge`, which means it will be included as context to every message to AI Assistant. 
+
+NOTE: When you enable knowledge base, it comes pre-populated with articles from https://www.elastic.co/security-labs[Elastic Security Labs], current through September 30, 2024. This allows AI Assistant to leverage Elastic's security research during your conversations. This enables it to answer questions such as, “Are there any new tactics used against Windows hosts that I should be aware of when investigating my alerts?”
+
+[discrete]
+[[knowledge-base-add-knowledge-document]]
+=== Add an individual document
+
+Add an individual document to knowledge base when you want AI Assistant to remember a specific piece of information.
+
+. Go to **Stack Management → AI Assistants → Elastic AI Assistant for Security → Knowledge Base**.
+. Click **New → Document**. 
+. Name the knowledge documet.
+. Under **Sharing**, select whether this knowledge should be **Global** or **Private**.
+. Write the knowledge in the **Markdown text** field.
+. Decide whether to make it **Required knowledge**.
+
+Alternatively, you can simply send a message to AI Assistant that instructs it to "Remember" the information. For example, "Remember that I changed my password today, October 24, 2024", "Remember that our primary data center is located in Austin, Texas", or "Remember we always use the 'Threat Hunting' Timeline template when investigating potential threats". Entries created in this way are private to you. By default they are not required knowledge, but you can make them required by instructing AI Assistant to "Always remember", for example "Always remember to address me as madam".
+
+[discrete]
+[[knowledge-base-add-knowledge-index]]
+=== Add an index
+
+Add an index as a knowledge source when you want information added to that index to automatically inform AI Assistant's responses. Common security examples include asset inventories, network configuration information, on-call matrices, threat intelligence reports, and vulnerability scans. 
+
+IMPORTANT: Indexes added to knowledge base must have at least one field mapped as {ref}/semantic-text.html[semantic text].
+
+. Go to **Stack Management → AI Assistants → Elastic AI Assistant for Security → Knowledge Base**.  
+. Click **New → Index**.
+. Name the knowledge source.
+. Under **Sharing**, select whether this knowledge should be **Global** or **Private**.
+. Under **Index** enter the name of the index you want to use as a knowledge source.
+. Under **Field**, enter the names of one or more semantic text fields within the index.
+. Under **Data Description**, describe when this information should be used by AI Assistant.
+. Under **Query Instruction**, describe how AI Assistant should query this index to retrieve relevant information.
+. Under **Output Fields**, list the fields which should be sent to AI Assistant. If none are listed, all fields will be sent.
+
+image::images/knowledge-base-add-index-config.png[Knowledge base's Edit index entry menu]
+

docs/serverless/AI-for-security/ai-for-security-landing-pg.mdx

@@ -1,6 +1,6 @@
 ---
 slug: /serverless/security/ai-for-security
-title: AI for security
+title: AI for Security
 description: Learn about Elastic's native AI security tools.
 tags: [ 'serverless', 'security', 'overview', 'LLM', 'artificial intelligence' ]
 status: in review