AI Data Sources - Elasticsearch

Feature Name	AI Data Sources - Elasticsearch
Feature ID	CrestApps.OrchardCore.AI.DataSources.Elasticsearch

Adds Elasticsearch support for AI data source document embeddings, vector search, and indexing.

Overview

This module provides Elasticsearch support for AI data source knowledge base indexes. It enables vector search and document embedding storage using Elasticsearch's k-NN capabilities, allowing AI tools to perform Retrieval-Augmented Generation (RAG) searches against Elasticsearch indexes.

When AI profiles are configured with data sources, the system needs to search and retrieve relevant documents to provide context to AI models. This module handles:

• Index schema management — Creates and manages Elasticsearch mappings for knowledge base indexes with dense vector fields for embeddings.
• Document indexing — Indexes source documents with their embeddings into Elasticsearch knowledge base indexes.
• Vector search — Performs k-NN similarity searches to find the most relevant documents for a given query.
• Filter execution — Executes Elasticsearch DSL filter queries translated from OData for filtered vector search.
• OData filter translation — Registers the CrestApps.Core Elasticsearch IODataFilterTranslator, which translates the OData filter of an AI profile, a chat interaction or a data source search tool instance into an Elasticsearch bool query on the filters. fields of the knowledge base index. See Filters on an Elasticsearch knowledge base.
• Batch document reading — Reads source documents from Elasticsearch indexes in batches for efficient indexing.
• Automatic source sync — When Orchard Core updates or deletes documents in Elasticsearch, the module forwards those provider notifications into the shared data-source indexing queue so mapped knowledge-base indexes stay aligned.
• External source connections — Adds Elasticsearch as a selectable AI data-source Source type so operators can read directly from an external Elasticsearch index by configuring either a self-managed endpoint URL or an Elastic Cloud ID, selecting the authentication type, setting the remote index name, and optionally supplying a certificate fingerprint.

Getting Started

1. Enable the AI Data Sources - Elasticsearch feature in the Orchard Core admin dashboard.
2. Create an Elasticsearch knowledge base index via Search > Indexing using the AI Knowledge Base Index (Elasticsearch) type.
3. Configure an AI data source under Artificial Intelligence > Data Sources, selecting either an Orchard-managed Elasticsearch source index profile or the Elasticsearch external source type and the knowledge base index.
4. For external Elasticsearch sources, choose the environment and authentication mode that matches the target cluster:
   • Self-managed with URL
   • Cloud-hosted with Elastic Cloud ID
   • None, Basic, API key, Base64 API key, or Key ID and key authentication
5. The module will automatically sync documents from the source index to the knowledge base index with embeddings.

The data source editor is described in the User Manual under Data sources.

Use a data connection

When the Data Connectors feature is on, the source editor also offers Data connection: pick an Elasticsearch connection from Data Connectors and the data source reads everything but the index name from the connection. Its own URL and credentials stay stored, unused, so switching back to its own settings loses nothing; a deployment plan leaves its own password and keys out while the connection is used. The URL, the authentication (none, Basic, an API key in a header, or a bearer token), the extra headers and the timeout are the connection's, so one set of credentials serves the AI data sources, the connected data sets and the Report Builder. The connection's secrets stay protected in the connection and are never written to the data source or to a deployment plan.

• Only people who may Manage data connections and their credentials see the picker. For anyone else, a data source that uses a connection keeps it when they save.
• A data source without a connection keeps its own settings, or the global connection below; existing data sources are unchanged.
• OAuth 2.0, OpenID Connect and query-string API keys are refused, because the Elasticsearch client can't send them.

Global connection

A connection can be configured once for the whole application instead of on every data source. When one is configured, the source editor offers Use the globally configured connection; a data source that uses it stores only the index name, and the connection fields are hidden.

Configure it under the shared Elasticsearch section, which every Elasticsearch feature reads:

{
  "OrchardCore": {
    "CrestApps": {
      "Elasticsearch": {
        "Url": "http://localhost:9200",
        "AuthenticationType": "Basic",
        "Username": "elastic",
        "Password": "elasticsearch"
      }
    }
  }
}

The section accepts Url, CloudId, AuthenticationType (None, Basic, ApiKey, Base64ApiKey or KeyIdAndKey), Username, Password, ApiKey, Base64ApiKey, ApiKeyId and CertificateFingerprint. Setting CloudId instead of Url connects to an Elastic Cloud deployment, and the authentication type is inferred from the supplied credentials when it is left out.

To point data sources at a different cluster than the rest of the application, override the shared values under the data source section:

{
  "OrchardCore": {
    "CrestApps": {
      "AI": {
        "DataSources": {
          "Elasticsearch": {
            "Url": "http://reporting:9200"
          }
        }
      }
    }
  }
}

The same values can be supplied as environment variables, which is how the Aspire host wires its local Elasticsearch container:

OrchardCore__CrestApps__Elasticsearch__Url
OrchardCore__CrestApps__Elasticsearch__Username
OrchardCore__CrestApps__Elasticsearch__Password

Connection settings stored on a data source always win over the configured values. This section is separate from OrchardCore:OrchardCore_Elasticsearch, which configures the Orchard Core Elasticsearch feature that owns the Orchard-managed indexes.

Filters on an Elasticsearch knowledge base

The feature registers the Elasticsearch IODataFilterTranslator with AddElasticsearchODataFilterTranslator(), keyed by the Elasticsearch provider name that the knowledge base index profile uses. It registers only the translator, because Orchard Core already provides the Elasticsearch client. For how a filter is set and validated, see Filtering retrieval.

The translator prefixes each field with filters., so category eq 'docs' becomes a term query on filters.category:

OData	Elasticsearch query
eq, ne	term (in a must_not for ne)
gt, ge, lt, le	range
contains(field, 'value'), endswith(field, 'value')	wildcard
startswith(field, 'value')	prefix
and, or, not, parentheses	bool with must, should or must_not

The knowledge base index maps the filters object with dynamic mapping, so Elasticsearch maps each string field as text with a keyword sub-field. A term query on the text field compares against the analyzed tokens, which are lower case and split into words, so category eq 'Support Docs' does not match. For an exact match on a string, use the keyword sub-field:

category.keyword eq 'Support Docs'

The same applies to contains, startswith and endswith when the value must match the whole stored string, including its case.