Avoiding source URL in search index created with Statiq.Lunr

[pascalberger]

I tried to integrate client-side search using Statiq.Lunr into a Statiq site. Therefore I added a pipeline to create the search index which looks like this:

using Statiq.Core;
using Statiq.Common;
using Statiq.Web.Modules;
using Statiq.Web.Pipelines;
using Statiq.Web;
using Statiq.SearchIndex;

public class SearchIndex : Pipeline
{
    public SearchIndex()
    {
        Dependencies.AddRange(
            nameof(Inputs),
            nameof(Content),
            nameof(Archives),
            nameof(Data));

        PostProcessModules = new ModuleList
        {
            new ExecuteIf(IExecutionContext.Current.GetBool("GenerateSearchIndex"))
            {
                new GetPipelineDocuments(ContentType.Content),

                new FilterDocuments(Config.FromDocument(doc => !Archives.IsArchive(doc))),

                new ForEachDocument
                {
                    new ExecuteConfig(Config.FromDocument((doc, ctx) =>
                    {
                        return new ModuleList
                        {
                            new SetMetadata(
                                GenerateLunrIndexKeys.LunrIndexItem,
                                new LunrIndexDocItem(
                                    doc,
                                    doc.GetTitle(),
                                    doc.GetContentStringAsync().Result))
                        };
                    }))
                },

                new GenerateLunrIndex(GenerateLunrIndexKeys.LunrIndexItem, "stopwords.txt", true)
                    .WithDestination("assets/js/searchindex.js")
            }
        };

        OutputModules = new ModuleList
        {
            new WriteFiles()
        };
    }
}

This basically works, but as URL the source file link is used, instead of the target file:

    a(
        {
            id:1,
            title:"Some Title",
            content:"Some content",
            description:'',
            tags:''
        },
        {
            url:'/foo/some-title.md',
            title:"Some Title",
            description:""
        }
    );

It seems like Statiq.Lunr just calls .GetLink (https://github.com/statiqdev/Statiq.Framework/blob/main/src/extensions/Statiq.Lunr/GenerateLunrIndex.cs#L202)
What do I need to change in my pipeline to get the output path like this:

    a(
        {
            id:1,
            title:"Some Title",
            content:"Some content",
            description:'',
            tags:''
        },
        {
            url:'/some-title',
            title:"Some Title",
            description:""
        }
    );

[daveaglick]

Ouch - right off the bat I see there's a mismatch in the package and namespace. I'm going to need to change the namespace to Statiq.Lunr to match the package. At one point I guess I was thinking about a more general "search index" package, but I switched to preferring specific technology packages a while ago (similar to Cake addins). I guess I missed renaming the namespace here with the package name.

Still researching the link problem though.

[daveaglick]

Seeing a few things it would be worth fixing up while I'm in here. For one, I don't like that you have to iterate documents to add the LunrIndexDocItem as metadata manually. I'll change it so the default is to do what one would expect from a document without having to manually create each index item (with the ability to override with your own index item if you want to).

As for the generated search paths, the issue here is one of ordering and sequencing. In the code above, the documents are coming from the GetPipelineDocuments module. This module is intended for internal use (though I've made it public because it can still be helpful externally). It gets documents from the Inputs pipeline in Statiq Web based on content type, so it's more of a peer to other pipelines like Content and Data as opposed to something to use after them. More specifically, because it gets documents as output by the Inputs pipeline (which is more "raw"), those document don't have their destination correctly set. That usually happens in one of the more content-type-specific pipelines like Content since we know better what modules are processing what files at that stage.

If you change the code above to the following I think it'll do what you're looking for:

public class SearchIndex : Pipeline
{
    public SearchIndex()
    {
        Dependencies.AddRange(nameof(Content));

        PostProcessModules = new ModuleList
        {
                new ReplaceDocuments(nameof(Content)),

                new FilterDocuments(Config.FromDocument(doc => !Archives.IsArchive(doc))),

                new ForEachDocument
                {
                    new ExecuteConfig(Config.FromDocument((doc, ctx) =>
                    {
                        return new ModuleList
                        {
                            new SetMetadata(
                                GenerateLunrIndexKeys.LunrIndexItem,
                                new LunrIndexDocItem(
                                    doc,
                                    doc.GetTitle(),
                                    doc.GetContentStringAsync().Result))
                        };
                    }))
                },

                new GenerateLunrIndex(GenerateLunrIndexKeys.LunrIndexItem, "stopwords.txt", true)
                    .WithDestination("assets/js/searchindex.js")
        };

        OutputModules = new ModuleList
        {
            new WriteFiles()
        };
    }
}

In the next version you'll be able to omit the ForEachDocument module and I might even work this into Statiq Web directly as an optional toggle (so no custom pipeline at all required).

I also added some comments to the GetPipelineDocuments module to make it clearer what it does and when to use it.

[daveaglick]

Code after the next Statiq Framework release can look like this:

public class SearchIndex : Pipeline
{
    public SearchIndex()
    {
        Dependencies.AddRange(nameof(Content));

        PostProcessModules = new ModuleList
        {
            new ReplaceDocuments(nameof(Content)),
            new GenerateLunrIndex("stopwords.txt", true)
                .WithDestination("assets/js/searchindex.js")
        };

        OutputModules = new ModuleList
        {
            new WriteFiles()
        };
    }
}

[pascalberger]

Thanks @daveaglick 👍 Works like a charm.

I think to have this in Statiq.Web behind a toggle would be nice. One thing which is missing in the sample code above, which should be part when added to Statiq Web, is setting the description.

I can do this by using the excerpt of the page like this:

searchItem.Description = doc.GetString(Statiq.Html.HtmlKeys.Excerpt);

While this works, it is not perfect for general content pages, as it will just return the first paragraph. In my case this is often just one or two sentences. Is there already a module available which could be used to extract a longer excerpt. Something like DocFx does in the default config:

[daveaglick]

I ended up adding a GenerateSearchIndex toggle to Statiq Web, so in the next release if you do something like this it'll create the searchindex.js file for you:

public class Program
{
    public static async Task<int> Main(string[] args) =>
        await Bootstrapper
            .Factory
            .CreateWeb(args)
            .AddSetting(WebKeys.GenerateSearchIndex, true)
            .RunAsync();
}

I also added some other settings to control where the search index is output and other search index generation properties.

That only creates the index file though, the client-side search functionality also needs to be added to the theme/layout. To test everything I went ahead and made CleanBlog search-aware so if you set GenerateSearchIndex it'll add a search box to the navigation bar and generate a search page (and won't do anything extra if GenerateSearchIndex is false). Still need to polish it up a little, but that'll be releasing momentarily too:

Is there already a module available which could be used to extract a longer excerpt. Something like DocFx does in the default config

Not that I know of, and extracting the excerpt or proximal search text would probably need to be done on the client. Alternatively I think you can add extra arbitrary parameters to the search index and then recall those in the results - though that'll require creating or implementing a custom LunrIndexItem to get the description how you want it.

Note that the GenerateExcerpt module is customizable too - it uses an excerpt separator by default (like <!--excerpt-->) and falls back to the first <p> only if one isn't found.