Using the Archive Pipeline with a custom pipeline

[juchom]

I'm still doing a toy project to learn Statiq and now comes the pagination problem.

Right now, I have a custom PostPipeline pipeline that looks like this :

public class PostPipeline : Pipeline
{
    public PostPipeline()
    {
        InputModules = new ModuleList
        {
            new ReadFiles("posts/**/*.md")
        };

       // Processing and PostProcessing modules goes here

        OutputModules = new ModuleList
        {
            new WriteFiles()
        };
}

Everything is fine, my post are rendered, my custom modules are working, all shortcodes were processed and extra metadata were added !

My problem now is I'm not using Statiq's default pipelines at the moment.

I want all the contents from posts/**/* to be ignored by the default pipelines so I can process them with my own pipelines and place them with my own rules.

Now the Archive pipeline depends on Inputs Content and Data pipelines and they are disabled.

So I'll try to split my problem in several simple questions :

1. Can I ask Statiq to ignore the posts/**/* directory and process all other contents normally and let me use my pipeline for this directory ?

2. Then what would be my options to do pagination with Outputs from the PostPipeline ?

Thanks in advance, and if it's not clear, let me know, I'll try to make things clearer !

[daveaglick]

Wow, you're making a ton of progress! Hopefully I've got some answers to this latest set of questions...

Can I ask Statiq to ignore the posts/**/* directory and process all other contents normally and let me use my pipeline for this directory ?

So there's a feature configured via the fluent API named .AddExcludedPath() that lets you exclude a directory entirely from Statiq. The impact of that though is that any pipeline will treat the path as excluded - it'll essentially appear invisible. This is useful when integration with other toolchains like webpack but isn't quite what you want.

What you'll probably need to do is adjust the globbing pattern that gets used by the built-in modules to exclude that folder. That's controlled by the InputFiles setting and has a default value of **/{!_,}*. Something like this using the fluent API should work:

.ConfigureFileSystem(fs =>
{
    fs.InputPaths.Clear();
    fs.InputPaths.Add("*{!posts}/**/{!_,}*");
})

I'm not entirely sure about the globbing pattern there so you might need to test that part out a bit.

Then what would be my options to do pagination with Outputs from the PostPipeline

There's a module for that - the PaginateDocuments module can be used when you want to do paging in your own pipelines. It'll output "parent" documents that contain all the documents-per-page as a documents list in the "Children" metadata. Let me know if it's not clear how to use that module and I can work up a little example.

[juchom]

Still no luck after 2 days, I don't get how you can do the pagination on Documents and render all the page with a specific file like in CleanBlog Index.cshtml.

Rendering a simple cshtml file is ok
Pagination of documents is probably ok (well, the number of generated files and their destination is correct)

But I don't get how I can tell the RazorRender to use blog/Index.cshtml file.

I was thinking to add as Input my blog/Index.cshtml file and then in the process phase merge the documents from my PostPipeline, but RenderRazor is not using it.

public AllPostsPipeline()
{
    Dependencies.Add(nameof(PostPipeline));
    InputModules = new ModuleList
    {
        new ReadFiles("blog/Index.cshtml")
    };

    ProcessModules = new ModuleList
    {
        new MergeDocuments(nameof(PostPipeline)),
        new PaginateDocuments(5),
        new AddTitle(
            Config.FromDocument(doc =>
            {
                // Default title
                string title = doc.GetString(Keys.Title, string.Empty);
                int index = doc.GetInt(Keys.Index);
                return index <= 1 ? title : (title + $" (Page {index})");
            })).KeepExisting(false),
        new SetDestination(Config.FromDocument((doc, ctx) =>
        {
            NormalizedPath destination = new NormalizedPath(Constants.BlogPath);
            int index = doc.GetInt(Keys.Index);
            if (index > 1)
            {
                destination = destination.Combine($"page/{index.ToString()}");
            }
            else
            {
                destination = destination.Combine($"index");
            }

            return destination.AppendExtension(ctx.Settings.GetPageFileExtensions()[0]);
        }))
    };

    PostProcessModules = new ModuleList
    {
        new RenderRazor()
    };

    OutputModules = new ModuleList { new WriteFiles() };
}

If you have any clue, I'll be interested !
Thanks a lot.

[daveaglick]

Just bumping this to let you know I haven't forgotten! Hoping to take a look soon.

[juchom]

Ok thanks a lot 👍

[daveaglick]

Okay, finally getting a chance to look at this! I think I see where the confusion is, and you were really close to begin with. In the AllPostsPipeline you're reading the index.cshtml file and then calling MergeDocuments with the posts. That's not quite right - what it'll do as-written is essentially overwrite the content and merge the metadata from the index.cshtml with each post. So you'll end up with essentially the same documents as if you had just brought in the post documents directly.

Instead, what you want to do is perform the merge with index.cshtml after the pagination so that the content from index.cshtml overwrites each of the generated pages before being rendered by Razor. Part of the trick there is that the MergeDocuments module can take child modules that produce the documents to be merged with the inputs (in this case the result of pagination). So inside the MergeDocuments module is where you want to read the index.cshtml file to produce the content that gets merged with the pages produced by PaginateDocuments. It's probably also a good idea to rename index.cshtml to _index.cshtml to represent that it's a template in the sense that you don't ever expect the actual _index.cshtml file to be rendered, it's just used to populate the content for other documents from the pagination process that don't actually exist on disk.

So with some small tweaks I think it'll do what you're looking for. Here's AllPostsPipeline:

using Statiq.Razor;

public class AllPostsPipeline : Pipeline
{
    public AllPostsPipeline()
    {
        Dependencies.Add(nameof(PostPipeline));

        ProcessModules = new ModuleList
        {
            new ReplaceDocuments(nameof(PostPipeline)),
            new PaginateDocuments(5),
            new MergeDocuments(
                new ReadFiles("blog/_index.cshtml")),
            new AddTitle(
                Config.FromDocument(doc =>
                {
                    // Default title
                    string title = doc.GetString(Keys.Title, string.Empty);
                    int index = doc.GetInt(Keys.Index);
                    return index <= 1 ? title : (title + $" (Page {index})");
                })).KeepExisting(false),
            new SetDestination(Config.FromDocument((doc, ctx) =>
            {
                NormalizedPath destination = new NormalizedPath("blog");
                int index = doc.GetInt(Keys.Index);
                if (index > 1)
                {
                    destination = destination.Combine($"page/{index.ToString()}");
                }
                else
                {
                    destination = destination.Combine($"index");
                }

                return destination.AppendExtension(ctx.Settings.GetPageFileExtensions()[0]);
            }))
        };

        PostProcessModules = new ModuleList
        {
            new RenderRazor()
        };

        OutputModules = new ModuleList { new WriteFiles() };
    }
}

And here's _index.cshtml:

<p>These are my posts (page @Document.GetString(Keys.Index)):</p>
@foreach (var post in @Document.GetDocuments(Keys.Children))
{
    <p>@Html.DocumentLink(post)</p>
}
@{
    var previous = Document.GetDocument(Keys.Previous);
    var next = Document.GetDocument(Keys.Next);
    if (previous != null)
    {
        <p>@Html.DocumentLink(previous)</p>
    }
    if (next != null)
    {
        <p>@Html.DocumentLink(next)</p>
    }
}

Which produces the following pages:

that look like this:

Did I answer the right question and get you back on the right track?

[juchom]

Wow, thanks for this very detailed answer, it's cristal clear and it works !

Thanks a lot again !