Using Data files in other documents

[ror3d]

I was attempting this because I thought it would be possible to do, but I didn't find the way it should work:

I have a yaml document with a list of names and emails:

users:
  -
    name: user1
    mail: [email protected]
  -
    name: user2
    mail: [email protected]

I want to iterate and show them from a cshtml file. Using the idea from https://www.statiq.dev/web/content-and-data/data I can get the document itself. What I am not able to find is how to get the users and iterate over the list.

Attempting to accomplish this, I have been mainly wishing for a few things in the documentation:

• one is an API documentation - classes and methods for each, because the way the C# code is structured, with classes in some files and extension methods in others, is not that easy to navigate
• more extensive examples in the current documentation - for example, in this case I have an example on how to get a data document, but an example of actually accessing and using the data would be a lot more practical.

[daveaglick]

The key here is knowing that YAML files with nested objects get represented as nested documents (I.e. IDocument) within the root document. So in this case, the users.yaml file (assuming that's what it's called) will end up in the Outputs collection and that document that represents the data file will have a key of "users" which contains a list of documents, each with a "name" and "mail" key.

One way to explore this is using the REPL that you get when you run in preview mode:

Granted, you kind of have to know what you're dealing with in terms of objects to really dig through it (or do a lot of .GetType().ToString() calls to figure it out). One enhancement that'll make this easier is the addition of a Dump() extension in the REPL that'll let you dump out entire object graphs to the console: statiqdev/Statiq.Framework#208

So now that we know what the structure looks like by exploring it via the REPL after an execution, I can write code like this in Razor:

@foreach(IDocument user in Outputs.Get("users.yaml").GetDocumentList("users"))
{
    <div>
        <div>Name: @user.GetString("name")</div>
        <div>Mail: @user.GetString("mail")</div>
    </div>
}

one is an API documentation

Totally agree! My main task right now is to finish up Statiq Docs which generates exactly that kind of site. It's not totally from scratch either - I'm porting over most of the code from Wyam, so the API docs will eventually look something like this: https://wyam.io/api/wyam.common.io/. I'm hoping to have this finished sometime in the next couple months.

more extensive examples in the current documentation

Also totally agree! There's a framework in place for actual runnable examples, but I've only gotten so far as to do some for archives. I really want to get Statiq Docs finished and get some API documentation up first, but examples will be the next thing I focus on after that.

[ror3d]

Okay yeah, this got it working.

I realise now that one of the problems I was having, that was causing this to not work, is that my yml file looked like:

users:
  - name: user1
    mail: [email protected]
  - name: user2
    mail: [email protected]
  - user3
  - user4

Apparently it doesn't like a document like that, because the result from GetDocumentList("users") gave me a Count of 0. When removing the last 2 items, I get a Count of 2.

The REPL functionality is great, by the way! I hadn't realised that that option was there, and it's very helpful to be able to test expressions and such, this will certainly help me going forward. Thanks for bringing it up!

[ror3d]

Okay, kinda related question:

I have a content file, homepage-notice.md which I would like to include in the index.html file. In the documentation for the frontmatter keys, there is a couple that hint at this: RenderPostProcessTemplates and ShouldOutput.

I would expect that setting both to false would generate the homepage-notice.html file but not save it to the output directory, but it doesn't seem to, the Outputs only has the .md document. I can get the rendered html by setting ShouldOutput to true, but that creates a homepage-notice.html file which I'd rather not have if possible.

On the other hand, I don't think I have got the proper way to include this content in the index.html file. I have accomplished it with

@Html.Raw(Outputs["homepage-notice.html"][0].GetContentProvider().GetTextReader().ReadToEnd())

But that seems overly complicated (and I'm not disposing of the textreader, as is said I should do in the comments for the GetTextReader method).

My first approach to this was to try to just load the markdown code from some metadata value and pass that through the markdown compiler, but I don't think that's actually doable?

Thanks!

[daveaglick]

If you want to process a document but not output it to disk, ShouldOutput is the correct front matter.

So given a file, homepage-notice.md:

ShouldOutput: false
---
# Hi!

This is a **markdown** file.

Will render the Markdown but not write a corresponding homepage-notice.html.

I think the challenge you're having is how to get that rendered Markdown content back out within another document. To some extent, this is exactly why there's a process and a separate post-process phase. Markdown is rendered in the former while Razor is rendered in the latter. Otherwise, there would be a chicken-and-egg problem with including one in the other. But that's not really what's causing problems here - it's more about how to find the right document in the Outputs collection that contains your rendered Markdown for inclusion.

The indexer for Outputs is shorthand for FilterDestinations(value) which itself iterates over an IEnumerable<IDocument> and returns every document that matches the pattern. To get an enumerator over documents (I.e. what the FilterDestinations() call and thus the indexer iterates over), the Outputs collection currently does a SelectMany() over every pipeline selecting the most recent phase with results, so you end up with multiple results for a given document if that document appears in multiple pipelines. Since Statiq Web uses one pipeline to initially read documents and then a second separate one to process them according to content type, calling Outputs["homepage-notice.html"][0] may be returning the document from the initial "Inputs" pipeline and not the one that was processed by the "Content" pipeline to render the Markdown. I say "may" because it's actually not defined, the pipelines are traversed in the SelectMany() with a dictionary iterator which doesn't have a defined order. So that all seems like a bug in retrospect so I've opened statiqdev/Statiq.Framework#226 to deal with it.

So that explains why you had trouble finding the right content, but let's look at what you could do now to get the expected results. The best approach is probably going to be getting the rendered Markdown document from the right pipeline. Something like this should work:

@Html.Raw(await Context.Outputs
    .FromPipeline("Content")
    .First(x => !x.Source.IsNull && x.Source.FileName.Equals("homepage-notice.md"))
    .GetContentStringAsync())

Did all that make sense? TL;DR: using the Razor code above along with ShouldOutput: false should get you what you're after. I also recognize that the syntax there isn't optimal, so I'll likely add some additional helpers to make this a little easier (like a GetSource() that can be used there instead of the First() call).

[ror3d]

Okay yes, all that makes sense. I was attempting to access it directly through Outputs without specifying the pipeline because I wasn't sure what pipeline it would have been. The syntax is indeed a bit more complex than I would have expected, so thank you for showing me how it should be done properly!

[daveaglick]

FWIW, after the next release this will work and provide the results you're expecting:

@Html.Raw(await Outputs.GetRelativeSource("homepage-notice.md").GetContentStringAsync())

[ror3d]

Awesome, thank you!