Custom UmbracoPageController using custom routes can no longer get UmbracoContext

[jamesrichardbrett]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

13.4.0

Bug summary

After upgrading to Umbraco 13.4.0 can no longer get the Umbraco Context in a Custom UmbracoPageController

An unhandled exception occurred while processing the request.
InvalidOperationException: Wasn't able to get an UmbracoContext
Umbraco.Extensions.UmbracoContextAccessorExtensions.GetRequiredUmbracoContext(IUmbracoContextAccessor umbracoContextAccessor)

Downgrading back to 13.3.1 fixes the issue

here is the code

public class RobotsTxtController : UmbracoPageController
{
    public static string[] RoutePatterns = new[] { "/robots.txt", "/{local}/robots.txt" };

    private readonly IPublishedValueFallback _publishedValueFallback;
    private readonly IUmbracoContextFactory _umbracoContextFactory;

    public RobotsTxtController(ICombinedLogger<RobotsTxtController> logger, ICompositeViewEngine compositeViewEngine,
            IPublishedValueFallback publishedValueFallback,
            IUmbracoContextFactory umbracoContextFactory)
        : base(logger.Logger, compositeViewEngine)
    {
        _publishedValueFallback = publishedValueFallback;
        _umbracoContextFactory = umbracoContextFactory;
    }

    public IActionResult Index()
    {
        var robotsDefaultContent = "User-agent: *\n Disallow: /";

        using (UmbracoContextReference umbracoContextReference = _umbracoContextFactory.EnsureUmbracoContext())
        {
            if (CurrentPage is SiteRoot root && root.Value<bool>("allowCrawlers"))
            {
                var overrideRobotstxtContent = root.Value<string>(_publishedValueFallback, "overrideRobotstxtContent");

                if (!string.IsNullOrWhiteSpace(overrideRobotstxtContent))
                {
                    robotsDefaultContent = overrideRobotstxtContent;
                }
                else
                {
                    robotsDefaultContent = "User-agent: *\nDisallow: /app_plugins/\nDisallow: /umbraco/";
                }

                var sitemap = CurrentPage?.FirstChild<Sitemap>();
                if (sitemap != null)
                {
                    robotsDefaultContent = robotsDefaultContent + $"\n\nSitemap: {sitemap.Url(mode: UrlMode.Absolute)}";
                }
            }
        }

        return Content(robotsDefaultContent, "text/plain", Encoding.UTF8);
    }
}

Composer

public class RobotsComposer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
    {
        builder.Services.Configure<UmbracoPipelineOptions>(options =>
        {
            options.AddFilter(new UmbracoPipelineFilter(nameof(RobotsTxtController))
            {
                Endpoints = app => app.UseEndpoints(endpoints =>
                {
                    for (int i = 0; i < RobotsTxtController.RoutePatterns.Length; i++)
                    {
                        endpoints.MapControllerRoute(
                            $"{nameof(RobotsTxtController)}_{i}",
                            RobotsTxtController.RoutePatterns[i],
                            new
                            {
                                Controller = ControllerExtensions.GetControllerName<RobotsTxtController>(),
                                Action = nameof(RobotsTxtController.Index)
                            })
                            .ForUmbracoPage(FindContent);
                    }
                })
            });
        });
    }

    private IPublishedContent FindContent(ActionExecutingContext actionExecutingContext)
    {
        // Resolve services from the container
        var umbracoContextFactory = actionExecutingContext.HttpContext.RequestServices
            .GetRequiredService<IUmbracoContextFactory>();

        using (UmbracoContextReference umbracoContextReference = umbracoContextFactory.EnsureUmbracoContext())
        {
            var umbracoContext = umbracoContextReference.UmbracoContext;

            var domain = DomainUtilities.SelectDomain(umbracoContext.Domains?.GetAll(false), umbracoContext.CleanedUmbracoUrl);

            if (domain == null)
                return DefaultToFirstRoot(umbracoContext);

            var content = umbracoContext.Content?.GetById(domain.ContentId);
            return content ?? DefaultToFirstRoot(umbracoContext);
        }
    }

    private IPublishedContent DefaultToFirstRoot(IUmbracoContext umbracoContext)
    {
        return umbracoContext.Content?.GetAtRoot().FirstOrDefault() ?? throw new InvalidOperationException("Umbraco has no content nodes to find");
    }
}

Specifics

This currently work fine in live which is still 13.3.1
DEV and UAT environments on 13.4.0 both error

Steps to reproduce

Visit the page /robots.txt

Expected result / actual result

No response

[github-actions]

Hi there @jamesrichardbrett!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[mpontin]

We are experiencing the same issue on 13.5.2 but only with Umbraco instances that contain multiple sites

[Migaroez]

Reproduced with slighly simplified code

public class RobotsTxtController : UmbracoPageController
{
    public static string[] RoutePatterns = new[] { "/robots.txt", "/{local}/robots.txt" };

    public RobotsTxtController(ILogger<RobotsTxtController> logger, ICompositeViewEngine compositeViewEngine)
        : base(logger, compositeViewEngine)
    {
    }

    public IActionResult Index()
    {
        var robotsDefaultContent = "User-agent: *\n Disallow: /";

        if (CurrentPage is SiteRoot { AllowCrawlers: true } root)
        {
            var overrideRobotsTxtContent = root.OverrideRobotstxtContent;

            if (!string.IsNullOrWhiteSpace(overrideRobotsTxtContent))
            {
                robotsDefaultContent = overrideRobotsTxtContent;
            }
            else
            {
                robotsDefaultContent = "User-agent: *\nDisallow: /app_plugins/\nDisallow: /umbraco/";
            }

            var sitemap = CurrentPage?.FirstChild<Sitemap>();
            if (sitemap != null)
            {
                robotsDefaultContent += $"\n\nSitemap: {sitemap.Url(mode: UrlMode.Absolute)}";
            }
        }

        return Content(robotsDefaultContent, "text/plain", Encoding.UTF8);
    }
}

public class RobotsComposer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
    {
        builder.Services.Configure<UmbracoPipelineOptions>(options =>
        {
            options.AddFilter(new UmbracoPipelineFilter(nameof(RobotsTxtController))
            {
                Endpoints = app => app.UseEndpoints(endpoints =>
                {
                    for (int i = 0; i < RobotsTxtController.RoutePatterns.Length; i++)
                    {
                        endpoints.MapControllerRoute(
                                $"{nameof(RobotsTxtController)}_{i}",
                                RobotsTxtController.RoutePatterns[i],
                                new
                                {
                                    Controller = ControllerExtensions.GetControllerName<RobotsTxtController>(),
                                    Action = nameof(RobotsTxtController.Index)
                                })
                            .ForUmbracoPage(FindContent);
                    }
                })
            });
        });
    }

    private IPublishedContent FindContent(ActionExecutingContext actionExecutingContext)
    {
        // Resolve services from the container
        var umbracoContextAccessor = actionExecutingContext.HttpContext.RequestServices
            .GetRequiredService<IUmbracoContextAccessor>();

        using IUmbracoContext umbracoContext = umbracoContextAccessor.GetRequiredUmbracoContext();
        
        var domain = DomainUtilities.SelectDomain(umbracoContext.Domains?.GetAll(false),
            umbracoContext.CleanedUmbracoUrl);

        if (domain == null)
            return DefaultToFirstRoot(umbracoContext);

        var content = umbracoContext.Content?.GetById(domain.ContentId);
        return content ?? DefaultToFirstRoot(umbracoContext);
    }

    private IPublishedContent DefaultToFirstRoot(IUmbracoContext umbracoContext)
    {
        return umbracoContext.Content?.GetAtRoot().FirstOrDefault() ??
               throw new InvalidOperationException("Umbraco has no content nodes to find");
    }
}

And also using the IVirtualPageController interface instead of a composer

public class RobotsTxtController :UmbracoPageController, IVirtualPageController
{
    
    public RobotsTxtController(ILogger<UmbracoPageController> logger, ICompositeViewEngine compositeViewEngine) : base(logger, compositeViewEngine)
    {
    }

    [HttpGet]
    [Route("/robots.txt")]
    [Route("/{local}/robots.txt")]
    public IActionResult Index()
    {
        var robotsDefaultContent = "User-agent: *\n Disallow: /";

        if (CurrentPage is SiteRoot { AllowCrawlers: true } root)
        {
            var overrideRobotsTxtContent = root.OverrideRobotstxtContent;

            if (!string.IsNullOrWhiteSpace(overrideRobotsTxtContent))
            {
                robotsDefaultContent = overrideRobotsTxtContent;
            }
            else
            {
                robotsDefaultContent = "User-agent: *\nDisallow: /app_plugins/\nDisallow: /umbraco/";
            }

            var sitemap = CurrentPage?.FirstChild<Sitemap>();
            if (sitemap != null)
            {
                robotsDefaultContent += $"\n\nSitemap: {sitemap.Url(mode: UrlMode.Absolute)}";
            }
        }

        return Content(robotsDefaultContent, "text/plain", Encoding.UTF8);
    }

    public IPublishedContent? FindContent(ActionExecutingContext actionExecutingContext)
    {
        // Resolve services from the container
        var umbracoContextAccessor = actionExecutingContext.HttpContext.RequestServices
            .GetRequiredService<IUmbracoContextAccessor>();

        using IUmbracoContext umbracoContext = umbracoContextAccessor.GetRequiredUmbracoContext();
        
        var domain = DomainUtilities.SelectDomain(umbracoContext.Domains?.GetAll(false),
            umbracoContext.CleanedUmbracoUrl);

        if (domain == null)
            return DefaultToFirstRoot(umbracoContext);

        var content = umbracoContext.Content?.GetById(domain.ContentId);
        return content ?? DefaultToFirstRoot(umbracoContext);
    }
    
    private IPublishedContent DefaultToFirstRoot(IUmbracoContext umbracoContext)
    {
        return umbracoContext.Content?.GetAtRoot().FirstOrDefault() ??
               throw new InvalidOperationException("Umbraco has no content nodes to find");
    }
}

[Migaroez]

Hey @jamesrichardbrett, we have optimized when how/when parts of the umbraco pipeline runs. One of such optimizations is to not do it when a request is considered a front end request (images, static files,...) by defining the rout as .txt you are implying to asp.net core that it is a client side file.

You can overwrite our ignoring client side files for specific cases by adding this to your composer as documented over here https://docs.umbraco.com/umbraco-cms/reference/routing/custom-routes#client-side-requests.

builder.Services.Configure<UmbracoRequestOptions>(options =>
        {
            options.HandleAsServerSideRequest = httpRequest => httpRequest.Path.Value?.EndsWith("/robots.txt") == true;
        });

Feel free to reopen the ticket if this would not solve your issue.