No Direct requests possible with helhum/typo3-secure-web

[valtissimo]

Bug Report

Current Behavior
When used together with helhum/typo3-secure-web, the crawler is not able to make direct requests. Requests are answered with "Called TYPO3 from the wrong document root!" and nothing gets indexed

Expected behavior/output
Direct requests should be possible when helhum/typo3-secure-web is used

Steps to reproduce
Install helhum/typo3-secure-web, set makeDirectRequests = '1' in crawler ext configuration

Environment

• Crawler version: 11.0.4
• TYPO3 version: 10.4.26
• Is your TYPO3 installation set up with Composer (Composer Mode): yes

Possible Solution
https://github.com/tomasnorre/crawler/blob/main/cli/bootstrap.php#L24 makes $typo3Root pointing to the /private/ directory, where executing index.php is not allowed. This mechanism needs to be replaced with something more stable.

Additional context

[github-actions]

Hi there, thank you for taking your time to create your first issue. Please give us a bit of time to review it.

[tomasnorre]

I have considered for a longer time, how to get rid of the Bootstrap file in general, and improve the direct request, but haven't come up with a good approach yet.

Will be happy to hear if you have any ideas/suggestions.

[tomasnorre]

I'm considering getting rid of the Direct request completely. I have that #974 will allow me to get rid of both the DirectRequest and the GuzzleRequest. That would make the code base smaller. And most likely also easier to maintain.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.