[Bug]: Login to a cloudflare domain w/ "Block AI Scrapers and Crawlers" enabled - 403: Forbidden

[therealsummoner]

⚠️ Before submitting, please verify the following: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
• I agree to follow Nextcloud's Code of Conduct

Bug description

Login to a cloudflare domain with desktop app is unsuccessful, resulting in 403:Forbidden error.
Turns out, one has to deactivate the "Block AI Scrapers and Crawlers" for the domain to get login working.

Steps to reproduce

Fresh install, try to logon to a cloudflare domain, where "Block AI Scrapers and Crawlers" are activated.

Expected behavior

Should work with "Block AI Scrapers and Crawlers" activated as well, just like desktop apps from other projects e.g. bitwarden.

Which files are affected by this bug

app

Operating system

Windows

Which version of the operating system you are running.

Windows 10 22H2

Package

Official Windows MSI

Nextcloud Server version

29.0.4

Nextcloud Desktop Client version

3.13.3

Is this bug present after an update or on a fresh install?

Fresh desktop client install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

• Default internal user-backend
• LDAP/ Active Directory
• SSO - SAML
• Other

Nextcloud Server logs

No response

Additional info

No response

[joshtrichards]

Have you tried submitting a query to Cloudflare about a false positive?

https://developers.cloudflare.com/bots/concepts/feedback-loop/

Without knowing why they're detecting your client connections in this category, there's very little we can do.

Aside: It's not exactly surprising. I can sort of see how sync traffic can look like a bot scraping a web site.

[therealsummoner]

Have you tried submitting a query to Cloudflare about a false positive?

https://developers.cloudflare.com/bots/concepts/feedback-loop/

Without knowing why they're detecting your client connections in this category, there's very little we can do.

Aside: It's not exactly surprising. I can sort of see how sync traffic can look like a bot scraping a web site.

No can do: "Bot Feedback Loop is available for Enterprise Bot Management customers." :(

[TheGuy920]

I am also having this problem. Even simple CURL requests from the terminal works so I am assuming its a simple change to the networking requests like changing the user-agent or adding additional headers

[joshtrichards]

The simple change is to turn off that feature on the CF side (or, possibly, use a CF service plan that allows you to adjust the rules for your use case). :)

Changing the UA/headers is not sufficient; CF says as much. Otherwise it would be very easy for the things this feature is supposed to block to simply lie to get around it[1].

Using a WAF in front of your infrastructure means tuning it to your in-use applications.

Nextcloud sync traffic does not resemble the typical web site traffic pattern; likely looks a lot like a bot: lots of highly rapid and very much automated, transactions across many URLs/files.

You could maybe apply to have your IP address add to the Verified Bot list, but that's between you and CF[2].

[1] https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
[2] https://developers.cloudflare.com/bots/troubleshooting/#what-is-cfbot_managementverified_bot