It's important to filter data types and allow only what you can actually use. Bots always search for files like .zip or .sql that could contain private or security information. For this, we identify first that it's a URI path with a dot (http.request.uri.path contains ".") and then by exclusion we define what we want to allow, by adding a negative match.
With this approach, it's very important to check often if you might use any data type that is not on the list and firewall will block false positives.
I don't include on allow list .txt and .xml, why? Because .txt is the main source of information for hackers on what version of theme/plugin you're using, therefore they can detect vulnerabilities and exploit them. Robots.txt, ads.txt and similar should be whitelisted if you don't set in Allow rulest known bot to get unrestricted access.
XML as well is very often used to identify pages that you can't crawl into and provides information for a hacker or malicious crawlers. Therefore I never allow unknown bots to access it, and I recommend making sure that for known bots you whitelist access by adding the path to Allow ruleset.
Data type .html in WordPress is not used, and accessing it will only required linke for Google or Pinterest for domain authorization purposes. ".html" should be whitelisted for known bots
(http.request.uri.path contains "." and not ends_with(http.request.uri.path, ".php") and not ends_with(http.request.uri.path, ".aac") and not ends_with(http.request.uri.path, ".css") and not ends_with(http.request.uri.path, ".eot") and not ends_with(http.request.uri.path, ".gif") and not ends_with(http.request.uri.path, ".jpeg") and not ends_with(http.request.uri.path, ".jpg") and not ends_with(http.request.uri.path, ".js") and not ends_with(http.request.uri.path, ".less") and not ends_with(http.request.uri.path, ".mp3") and not ends_with(http.request.uri.path, ".mp4") and not ends_with(http.request.uri.path, ".ogg") and not ends_with(http.request.uri.path, ".otf") and not ends_with(http.request.uri.path, ".pdf") and not ends_with(http.request.uri.path, ".png") and not ends_with(http.request.uri.path, ".svg") and not ends_with(http.request.uri.path, ".ico") and not ends_with(http.request.uri.path, ".ttf") and not ends_with(http.request.uri.path, ".webp") and not ends_with(http.request.uri.path, ".woff") and not ends_with(http.request.uri.path, ".woff2"))
Here is my ruleset to block bad bots with the Cloudflare firewall. I'm filtering SEO bots like Ahrefs or Moz, if you're using them then of course remove them from the ruleset.
(http.user_agent contains "Semrush") or (http.user_agent contains "ahrefs") or (http.user_agent contains "moz.com") or (http.user_agent contains "80legs") or (http.user_agent contains "Abonti") or (http.user_agent contains "admantx") or (http.user_agent contains "aipbot") or (http.user_agent contains "AllSubmitter") or (http.user_agent contains "Backlink") or (http.user_agent contains "backlink") or (http.user_agent contains "Badass") or (http.user_agent contains "Bigfoot") or (http.user_agent contains "BLEXBot") or (http.user_agent contains "Buddy") or (http.user_agent contains "CherryPicker") or (http.user_agent contains "cloudsystemnetwork") or (http.user_agent contains "cognitiveseo") or (http.user_agent contains "Collector") or (http.user_agent contains "cosmos") or (http.user_agent contains "CrazyWebCrawler") or (http.user_agent contains "Crescent") or (http.user_agent contains "Devil") or (http.user_agent contains "DittoSpyder") or (http.user_agent contains "Konqueror") or (http.user_agent contains "getintent") or (http.user_agent contains "Grabber") or (http.user_agent contains "GrabNet") or (http.user_agent contains "HEADMasterSEO") or (http.user_agent contains "heritrix") or (http.user_agent contains "htmlparser") or (http.user_agent contains "hubspot") or (http.user_agent contains "Jyxobot") or (http.user_agent contains "larbin") or (http.user_agent contains "ltx71") or (http.user_agent contains "leiki") or (http.user_agent contains "LinkScan") or (http.user_agent contains "Magnet") or (http.user_agent contains "Mag-Net") or (http.user_agent contains "Mechanize") or (http.user_agent contains "MegaIndex") or (http.user_agent contains "Metasearch") or (http.user_agent contains "MJ12bot") or (http.user_agent contains "Navroad") or (http.user_agent contains "Netcraft") or (http.user_agent contains "niki-bot") or (http.user_agent contains "NimbleCrawler") or (http.user_agent contains "Nimbostratus") or (http.user_agent contains "Ninja") or (http.user_agent contains "Openfind") or (http.user_agent contains "Pixray") or (http.user_agent contains "probethenet") or (http.user_agent contains "proximic") or (http.user_agent contains "psbot") or (http.user_agent contains "RankActive") or (http.user_agent contains "RankingBot") or (http.user_agent contains "RankurBot") or (http.user_agent contains "Reaper") or (http.user_agent contains "SalesIntelligent") or (http.user_agent contains "SEOkicks") or (http.user_agent contains "spbot") or (http.user_agent contains "SEOstats") or (http.user_agent contains "Snapbot") or (http.user_agent contains "Stripper") or (http.user_agent contains "Siteimprove") or (http.user_agent contains "sitesell") or (http.user_agent contains "Siphon") or (http.user_agent contains "Sucker") or (http.user_agent contains "TenFourFox") or (http.user_agent contains "TurnitinBot") or (http.user_agent contains "trendiction") or (http.user_agent contains "twingly") or (http.user_agent contains "VidibleScraper") or (http.user_agent contains "WebLeacher") or (http.user_agent contains "WebmasterWorldForum") or (http.user_agent contains "webmeup") or (http.user_agent contains "Widow") or (http.user_agent contains "Xaldon") or (http.user_agent contains "Xenu") or (http.user_agent contains "xtractor") or (http.user_agent contains "Zermelo") or (http.user_agent contains "ZoominfoBot") or (http.user_agent contains "brands-bot") or (http.user_agent contains "bbot") or (http.user_agent contains "brands-bot-logo") or (http.user_agent contains "python") or (http.user_agent contains "java") or (http.user_agent contains "Turnitin") or (http.user_agent contains "bit.ly") or (http.user_agent contains "PulsePoint-Ads") or (http.user_agent contains "Bloglovin") or (http.user_agent contains "PetalBot") or (http.user_agent contains "webmeup-crawler") or (http.user_agent contains "wp_is_mobile") or (http.user_agent contains "Seekport") or (http.user_agent contains "DotBot") or (http.user_agent contains "BLEXBot") or (http.user_agent contains "paloaltonetworks") or (http.user_agent contains "Go-http-client")
Here is part 2 because the Cloudflare rule has a limited length. For that, you need to create another ruleset.
(http.user_agent contains "Barkrowler") or (http.user_agent contains "YahooMailProxy") or (http.user_agent contains "SentiBot") or (http.user_agent contains "IonCrawl") or (http.user_agent contains "netEstate") or (http.user_agent contains "Geedo") or (http.user_agent contains "DataForSeoBot")
Important!
After you apply any of these rules, make sure you monitor what requests have been blocked and try to identify false positives.