Does Bing dislike Cloudflare?

I started migrating my writings off WordPress and over to this GitHub Pages site towards the end of 2021. And since I've done that I've not been able to get any search coverage of this site in Bing. But suddenly it seems to be starting to work. And it's looking like it might be related to CloudFlare...

The issue url copied!

When I initially set this site up I set up my DNS in a free CloudFlare account. GitHub Pages requires you to configure a CNAME record that points your domain name at the name of your GitHub repo. I'd been using CloudFlare in the past for some other sites I expose through my home server, so it seemed a sensible thing to do at that point:

The "proxy" mode that's enabled here means that you get more than just DNS resoulution. There's data analytics, security features and some basic caching options exposed by CloudFlare's edge servers too. Having that on is the default, and I didn't think much of it at the time because I'd not had issues with my existing sites.

But once I started migrating content and submitting things to search engines I immediately noticed a difference. Google started to crawl and index things. Admittedly slowly - as my new domain would have had no "reputation" with them at that point. But Bing started to crawl and refused to index anything. Whatever I looked at via their search console, I saw the same thing:

Lots of discovered pages, but nothing ever added to the index. The structure was definitely crawled, and it showed the right list of pages but they were all marked as excluded:

But drilling down to the details of that URL, the "inspect" button would not show a specific issue:

But that phrase "if it passes quality checks" might have been an important thing here...

Working on it url copied!

I spent a lot of time editing metadata, requesting reindexing, messing with robots.txt based on looking through Bing's help. All on the assumption that the "quality" issue might have been related to the content I was serving. The best info to guide this that I could find was the Bing Webmaster Guide page but none of the things I read here seemed to help.

Google continued to index more and more over time, and show more results. But Bing stayed stubbornly at zero.

Doing some research I did find that there were others reporting issues on CloudFlare's forums. There are a few possibilities discussed - about CloudFlare blocking the Bing Bot, and the bot dislinking sites behind CloudFlare.

I was pretty sure I could show the problem wasn't blocking the Bing Bot - There was no mention of it in the CloudFlare security logs, and I could create an "explicit allow" rule for Bing's crawler IP ranges ok.

But after well over a year of experiments with metadata and settings, the number of indexed pages on Bing stayed stubbornly at zero.

Some progress? url copied!

But recently I decided to try changing CloudFlare from "proxy" to "just DNS" mode for my blog. That took a bit of fiddling, as initially I got some "this certificate has changed - you're probably being MITM attacked?" style errors from Chrome when trying to browse the site. But having resolved that, I saw two immediate changes:

Firstly, GitHub Pages' DNS checking for custom domains suddenly started to pass:

This had never worked, in the time since I'd set up this repo to publish my generated pages. In the past it had sat permenantly in the "checking" state.

And secondly I fairly quickly noticed I finally got pages in Bing's index:

A small, but steady increase which went from 20-odd in the minutes after I made the change to 50-odd the next morning, and a few days later it's north of 150.

Now, I had not checked the state of Bing directly just before I made the change to CloudFlare, so maybe this is just a coincidence. But the timing correlation is interesting...

But either way, it's a big step forward for my SEO!

↑ Back to top