A year or two ago I noticed that BingBot was messing up my stats - it was a considerable percentage of traffic to some of my sites. It made no sense to me that Google would report on bots. That’s sorta the role of server log files. Javascript is supposed to be for humans, but since some bots now execute Javascript it was messing up the analytics data. I posted the problem on Google’s discussion forums but other than a few people agreeing it was a problem - there was no word from Google. Until now. As is typical of them they’ve solved the problem in a big way - not just for BingBot, but for all known spiders.
Unless you purposely want to inflate your data (say to sell ads), you’ll want to enable new feature. But realize that your traffic will drop as a result (though your ratios will improve since bots don’t buy memberships). The smaller the site the more your traffic is likely to drop. And because this is a filter it will only affect data collected after you make the change. I wish they had the ability to get bot data out of old stats, but that apparently isn’t being offered.
I’ll post my findings in a few days when I have some real data to show.
That’s interesting. I had noticed the huge amount of requests that Bing does to my sites. Far more requests than google so I can believe that numbers are inflated a lot. I hate to switch it off and start feeling I lost a lot of traffic though…
Well, Google is not actually filtering bots. I was looking at some stats and noticed the followingâ¦
When I dug into it and compared the two high days to the same days the prior week, I saw thisâ¦
So the increase is completely because of traffic from Microsoft. Now Google will say the problem is because it’s coming through as IE rather than bingbot, but my guess is what they’re doing is page renders - so having a User Agent of IE makes sense in that context.
The reason why it happened this day with this site is because the site was converted to SSL, so Microsoft wanted to recrawl the site.
Still, given that Google is supposed to now be filtering bot traffic I think that’s a huge #FAIL on their part.
