Фейковые поисковые боты Гугла и Яндекса

admin · 27 Дек 2019

Написал функцию для проверки подлинности поисковых ботов для повышения эффективности антиддос скрипта.

PHP:

function isSearchBot()
{
  $bots = array('Google'=>array('.googlebot.com', '.google.com'), 'Yandex'=>array('.yandex.com', '.yandex.ru', '.yandex.net'), 'mail.ru'=>array('.mail.ru'), 'msn.com'=>array('.msn.com'), 'bing.com'=>array('.msn.com'));

  foreach($bots as $u=>$h)
  {
      if( strpos($_SERVER['HTTP_USER_AGENT'], $u) == true )
      {
         $iphost=gethostbyaddr($_SERVER['REMOTE_ADDR']);

         foreach($h as $host)
         {
             if (substr($iphost, -strlen($host) )==$host) return true;
         }

         return 'фейковый бот';
     }
  }

  return 'не бот';
}

Обнаружил, что много спам-ботов притворяется поисковыми ботами.

Фейковые боты Гугла

Код:

107.179.75.186 - US - / - Googlebot/2.1 ( http://www.googlebot.com/bot.html) (fake bot)
107.179.75.2 - US - / - Googlebot/2.1 ( http://www.googlebot.com/bot.html) (fake bot)
12.244.119.126 - US - / - Googlebot/2.1 ( http://www.googlebot.com/bot.html) (fake bot)
144.91.78.42 vmi304834.contaboserver.net
167.71.8.249 - NL - / - Googlebot/2.1 ( http://www.googlebot.com/bot.html) (fake bot)
176.62.80.38 - RU - /threads/601/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) (fake bot)
178.155.5.93 - RU - /threads/637/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) (fake bot)
95.26.114.231 - RU - / - Googlebot/2.1 ( http://www.googlebot.com/bot.html) (fake bot)
64.137.110.237 - US - / - Googlebot/2.1 ( http://www.googlebot.com/bot.html) (fake bot)
71.167.120.152 - US - /threads/678/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) (host pool-71-167-120-152.nycmny.fios.verizon.net)
46.45.34.93 - RU - /threads/408/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Фейковые боты Яндекса

Код:

54.37.137.122 - PL- / - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
94.127.70.125 - RU - /threads/374/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) (host s094127070125.m.truevds.ru)

Появился бот YANDEX ENTERPRISE NETWORK из корпоративной сети Яндекса. Проверку скриптом не проходит, так как не имеет хоста.
Значит не обязательно пускать его на сервер. Тем более, если он ведет себя неадекватно - делает несколько запросов к странице в минуту - зачем?

Но можете добавить в белый лист через админку.

Код:

213.180.203.113 - RU - /threads/401/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 (fake bot, 2019-12-28 12:15:04)
213.180.203.124 - RU - /threads/64/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 (fake bot, 2019-12-28 08:42:54)
213.180.203.141 - RU - /threads/180/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 (fake bot, 2019-12-28 02:16:16)
213.180.203.159 - RU - /threads/702/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) (fake bot, 2019-12-28 09:07:12)
213.180.203.97 - RU - /threads/129/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 (fake bot, 2019-12-28 12:36:41)

Возможно это проверка на клоакинг.

Полный лог работы скрипта за несколько месяцев.

Код:

95.30.246.39 - [2022-11-24 05:42:56] /threads/579/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 95-30-246-39.broadband.corbina.ru
46.8.222.167 - [2022-12-19 07:59:16] /threads/579/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 46.8.222.167
188.243.0.8 - [2022-12-23 10:21:44] /login/keep-alive - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 188.243.0.8.pool.sknt.ru
93.100.118.135 - [2023-02-03 05:31:32] /threads/579/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 93.100.118.135.pool.sknt.ru
188.170.82.1 - [2023-02-06 04:14:23] /threads/155/page-5 - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) Page Audit (https://page-audit.com/bot.txt) - 188.170.82.1
84.38.132.35 - [2023-02-16 02:36:53] /threads/1045/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 84.38.132.35
213.180.203.254 - [2023-03-11 07:09:23] /threads/170/post-1959 - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 - 213.180.203.254
188.120.251.252 - [2023-03-25 11:08:51] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - zwrk104-chckr-p.fvds.ru
178.158.38.37 - [2023-03-29 12:15:47] / - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.64 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 178.158.38.37
82.146.35.212 - [2023-04-22 06:39:34] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - zwrk101-chckr-p.fvds.ru
212.109.193.109 - [2023-04-29 11:34:11] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - zwrk102-chckr-p.fvds.ru
176.59.207.218 - [2023-05-05 03:06:39] /threads/4/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Page Audit (https://page-audit.com/bot.txt) - 176.59.207.218
212.193.186.71 - [2023-05-08 06:00:23] /threads/1082/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 212.193.186.71
45.134.25.219 - [2023-05-08 06:52:14] /threads/534/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 45.134.25.219
94.154.189.24 - [2023-05-08 10:32:37] /threads/958/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 94.154.189.24
45.142.254.123 - [2023-05-09 10:38:07] /threads/1053/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 45.142.254.123
45.139.126.232 - [2023-05-10 05:10:32] /threads/958/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 45.139.126.232
193.58.170.154 - [2023-05-10 05:52:49] /threads/1053/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 193.58.170.154
194.55.105.50 - [2023-05-11 07:05:11] /threads/355/ - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 194.55.105.50
95.79.145.24 - [2023-05-15 06:46:37] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - dynamicip-95-79-145-24.pppoe.nn.ertelecom.ru
212.109.195.88 - [2023-05-17 11:08:27] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - zwrk103-chckr-p.fvds.ru
194.26.229.64 - [2023-05-24 03:00:19] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - ds-5254a203.aeza.network
91.121.150.229 - [2023-05-30 06:10:09] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - ns358486.ip-91-121-150.eu
5.53.124.184 - [2023-06-08 02:26:37] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - gojobgenie.com
5.130.144.139 - [2023-06-12 10:38:57] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - l5-130-144-139.novotelecom.ru
91.215.110.53 - [2023-07-21 01:00:54] /threads/993/ - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Page Audit (https://page-audit.com/bot.txt) - 91.215.110.53
91.227.155.101 - [2023-08-04 03:25:25] /threads/652/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 91.227.155.101
91.243.188.207 - [2023-08-04 05:14:10] /threads/492/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 91.243.188.207
176.119.140.59 - [2023-08-04 09:01:09] /threads/155/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 176.119.140.59
176.119.140.140 - [2023-08-04 09:44:36] / - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 176.119.140.140
91.243.188.112 - [2023-08-05 08:28:17] /threads/637/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 91.243.188.112
91.243.188.180 - [2023-08-06 12:11:07] /threads/1024/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 91.243.188.180
91.227.155.249 - [2023-08-10 07:25:47] /forums/antiddos-hosting/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 91.227.155.249
91.227.155.38 - [2023-08-10 12:46:09] /threads/672/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 91.227.155.38
176.119.140.20 - [2023-08-11 04:13:56] /threads/733/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 176.119.140.20
176.119.140.247 - [2023-08-11 02:39:05] /threads/96/ - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 176.119.140.247
2a0d:1ac3:7935:: - [2023-08-13 11:49:10] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 2a0d:1ac3:7935::
178.64.251.44 - [2023-08-14 12:49:19] /sitemap.xml - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 178.64.251.44
178.69.156.187 - [2023-08-14 10:03:26] /sitemap.xml - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) - 178.69.156.187
65.21.200.52 - [2023-08-16 11:15:57] / - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 65-21-200-52.ptr

ivkuza · 25 Окт 2021

Вообще понятие клоакинг, помоему явно устарело и ушло в прошлое.
Уже давно принято вначале определять того кто к тебе зашел и в зависимости от этого выдавать контент, к примеру версию сайта для мобильных устройств или для пк, либо язык пользователя, либо под определенный браузер, причем впереди всех в этом направлении идут сами поисковики.
Так что думаю врятле это проверка на клоакинг

Фейковые поисковые боты Гугла и Яндекса

admin

Администратор

ivkuza

New Member