últimos mensajes de: Diabliyo

Buen dia.

Estoy refinando mi sistema de visitas, ya que recibo un montonon de visitas pero me doy cuenta que el 99% de las visitas no puedes asegurar que un webcrawler setee la variable del Navegador con su nombre del bot, asi que no queda de otra mas que ir armandome mi base de datos de IPs propiedad de los bots... Pero, claro esto es una locura porque en menos de una semana tengo ya identiifcadas muchisimos rangos de IPs...

Un ejemplo de mi procedimiento:

1) Recibi una visita de esta ip: 66.249.65.149
2) Abro mi consola y hago "whois 66.249.65.149", inmediatamente en el NetName valido que es "Google", osea, en conclusion es google bot, OJO, pero en la variable del navegador NO habia rastro de que era googlebot...

Citar

NetRange: 66.249.64.0 - 66.249.95.255
CIDR: 66.249.64.0/19
NetName: GOOGLE

3) Tomo el valor de NetRange y me creo una bdd personal.

Código:

function is_robot($ip)
          {
          $r=0; # inicializamos 
          $arr=array( # array de bots y crawlers
                    "google"=>array(
                              "66.249.64.0"=>"66.249.95.255"
                              )
                    );

          foreach( $arr as $key=>$val ) # recorremos
                    {
                    if( !$r && is_array($val) && count($val) ) # si hay datos
                              {
                              foreach( $val as $key2=>$val2 )
                                        {
                                        if( !$r && ip2long($obj)>=ip2long($key2) && ip2long($obj)<=ip2long($val2) )
                                                  $r= $key; # obtenemos el bot
                                        }
                              }
                    }

          return $r; # retornamos resultado
          }

4) Cuando la funcion is_robot($ip) retorna 0 (cero), entonces es una visita de humano, de lo contrario obtendria la "marca" del bot o crawler.

Quisiera saber si existe alguna base de datos de IPs ya identificadas como bots y crawlers ??? porque sinceramente esto ya se volvio una locura estar checando a diario y checando las IPs... claro esta, que hasta ahorita he identificado y que los servers que son bots llevan siempre el NetName de: APNIC, OVH, RIPE, Amazon, Alibaba, DigitalOcean, VegasNAP, Choopa, etc....

Y los webcrawler que he identiifcado usan un NetName como: Facebook, Google, Yandex, Bing.

Espero alguien me pueda dar sus comentarios !

Les comparto mi lista de IPs detectadas de Enero 2017 a la feche de hoy !.

Código

<?php
	$botips= array(
		"facebook"=>array(
			"31.13.66.0"=>"31.13.66.255", 
			"69.171.224.0"=>"69.171.255.255", 
			"157.240.0.0"=>"157.240.255.255", 
			"173.252.64.0"=>"173.252.127.255", 
			"66.220.144.0"=>"66.220.159.255"
			), 
		"google"=>array(
			"64.233.160.0"=>"64.233.191.255", 
			"66.102.0.0"=>"66.102.15.255", 
			"66.249.64.0"=>"66.249.95.255", 
			"74.125.0.0"=>"74.125.255.255", 
			"108.177.0.0"=>"108.177.127.255", 
			"173.194.0.0"=>"173.194.255.255", 
			"216.239.32.0"=>"216.239.63.255", 
			"216.58.192.0"=>"216.58.223.255", 
			"72.14.192.0"=>"72.14.255.255", 
			"209.85.128.0"=>"209.85.255.255"
			), 
		"bing"=>array(
			"157.54.0.0"=>"157.60.255.255", 
			"207.46.0.0"=>"207.46.255.255", 
			"40.74.0.0"=>"40.125.127.255", 
			"23.96.0.0"=>"23.103.255.255", 
			"40.74.0.0"=>"40.125.127.255", 
			"13.64.0.0"=>"13.107.255.255", 
			"137.135.0.0"=>"137.135.255.255", 
			"104.40.0.0"=>"104.47.255.255"
			), 
		"ovh"=>array(
			"149.56.0.0"=>"149.56.255.255", 
			"178.32.96.0"=>"178.32.127.255", 
			"188.165.32.0"=>"188.165.47.255", 
			"51.254.0.0"=>"51.255.255.255", 
			"37.187.160.0"=>"37.187.175.255", 
			"164.132.0.0"=>"164.132.255.255", 
			"151.80.32.0"=>"151.80.47.255", 
			"94.23.112.220"=>"94.23.112.223", 
			"91.134.0.0"=>"91.134.255.255", 
			"91.134.0.0"=>"91.134.255.255", 
			"158.69.0.0"=>"158.69.255.255", 
			"192.99.0.0"=>"192.99.255.255", 
			"142.4.192.0"=>"142.4.223.255", 
			"167.114.0.0"=>"167.114.255.255", 
			"192.99.0.0"=>"192.99.255.255", 
			"151.80.16.0"=>"151.80.31.255", 
			"198.27.64.0"=>"198.27.127.255", 
			"176.31.59.228"=>"176.31.59.231", 
			"178.33.191.168"=>"178.33.191.171", 
			"66.70.128.0"=>"66.70.255.255"
			), 
		"duckduckgo"=>array(
			"52.0.0.0"=>"52.31.255.255", 
			"50.16.0.0"=>"50.19.255.255"
			), 
		"ripe"=>array(
			"46.229.168.0"=>"46.229.169.255", 
			"62.210.128.0"=>"62.210.255.255", 
			"144.76.0.0"=>"144.76.255.255", 
			"137.74.0.0"=>"137.74.255.255", 
			"77.67.54.0"=>"77.67.54.15", 
			"79.184.0.0"=>"79.184.255.255", 
			"195.22.124.0"=>"195.22.127.255", 
			"2.138.0.0"=>"2.139.255.255", 
			"194.60.69.0"=>"194.60.69.255", 
			"88.0.0.0"=>"88.15.255.255", 
			"84.78.0.0"=>"84.79.255.255", 
			"188.226.128.0"=>"188.226.191.255", 
			"163.172.0.0"=>"163.172.255.255", 
			"31.13.114.0"=>"31.13.114.255", 
			"93.73.0.0"=>"93.73.127.255", 
			"85.105.0.0"=>"85.105.255.255", 
			"73.0.0.0"=>"73.255.255.255", 
			"73.58.128.0"=>"73.58.255.255", 
			"31.13.114.0"=>"31.13.114.255", 
			"88.99.0.0"=>"88.99.255.255", 
			"46.165.192.0"=>"46.165.199.255", 
			"70.39.128.0"=>"70.39.255.255", 
			"70.39.157.192"=>"70.39.157.223", 
			"77.66.1.96"=>"77.66.1.103", 
			"5.9.88.96"=>"5.9.88.127", 
			"217.182.0.0"=>"217.182.255.255", 
			"194.187.168.0"=>"194.187.171.255", 
			"51.36.0.0"=>"51.36.255.255", 
			"62.210.0.0"=>"62.210.127.255", 
			"94.199.151.16"=>"94.199.151.31", 
			"148.251.0.0"=>"148.251.255.255", 
			"176.10.99.192"=>"176.10.99.223", 
			"212.62.42.0"=>"212.62.42.255", 
			"178.33.169.232"=>"178.33.169.235", 
			"213.239.210.0"=>"213.239.211.255", 
			"185.170.42.0"=>"185.170.42.255", 
			"217.69.128.0"=>"217.69.135.255", 
			"83.32.0.0"=>"83.39.255.255", 
			"79.168.0.0"=>"79.168.255.255", 
			"5.196.0.0"=>"5.196.255.255", 
			"94.228.205.64"=>"94.228.205.127", 
			"93.174.123.128"=>"93.174.123.255", 
			"94.23.168.0"=>"94.23.175.255", 
			"31.177.95.0"=>"31.177.95.255", 
			"80.248.225.128"=>"80.248.225.191", 
			"95.221.0.0"=>"95.221.255.255", 
			"89.144.0.0"=>"89.144.63.255", 
			"93.160.60.0"=>"93.160.60.255", 
			"176.10.104.239"=>"176.10.104.241", 
			"176.126.252.8"=>"176.126.252.15", 
			"188.165.192.0"=>"188.165.255.255", 
			"95.83.160.0"=>"95.83.191.255", 
			"46.105.96.0"=>"46.105.127.255", 
			"185.117.118.0"=>"185.117.118.255", 
			"37.187.32.0"=>"37.187.35.255", 
			"185.87.185.0"=>"185.87.185.255", 
			"5.135.193.212"=>"5.135.193.215", 
			"87.116.176.0"=>"87.116.191.255", 
			"64.119.157.112"=>"64.119.157.127", 
			"64.119.128.0"=>"64.119.159.255", 
			"91.247.38.0"=>"91.247.38.255", 
			"77.75.76.0"=>"77.75.76.255", 
			"138.201.0.0"=>"138.201.255.255", 
			"185.163.0.0"=>"185.163.3.255", 
			"85.102.40.0"=>"85.102.255.255", 
			"83.56.0.0"=>"83.59.255.255", 
			"51.15.0.0"=>"51.15.255.255", 
			"178.17.168.0"=>"178.17.175.255", 
			"93.159.230.0"=>"93.159.231.255", 
			"78.109.16.0"=>"78.109.31.255", 
			"5.199.130.0"=>"5.199.130.255", 
			"78.24.220.0"=>"78.24.223.255", 
			"93.115.92.0"=>"93.115.95.255", 
			"138.246.253.0"=>"138.246.253.255", 
			"37.187.72.0"=>"37.187.79.255", 
			"83.140.112.0"=>"83.140.112.31", 
			"146.185.223.0"=>"146.185.223.255", 
			"185.31.172.234"=>"185.31.172.234", 
			"185.70.8.0"=>"185.70.11.255", 
			"188.40.126.64"=>"188.40.126.127"
			), 
		"apnich"=>array(
			"58.21.0.0"=>"58.21.255.255", 
			"36.32.0.0"=>"36.35.255.255", 
			"123.125.71.0"=>"123.125.71.255", 
			"220.181.0.0"=>"220.181.255.255", 
			"180.152.0.0"=>"180.159.255.255", 
			"180.76.0.0"=>"180.76.255.255", 
			"116.252.0.0"=>"116.253.255.255", 
			"123.144.0.0"=>"123.147.255.255", 
			"171.36.0.0"=>"171.39.255.255", 
			"114.240.0.0"=>"114.255.255.255", 
			"139.170.0.0"=>"139.170.255.255", 
			"175.16.0.0"=>"175.23.255.255", 
			"123.188.0.0"=>"123.191.255.255", 
			"125.76.0.0"=>"125.76.127.255", 
			"111.160.0.0"=>"111.167.255.255", 
			"106.45.0.0"=>"106.45.255.255", 
			"125.211.0.0"=>"125.211.255.255", 
			"61.146.178.0"=>"61.146.178.255", 
			"42.224.0.0"=>"42.239.255.255", 
			"106.37.0.0"=>"106.39.255.255", 
			"136.243.0.0"=>"136.243.255.255", 
			"110.80.0.0"=>"110.87.255.255", 
			"218.30.96.0"=>"218.30.127.255", 
			"106.11.0.0"=>"106.11.255.255", 
			"210.245.0.0"=>"210.245.15.255", 
			"202.46.32.0"=>"202.46.63.255", 
			"103.16.46.0"=>"103.16.46.255", 
			"118.99.96.0"=>"118.99.96.255", 
			"182.74.166.152"=>"182.74.166.159",
			"121.201.0.0"=>"121.201.127.255", 
			"58.19.0.0"=>"58.19.255.255", 
			"120.32.0.0"=>"120.39.255.255", 
			"60.0.0.0"=>"60.10.255.255", 
			"106.80.0.0"=>"106.95.255.255", 
			"123.51.128.0"=>"123.51.255.255", 
			"113.112.0.0"=>"113.119.255.255", 
			"119.23.0.0"=>"119.23.255.255", 
			"175.42.0.0"=>"175.42.63.255", 
			"123.112.0.0"=>"123.127.255.255", 
			"47.92.0.0"=>"47.95.255.255", 
			"27.192.0.0"=>"27.223.255.255", 
			"106.51.192.0"=>"106.51.223.255", 
			"106.128.0.0"=>"106.191.255.255"
			), 
		"amazon"=>array(
			"54.144.0.0"=>"54.159.255.255", 
			"52.32.0.0"=>"52.63.255.255", 
			"34.192.0.0"=>"34.255.255.255", 
			"54.208.0.0"=>"54.209.255.255", 
			"35.160.0.0"=>"35.167.255.255", 
			"35.152.0.0"=>"35.183.255.255", 
			"52.192.0.0"=>"52.223.255.255", 
			"54.224.0.0"=>"54.239.255.255", 
			"52.84.0.0"=>"52.95.255.255", 
			"54.160.0.0"=>"54.175.255.255", 
			"54.72.0.0"=>"54.95.255.255", 
			"54.176.0.0"=>"54.191.255.255", 
			"54.210.0.0"=>"54.211.255.255", 
			"54.208.0.0"=>"54.221.255.255", 
			"54.204.0.0"=>"54.205.255.255", 
			"54.192.0.0"=>"54.207.255.255", 
			"107.20.0.0"=>"107.23.255.255", 
			"13.52.0.0"=>"13.59.255.255", 
			"184.72.0.0"=>"184.73.255.255", 
			"54.64.0.0"=>"54.71.255.255", 
			"18.219.0.0"=>"18.228.255.255", 
			"79.125.0.0"=>"79.125.63.255", 
			"18.194.0.0"=>"18.195.255.255", 
			"18.194.0.0"=>"18.197.255.255"
			), 
		"alibaba"=>array(
			"47.88.0.0"=>"47.91.255.255"
			), 
		"generalcrawler"=>array(
			"162.210.192.0"=>"162.210.199.255", 
			"173.249.0.0"=>"173.249.63.255"
			), 
		"yahoo"=>array(
			"68.180.128.0"=>"68.180.255.255", 
			"72.30.0.0"=>"72.30.255.255"
			), 
		"exaled"=>array(
			"178.255.208.0"=>"178.255.215.255", 
			"67.227.128.0"=>"67.227.255.255", 
			"69.39.224.0"=>"69.39.239.255"
			), 
		"twitter"=>array(
			"199.16.156.0"=>"199.16.159.255"
			), 
		"apple"=>array(
			"17.0.0.0"=>"17.255.255.255"
			), 
		"yandex"=>array(
			"77.88.47.0"=>"77.88.47.255", 
			"5.255.250.0"=>"5.255.250.255", 
			"84.201.133.0"=>"84.201.133.127", 
			"141.8.144.0"=>"141.8.144.255", 
			"93.158.161.0"=>"93.158.161.255", 
			"141.8.143.128"=>"141.8.143.255"
			), 
		"MIT"=>array(
			"128.30.0.0"=>"128.30.255.255"
			), 
		"DigitalOcean"=>array(
			"104.236.0.0"=>"104.236.255.255", 
			"192.241.128.0"=>"192.241.255.255", 
			"104.131.0.0"=>"104.131.255.255", 
			"138.68.0.0"=>"138.68.255.255", 
			"138.197.0.0"=>"138.197.255.255"
			), 
		"VegasNAP"=>array(
			"104.128.64.0"=>"104.128.75.255"
			), 
		"Leaseweb"=>array(
			"108.59.0.0"=>"108.59.15.255", 
			"199.58.84.0"=>"199.58.87.255"
			), 
		"Choopa"=>array(
			"104.207.128.0"=>"104.207.129.255", 
			"104.207.128.0"=>"104.207.159.255", 
			"104.238.180.0"=>"104.238.181.255", 
			"104.238.128.0"=>"104.238.191.255", 
			"45.32.0.0"=>"45.32.255.255", 
			"45.32.172.0"=>"45.32.173.255"
			), 
		"paypal"=>array(
			"173.0.80.0"=>"173.0.95.255"
			), 
		"Otros"=>array(
			"198.240.100.0"=>"198.240.103.255", 
			"198.240.64.0"=>"198.240.127.255", 
			"172.82.160.0"=>"172.82.191.255", 
			"172.82.128.0"=>"172.82.191.255", 
			"204.12.206.0"=>"204.12.206.255", 
			"204.12.192.0"=>"204.12.255.255", 
			"199.191.56.0"=>"199.191.59.255", 
			"199.191.56.132"=>"199.191.56.135", 
			"173.205.33.16"=>"173.205.33.31", 
			"173.205.0.0"=>"173.205.127.255", 
			"104.223.112.0"=>"104.223.127.255", 
			"104.223.0.0"=>"104.223.127.255", 
			"204.79.180.0"=>"204.79.180.255", 
			"104.144.89.176"=>"104.144.89.191", 
			"104.144.0.0"=>"104.144.255.255", 
			"69.30.213.136"=>"69.30.213.143", 
			"69.30.192.0"=>"69.30.255.255"
			)
		);
 
?>