Microsoftのチンピラロボット

このサイトでは5~6年前からMicosoft/bingの検索拒否設定をしている。robots.txtの冒頭で次のようなことを書いている。Microsoft/bingの低能ゴミロボットでもよくわかるように冒頭に置いているのである。

SAMPLE
User-agent: bingbot
Disallow: /

しかし、そんなことにはお構いなくbingのチンピラロボットはやってきている。さすがならず者のボス。ゴミ会社のMicrosoftのやりそうなことである。この会社は信用できない。徹頭徹尾腐っていることがよくわかる。アメリカの多くの会社と同じように下品でも「大きすぎて潰せない」ので生きながらえているだけである。

GoogleMicrosoftなどでは、ボス配下の子分の闇ロボットを使ってデータを収集していることは公知の事実だが、親分自らがWEB世界のルール(robots.txt)を公然と無視してくるのは悪質である。Googleも信用できないが、まだMicrosoftよりはマシである。Googleの検索拒否をしていたときは、親分のGoogleのロボットが来たことはなかったのに比べると、Microsoft/bingのチンピラぶりは顕著である。

下記は今週の例(2020/06/07-2020/06/09)である。さすがMicrosoft。世界を股にかけたゴミ会社。裏では悪徳三昧である。カネ儲けはできても品位のないカス会社である。
ちなみに、中国のbaiduもルール無視はひどいがMicrosoftほどではない。どちらも似た者同士、ゴミとカスの米中戦争であるが(笑)。

この同類に40.77.***.***系のBingPreviewがある。*.js, *.cssや画像取り専用のようであるがこのバカロボットはrobots.txtを読まない。もっとも、読んでも守らないが。
こんな会社がやっている「クラウドストレージ」などとうてい信用できるわけがない。データを勝手につまみ食いすることは目に見えている。
LOG
157.55.39.39 - - [07/Jun/2020:05:43:59 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
これを見ているので巡回しないのかと思ったら大間違い。無視してあちこちデータを収集している。
157.55.39.39 - - [07/Jun/2020:09:44:15 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://wingww.bing.com/bingbot.htm)"
157.55.39.18 - - [07/Jun/2020:23:42:05 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.96 - - [08/Jun/2020:01:50:18 0900] "GET /tomato/pc6/ac1_0139p.html HTTP/1.1" 200 2089 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.96 - - [08/Jun/2020:03:52:16 0900] "GET /pluto/tpc/tp1/tpc_0093_2.html HTTP/1.1" 200 25285 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.96 - - [08/Jun/2020:10:07:34 0900] "GET /taro/idt/id2/tips_0023.html HTTP/1.1" 404 298 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.96 - - [08/Jun/2020:12:02:42 0900] "GET /pluto/tpc/tp1/tpc_0029.html HTTP/1.1" 200 1621 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.37 - - [08/Jun/2020:14:31:02 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.96 - - [08/Jun/2020:14:31:09 0900] "GET /pluto/tpc/tp1/tpc_0018_a1.html HTTP/1.1" 200 1297 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.96 - - [08/Jun/2020:19:09:02 0900] "GET /tomato/pc1/mfc6_0905p.html HTTP/1.1" 200 4206 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.96 - - [08/Jun/2020:19:09:40 0900] "GET /tomato/text.css HTTP/1.1" 200 2276 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.96 - - [08/Jun/2020:19:34:21 0900] "GET /tomato/pc1/mfc2_1104p.html HTTP/1.1" 200 3694 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.17 - - [09/Jun/2020:02:10:30 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.32 - - [09/Jun/2020:08:04:55 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.32 - - [09/Jun/2020:11:07:54 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.150 - - [09/Jun/2020:15:27:45 0900] "GET /pluto/tpc/tp2/tpc_0149.html HTTP/1.1" 200 4041 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.38 - - [09/Jun/2020:17:10:43 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.38 - - [09/Jun/2020:18:39:50 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.150 - - [09/Jun/2020:18:40:09 0900] "GET /pluto/tpc/tp1/tpc_0066.html HTTP/1.1" 200 1235 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.38 - - [09/Jun/2020:19:21:50 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.38 - - [09/Jun/2020:20:21:31 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.150 - - [09/Jun/2020:20:23:52 0900] "GET /pluto/tpc/tp1/tpc_0008.html HTTP/1.1" 200 2218 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.38 - - [09/Jun/2020:20:55:09 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.150 - - [09/Jun/2020:22:27:20 0900] "GET /taro/iec/ie1/ts1_0011.html HTTP/1.1" 200 2687 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
157.55.39.32 - - [09/Jun/2020:23:50:00 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"

207.46.13.109 - - [08/Jun/2020:03:01:57 0900] "GET /tomato/pc2/c2_0022p.html HTTP/1.1" 200 1881 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.44 - - [08/Jun/2020:14:26:50 0900] "GET /edit/edt/ed1/news.html HTTP/1.1" 200 20717 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:01:38:08 0900] "GET /tavern/sjk/ns1/2002_01.html HTTP/1.1" 200 8847 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:02:27:13 0900] "GET /tomato/pcy/items.html HTTP/1.1" 404 245 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:03:40:35 0900] "GET /taro/idt/id2/wks_0096.html HTTP/1.1" 200 3219 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.44 - - [09/Jun/2020:04:47:27 0900] "GET /tomato/pcy/tips_0003.html HTTP/1.1" 404 249 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.44 - - [09/Jun/2020:04:48:47 0900] "GET /pluto/tpc/tp5/items.html HTTP/1.1" 200 2162 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:04:48:51 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:04:48:52 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.44 - - [09/Jun/2020:04:48:57 0900] "GET /pluto/tpc/mscr.js HTTP/1.1" 200 3827 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.44 - - [09/Jun/2020:05:23:44 0900] "GET /tavern/cfb/cb1/cb_0046.html HTTP/1.1" 200 4399 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.44 - - [09/Jun/2020:09:04:23 0900] "GET /edit/edt/ed1/seso.html HTTP/1.1" 404 293 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.44 - - [09/Jun/2020:09:32:25 0900] "GET /pluto/tpc/tp1/tpc_0008.html HTTP/1.1" 200 2218 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:09:44:04 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.239 - - [09/Jun/2020:15:18:37 0900] "GET /book/bk2/s1/bk2_0079.html HTTP/1.1" 404 251 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:16:59:27 0900] "GET /pluto/tpc/tp4/tpc_0304.html HTTP/1.1" 200 3112 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:18:32:23 0900] "GET /pluto/tpc/tp1/tpc_0002.html HTTP/1.1" 200 2081 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:18:49:58 0900] "GET /tomato/pc1/mfc4_0207p.html HTTP/1.1" 200 1892 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:19:22:18 0900] "GET /tomato/pcy/excel_0004.html HTTP/1.1" 404 249 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.239 - - [09/Jun/2020:20:21:42 0900] "GET /tavern/cfb/cb1/cb_0099.html HTTP/1.1" 200 3808 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.239 - - [09/Jun/2020:20:28:58 0900] "GET /pluto/tpc/tp1/tpc_0015.html HTTP/1.1" 200 3472 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.239 - - [09/Jun/2020:20:55:33 0900] "GET /pluto/tpc/tp1/tpc_0060.html HTTP/1.1" 200 6654 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.116 - - [09/Jun/2020:22:54:11 0900] "GET /pluto/tpc/tp1/tpc_0078.html HTTP/1.1" 200 1123 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.239 - - [09/Jun/2020:23:29:14 0900] "GET /tomato/pcy/tips_0063.html HTTP/1.1" 404 249 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
207.46.13.239 - - [09/Jun/2020:23:50:07 0900] "GET /pluto/tpc/tp1/tpc_0093_2.html HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"

13.66.139.1 - - [08/Jun/2020:17:04:36 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
13.66.139.0 - - [08/Jun/2020:17:05:50 0900] "GET /tavern/cfb/cb2/cb_0135.html HTTP/1.1" 200 8014 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
13.66.139.1 - - [09/Jun/2020:13:11:01 0900] "GET /robots.txt HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
13.66.139.0 - - [09/Jun/2020:13:11:23 0900] "GET /taro/ipc/ip1/ipc_0016_b1.html HTTP/1.1" 200 4796 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"

40.77.167.236 - - [08/Jun/2020:23:32:11 0900] "GET /tomato/pc3/pc3_0002.html HTTP/1.1" 200 1616 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
40.77.167.236 - - [09/Jun/2020:02:05:15 0900] "GET /inet/site/site_policy.html HTTP/1.1" 200 2086 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
40.77.167.5 - - [09/Jun/2020:11:19:54 0900] "GET /tavern/sjk/ns1/2004_01.html HTTP/1.1" 200 10896 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
40.77.167.5 - - [09/Jun/2020:12:17:06 0900] "GET /tomato/pc6/vba2_0134p.html HTTP/1.1" 200 2683 "-" "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
40.77.167.236 - - [09/Jun/2020:21:42:01 0900] "GET /edit/info/inf1/ccp_01.html HTTP/1.1" 200 5035 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)"
- 2020/06/09 -



bingから飛んできたようなものがある。
LOG
●IPアドレス: 36.2.169.189
●ホスト名 : 36-2-169-189.kanagawa.fdn.vectant.ne.jp

36.2.169.189 - - [16/Jun/2020:16:48:19 0900] "GET / HTTP/1.1" 302 20 "-" "Mozilla/5.0 (Linux; Android 7.0; SO-01J) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Mobile Safari/537.36"
36.2.169.189 - - [16/Jun/2020:16:48:35 0900] "GET / HTTP/1.1" 302 20 "-" "Mozilla/5.0 (Linux; Android 7.0; SO-01J) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Mobile Safari/537.36"
36.2.169.189 - - [16/Jun/2020:16:48:53 0900] "GET /index.html HTTP/1.1" 200 7188 "http://www.bing.com/search?q=mermaid-tavern.com&form=QBLH&sp=-1&pq=mermaid-tavern.com&sc=2-18&qs=n&sk=&cvid=0FBADA6ACE5A48C7AEC4F03DF6484AE0" "Mozilla/5.0 (Linux; Android 7.0; SO-01J) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Mobile Safari/537.36"
このオトコ本当にbingから飛んできたのかあやしい点がある。初め二つが「http://www.mermaid-tavern.com/」または「http://mermaid-tavern.com/」で来ているからである。すなわち、このURLを知っていたということである。それでbingに飛ばされた。だからbingからやってきたというわけでもあるまい(bingからは呼び出せないようになっているはずだが)。