nginx设置屏蔽bytespider垃圾蜘蛛-凯发天生赢家一触即发官网

字节跳动蜘蛛bytespider每日过分抓取我们的网站内容,公然不遵守robots协议,致使服务器高负载状态,下面就教大家如何设置nginx。

7vpbd6m4ev41pcyhjldhkatn1vgso ojx zgicamxgs4yh79qgdhdnbpevyk29nttienvjjkun2lrwqsg9w4hkt2lras14syxlolgzugiq4jin2a5nbiba43aj8lbinit4i5qby25vg6j66xtbjglcdjljpdpdbjtlvpys9kdpomxww1lyrylwq72/c6grljr13pkrh50eglkt3jhx7xg7bnntqwxpaxbqshc2w3kc5mjhugq2ms5e0qlluvattdmsx4qenbufjvm/9mg11vlvkm2azgwku6iaa7px8f7losxu1o386x0fumrmkvzjokyncj6gkjciyi2nnkh6nt0ms/dt3qzzjykqkse/od7ubpqb2byoi8jmioo8kxekvqeivp3rybsizpapvnezpsvlmsjvv1huhj0wkg3y/slgvt2cblnadnvctb/kw1j8m/ku/apzxzq5fmxnkmau018jlyy9mdrxisfysmseu7sow1 elkcag1yxdmbbxqnd5unc9/u30ciczajk7jnz3/cxvzulvblvz3ftydir3no 4dvgrgvp v8eqacwxch kf n43wwt9vl6k3xcvjpvfdk9efy/ui/lwdbdg7ny g/rdljp1jd6/90dh1b4gydmygn6rorugiwmmokocii0qadadpstv8gumhnf9nr cq n58eytdqlkmt5b5cfcjqdqkheqphersvb5x96y81jctealp0btk0hz2xm060pnjcm5sbyi/ofeqf2qidi0kp k69j ayjdjklugwq ltfktuqsclp/fm/hshd/jpsftum7pdt1hyy yhxejuseeb5jdjgrdzg3ae/tn wolt8no6mdhdnbkpv1rhgo/gwallk9an93cdhx/24qht8xbqh2kml8mwr m1dlxdvlsowkmkqnnmgbwvlazath8ndfcjwrfrsld xt8ucwkbh z1babosaf2svvwh6sitd7orjffzbjogpbxkconbn5wzxvohdwrgpmrdswrpv2h402kexwg6yymhefqypjxudpe6hy7q6kpx6we6diix28jf1tor1hf3shgrshyrxj3zervaulfwq3ngh5navodvxbo14yqx1yo9wf7rqgvs0l/uwkrot s/0v0shei7lalr8cj5rvnviivvhm8mmhcg9yiwpyf5y8 gptmlz6a30vqnkvqmvh9vyzjpd0kljkyjcdg4ejnoqm46ytcyss5krydodmayyf0ubd1dxzwndzlwmtvaq9hoazftxsymdyhkgelvzdoi/qldvekqvta6m7y1yyluwmqer3xpqscaxcquk1mffe52bu1snhxpom9bw7cefuxukwmibhbkb8kdg/pmia83 dpgqpeoz79hrwugbzc5hmitl/seax63msaz/qhnye5qvgfruzkp ag9dmzgazpflcxuponv84etruess1r4 vasr6fjn3w5q6zdknxke5wndu0e0un5crirbufbsanqh/p4fd5nfqf1fpu9rqru6lranu9nupvwtjjtaftitkd5a4/gorlafeiyqau/0okq6mjhmpxzc39p yto3 viu0jaxwd5 mbw0dd94lk1o9b5ohoslb fzaukflgn4ymtrpcmdnzwldhelhm1xghpljdbphhuuzbt cpygnggrkjhtodu5tac5 my87hksg7mwdp3exwej7ro0c6vqpw4yp fvdromolre8g4obzr2btasrxhukyrhtjjz5j1xchkbvyzjpr1kbdbarmt2m hyohybrwjbgjlxs7/8l8wqfkrevqe09agcgkfft8zmp1/3tmrbjot925j04rslocdaqq6ul4n6ji4luhwlrdq/z10n4bu7/i7eba5dlkyvblzeb4kkeewkt8rum7poka5xjxzs0q rkmeou3kvkx7uizksubr3h1adbyzhhpdmu jioxgp a5sn5zyicswquzzmkdubagvijt5snrq4s6jdp8n p0y/zs0ztp6lhbglh62hnqhgzbzc/dxix4rjqvv2102h3ueb o4af8hlxlj0c1vaxtufu08qza41etacdevvl8er6 9nxxxgi3frixwnexaopl34wf jcwulhajqx pyya w3y4hucdgnbw4scgobr2qbvsd9lxzavw6t3rnaz7geh2tqpsgu884pms6opdrfbi6oovw7c94 12q72yfbfwgmoeetjbxra4kpfxgrwdp/bscnr95flknpybkwgxhbzpus2rh3v5bk t0wxy8gw a//lyhe12wj6cqnw/2tqovxm2k2jiizmhcbryjkcxf8rudiob3h 74x/vhdl15zqebkhdvdq526pdbxovrddi2lhga728u5ekg2fnum9shftjkli0af4or8zx7it1putanvcqpfbkfjijhrjcgxuy7lrh01idx0xrnz1e70fe k3fjgn9deu2gjc59uox7t5lsnsyd1po2y7r0fts990xu kf9c16am4kx geoxvd0/7jd1p6tdr3srr7i2wvc3i9o6buu9/6a/evvjvx79 jyqjyxpi4fi4e7huieuzl6565ms6drxhxz2v1srp8h"}">

如上图所示,第一种对于不遵守robots协议无效;第二种亲测非常有效;第三种容易误伤;第四种系统版本不一定有效。

本次只讲第二种,步骤如下:

① 打开宝塔面板,点击网站 → 设置 → 配置文件;

宝塔面板nginx配置文件

② 约8行除,添加如下代码并保存

    if ($http_user_agent ~* "feeddemon|indy library|alexa toolbar|asktbfxtv|ahrefsbot|crawldaddy|coolpadwebkit|java|feedly|universalfeedparser|apachebench|microsoft url control|swiftbot|zmeu|obot|jaunty|python-urllib|lightdeckreports bot|yyspider|digext|httpclient|mj12bot|heritrix|bytespider|ezooms|googlebot|jikespider|semrushbot|^$" ) {
      return 403;
    }

代码中我屏蔽谷歌蜘蛛|googlebot| ,同时也屏蔽了垃圾搜索引擎蜘蛛和目前大部分爬虫,后续可以查看网站日志,返回状态码为403就可以了。初次想要验证结果,可将百度蜘蛛|baiduspider| 添加上,到百度搜索资源平台试着抓取诊断一下,基本上都是抓取失败的。

原创文章,作者:霍欣标,如若转载,请注明出处:https://www.yijing21.cn/seo/1151.html

上一篇 2024 年 5 月 26 日
下一篇 2024 年 5 月 18 日

相关推荐

网站地图