webClient.getPage爬取csdn页面,卡死

最近在写一下简单的爬虫,然后使用webClient.getPage的时候,一请求就直接挂掉,很长时间没有反应,让人很头疼,然后去掉js,返回的又是一些看不懂的:
获取列表页的文章列表:
https://blog.csdn.net/qq_38366063/article/list/1

正确的应该是:
在这里插入图片描述

这是获取列表页返回错误的:
在这里插入图片描述

<html><script>
var arg1='0BC3DE07BEF4716BE7116BB1EE989B8E4EDBB0D5';
var _0x4818=['x63x73x4bx48x77x71x4dx49','x5ax73x4bx4ax77x72x38x56x65x41x73x79','x55x63x4bx69x4ex38x4fx2fx77x70x6cx77x4dx41x3dx3d','x4ax52x38x43x54x67x3dx3d','x59x73x4fx6ex62x53x45x51x77x37x6fx7ax77x71x5ax4bx65x73x4bx55x77x37x6bx77x58x38x4fx52x49x51x3dx3d','x77x37x6fx56x53x38x4fx53x77x6fx50x43x6cx33x6...
function reload(x) {setCookie("acw_sc__v2", x);document.location.reload();}
</script></html>

然后就很烦,然后想了想,要是直接get请求返回流将流转字符串转dom对象是不是就可以了,然后试了一下,真可以,美滋滋,解决了爬虫最关键问题:

 public static InputStream doGet(String urlstr, Map<String, String> headers) throws IOException {
        URL url = new URL(urlstr);
        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
        conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 " +
                "(KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
        conn.setRequestProperty("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp," +
                "image/apng,*/*;q=0" +
                ".8");
        if (headers != null) {
            Set<String> keys = headers.keySet();
            for (String key : keys) {
                conn.setRequestProperty(key, headers.get(key));
            }
        }
        Random random = new Random();
        String ip =
                (random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100);
        conn.setRequestProperty("x-forwarded-for", ip);
        InputStream inputStream = conn.getInputStream();
        return inputStream;
    }

返回的InputStream直接转String,然后直接Jsoup解析即可:

   public static String inputStreamToString(InputStream is, String charset) throws IOException {
        byte[] bytes = new byte[1024];
        int byteLength = 0;
        StringBuffer sb = new StringBuffer();
        while ((byteLength = is.read(bytes)) != -1) {
            sb.append(new String(bytes, 0, byteLength, charset));
        }
        return sb.toString();
    }

在这里插入图片描述
还是有问题,有的页面可以直接流的方式直接写,有的不行,不知道什么原因=-=.
博客的详情页面就不行…

csdn有验证token,打开控制台,找到对应token,加上去就好了,token一般更新时间为半个小时…否则会被屏蔽…
,csdn爬取,加上token即可,自己通过webClient.getPage,就可以获取.,或者直接通过httpRequest,直接get请求也是可以的…记得传token即可.

世界上所有的不公平都是由于当事人能力不足造成的.