PHP登入网站抓取并且抓取数据

有时候需要登入网站，然后去抓取一些有用的信息，人工做的话，太累了。有的人可以很快的做到登入，但是需要在登入后再去访问其他页面始终都访问不了，因为他们没有带Cookie进去而被当做是两次会话。下面看看代码

<?php  //test.php
function getWebContent($host,$page="/",$paramstr="",$cookies='',$medth="POST",$port=80){
    $fp = fsockopen($host,$port);
    if(!$fp){
        return false;
    }
    $medth = strtoupper($medth);
    $medth = $medth=="POST" ? "POST":"GET";
    $length = strlen($paramstr);
    if($medth == "GET" && $paramstr){
        $page .= "?".$paramstr;
    }
    $out = "$medth $page  HTTP/1.1 ";
    $out .= "Accept: */* ";
    $out .= "Host: www.exaple.com ";
    $out .= "Content-Length: ".$length." ";
    $out .= "Content-Type: application/x-www-form-urlencoded ";
    if($cookies){
        $out .= "Cookie: ".$cookies." ";
    }
    $out .= "Connection: Keep-Alive ";
    if($medth=='POST' && $paramstr){
        $out .= $paramstr." ";
    }
    fwrite($fp, $out);
    $cookie = "";
    $content = "";
    while (!feof($fp)) {
        $str = fgets($fp);
        if(preg_match("/Set-Cookie:([^ ]*)/",$str,$matchs)){
            if($cookie){
                $cookie .= ";".$matchs[1];
            }else{
                $cookie = $matchs[1];
            }
        }
        $content .= $str;
        echo $str;
    }
    fclose($fp);
    return array('content'=>$content,'cookie'=>$cookie);
}

$params = "name=admin&pwd=admin";
$rs = getWebContent("127.0.0.1","/test/login.php",$params,"","POST",8080);
echo $rs['content'];
$rs = getWebContent("127.0.0.1","/test/index.php","",$rs['cookie'],"POST",8080);
//这里传入上次cookie是关键,否则会被当成两次会话
echo $rs['content'];
?>

<?php //login.php
    $name = $_REQUEST['name'];
    $pwd = $_REQUEST['pwd'];
    if($name == "admin" && $pwd == "admin"){
        setcookie("cname",$name);
        echo "success";
    }else{
        echo "failed";
    }
?>

<?php //index.php
if(isset($_COOKIE['cname']) && $_COOKIE['cname']){
    echo "<ul><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li></ul>";
}else{
    echo "please login first!";
}
?>

将上面三个文件分别保存，login.php和index.php放在root目录下的test目录下。然后test.php放在任意目录，然后去命令行运行php test.php，结果就能出来。

还有一种更简单的方式，就是用curl,代码如下，可以用下面的代码替换test.php
<?php
$post_data = array (
    "name" => "admin",
    "pwd" => "admin",
);
$cookie_jar = tempnam('./', 'cookie');//新建cookie文件
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://localhost:8080/test/login.php");
//设定返回的数据是否自动显示
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// 我们在POST数据哦！
curl_setopt($ch, CURLOPT_POST, 1);
// 把post的变量加上
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
//把返回来的cookie信息保存在$cookie_jar文件中
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_jar);
echo curl_exec($ch);
curl_close($ch);

$ch2 = curl_init();
curl_setopt($ch2, CURLOPT_URL, "http://localhost:8080/test/index.php");
curl_setopt($ch2, CURLOPT_HEADER, false);
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch2, CURLOPT_COOKIEFILE, $cookie_jar);
echo curl_exec($ch2);
unlink($cookie_jar);
curl_close($ch2);
?>