Java代码实现远程网页抓取
一、返回头信息的获取
步骤：
1、定义URL对象并初始化；
2、定义URLConnection对象，并通过URL对象的openConnection()方法获取该对象；
3、调用URLConnection对象的connect()方法实现和服务器的连接；
4、通过URLConnection对象获取请求头的域信息（getHeaderFields()、getHeaderField(key)）；
5、使用URLConnection对象的方法来获取信息。
示例：
String urlName = "http://www....com ";
try {
URL url = new URL(urlName);
URLConnection connection = url.openConnection();
connection.connect();
// print header fields
Map<String, List<String>> headers = connection.getHeaderFields();
for (Map.Entry<String, List<String>> entry : headers.entrySet()) {
String key = entry.getKey();
for (String value : entry.getValue()) {
System.out.println(key + ": " + value);
}
}
// print convenience functions
System.out.println("------------------------");
System.out.println("getContentType:" + connection.getContentType());
System.out.println("getContentLength:"
+ connection.getContentLength());
System.out.println("getContentEncoding:"
+ connection.getContentEncoding());
System.out.println("getDate:" + connection.getDate());
System.out.println("getExpiration:" + connection.getExpiration());
System.out.println("getLastModified:"
+ connection.getLastModified());
System.out.println("------------------------");
Scanner in = new Scanner(connection.getInputStream());
// print first ten lines of contents
for (int n = 1; in.hasNextLine() && n <= 10; n++) {
System.out.println(in.nextLine());
}
if (in.hasNextLine())
System.out.println("...");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
二、带参数的请求
在默认情况下，建立的连接只有从服务器读取信息的输入流，并没有任何之行写操作的输出流。如果想获取输出流（例如，想一个Web服务器提交数据），那么需要调用：connection.setDoOutput(true);
示例：
String urlName = "……";
Map<String, String> paras = new HashMap<String, String>();
paras.put("flightway", "Single");
String result;
try {
result = doPost(urlName, paras);
System.out.println(result);
} catch (IOException e) {
e.printStackTrace();
}
public static String doPost(String rlString,
Map<String, String> nameValuePairs) throws IOException {
URL url = new URL(rlString);
URLConnection connection = url.openConnection();
connection.setDoOutput(true);
PrintWriter out = new PrintWriter(connection.getOutputStream());
boolean first = true;
for (Map.Entry<String, String> pair : nameValuePairs.entrySet()) {
if (first)
first = false;
else
out.print('&');
String name = pair.getKey();
String value = pair.getValue();
out.print(name);
out.print('=');
out.print(URLEncoder.encode(value, "GB2312"));//UTF-8
}
out.close();
Scanner in;
StringBuffer response = new StringBuffer();
try {
in = new Scanner(connection.getInputStream());
} catch (IOException e) {
if (!(connection instanceof HttpURLConnection))
throw e;
InputStream err = ((HttpURLConnection) connection).getErrorStream();
if (err == null)
throw e;
in = new Scanner(err);
}
while (in.hasNextLine()) {
response.append(in.nextLine());
response.append("\n");
}
in.close();
return response.toString();
}
备注：
import java.io.*;import java.net.*;import java.util.*;
huc.setDoOutput(true);
// 设置为post方式
huc.setRequestMethod("POST");
huc.setRequestProperty("user-agent", "mozilla/4.7 [en] (win98; i)");
转载自：http://incan.iteye.com/blog/279000