hadoop 遇到java.net.ConnectException: to 0.0.0.0:10020 failed on connection

这个问题一般是在hadoop2.x版本里会出现，Hadoop的datanode需要访问namenode的jobhistory server,如果没有修改，则默认为0.0.0.0:10020，则可以修改mapred-site.xml文件：

<property>
<name>mapreduce.jobhistory.address</name>

<value>0.0.0.0:10020</value>
</property>

<property>
<name>mapreduce.jobhistory.webapp.address</name>

<value>0.0.0.0:19888</value>
</property>

最后不要忘记了启动jobhistory

$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver

********************************************************************

a:C:words.txt 文件内容:

hello alamp s

hello qq

hello xx

hello aa

hello swk

hello zbj

hello blm

blm xixi

zbj hehe

swk haha

b. java

@Test

public void testUpload() throws IllegalArgumentException, IOException {

FSDataOutputStream out = fs.create(new Path(

"hdfs://itcast01:9000/upload2"));

FileInputStream in = new FileInputStream(new File("c:/words.txt"));

IOUtils.copyBytes(in, out, 2048, true);

}

package cn.itcast.hadoop.mr;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();

conf.setInt("mapreduce.client.submit.file.replication", 20);

Job job = Job.getInstance(conf);

//notice

job.setJarByClass(WordCount.class);

//set mapper`s property

job.setMapperClass(WCMapper.class);

job.setMapOutputKeyClass(Text.class);

job.setMapOutputValueClass(LongWritable.class);

FileInputFormat.setInputPaths(job, new Path("/upload2/"));

//set reducer`s property

job.setReducerClass(WCReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(LongWritable.class);

FileOutputFormat.setOutputPath(job, new Path("/usr/mapperReduce"));

//submit

job.waitForCompletion(true);

}

c.打jar包

[root@itcast01 usr]# hadoop jar mr.jar

17/05/20 16:44:09 INFO client.RMProxy: Connecting to ResourceManager at itcast01/192.168.233.128:8032

17/05/20 16:44:10 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.

17/05/20 16:44:11 INFO input.FileInputFormat: Total input paths to process : 1

17/05/20 16:44:11 INFO mapreduce.JobSubmitter: number of splits:1

17/05/20 16:44:11 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1495288787912_0006

17/05/20 16:44:12 INFO impl.YarnClientImpl: Submitted application application_1495288787912_0006

17/05/20 16:44:12 INFO mapreduce.Job: The url to track the job: http://itcast01:8088/proxy/application_1495288787912_0006/

17/05/20 16:44:12 INFO mapreduce.Job: Running job: job_1495288787912_0006

17/05/20 16:44:55 INFO mapreduce.Job: Job job_1495288787912_0006 running in uber mode : false

17/05/20 16:44:55 INFO mapreduce.Job: map 0% reduce 0%

17/05/20 16:46:24 INFO mapreduce.Job: map 100% reduce 0%

17/05/20 16:47:02 INFO mapreduce.Job: map 100% reduce 100%

17/05/20 16:47:03 INFO mapreduce.Job: Job job_1495288787912_0006 completed successfully

17/05/20 16:47:06 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=435

FILE: Number of bytes written=186471

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=210

HDFS: Number of bytes written=78

HDFS: Number of read operations=6

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=1

Launched reduce tasks=1

Data-local map tasks=1

Total time spent by all maps in occupied slots (ms)=65092

Total time spent by all reduces in occupied slots (ms)=32649

Total time spent by all map tasks (ms)=65092

Total time spent by all reduce tasks (ms)=32649

Total vcore-seconds taken by all map tasks=65092

Total vcore-seconds taken by all reduce tasks=32649

Total megabyte-seconds taken by all map tasks=66654208

Total megabyte-seconds taken by all reduce tasks=33432576

Map-Reduce Framework

Map input records=17

Map output records=32

Map output bytes=365

Map output materialized bytes=435

Input split bytes=93

Combine input records=0

Combine output records=0

Reduce input groups=13

Reduce shuffle bytes=435

Reduce input records=32

Reduce output records=13

Spilled Records=64

Shuffled Maps =1

Failed Shuffles=0

Merged Map outputs=1

GC time elapsed (ms)=290

CPU time spent (ms)=5530

Physical memory (bytes) snapshot=284258304

Virtual memory (bytes) snapshot=1685770240

Total committed heap usage (bytes)=136515584

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=117

File Output Format Counters

Bytes Written=78

[root@itcast01 usr]# hadoop fs -ls /

Found 9 items

-rw-r--r-- 1 root supergroup 153512879 2017-05-17 05:34 /jdk

-rw-r--r-- 1 root supergroup 62 2017-05-20 13:54 /score_in

drwx------ - root supergroup 0 2017-05-17 06:15 /tmp

-rw-r--r-- 3 root supergroup 32 2017-05-17 14:47 /upload

-rw-r--r-- 3 root supergroup 117 2017-05-20 16:35 /upload2

drwxr-xr-x - root supergroup 0 2017-05-20 16:44 /usr

drwxr-xr-x - root supergroup 0 2017-05-17 06:18 /wcout

-rw-r--r-- 1 root supergroup 70 2017-05-17 06:12 /words

drwxr-xr-x - root supergroup 0 2017-05-20 15:23 /xx

[root@itcast01 usr]# hadoop fs -cat /usr/upload2/part-r-00000

cat: `/usr/upload2/part-r-00000': No such file or directory

[root@itcast01 usr]# hadoop fs -ls /usr/

Found 4 items

drwxr-xr-x - root supergroup 0 2017-05-17 14:54 /usr/local

drwxr-xr-x - root supergroup 0 2017-05-20 16:47 /usr/mapperReduce

drwxr-xr-x - root supergroup 0 2017-05-20 16:08 /usr/swk

drwxr-xr-x - root supergroup 0 2017-05-17 11:25 /usr/test

[root@itcast01 usr]# hadoop fs -ls /usr/mapperReduce

Found 2 items

-rw-r--r-- 1 root supergroup 0 2017-05-20 16:47 /usr/mapperReduce/_SUCCESS

-rw-r--r-- 1 root supergroup 78 2017-05-20 16:46 /usr/mapperReduce/part-r-00000

[root@itcast01 usr]# hadoop fs -cat /usr/mapperReduce/part-r-00000

aa 1

alamp 1

blm 2

haha 1

hehe 1

hello 7

qq 1

s 1

swk 2

xixi 1

xx 1

zbj 2

[root@itcast01 usr]#