安装包
-
Hadoop 3.1.3
-
Zookeeper 3.5.7
-
Hbase 2.2.2
所需工具链接：
链接：https://pan.baidu.com/s/1jcenv7SeGX1gjPT9RnBsIQ
提取码：rkca
伪分布式并无用处，只作为课堂测试环境使用，配置所做笔记。
配置
Hadoop
core-site.xml 填加配置项
<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<!-- 根据实际地址进行配置 -->
<value>hdfs://hadoop104:8020</value>
</property>
<!-- 指定Hadoop运行时产生文件的存储目录 -->
<property>
<name>hadoop.data.dir</name>
<value>/opt/module/hadoop-3.1.3/data</value>
</property>
<!-- 修改访问web端的用户 -->
<property>
<name>hadoop.http.staticuser.user</name>
<value>nevesettle</value>
</property>
hdfs-site.xml 添加配置项
<!--指定存储的副本个数 -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<!-- 指定namenode数据的存储目录 -->
<property>
<name>dfs.namenode.name.dir</name>
<value>file://${hadoop.data.dir}/name</value>
</property>
<!-- 指定datanode数据的存储目录 -->
<property>
<name>dfs.datanode.data.dir</name>
<value>file://${hadoop.data.dir}/data</value>
</property>
<!-- 指定secondarynode数据的存储目录 -->
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file://${hadoop.data.dir}/namesecondary</value>
</property>
<!-- 兼容配置-->
<property>
<name>dfs.client.datanode-restart.timeout</name>
<value>30s</value>
</property>
<!--2nn web端访问地址-->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop104:9868</value>
</property>
<!--nn web端访问地址-->
<property>
<name>dfs.namenode.http-address</name>
<value>hadoop104:9870</value>
</property>
yarn-site.xml 添加配置项
<!-- Reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop104</value>
</property>
<!-- 环境变量的继承 -->
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>
</property>
<!--开启日志聚集功能-->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!--查看日志的路径-->
<property>
<name>yarn.log.server.url</name>
<value>http://hadoop104:19888/jobhistory/logs</value>
</property>
<!--日志存储的时间-->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
<!-- 找不到主类报错 -->
<property>
<name>yarn.application.classpath</name>
<value>/opt/module/hadoop-3.1.3/etc/hadoop:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/*:/opt/module/hadoop-3.1.3/share/hadoop/common/*:/opt/module/hadoop-3.1.3/share/hadoop/hdfs:/opt/module/hadoop-3.1.3/share/hadoop/hdfs/lib/*:/opt/module/hadoop-3.1.3/share/hadoop/hdfs/*:/opt/module/hadoop-3.1.3/share/hadoop/mapreduce/lib/*:/opt/module/hadoop-3.1.3/share/hadoop/mapreduce/*:/opt/module/hadoop-3.1.3/share/hadoop/yarn:/opt/module/hadoop-3.1.3/share/hadoop/yarn/lib/*:/opt/module/hadoop-3.1.3/share/hadoop/yarn/*
</value>
</property>
mapred-site.xml 添加配置项
<!-- 指定MR运行在Yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- 历史服务器端地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop104:10020</value>
</property>
<!-- 历史服务器web端地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop104:19888</value>
</property>
workers (hadoop根目录/etc/hadoop/workers)
hadoop104(改为自己的地址或映射)
zookeeper
zookeeper/conf 下
zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/module/zookeeper-3.5.7/zkData
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
4lw.commands.whitelist=*
server.1=hadoop104:2888:3888
zookeeper根目录下创建目录 zkData（根据上边的配置命名）
该目录下创建文件 myid（名称不可变）
内容为 1 （该节点的id，进行选举时使用，虽然只有一台，但是也要写）
hbase
hbase-env.sh 修改与上篇博客相同，不再累述
hbase-site.xml 添加配置项
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoop104:8020/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hadoop104:2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/opt/module/zookeeper-3.5.7/zkData</value>
</property>
<property>
<name>hbase.unsafe.stream.capability.enforce</name>
<value>false</value>
</property>
regionservers 添加
hadoop104
编写启动、停止脚本
依次开启、停止比较麻烦，写好了脚本供大家使用。
#!/bin/bash
if [ $# -lt 1 ]
then
echo '输入参数有误'
exit
fi
case $1 in
start)
echo '========== start hdfs =========='
/opt/module/hadoop-3.1.3/sbin/start-dfs.sh
echo '========== start yarn =========='
/opt/module/hadoop-3.1.3/sbin/start-yarn.sh
# echo '========== start history =========='
# /opt/module/hadoop-3.1.3/bin/mapred --daemon start historyserver
echo '========== start zookeeper ============'
/opt/module/zookeeper-3.5.7/bin/zkServer.sh start
echo '========== start hbase ============'
/opt/module/hbase-2.2.2/bin/start-hbase.sh
;;
stop)
echo '========== stop hbase ============'
/opt/module/hbase-2.2.2/bin/stop-hbase.sh
echo '========== stop zookeeper ============'
/opt/module/zookeeper-3.5.7/bin/zkServer.sh stop
echo '========== stop yarn =========='
/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh
echo '========== stop hdfs =========='
/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh
;;
esac
总结
以上就是所有的配置项了，配置完成即可使用idea进行API操作了。