hadoop 完全分布式环境搭建

1、虚拟机角色分配：

192.168.44.184 hadoop02 NameNode/DataNode ResourceManager/NodeManager
192.168.44.185 hadoop03 DataNode NodeManager
192.168.44.186 hadoop04 DataNode NodeManager

每台机器配置好hostname,hosts：/etc/hostname 中配置好hostname

/etc/hosts 中配置好hosts：

192.168.44.184 hadoop02
192.168.44.185 hadoop03
192.168.44.186 hadoop04

2、sskey免登录设置

1）ssh-keygen -t rsa （会在.ssh目录生成公钥和私钥,每台机器都要生成）

2）以hadoop02为主，将公钥copy到各个机器

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop02
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop03
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop04

3、jdk的安装(在hadoop02上配置，另外两台机器稍后复制过去)

1） 直接下载jdk包放到安装目录即可（jdk-8u131-linux-x64.tar.gz解压到安装目录  tar zxvf ./jdk-8u131-linux-x64.tar.gz -C /home/hadoop/app/）

2） 配置环境变量 直接在 .bash_profile配置

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131
export PATH=$JAVA_HOME/bin:$PATH

4.下载安装hadoop-2.6.0-cdh5.7.0(在hadoop02上配置，另外两台机器稍后复制过去)

cd__h下载地址：（http://archive.cloudera.com/cdh5/cdh/5/)

1）直接下载hadoop-2.6.0-cdh5.7.0.tar.gz包放到安装目录即可

2）配置环境变量

export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$PATH

5、配置修改hadoop相关配置文件(在hadoop02上配置，另外两台机器稍后复制过去)

1）hadoop_home/etc/hadoop/hadoop-env.sh 配置jdk

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131

2）etc/hadoop/core-site.xml

fs.defaultFS hdfs://hadoop02:8020

    <property>  
        <name>hadoop.tmp.dir</name>  
        <value>/home/hadoop/app/tmp</value>  
    </property>

3）etc/hadoop/hdfs-site.xml  设置副本系数和name、data的存放路径

dfs.replication 3
dfs.namenode.name.dir /home/hadoop/app/tmp/dfs/name
dfs.datanode.data.dir /home/hadoop/app/tmp/dfs/data

4）etc/hadoop/yarn-site.xml

yarn.nodemanager.aux-services mapreduce_shuffle
yarn.resourcemanager.hostname hadoop02

5）etc/hadoop/mapred-site.xml

mapreduce.framework.name yarn

6）/etc/hadoop/slaves(标明从节点)

hadoop02
hadoop03
hadoop04

6、分发安装包到 hadoop03、hadoop03节点上

scp -r ~/app hadoop@hadoop03:~/
scp -r ~/app hadoop@hadoop04:~/
scp ~/.bash_profile hadoop@hadoop03:~/
scp ~/.bash_profile hadoop@hadoop04:~/

在hadoop03,hadoop04上分别source下.bash_profile文件

7、NameNode格式化：只要在hadoop02上格式化

bin/hdfs namenode -format

8、启动hadoop集群。在NameNode启动

sbin/start-all.sh

启动日志：

[hadoop@hadoop02 hadoop-2.6.0-cdh5.7.0]$ ./sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
18/08/19 23:20:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Starting namenodes on [hadoop02]
hadoop02: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop02.out
hadoop02: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop02.out
hadoop04: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop04.out
hadoop03: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop03.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 78:05:94:d6:0b:e2:78:7d:93:3c:90:d0:ab:7c:ed:e2.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop02.out
18/08/19 23:21:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop02.out
hadoop03: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop03.out
hadoop04: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop04.out
hadoop02: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop02.out

9、验证

1）jps验证：

[hadoop@hadoop02 hadoop-2.6.0-cdh5.7.0]$ jps
3862 SecondaryNameNode
4086 NodeManager
3591 NameNode
3991 ResourceManager
3678 DataNode
4399 Jps

[hadoop@hadoop03 ~]$ jps
3460 NodeManager
3576 Jps
3372 DataNode

[hadoop@hadoop04 ~]$ jps
3284 NodeManager
3400 Jps
3196 DataNode

2）web页面验证：http://hadoop02:50070 （hdfs）

http://hadoop02:8088 （yarn）

10、停止

sbin/stop-all.sh