Hadoop-2.7.3-本地模式安装-wordcount例子
阅读原文时间:2023年07月16日阅读:1
  1. 准备虚拟机:linux-rhel-7.4-server,由于不使用虚拟机进行联网,所以选择host-only网络模式。此处,需要再VitralBox的管理菜单中的主机网络管理器新建一个虚拟网卡。安装完成虚拟机之后,默认网卡是关闭的,需要进行开启,指令如下:

    [root@hadoop-01 network-scripts]# vi ifcfg-enp0s3 #默认网卡配置

    TYPE=Ethernet
    PROXY_METHOD=none
    BROWSER_ONLY=no
    BOOTPROTO=dhcp
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=no
    IPV6INIT=yes
    IPV6_AUTOCONF=yes
    IPV6_DEFROUTE=yes
    IPV6_FAILURE_FATAL=no
    IPV6_ADDR_GEN_MODE=stable-privacy
    NAME=enp0s3
    UUID=9e448496-ecd5-4122-a91f-91f91bd15f5e
    DEVICE=enp0s3
    ONBOOT=yes #修改为 yes,默认是no然后重启虚拟机

  2. 此时再来查看本机网络配置如下,就可以与宿主机的同网段的虚拟网卡进行通讯了,如果宿主机启用的网络共享,那么可以让虚拟机进行联网

    [root@hadoop-01 network-scripts]# ifconfig
    enp0s3: flags=4163 mtu 1500 #已经有IP分配进来
    inet 192.168.56.101 netmask 255.255.255.0 broadcast 192.168.56.255
    inet6 fe80::bcf9:1d0d:e75d:500f prefixlen 64 scopeid 0x20
    ether 08:00:27:fb:11:51 txqueuelen 1000 (Ethernet)
    RX packets 5763894 bytes 8204104505 (7.6 GiB)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 310622 bytes 23522131 (22.4 MiB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

    lo: flags=73 mtu 65536
    inet 127.0.0.1 netmask 255.0.0.0
    inet6 ::1 prefixlen 128 scopeid 0x10
    loop txqueuelen 1 (Local Loopback)
    RX packets 1698 bytes 134024 (130.8 KiB)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 1698 bytes 134024 (130.8 KiB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

  3. 上传准备好的程序包

    ZBMAC-C03VQ091H:实验介质 hadoop$ ls
    ZooInspector.zip mysql-5.7.19-1.el7.x86_64.rpm-bundle.tar
    ZooViewer.zip mysql-connector-java-5.1.43-bin.jar
    apache-flume-1.7.0-bin.tar.gz pig-0.17.0.tar.gz
    apache-hive-2.3.0-bin.tar.gz sqoop-1.4.5.bin__hadoop-0.23.tar.gz
    hadoop-2.7.3.tar.gz virtualbox
    hbase-1.3.1-bin.tar.gz winscp513setup.exe
    hue-4.0.1.tgz zookeeper-3.4.10.tar.gz
    jdk-8u144-linux-x64.tar.gz

    #使用SCP命令:
    scp ./* hadoop-01@192.168.56.101:/home/hadoop-01/

  4. 安装JDK1.8并解压:

    tar -zxvf jdk-8u144-linux-x64.tar.gz

  5. 设置当前用户的java_home 编辑~/.bash_profile

    JAVA_HOME=/home/hadoop-02/sdk-home/jdk1.8.0_144
    export JAVA_HOME

    PATH=$JAVA_HOME/bin:$PATH
    export PATH

  6. java环境变量设置成功后,使用 :java -version查看版本是否正确

  7. 解压hadoop

    tar -zxvf hadoop-2.7.3.tar.gz

  8. hadoop目录结构解释:

    [hadoop-02@hadoop-02 ~]$ tree -L 3 /home/hadoop-02/sdk-home/hadoop-2.7.3/
    /home/hadoop-02/sdk-home/hadoop-2.7.3/
    |-- bin # 可执行命令
    | |-- container-executor
    | |-- hadoop| |-- yarn
    | `-- yarn.cmd
    |-- etc
    | `-- hadoop # 配置文件目录
    | |-- capacity-scheduler.xml
    | |-- configuration.xsl
    | | |-- yarn-env.sh
    | `-- yarn-site.xml
    |-- include
    | |-- hdfs.h
    | | `-- TemplateFactory.hh
    |-- lib
    | `-- native
    | |-- libhadoop.a
    | |-- libhadooppipes.a
    |
    | `-- libhdfs.so.0.0.0
    |-- libexec
    | |-- hadoop-config.cmd
    | |-- hadoop-config.sh
    | |-- LICENSE.txt
    |-- logs
    | |-- hadoop-hadoop-02-datanode-hadoop-02.log
    | |-- hadoop-hadoop-02-datanode-hadoop-02.out
    | |-- |-- NOTICE.txt
    |-- README.txt
    |-- sbin # 启停脚本
    | |-- distribute-exclude.sh
    | |-- hadoop-daemon.sh| `-- yarn-daemons.sh
    `-- share
    |-- doc #文档目录
    | `-- hadoop
    `-- hadoop #所有jar包
    |-- common
    |-- hdfs
    |-- httpfs
    |-- kms
    |-- mapreduce #内含示例jar包
    |-- tools
    `-- yarn

  9. 设置hadoop的环境变量:

    # /hadoop-2.7.3/etc/hadoop/hadoop-env.sh

    #修改JAVA_HOME为实际对应目录:

    The java implementation to use.

    export JAVA_HOME=/home/hadoop-02/sdk-home/jdk1.8.0_144/

  10. 至此本机环境已经准备好找到hadoop的sbin目录执行start-all.sh

    [hadoop-02@hadoop-02 sbin]$ ./start-all.sh
    This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
    Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
    Starting namenodes on []
    hadoop-02@localhost's password:
    localhost: starting namenode, logging to /home/hadoop-02/sdk-home/hadoop-2.7.3/logs/hadoop-hadoop-02-namenode-hadoop-02.out
    hadoop-02@localhost's password:
    localhost: starting datanode, logging to /home/hadoop-02/sdk-home/hadoop-2.7.3/logs/hadoop-hadoop-02-datanode-hadoop-02.out
    Starting secondary namenodes [0.0.0.0]
    hadoop-02@0.0.0.0's password:
    0.0.0.0: starting secondarynamenode, logging to /home/hadoop-02/sdk-home/hadoop-2.7.3/logs/hadoop-hadoop-02-secondarynamenode-hadoop-02.out
    0.0.0.0: Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
    0.0.0.0: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:472)
    0.0.0.0: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:462)
    0.0.0.0: at org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress(NameNode.java:455)
    0.0.0.0: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:229)
    0.0.0.0: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.(SecondaryNameNode.java:192)
    0.0.0.0: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:671)
    starting yarn daemons
    starting resourcemanager, logging to /home/hadoop-02/sdk-home/hadoop-2.7.3/logs/yarn-hadoop-02-resourcemanager-hadoop-02.out
    hadoop-02@localhost's password:
    localhost: starting nodemanager, logging to /home/hadoop-02/sdk-home/hadoop-2.7.3/logs/yarn-hadoop-02-nodemanager-hadoop-02.out
    [hadoop-02@hadoop-02 sbin]$

  11. 中间如果没有配置免密登录,会出现四次输入密码,注意观察日志在启动对应的服务。

  12. 检查服务是否正常,包含如下服务:

    [hadoop-02@hadoop-02 sbin]$ jps
    6305 Jps
    6178 NodeManager
    5883 ResourceManager
    [hadoop-02@hadoop-02 sbin]$

  13. 运行wordcount 示例:

    [hadoop-02@hadoop-02 sbin]$ hadoop jar /home/hadoop-02/sdk-home/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /home/hadoop-02/test_hadoop/wordcount.txt /home/hadoop-02/test_hadoop/wordcount_output/

  14. 到输出目录就能看到结果文件如下:

    [hadoop-02@hadoop-02 sbin]$ cd /home/hadoop-02/test_hadoop/wordcount_output
    [hadoop-02@hadoop-02 wordcount_output]$ ls
    _SUCCESS part-r-00000

  15. 至此本地环境搭建就介绍到这里