CentOS6.5如何实现Hadoop_2.8.1编译及HDFS伪分布式部署

这篇文章主要介绍了Centos6.5如何实现hadoop_2.8.1编译及hdfs伪分布式部署，具有一定借鉴价值，感兴趣的朋友可以参考下，希望大家阅读完这篇文章之后大有收获，下面让小编带着大家一起了解一下。

CentOS6.5实现Hadoop编译及HDFS伪分布式部署

Hadoop 2.x  三大组件
mapReduce(Others)      -- 计算
YARN                     -- 资源和作业调度平台
HDFS                     -- 存储

环境准备：
[root@hadoop001 software]# pwd
/opt/software
-rw-r--r--.  1 root root   8617253 May 14 07:05 apache-Maven-3.3.9-bin.zip
-rw-r--r--.  1 root root   7546219 May 14 07:05 findbugs-1.3.9.zip
-rw-r--r--.  1 root root  34523353 May 14 07:05 hadoop-2.8.1-src.tar.gz
-rw-r--r--.  1 root root 424555111 May 14 07:09 hadoop-2.8.1.tar.gz
-rw-r--r--.  1 root root 173271626 May 14 07:09 jdk-8u45-linux-x64.gz
-rw-r--r--.  1 root root  96721446 May 14 07:07 .m2.tar.gz
-rw-r--r--.  1 root root   2401901 May 14 07:04 protobuf-2.5.0.tar.gz



1.Hadoop源代码下载
[root@hadoop001 software]# tar -xzvf hadoop-2.8.1-src.tar.gz
# 检查源码包环境
[root@hadoop001 hadoop-2.8.1-src]# cat BUILDING.txt

----------------------------------------------------------------------------------
Requirements:
* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption perfORMance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)


2.JAVA安装
[root@hadoop001 ~]# mkdir -p /usr/java
[root@hadoop001 ~]# mv jdk-8u45-linux-x64.gz /usr/java
[root@hadoop001 ~]# cd /usr/java
[root@hadoop001 ~]# tar -xzvf jdk-8u45-linux-x64.gz

# 修改用户和用户组
[root@hadoop001 java]# ll
total 169388
drwxr-xr-x. 8 uucp  143      4096 Apr 10  2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 May 14 07:09 jdk-8u45-linux-x64.gz
[root@hadoop002 java]# chown -R root:root jdk1.8.0_45

[root@hadoop002 java]# ll
total 169388
drwxr-xr-x. 8 root root      4096 Apr 11  2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz

# JAVA全局变量配置
[root@hadoop001 java]# vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
注：不用卸载原有JDK,环境变量可以覆盖原来JDK

[root@hadoop001 java]# source /etc/profile
[root@hadoop001 java]# which java
/usr/java/jdk1.8.0_45/bin/java
[root@hadoop001 java]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

3.Maven安装与部署
[root@hadoop001 software]# ll
total 466948
-rw-r--r--.  1 root root   8617253 May 14 07:05 apache-maven-3.3.9-bin.zip

[root@hadoop001 software]# unzip apache-maven-3.3.9-bin.zip
[root@hadoop001 software]# ll
total 466952
drwxr-xr-x.  6 root root      4096 Nov 10  2015 apache-maven-3.3.9
-rw-r--r--.  1 root root   8617253 May 14 07:05 apache-maven-3.3.9-bin.zip

#Maven全局环境变量
[root@hadoop001 java]# vi /etc/profile
export MAVEN_HOME=/opt/software/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 software]# source /etc/profile

[root@hadoop001 software]# which mvn
/opt/software/apache-maven-3.3.9/bin/mvn

[root@hadoop001 software]# mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323Dc5; 2015-11-10T11:41:47-05:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"

#Maven仓库文件手动导入
[root@hadoop001 software]# mv .m2.tar.gz ~
[root@hadoop001 software]# cd ~
[root@hadoop001 ~]# tar -xzvf .m2.tar.gz


# Maven仓库默认路径和修改
[root@hadoop001 conf]# pwd
/opt/software/apache-maven-3.3.9/conf/settings.xml

  <!-- localRepository
   | The path to the local repository maven will use to store artifacts.
   |
   | Default: ${user.home}/.m2/repository
  /path/to/local/repo
  -->
-- pro.xml以maven仓库的项目，mvn 编译、打包、测试


4.protobuf源码安装
[root@hadoop001 software]# pwd
/opt/software
[root@hadoop001 software]# tar -xzvf protobuf-2.5.0.tar.gz


[root@hadoop001 software]# ll
total 466956
drwxr-xr-x. 10 109965 5000      4096 Feb 26  2013 protobuf-2.5.0
-rw-r--r--.  1 root   root   2401901 May 14 07:04 protobuf-2.5.0.tar.gz
[root@hadoop001 software]# chown -R root:root protobuf-2.5.0

[root@hadoop001 software]# cd protobuf-2.5.0
[root@hadoop001 protobuf-2.5.0]# yum install -y GCc gcc-c++ make cmake
[root@hadoop001 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
[root@hadoop001 protobuf-2.5.0]# make && make install

# protobuf全局环境变量
[root@hadoop001 java]# vi /etc/profile
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 protobuf-2.5.0]# source /etc/profile

[root@hadoop001 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
[root@hadoop001 protobuf-2.5.0]# 


5.Findbugs安装
[root@hadoop001 software]# pwd
/opt/software
[root@hadoop001 software]# unzip findbugs-1.3.9.zip
[root@hadoop001 software]# ll
total 466960
drwxr-xr-x.  7 root   root      4096 Aug 21  2009 findbugs-1.3.9
-rw-r--r--.  1 root   root   7546219 May 14 07:05 findbugs-1.3.9.zip

#Findbugs全局环境变量
[root@hadoop002 software]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 software]# 
[root@hadoop001 software]# source /etc/profile
[root@hadoop001 software]# findbugs -version
1.3.9

6.其他依赖
yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

7.Hadoop源代码编译
[root@hadoop001 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar
[INFO] Apache Hadoop Extras ............................... SUCCESS [  3.050 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  7.795 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  5.657 s]
[INFO] Apache Hadoop Amazon WEB Services support .......... SUCCESS [  6.914 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  5.264 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  7.341 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.698 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  6.050 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  6.549 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  3.666 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.016 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.066 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:39 min
[INFO] Finished at: 2018-05-14T22:30:38-04:00
[INFO] Final Memory: 190M/454M
[INFO] ------------------------------------------------------------------------




----------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------
hodoop部署
单机               --无进程
伪分布式部署   -- 进程存在+1节点   开发
集群部署         -- 进程存在+n阶段   开发/生产

[root@hadoop001 software]# tar -xzvf hadoop-2.8.1.tar.gz
[root@hadoop001 software]# chown -R root:root hadoop-2.8.1

[root@hadoop001 hadoop-2.8.1]# ll
drwxrwxr-x. 2 root root  4096 Jun  2  2017 bin                -- 执行命令shell
drwxrwxr-x. 3 root root  4096 Jun  2  2017 etc                -- 配置文件                 
drwxrwxr-x. 2 root root  4096 Jun  2  2017 include
drwxrwxr-x. 3 root root  4096 Jun  2  2017 lib                  -- 库
drwxrwxr-x. 2 root root  4096 Jun  2  2017 libexec
-rw-rw-r--. 1 root root 99253 Jun  2  2017 LICENSE.txt
-rw-rw-r--. 1 root root 15915 Jun  2  2017 NOTICE.txt
-rw-r--r--. 1 root root  1366 Jun  2  2017 README.txt
drwxrwxr-x. 2 root root  4096 Jun  2  2017 sbin                -- 启动和关闭hadoop                 
drwxrwxr-x. 4 root root  4096 Jun  2  2017 share              --  jar

#hadoop全局环境变量
[root@hadoop001 hadoop-2.8.1]# vim /etc/profile
export HADOOP_HOME=/opt/software/hadoop-2.8.1
export PATH=$HADOOP_HOME/bin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop001 hadoop-2.8.1]# source /etc/profile
[root@hadoop001 hadoop-2.8.1]# which hadoop
/opt/software/hadoop-2.8.1/bin/hadoop

[root@hadoop001 bin]# rm -rf *.cmd
[root@hadoop001 sbin]# rm -rf *.cmd

#配置core-site文件
[root@hadoop001 hadoop]# vim core-site.xml

点击(此处)折叠或打开

1. <configuration>
   

2.     <property>
   

3.         <name>fs.defaultFS</name>
   

4.         <value>hdfs://localhost:9000</value>    --HDFD端口和路径
   

5.     </property>
   

6. </configuration>
   

9. <configuration>
   

10.     <property>
    

11.         <name>dfs.replication</name>
    

12.         <value>1</value>             -- 集群主机数量
    

13.     </property>
    

14. </configuration>
    

15. <configuration>

#本机ssh验证配置
[root@hadoop001 hadoop]# cd ~
[root@hadoop001 ~]# cd .ssh
[root@hadoop001 .ssh]# ll
total 0
[root@hadoop001 ~]# rm -rf .ssh
[root@hadoop001 ~]# ssh-keygen
[root@hadoop001 ~]# cd .ssh
[root@hadoop001 .ssh]# ll
total 8
-rw-------. 1 root root 1675 May 14 23:38 id_rsa
-rw-r--r--. 1 root root  396 May 14 23:38 id_rsa.pub
[root@hadoop001 .ssh]# cat id_rsa.pub >> authorized_keys
[root@hadoop001 .ssh]# ll
total 12
-rw-r--r--. 1 root root  396 May 14 23:40 authorized_keys
-rw-------. 1 root root 1675 May 14 23:38 id_rsa
-rw-r--r--. 1 root root  396 May 14 23:38 id_rsa.pub

#第一次必须操作，本机修改过端口
[root@hadoop001 ~]# ssh localhost date
ssh: connect to host localhost port 22: Connection refused
[root@hadoop001 ~]# ssh -p2222 localhost date
The authenticity of host '[localhost]:2222 ([::1]:2222)' can't be established.
RSA key fingerprint is 09:b9:67:65:cb:e6:ca:31:5d:33:6c:3b:92:9e:c2:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:2222' (RSA) to the list of known hosts.
Mon May 14 23:42:37 EDT 2018

#格式化HDFS文件系统
[root@hadoop001 ~]# which hdfs
/opt/software/hadoop-2.8.1/bin/hdfs
[root@hadoop001 ~]# hdfs namenode -format

#启动HDFS文件系统
[root@hadoop001 sbin]# /opt/software/hadoop-2.8.1/sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
Starting secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused

 #解决port 22错误

[root@hadoop000 hadoop]# vi  hadoop-env.sh

添加：export HADOOP_SSH_OPTS="-p 2222"

 #解决 ERROR:JAVA_HOME
[root@hadoop000 hadoop]# vi  hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_45

#启动OK
[root@hadoop001 sbin]# /opt/software/hadoop-2.8.1/sbin/start-dfs.sh
[root@hadoop001 hadoop]# /opt/software/hadoop-2.8.1/sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-namenode-hadoop001.out
localhost: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-datanode-hadoop001.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-secondarynamenode-hadoop001.out

#检测启动
[root@hadoop000 hadoop-2.8.1]# jps
16243 Jps
15943 DataNode
5127 Launcher
16139 SecondaryNameNode
15853 NameNode

感谢你能够认真阅读完这篇文章，希望小编分享的“CentOS6.5如何实现Hadoop_2.8.1编译及HDFS伪分布式部署”这篇文章对大家有帮助，同时也希望大家多多支持编程网，关注编程网精选频道，更多相关知识等着你来学习!