澳门皇冠金沙网站-澳门皇冠844网站

热门关键词: 澳门皇冠金沙网站,澳门皇冠844网站

【皇冠体育投注】6安装部署

1. 主机规划

主机名称

外网IP

内网IP

操作系统

备注

安装软件

mini01

10.0.0.11

172.16.1.11

CentOS 7.4

ssh port:22

Hadoop 【NameNode  SecondaryNameNode】

mini02

10.0.0.12

172.16.1.12

CentOS 7.4

ssh port:22

Hadoop 【ResourceManager】

mini03

10.0.0.13

172.16.1.13

CentOS 7.4

ssh port:22

Hadoop 【DataNode  NodeManager】

mini04

10.0.0.14

172.16.1.14

CentOS 7.4

ssh port:22

Hadoop 【DataNode  NodeManager】

mini05

10.0.0.15

172.16.1.15

CentOS 7.4

ssh port:22

Hadoop 【DataNode  NodeManager】

皇冠体育投注 1

加多hosts音讯,有限支撑每台都得以并行ping通

[root@mini01 ~]# cat /etc/hosts 
127.0.0.1  localhost localhost.localdomain localhost4 localhost4.localdomain4
::1        localhost localhost.localdomain localhost6 localhost6.localdomain6
 
10.0.0.11    mini01
10.0.0.12    mini02
10.0.0.13    mini03
10.0.0.14    mini04
10.0.0.15    mini05

  1. 累加用户账号

# 使用三个特意的用户,幸免直接利用root用户
# 加多用户、内定家目录并钦命用户密码
useradd -d /app yun && echo '123456' | /usr/bin/passwd --stdin yun
# sudo提权
echo "yun  ALL=(ALL)      NOPASSWD: ALL" >>  /etc/sudoers
# 让其他普通用户能够步向该目录查看新闻
chmod 755 /app/

  1. 福寿无疆yun用户免秘钥登入

务求:依照设计落到实处 mini01 到 mini01、mini02、mini03、mini04、mini05 免秘钥登陆
              实现 mini02 到 mini01、mini02、mini03、mini04、mini05 免秘钥登入
# 能够动用ip也得以是hostname  可是出于大家布置采纳的是 hostname 格局互相,所以使用hostname
# 同临时候hostname格局分发,能够因此hostname远程登陆,也能够IP远程登入

3.1. 创办密钥

# 完成 mini01 到 mini02、mini03、mini04、mini05 免秘钥登陆
[yun@mini01 ~]$ ssh-keygen -t rsa  # 一路回车就能够
Generating public/private rsa key pair.
Enter file in which to save the key (/app/.ssh/id_rsa):
Created directory '/app/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /app/.ssh/id_rsa.
Your public key has been saved in /app/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rAFSIyG6Ft6qgGdVl/7v79DJmD7kIDSTcbiLtdKyTQk yun@mini01
The key's randomart image is:
---[RSA 2048]----
|. o.o    .      |
|.. o .  o..      |
|... . . o=      |
|..o. oE B        |
|.o .. .*S*      |
|o ..  oB.. .= . |
|o.o  .* ..   |
|oo    . .  oo.  |
|.          . o  |
----[SHA256]-----
 
# 生成之后会在用户的根目录生成三个 “.ssh”的文本夹
[yun@mini01 ~]$ ll -d .ssh/
drwx------ 2 yun yun 38 Jun  9 19:17 .ssh/
[yun@mini01 ~]$ ll .ssh/
total 8
-rw------- 1 yun yun 1679 Jun  9 19:17 id_rsa
-rw-r--r-- 1 yun yun  392 Jun  9 19:17 id_rsa.pub

3.2. 散发密钥

# 能够利用ip也足以是hostname  不过出于大家选择的是 hostname 情势通讯,所以使用hostname
[yun@mini01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub 172.16.1.11  # IP格局【这里而不是】
# 分发
[yun@mini01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini03  # 主机名形式【全部的都那样 从mini01到mini05】
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/app/.ssh/id_rsa.pub"
The authenticity of host '[mini03]:22 ([10.0.0.13]:22)' can't be established.
ECDSA key fingerprint is SHA256:pN2NUkgCTt b9P5TfQZcTh4PF4h7iUxAs6 V7Slp1YI.
ECDSA key fingerprint is MD5:8c:f0:c7:d6:7c:b1:a8:59:1c:c1:5e:d7:52:cb:5f:51.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
yun@mini03's password:
 
Number of key(s) added: 1
 
Now try logging into the machine, with:  "ssh -p '22' 'mini03'"
and check to make sure that only the key(s) you wanted were added.

mini01分发密钥

[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini01
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini02
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini03
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini04
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini05

mini02分发密钥

[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini01
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini02
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini03
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini04
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini05

长距离登陆测验【最好都测量检验一下】

 [yun@mini02 ~]$ ssh mini05
Last login: Sat Jun  9 19:47:43 2018 from 10.0.0.11
 
Welcome You Login
 
[yun@mini05 ~]$            # 表示远程登入成功

1. Hadoop的HA机制

  前言:正式引进HA机制是从hadoop2.0起头,以前的本子中绝非HA机制

创设三台虚构机,IP地址为:192.168.169.101,192.168.169.102,192.168.169.103

3.3. 长距离免密登入原理图

皇冠体育投注 2

 

1.1. HA的运维机制

(1)hadoop-HA集群运作机制介绍

  所谓HA,即高可用(7*24钟头不暂停服务)

  完成高可用最关键的是扫除单点故障

  hadoop-ha严刻来讲应该分为种种零部件的HA机制——HDFS的HA、YA汉兰达N的HA

(2)HDFS的HA机制详解

  通过双namenode解决单点故障

  双namenode和煦工作的要点:

     A、元数据管理章程须要更改:

        内部存款和储蓄器中各自小编保护存一份元数据

        Edits日志只能有一份,只有Active状态的namenode节点可以做写操作

        七个namenode都能够读取edits

        分享的edits放在三个分享存储中管理(qjournal和NFS两个主流完毕)

      B、须要三个气象管理功用模块

        达成了二个zkfailover,常驻在每四个namenode所在的节点

        每叁个zkfailover负担监察和控制本人所在namenode节点,利用zk实行状态标志

        当供给开始展览情状切换时,由zkfailover来肩负切换

        切换时索要防守brain split现象的发出

1.2. HDFS-HA图解

皇冠体育投注 3

 

将192.168.169.102为namenode,192.168.169.101,192.168.169.103为datanode

3.4. .ssh目录中的文件注解

 [yun@mini01 .ssh]$ pwd
/app/.ssh
[yun@mini01 .ssh]$ ll
total 16
-rw------- 1 yun yun  784 Jun  9 19:43 authorized_keys
-rw------- 1 yun yun 1679 Jun  9 19:17 id_rsa
-rw-r--r-- 1 yun yun  392 Jun  9 19:17 id_rsa.pub
-rw-r--r-- 1 yun yun 1332 Jun  9 19:41 known_hosts
########################################################################################
  authorized_keys:存放远程免密登陆的公钥,主要透过那么些文件记录多台机器的公钥
  id_rsa : 生成的私钥文件
  id_rsa.pub : 生成的公钥文件
know_hosts : 已知的主机公钥清单

2. 主机规划

主机名称

外网IP

内网IP

操作系统

备注

安装软件

运行进程

mini01

10.0.0.111

172.16.1.111

CentOS 7.4

ssh port:22

jdk、hadoop

NameNode、DFSZKFailoverController(zkfc)

mini02

10.0.0.112

172.16.1.112

CentOS 7.4

ssh port:22

jdk、hadoop

NameNode、DFSZKFailoverController(zkfc)

mini03

10.0.0.113

172.16.1.113

CentOS 7.4

ssh port:22

jdk、hadoop、zookeeper

ResourceManager

mini04

10.0.0.114

172.16.1.114

CentOS 7.4

ssh port:22

jdk、hadoop、zookeeper

ResourceManager

mini05

10.0.0.115

172.16.1.115

CentOS 7.4

ssh port:22

jdk、hadoop、zookeeper

DataNode、NodeManager、JournalNode、QuorumPeerMain

mini06

10.0.0.116

172.16.1.116

CentOS 7.4

ssh port:22

jdk、hadoop、zookeeper

DataNode、NodeManager、JournalNode、QuorumPeerMain

mini07

10.0.0.117

172.16.1.117

CentOS 7.4

ssh port:22

jdk、hadoop、zookeeper

DataNode、NodeManager、JournalNode、QuorumPeerMain

留神:针对HA情势,就无需SecondaryNameNode了,因为STANDBY状态的namenode会担负做checkpoint

Linux增加hosts消息,保障每台都足以互相ping通 

[root@mini01 ~]# cat /etc/hosts 
127.0.0.1  localhost localhost.localdomain localhost4 localhost4.localdomain4
::1        localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.0.111    mini01
10.0.0.112    mini02
10.0.0.113    mini03
10.0.0.114    mini04
10.0.0.115    mini05
10.0.0.116    mini06
10.0.0.117    mini07

Windows的hosts文件修改

 # 文件地点C:WindowsSystem32driversetc  在hosts中增添如下内容
…………………………………………
10.0.0.111    mini01
10.0.0.112    mini02
10.0.0.113    mini03
10.0.0.114    mini04
10.0.0.115    mini05
10.0.0.116    mini06
10.0.0.117    mini07

  1. 充分用户账号

 # 使用八个专程的用户,制止间接行使root用户
# 加多用户、钦定家目录并内定用户密码
useradd -d /app yun && echo '123456' | /usr/bin/passwd --stdin yun
# sudo提权
echo "yun  ALL=(ALL)      NOPASSWD: ALL" >>  /etc/sudoers
# 让别的普通用户能够进来该目录查看消息
chmod 755 /app/

  1. 贯彻yun用户免秘钥登入

务求:依照安顿达成 mini01 到 mini01、mini02、mini03、mini04、mini05、mini06、mini07 免秘钥登入
              完结 mini02 到 mini01、mini02、mini03、mini04、mini05、mini06、mini07 免秘钥登入
              实现 mini03 到 mini01、mini02、mini03、mini04、mini05、mini06、mini07 免秘钥登入
              完毕 mini04 到 mini01、mini02、mini03、mini04、mini05、mini06、mini07 免秘钥登入
              完结 mini05 到 mini01、mini02、mini03、mini04、mini05、mini06、mini07 免秘钥登入
              达成 mini06 到 mini01、mini02、mini03、mini04、mini05、mini06、mini07 免秘钥登入
              完毕 mini07 到 mini01、mini02、mini03、mini04、mini05、mini06、mini07 免秘钥登入

# 可以接纳ip也得以是hostname  但是由于大家安排使用的是 hostname 格局相互,所以利用hostname
# 同一时间hostname格局分发,能够通过hostname远程登陆,也能够IP远程登录

现实经过就非常少说了,请参见 

  1. Jdk【java8】

具体进度就十分少说了,请参见

  1. Zookeeper部署

依据设计zookeeper安插在mini03、mini04、mini05、mini06、mini07上

6.1. 布署消息

[yun@mini03 conf]$ pwd
/app/zookeeper/conf
[yun@mini03 conf]$ vim zoo.cfg
#单个客户端与单台服务器之间的连接数的限定,是ip级其他,暗中认可是60,假诺设置为0,那么申明不作任何限制。
maxClientCnxns=1500
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
# dataDir=/tmp/zookeeper
dataDir=/app/bigdata/zookeeper/data
# the port at which the clients will connect
clientPort=2181
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

# leader和follow通讯端口和投票大选端口 
server.3=mini03:2888:3888
server.4=mini04:2888:3888
server.5=mini05:2888:3888
server.6=mini06:2888:3888
server.7=mini07:2888:3888

6.2. 添加myid文件

[yun@mini03 data]$ pwd
/app/bigdata/zookeeper/data
[yun@mini03 data]$ vim myid  # 其中mini03的myid 为3;mini04的myid 为4;mini05的myid 为5;mini06的myid 为6;mini07的myid 为7 
3

6.3. 启动zk服务

# 依次在开发银行mini03、mini04、mini05、mini06、mini07 zk服务
[yun@mini03 ~]$ cd zookeeper/bin/
[yun@mini03 bin]$ pwd
/app/zookeeper/bin
[yun@mini03 bin]$ ll
total 56
-rwxr-xr-x 1 yun yun  238 Oct  1  2012 README.txt
-rwxr-xr-x 1 yun yun  1909 Oct  1  2012 zkCleanup.sh
-rwxr-xr-x 1 yun yun  1049 Oct  1  2012 zkCli.cmd
-rwxr-xr-x 1 yun yun  1512 Oct  1  2012 zkCli.sh
-rwxr-xr-x 1 yun yun  1333 Oct  1  2012 zkEnv.cmd
-rwxr-xr-x 1 yun yun  2599 Oct  1  2012 zkEnv.sh
-rwxr-xr-x 1 yun yun  1084 Oct  1  2012 zkServer.cmd
-rwxr-xr-x 1 yun yun  5467 Oct  1  2012 zkServer.sh
-rw-rw-r-- 1 yun yun 17522 Jun 28 21:01 zookeeper.out
[yun@mini03 bin]$ ./zkServer.sh start 
JMX enabled by default
Using config: /app/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

6.4. 询问运营状态

# 其中mini03、mini04、mini06、mini07状态如下
[yun@mini03 bin]$ ./zkServer.sh status
JMX enabled by default
Using config: /app/zookeeper/bin/../conf/zoo.cfg
Mode: follower

# 个中mini05 状态如下
[yun@mini05 bin]$ ./zkServer.sh status
JMX enabled by default
Using config: /app/zookeeper/bin/../conf/zoo.cfg
Mode: leader

PS:4个follower 1个leader

  1. Hadoop安插与配置修改

  注意:每台机器的Hadoop以及配置同样

7.1. 部署

[yun@mini01 software]$ pwd
/app/software
[yun@mini01 software]$ ll
total 194152
-rw-r--r-- 1 yun yun 198811365 Jun  8 16:36 CentOS-7.4_hadoop-2.7.6.tar.gz
[yun@mini01 software]$ tar xf CentOS-7.4_hadoop-2.7.6.tar.gz
[yun@mini01 software]$ mv hadoop-2.7.6/ /app/
[yun@mini01 software]$ cd
[yun@mini01 ~]$ ln -s hadoop-2.7.6/ hadoop
[yun@mini01 ~]$ ll
total 4
lrwxrwxrwx  1 yun yun  13 Jun  9 16:21 hadoop -> hadoop-2.7.6/
drwxr-xr-x  9 yun yun  149 Jun  8 16:36 hadoop-2.7.6
lrwxrwxrwx  1 yun yun  12 May 26 11:18 jdk -> jdk1.8.0_112
drwxr-xr-x  8 yun yun  255 Sep 23  2016 jdk1.8.0_112

7.2. 情状变量

[root@mini01 profile.d]# pwd
/etc/profile.d
[root@mini01 profile.d]# vim hadoop.sh
export HADOOP_HOME="/app/hadoop"
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
 
[root@mini01 profile.d]# source /etc/profile  # 生效

7.3. core-site.xml

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
……………………
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <!-- 指定hdfs的nameservice为bi -->
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://bi/</value>
  </property>

  <!-- 钦命hadoop偶尔目录 -->
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/app/hadoop/tmp</value>
  </property>

  <!-- 指定zookeeper地址 -->
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>mini03:2181,mini04:2181,mini05:2181,mini06:2181,mini07:2181</value>
  </property>

</configuration>

7.4. hdfs-site.xml

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim hdfs-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  ……………………
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <!--钦点hdfs的nameservice为bi,需求和core-site.xml中的保持一致 -->
  <property>
    <name>dfs.nameservices</name>
    <value>bi</value>
  </property>

  <!-- bi上面有三个NameNode,分别是nn1,nn2 -->
  <property>
    <name>dfs.ha.namenodes.bi</name>
    <value>nn1,nn2</value>
  </property>

  <!-- nn1的RPC通讯地方 -->
  <property>
    <name>dfs.namenode.rpc-address.bi.nn1</name>
    <value>mini01:9000</value>
  </property>
  <!-- nn1的http通讯地方 -->
  <property>
    <name>dfs.namenode.http-address.bi.nn1</name>
    <value>mini01:50070</value>
  </property>

  <!-- nn2的RPC通讯地方 -->
  <property>
    <name>dfs.namenode.rpc-address.bi.nn2</name>
    <value>mini02:9000</value>
  </property>
  <!-- nn2的http通讯地方 -->
  <property>
    <name>dfs.namenode.http-address.bi.nn2</name>
    <value>mini02:50070</value>
  </property>

  <!-- 钦点NameNode的edits元数据在JournalNode上的贮存地方 -->
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://mini05:8485;mini06:8485;mini07:8485/bi</value>
  </property>

  <!-- 钦赐JournalNode在地方磁盘寄放数据的任务 -->
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/app/hadoop/journaldata</value>
  </property>

  <!-- 开启NameNode败北自动切换 -->
  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>

  <!-- 配置退步自动切换完成方式 -->
  <property>
    <name>dfs.client.failover.proxy.provider.bi</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>

  <!-- 配置隔绝机制措施,多少个机制用换行分割,即各类建制暂用一行-->
  <!-- 在那之中shell(/bin/true) 表示可进行叁个剧本  例如shell(/app/yunwei/hadoop_fence.sh) -->
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>
      sshfence
      shell(/bin/true)
    </value>
  </property>

  <!-- 使用sshfence隔绝机制时索要ssh免登录 -->
  <property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/app/.ssh/id_rsa</value>
  </property>

  <!-- 配置sshfence隔断机制超时时间 单位:阿秒 -->
  <property>
    <name>dfs.ha.fencing.ssh.connect-timeout</name>
    <value>30000</value>
  </property>

</configuration>

7.5. mapred-site.xml

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ cp -a mapred-site.xml.template mapred-site.xml
[yun@mini01 hadoop]$ vim mapred-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  ……………………
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <!-- 指定mr框架为yarn方式 -->
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>

</configuration>

7.6. yarn-site.xml

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim yarn-site.xml
<?xml version="1.0"?>
<!--
  ……………………
-->
<configuration>

<!-- Site specific YARN configuration properties -->
  <!-- 开启RM高可用 -->
  <property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
  </property>

  <!-- 指定RM的cluster id -->
  <property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>yrc</value>
  </property>

  <!-- 指定RM的名字 -->
  <property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
  </property>

  <!-- 分别钦定RM的地点 -->
  <property>
    <name>yarn.resourcemanager.hostname.rm1</name>
    <value>mini03</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.rm2</name>
    <value>mini04</value>
  </property>

  <!-- 钦点zk集群地址 -->
  <property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>mini03:2181,mini04:2181,mini05:2181,mini06:2181,mini07:2181</value>
  </property>

  <!-- reduce 获取数据的办法 -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

</configuration>

7.7. 修改slaves

      slaves是钦点子节点的地点,因为要在mini01上运行HDFS、在mini03运营yarn,所以mini01上的slaves文件钦赐的是datanode的职责,mini03上的slaves文件钦点的是nodemanager的职位

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim slaves
mini05
mini06
mini07

PS:改后配备后,将那几个布置拷到别的Hadoop机器

  1. 开发银行有关服务

  注意:第一遍运营时严刻依据上边包车型地铁步骤!!!!!!!

8.1. 启动zookeeper集群

前面早就运维了,这里就不说了

8.2. 启动journalnode

# 依据规划在mini05、mini06、mini07 运行   
# 在率先次格式化的时候必要先运行journalnode  之后就不要了
[yun@mini05 ~]$ hadoop-daemon.sh start journalnode  # 已经陈设景况变量,所以不用进入到响应的目录 
starting journalnode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-journalnode-mini05.out
[yun@mini05 ~]$ jps
1281 QuorumPeerMain
1817 Jps
1759 JournalNode

8.3. 格式化HDFS

# 在mini01上实践命令
[yun@mini01 ~]$ hdfs namenode -format 
18/06/30 18:29:12 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:  host = mini01/10.0.0.111
STARTUP_MSG:  args = [-format]
STARTUP_MSG:  version = 2.7.6
STARTUP_MSG:  classpath = ………………
STARTUP_MSG:  build = Unknown -r Unknown; compiled by 'root' on 2018-06-08T08:30Z
STARTUP_MSG:  java = 1.8.0_112
************************************************************/
18/06/30 18:29:12 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
18/06/30 18:29:12 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-2385f26e-72e6-4935-aa09-47848b5ba4be
18/06/30 18:29:13 INFO namenode.FSNamesystem: No KeyProvider found.
18/06/30 18:29:13 INFO namenode.FSNamesystem: fsLock is fair: true
18/06/30 18:29:13 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
18/06/30 18:29:13 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
18/06/30 18:29:13 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
18/06/30 18:29:13 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
18/06/30 18:29:13 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Jun 30 18:29:13
18/06/30 18:29:13 INFO util.GSet: Computing capacity for map BlocksMap
18/06/30 18:29:13 INFO util.GSet: VM type      = 64-bit
18/06/30 18:29:13 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
18/06/30 18:29:13 INFO util.GSet: capacity      = 2^21 = 2097152 entries
18/06/30 18:29:13 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
18/06/30 18:29:13 INFO blockmanagement.BlockManager: defaultReplication        = 3
18/06/30 18:29:13 INFO blockmanagement.BlockManager: maxReplication            = 512
18/06/30 18:29:13 INFO blockmanagement.BlockManager: minReplication            = 1
18/06/30 18:29:13 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
18/06/30 18:29:13 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
18/06/30 18:29:13 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
18/06/30 18:29:13 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
18/06/30 18:29:13 INFO namenode.FSNamesystem: fsOwner            = yun (auth:SIMPLE)
18/06/30 18:29:13 INFO namenode.FSNamesystem: supergroup          = supergroup
18/06/30 18:29:13 INFO namenode.FSNamesystem: isPermissionEnabled = true
18/06/30 18:29:13 INFO namenode.FSNamesystem: Determined nameservice ID: bi
18/06/30 18:29:13 INFO namenode.FSNamesystem: HA Enabled: true
18/06/30 18:29:13 INFO namenode.FSNamesystem: Append Enabled: true
18/06/30 18:29:13 INFO util.GSet: Computing capacity for map INodeMap
18/06/30 18:29:13 INFO util.GSet: VM type      = 64-bit
18/06/30 18:29:13 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
18/06/30 18:29:13 INFO util.GSet: capacity      = 2^20 = 1048576 entries
18/06/30 18:29:13 INFO namenode.FSDirectory: ACLs enabled? false
18/06/30 18:29:13 INFO namenode.FSDirectory: XAttrs enabled? true
18/06/30 18:29:13 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
18/06/30 18:29:13 INFO namenode.NameNode: Caching file names occuring more than 10 times
18/06/30 18:29:13 INFO util.GSet: Computing capacity for map cachedBlocks
18/06/30 18:29:13 INFO util.GSet: VM type      = 64-bit
18/06/30 18:29:13 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
18/06/30 18:29:13 INFO util.GSet: capacity      = 2^18 = 262144 entries
18/06/30 18:29:13 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
18/06/30 18:29:13 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
18/06/30 18:29:13 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension    = 30000
18/06/30 18:29:13 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
18/06/30 18:29:13 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
18/06/30 18:29:13 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
18/06/30 18:29:13 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
18/06/30 18:29:13 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
18/06/30 18:29:13 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/06/30 18:29:13 INFO util.GSet: VM type      = 64-bit
18/06/30 18:29:13 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
18/06/30 18:29:13 INFO util.GSet: capacity      = 2^15 = 32768 entries
18/06/30 18:29:14 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1178102935-10.0.0.111-1530354554626
18/06/30 18:29:14 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
18/06/30 18:29:14 INFO namenode.FSImageFormatProtobuf: Saving image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
18/06/30 18:29:14 INFO namenode.FSImageFormatProtobuf: Image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 320 bytes saved in 0 seconds.
18/06/30 18:29:15 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/06/30 18:29:15 INFO util.ExitUtil: Exiting with status 0
18/06/30 18:29:15 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at mini01/10.0.0.111
************************************************************/

拷贝到mini02

#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文本,这里笔者布置的是/app/hadoop/tmp,然后将/app/hadoop/tmp拷贝到mini02的/app/hadoop/下。
# 方法1:
[yun@mini01 hadoop]$ pwd
/app/hadoop
[yun@mini01 hadoop]$ scp -r tmp/ yun@mini02:/app/hadoop 
VERSION                          100%  202  189.4KB/s  00:00
seen_txid                        100%    2    1.0KB/s  00:00
fsimage_0000000000000000000.md5  100%  62    39.7KB/s  00:00
fsimage_0000000000000000000      100%  320  156.1KB/s  00:00

##########################3
# 方法2:##也足以如此,提出hdfs namenode -bootstrapStandby  # 可是须求mini02的Hadoop起来才行

8.4. 格式化ZKFC

#在mini01上推行壹回就能够
[yun@mini01 ~]$ hdfs zkfc -formatZK 
18/06/30 18:54:30 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at mini01/10.0.0.111:9000
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:host.name=mini01
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_112
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:java.home=/app/jdk1.8.0_112/jre
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:java.class.path=……………………
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/app/hadoop-2.7.6/lib/native
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-693.el7.x86_64
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:user.name=yun
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:user.home=/app
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Client environment:user.dir=/app/hadoop-2.7.6
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=mini03:2181,mini04:2181,mini05:2181,mini06:2181,mini07:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@7f3b84b8
18/06/30 18:54:30 INFO zookeeper.ClientCnxn: Opening socket connection to server mini04/10.0.0.114:2181. Will not attempt to authenticate using SASL (unknown error)
18/06/30 18:54:30 INFO zookeeper.ClientCnxn: Socket connection established to mini04/10.0.0.114:2181, initiating session
18/06/30 18:54:30 INFO zookeeper.ClientCnxn: Session establishment complete on server mini04/10.0.0.114:2181, sessionid = 0x4644fff9cb80000, negotiated timeout = 5000
18/06/30 18:54:30 INFO ha.ActiveStandbyElector: Session connected.
18/06/30 18:54:30 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/bi in ZK.
18/06/30 18:54:30 INFO zookeeper.ZooKeeper: Session: 0x4644fff9cb80000 closed
18/06/30 18:54:30 INFO zookeeper.ClientCnxn: EventThread shut down

8.5. 启动HDFS

# 在mini01上执行
[yun@mini01 ~]$ start-dfs.sh 
Starting namenodes on [mini01 mini02]
mini01: starting namenode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-namenode-mini01.out
mini02: starting namenode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-namenode-mini02.out
mini07: starting datanode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-datanode-mini07.out
mini06: starting datanode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-datanode-mini06.out
mini05: starting datanode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-datanode-mini05.out
Starting journal nodes [mini05 mini06 mini07]
mini07: journalnode running as process 1691. Stop it first.
mini06: journalnode running as process 1665. Stop it first.
mini05: journalnode running as process 1759. Stop it first.
Starting ZK Failover Controllers on NN hosts [mini01 mini02]
mini01: starting zkfc, logging to /app/hadoop-2.7.6/logs/hadoop-yun-zkfc-mini01.out
mini02: starting zkfc, logging to /app/hadoop-2.7.6/logs/hadoop-yun-zkfc-mini02.out

8.6. 启动YARN

#####注意#####:是在mini03上实行start-yarn.sh,把namenode和resourcemanager分开是因为质量难题
# 因为她们都要占用多量能源,所以把他们分别了,他们分别了就要分别在差别的机器上运维
[yun@mini03 ~]$ start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-resourcemanager-mini03.out
mini06: starting nodemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-nodemanager-mini06.out
mini07: starting nodemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-nodemanager-mini07.out
mini05: starting nodemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-nodemanager-mini05.out

################################
# 在mini04启动 resourcemanager
[yun@mini04 ~]$ yarn-daemon.sh start resourcemanager  # 也可用start-yarn.sh 
starting resourcemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-resourcemanager-mini04.out

8.7. 起步表达

# 第叁次开发银行的时候请严刻依照地点的步调【第二回提到格式化难题】

# 第三回以及未来,步骤为: 运转zookeeper、HDFS、YA奥迪Q3N

  1. 浏览访问

9.1. Hdfs访问

9.1.1. 正规情形拜望

   

皇冠体育投注 4

皇冠体育投注 5

 

 9.1.2. mini01挂了Active自动切换

# mini01操作
[yun@mini01 ~]$ jps
3584 DFSZKFailoverController
3283 NameNode
5831 Jps
[yun@mini01 ~]$ kill 3283
[yun@mini01 ~]$ jps
3584 DFSZKFailoverController
5893 Jps

Namenode挂了所以mini01无法访谈
   

皇冠体育投注 6

可知Hadoop已经切换过去了,之后mini01纵然起来了,状态也只可以为standby 。

9.2. Yarn访问

皇冠体育投注 7


会直接跳转到

皇冠体育投注 8

# 该图从别的地点截取,所以有个别相配

# Linux下访问
[yun@mini01 ~]$ curl mini04:8088
This is standby RM. The redirect url is:

HA完毕

  1. 集群运转测量检验

10.1. Haadmin与气象切换管理

[yun@mini01 ~]$ hdfs haadmin
Usage: haadmin
    [-transitionToActive [--forceactive] <serviceId>]
    [-transitionToStandby <serviceId>]
    [-failover [--forcefence] [--forceactive] <serviceId> <serviceId>]
    [-getServiceState <serviceId>]
    [-checkHealth <serviceId>]
    [-help <command>]

Generic options supported are
-conf <configuration file>    specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

能够观察,状态操作的下令示例:

# 查看namenode工作意况 
hdfs haadmin -getServiceState nn1

# 将standby状态namenode切换到active
hdfs haadmin -transitionToActive nn1

# 将active状态namenode切换到standby
hdfs haadmin -transitionToStandby nn2

10.2. 测量检验集群职业情景的部分下令

测量检验集群工作状态的有的指令 :
hdfs dfsadmin -report    查看hdfs的各节点状态信息
hdfs haadmin -getServiceState nn1  # hdfs haadmin -get瑟维斯State nn2      获取叁个namenode节点的HA状态
hadoop-daemon.sh start namenode  单独运维一个namenode进度
hadoop-daemon.sh start zkfc  单独运行八个zkfc进度

10.3. Datanode动态上下线

Datanode动态上下线不会细小略,步骤如下:

  a)  企图一台服务器,设置好条件

  b)  布署hadoop的安装包,并伙同集群配置

  c)  联英特网线,新datanode会自动步入集群

  d)  倘使是二遍扩大巨额datanode,还应有做集群负载重均衡

10.4. 数据块的balance

启动balancer的命令:

start-balancer.sh -threshold 8

运作之后,会有Balancer进度现身:

皇冠体育投注 9

上述命令设置了Threshold为8%,那么实行balancer命令的时候,首先计算全数DataNode的磁盘利用率的均值,然后决断如果某八个DataNode的磁盘利用率超越那一个均值Threshold,那么将会把这一个DataNode的block转移到磁盘利用率低的DataNode,那对于新节点的踏一贯讲特别卓有功能。Threshold的值为1到100里边,不出示的张开参数设置的话,私下认可是10。

Hadoop2.3-HA高可用集群处境搭建 
Hadoop项目之基于CentOS7的Cloudera 5.10.1(CDH)的装置配备 
Hadoop2.7.2集群搭建详解(高可用) 
接纳Ambari来布置Hadoop集群(搭建内网HDP源) 
Ubuntu 14.04下Hadoop集群安装 
CentOS 6.7安装Hadoop 2.7.2 
Ubuntu 16.04上构建遍布式Hadoop-2.7.3集群 
CentOS 7 下 Hadoop 2.6.4 布满式集群景况搭建 
Hadoop2.7.3 Spark2.1.0完全分布式集群搭建进程 
CentOS 7.4下编写翻译安装Hadoop 2.7.6 及所需文件 
Ubuntu 16.04.3 下安装配备 Hadoop

越来越多Hadoop相关音讯见Hadoop 专项论题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

Linux公社的RSS地址:

本文永恒更新链接地址

皇冠体育投注 10

闭馆防火墙,安装JDK1.8,设置SSH无密码登入,下载Hadoop-2.8.2.tar.gz到/hadoop目录下。

4. Jdk【java8】

1 安装namenode结点

4.1. 软件设置

 [yun@mini01 software]# pwd
/app/software
[yun@mini01 software]# tar xf jdk1.8.0_112.tar.gz
[yun@mini01 software]# ll
total 201392
drwxr-xr-x 8  10  143      4096 Dec 20 13:27 jdk1.8.0_112
-rw-r--r-- 1 root root 189815615 Mar 12 16:47 jdk1.8.0_112.tar.gz
[yun@mini01 software]# mv jdk1.8.0_112/ /app/
[yun@mini01 software]# cd /app/
[yun@mini01 app]# ll
total 8
drwxr-xr-x  8  10  143 4096 Dec 20 13:27 jdk1.8.0_112
[yun@mini01 app]# ln -s jdk1.8.0_112/ jdk
[yun@mini01 app]# ll
total 8
lrwxrwxrwx  1 root root    13 May 16 23:19 jdk -> jdk1.8.0_112/
drwxr-xr-x  8  10  143 4096 Dec 20 13:27 jdk1.8.0_112

4.2. 意况变量

 [root@mini01 ~]$ pwd
/app
[root@mini01 ~]$ ll -d jdk*  # 能够依靠实际处境选用jdk版本,个中jdk1.8 能够相配 jdk1.7 
lrwxrwxrwx 1 yun yun  11 Mar 15 14:58 jdk -> jdk1.8.0_112
drwxr-xr-x 8 yun yun 4096 Dec 20 13:27 jdk1.8.0_112
[root@mini01 profile.d]$ pwd
/etc/profile.d
[root@mini01 profile.d]$ cat jdk.sh # java景况变量 
export JAVA_HOME=/app/jdk
export JRE_HOME=/app/jdk/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH
 
[root@mini01 profile.d]# source /etc/profile
[root@mini01 profile.d]$ java -version 
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

  1. Hadoop配置修改并运行【全数机器配置文件一律】

[yun@mini01 software]$ pwd
/app/software
[yun@mini01 software]$ ll
total 194152
-rw-r--r-- 1 yun yun 198811365 Jun  8 16:36 CentOS-7.4_hadoop-2.7.6.tar.gz
[yun@mini01 software]$ tar xf CentOS-7.4_hadoop-2.7.6.tar.gz
[yun@mini01 software]$ mv hadoop-2.7.6/ /app/
[yun@mini01 software]$ cd
[yun@mini01 ~]$ ln -s hadoop-2.7.6/ hadoop
[yun@mini01 ~]$ ll
total 4
lrwxrwxrwx  1 yun yun  13 Jun  9 16:21 hadoop -> hadoop-2.7.6/
drwxr-xr-x  9 yun yun  149 Jun  8 16:36 hadoop-2.7.6
lrwxrwxrwx  1 yun yun  12 May 26 11:18 jdk -> jdk1.8.0_112
drwxr-xr-x  8 yun yun  255 Sep 23  2016 jdk1.8.0_112

5.1. 碰着变量

[root@mini01 profile.d]# pwd
/etc/profile.d
[root@mini01 profile.d]# vim hadoop.sh 
export HADOOP_HOME="/app/hadoop"
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
 
[root@mini01 profile.d]# source /etc/profile  # 生效    

5.2. core-site.xml

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
……………………
<!-- Put site-specific property overrides in this file. -->

<configuration>
  <!-- 钦点HADOOP所使用的文件系统schema(UEnclaveI),HDFS的要命(NameNode)的地址 -->
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://mini01:9000</value>  <!-- mini01 是hostname信息 -->
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/app/hadoop/tmp</value>
  </property>

  <!-- 启用垃圾箱效率,单位分钟 -->
  <property>
    <name>fs.trash.interval </name>
    <value>1440</value>
  </property>

</configuration>

5.3. hdfs-site.xml

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
………………
<!-- Put site-specific property overrides in this file. -->
 
<configuration>
  <!-- 钦定HDFS别本的数目 -->
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
 
  <property>
    <!-- 三个name标签都得以。周期性的统一fsimage和edits log文件同时使edits log保持在鲜明限制内。最佳和namenode不在一台机械,因为所需内部存储器和namenode同样-->
    <!-- <name>dfs.secondary.http.address</name> -->
    <name>dfs.namenode.secondary.http-address</name>
    <value>mini01:50090</value>
  </property>
 
  <!-- dfs namenode 的目录,能够有多少个目录,然后各种目录挂不一样的磁盘,每种目录下的文本新闻是大同小异的,也等于备份 -->
  <!-- 如有要求,放手注释就能够
  <property>
    <name>dfs.namenode.name.dir</name>
    <value> file://${hadoop.tmp.dir}/dfs/name,file://${hadoop.tmp.dir}/dfs/name1,file://${hadoop.tmp.dir}/dfs/name2</value>
  </property>
  -->
 
  <!-- 也得以配备dfs.datanode.data.dir  配置为四个目录,相当于扩大容积-->
 
</configuration>  

5.4. mapred-site.xml

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ mv mapred-site.xml.template mapred-site.xml 
[yun@mini01 hadoop]$ vim mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
………………
<!-- Put site-specific property overrides in this file. -->
 
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
 
</configuration>

5.5. yarn-site.xml

[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim yarn-site.xml
<?xml version="1.0"?>
……………………
<configuration>
 
<!-- Site specific YARN configuration properties -->
  <!-- 指定YARN的老大(ResourceManager)的地址 -->
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>mini02</value>  <!-- 依据设计 mini02 为ResourceManager -->
  </property>
 
  <!-- reducer获取数据的章程 -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
 
</configuration>

5.6. slaves

# 该配置和Hadoop服务无关,只是用来Hadoop脚本的批量利用
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ cat slaves
mini03
mini04
mini05

5.7. 格式化namenode
(是对namenode进行早先化)

[yun@mini01 hadoop]$ hdfs namenode -format 
18/06/09 17:44:56 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:  host = mini01/10.0.0.11
STARTUP_MSG:  args = [-format]
STARTUP_MSG:  version = 2.7.6
………………
STARTUP_MSG:  java = 1.8.0_112
************************************************************/
18/06/09 17:44:56 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
18/06/09 17:44:56 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-72e356f5-7723-4960-885a-72e522e19be1
18/06/09 17:44:57 INFO namenode.FSNamesystem: No KeyProvider found.
18/06/09 17:44:57 INFO namenode.FSNamesystem: fsLock is fair: true
18/06/09 17:44:57 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
18/06/09 17:44:57 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
18/06/09 17:44:57 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
18/06/09 17:44:57 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
18/06/09 17:44:57 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Jun 09 17:44:57
18/06/09 17:44:57 INFO util.GSet: Computing capacity for map BlocksMap
18/06/09 17:44:57 INFO util.GSet: VM type      = 64-bit
18/06/09 17:44:57 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
18/06/09 17:44:57 INFO util.GSet: capacity      = 2^21 = 2097152 entries
18/06/09 17:44:57 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
18/06/09 17:44:57 INFO blockmanagement.BlockManager: defaultReplication        = 3
18/06/09 17:44:57 INFO blockmanagement.BlockManager: maxReplication            = 512
18/06/09 17:44:57 INFO blockmanagement.BlockManager: minReplication            = 1
18/06/09 17:44:57 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
18/06/09 17:44:57 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
18/06/09 17:44:57 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
18/06/09 17:44:57 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
18/06/09 17:44:57 INFO namenode.FSNamesystem: fsOwner            = yun (auth:SIMPLE)
18/06/09 17:44:57 INFO namenode.FSNamesystem: supergroup          = supergroup
18/06/09 17:44:57 INFO namenode.FSNamesystem: isPermissionEnabled = true
18/06/09 17:44:57 INFO namenode.FSNamesystem: HA Enabled: false
18/06/09 17:44:57 INFO namenode.FSNamesystem: Append Enabled: true
18/06/09 17:44:58 INFO util.GSet: Computing capacity for map INodeMap
18/06/09 17:44:58 INFO util.GSet: VM type      = 64-bit
18/06/09 17:44:58 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
18/06/09 17:44:58 INFO util.GSet: capacity      = 2^20 = 1048576 entries
18/06/09 17:44:58 INFO namenode.FSDirectory: ACLs enabled? false
18/06/09 17:44:58 INFO namenode.FSDirectory: XAttrs enabled? true
18/06/09 17:44:58 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
18/06/09 17:44:58 INFO namenode.NameNode: Caching file names occuring more than 10 times
18/06/09 17:44:58 INFO util.GSet: Computing capacity for map cachedBlocks
18/06/09 17:44:58 INFO util.GSet: VM type      = 64-bit
18/06/09 17:44:58 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
18/06/09 17:44:58 INFO util.GSet: capacity      = 2^18 = 262144 entries
18/06/09 17:44:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
18/06/09 17:44:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
18/06/09 17:44:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension    = 30000
18/06/09 17:44:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
18/06/09 17:44:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
18/06/09 17:44:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
18/06/09 17:44:58 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
18/06/09 17:44:58 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
18/06/09 17:44:58 INFO util.GSet: Computing capacity for map NameNodeRetryCache
18/06/09 17:44:58 INFO util.GSet: VM type      = 64-bit
18/06/09 17:44:58 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
18/06/09 17:44:58 INFO util.GSet: capacity      = 2^15 = 32768 entries
18/06/09 17:44:58 INFO namenode.FSImage: Allocated new BlockPoolId: BP-925531343-10.0.0.11-1528537498201
18/06/09 17:44:58 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
18/06/09 17:44:58 INFO namenode.FSImageFormatProtobuf: Saving image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
18/06/09 17:44:58 INFO namenode.FSImageFormatProtobuf: Image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 319 bytes saved in 0 seconds.
18/06/09 17:44:58 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/06/09 17:44:58 INFO util.ExitUtil: Exiting with status 0
18/06/09 17:44:58 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at mini01/10.0.0.11
************************************************************/
[yun@mini01 hadoop]$ pwd
/app/hadoop
[yun@mini01 hadoop]$ ll
total 112
drwxr-xr-x 2 yun yun  194 Jun  8 16:36 bin
drwxr-xr-x 3 yun yun    20 Jun  8 16:36 etc
drwxr-xr-x 2 yun yun  106 Jun  8 16:36 include
drwxr-xr-x 3 yun yun    20 Jun  8 16:36 lib
drwxr-xr-x 2 yun yun  239 Jun  8 16:36 libexec
-rw-r--r-- 1 yun yun 86424 Jun  8 16:36 LICENSE.txt
-rw-r--r-- 1 yun yun 14978 Jun  8 16:36 NOTICE.txt
-rw-r--r-- 1 yun yun  1366 Jun  8 16:36 README.txt
drwxr-xr-x 2 yun yun  4096 Jun  8 16:36 sbin
drwxr-xr-x 4 yun yun    31 Jun  8 16:36 share
drwxrwxr-x 3 yun yun    17 Jun  9 17:44 tmp  # 该目录在此之前是平昔不的
[yun@mini01 hadoop]$ ll tmp/
total 0
drwxrwxr-x 3 yun yun 18 Jun  9 17:44 dfs
[yun@mini01 hadoop]$ ll tmp/dfs/
total 0
drwxrwxr-x 3 yun yun 21 Jun  9 17:44 name
[yun@mini01 hadoop]$ ll tmp/dfs/name/
total 0
drwxrwxr-x 2 yun yun 112 Jun  9 17:44 current
[yun@mini01 hadoop]$ ll tmp/dfs/name/current/
total 16
-rw-rw-r-- 1 yun yun 319 Jun  9 17:44 fsimage_0000000000000000000
-rw-rw-r-- 1 yun yun  62 Jun  9 17:44 fsimage_0000000000000000000.md5
-rw-rw-r-- 1 yun yun  2 Jun  9 17:44 seen_txid
-rw-rw-r-- 1 yun yun 199 Jun  9 17:44 VERSION

  

5.8. 启动namenode

# 在mini01上启动
[yun@mini01 sbin]$ pwd
/app/hadoop/sbin
[yun@mini01 sbin]$ ./hadoop-daemon.sh start namenode  # 截至使用: hadoop-daemon.sh stop namenode
starting namenode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-namenode-mini01.out
[yun@mini01 sbin]$ jps
6066 Jps
5983 NameNode
[yun@mini01 sbin]$ ps -ef | grep 'hadoop'
yun        5983      1  6 17:55 pts/0    00:00:07 /app/jdk/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/app/hadoop-2.7.6/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/app/hadoop-2.7.6 -Dhadoop.id.str=yun -Dhadoop.root.logger=INFO,console -Djava.library.path=/app/hadoop-2.7.6/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/app/hadoop-2.7.6/logs -Dhadoop.log.file=hadoop-yun-namenode-mini01.log -Dhadoop.home.dir=/app/hadoop-2.7.6 -Dhadoop.id.str=yun -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/app/hadoop-2.7.6/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
yun        6160  2337  0 17:57 pts/0    00:00:00 grep --color=auto hadoop
[yun@mini01 sbin]$ netstat -lntup | grep '5983'
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:50070          0.0.0.0:*              LISTEN      5983/java         
tcp        0      0 10.0.0.11:9000          0.0.0.0:*              LISTEN      5983/java 

5.8.1.  浏览器访问

皇冠体育投注 11

皇冠体育投注 12

5.9. 启动datanode

# mini03、mini04、mini05 启动datanode
# 由于增加情况变量,所以能够在其余目录运转
[yun@mini02 ~]$ hadoop-daemon.sh start datanode  # 截至使用: hadoop-daemon.sh stop datanode
starting datanode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-datanode-mini02.out
[yun@mini02 ~]$ jps
5349 Jps
5263 DataNode

5.9.1. 刷新浏览器

皇冠体育投注 13

5.10. 使用脚本批量起步hdf
# 依据设计在mini01运行
[yun@mini01 hadoop]$ start-dfs.sh
Starting namenodes on [mini01]
mini01: starting namenode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-namenode-mini01.out
mini04: starting datanode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-datanode-mini04.out
mini03: starting datanode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-datanode-mini03.out
mini05: starting datanode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-datanode-mini05.out
Starting secondary namenodes [mini01]
mini01: starting secondarynamenode, logging to /app/hadoop-2.7.6/logs/hadoop-yun-secondarynamenode-mini01.out

U福睿斯L地址:(HDFS管理分界面)

5.11. 行使脚本批量起动yarn

# 依据设计在mini02运营
# 启动yarn
[yun@mini02 hadoop]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-resourcemanager-mini02.out
mini05: starting nodemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-nodemanager-mini05.out
mini04: starting nodemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-nodemanager-mini04.out
mini03: starting nodemanager, logging to /app/hadoop-2.7.6/logs/yarn-yun-nodemanager-mini03.out   

U揽胜L地址:(MLacrosse管理分界面)
   

5.12. 最终结果

##### mini01
[yun@mini01 hadoop]$ jps
16336 NameNode
16548 SecondaryNameNode
16686 Jps
 
##### mini02
[yun@mini02 hadoop]$ jps
10936 ResourceManager
11213 Jps
 
##### mini03
[yun@mini03 ~]$ jps
9212 Jps
8957 DataNode
9039 NodeManager
 
##### mini04
[yun@mini04 ~]$ jps
4130 NodeManager
4296 Jps
4047 DataNode
 
##### mini05
[yun@mini05 ~]$ jps
7011 DataNode
7091 NodeManager
7308 Jps

Hadoop2.3-HA高可用集群情状搭建 
Hadoop项目之基于CentOS7的Cloudera 5.10.1(CDH)的设置配置 
Hadoop2.7.2集群搭建详解(高可用) 
使用Ambari来布署Hadoop集群(搭建内网HDP源) 
Ubuntu 14.04下Hadoop集群安装 
CentOS 6.7安装Hadoop 2.7.2 
Ubuntu 16.04上营造布满式Hadoop-2.7.3集群 
CentOS 7 下 Hadoop 2.6.4 布满式集群遇到搭建 
Hadoop2.7.3 Spark2.1.0通通布满式集群搭建进程 
CentOS 7.4下编写翻译安装Hadoop 2.7.6 及所需文件 
Ubuntu 16.04.3 下安装配置 Hadoop

越来越多Hadoop相关新闻见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

Linux公社的RSS地址:

正文恒久更新链接地址

皇冠体育投注 14

  将hadoop-2.8.2.tar.gz解压到192.168.169.102的hadoop用户的home目录/hadoop下

[hadoop@hadoop02 ~]$ pwd
/hadoop
[hadoop@hadoop02 ~]$ tar zxvf hadoop-2.8.2.tar.gz
... ...
[hadoop@hadoop02 ~]$ cd hadoop-2.8.2/
[hadoop@hadoop02 hadoop-2.8.2]$ pwd
/hadoop/hadoop-2.8.2
[hadoop@hadoop02 hadoop-2.8.2]$ ls -l
总用量 132
drwxr-xr-x 2 hadoop hadoop  4096 10月 20 05:11 bin
drwxr-xr-x 3 hadoop hadoop    19 10月 20 05:11 etc
drwxr-xr-x 2 hadoop hadoop  101 10月 20 05:11 include
drwxr-xr-x 3 hadoop hadoop    19 10月 20 05:11 lib
drwxr-xr-x 2 hadoop hadoop  4096 10月 20 05:11 libexec
-rw-r--r-- 1 hadoop hadoop 99253 10月 20 05:11 LICENSE.txt
-rw-r--r-- 1 hadoop hadoop 15915 10月 20 05:11 NOTICE.txt
-rw-r--r-- 1 hadoop hadoop  1366 10月 20 05:11 README.txt
drwxr-xr-x 2 hadoop hadoop  4096 10月 20 05:11 sbin
drwxr-xr-x 4 hadoop hadoop    29 10月 20 05:11 share
[hadoop@hadoop02 hadoop-2.8.2]$ 

 2 配置Hadoop情形变量

[hadoop@hadoop02 bin]$ vi /hadoop/.bash_profile
export HADOOP_HOME=/hadoop/hadoop-2.8.2
export PATH=$PATH:$HADOOP_HOME/bin

 注意:另两台虚构机也要平等配备。

 执行source ~./.bash_profile使配置生效,并表明:

[hadoop@hadoop02 bin]$ source ~/.bash_profile
[hadoop@hadoop02 bin]$ echo $HADOOP_HOME
/hadoop/hadoop-2.8.2
[hadoop@hadoop02 bin]$ echo $PATH
/usr/java/jdk1.8.0_151/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/hadoop/.local/bin:/hadoop/bin:/hadoop/.local/bin:/hadoop/bin:/hadoop/hadoop-2.8.2/bin
[hadoop@hadoop02 bin]$ 

 3 创立hadoop工作目录

[hadoop@hadoop02 bin]$ mkdir -p /hadoop/hadoop/dfs/name /hadoop/hadoop/dfs/data /hadoop/hadoop/tmp

 4 修改hadoop配制文件

    共修改7个配制文件:

    hadoop-env.sh: java意况变量
    yarn-env.sh:  制订yarn框架的java运营情形,yarn它将能源管理和处理组件分开。基于yarn的架构不受MapReduce约束。
    slaves: 钦赐datanode数据存款和储蓄服务器
    core-site.xml:  钦命访谈hadoop web分界面包车型地铁路子
    hdfs-site.xml:  文件系统的安排文件
    mapred-site.xml:  MapReducer任务布署文件
    yarn-site.xml: yarn框架配置,首就算一些职务的启航地点

4.1 /hadoop/hadoop-2.8.2/etc/hadoop/hadoop-env.sh

[hadoop@hadoop02 hadoop]$ vi /hadoop/hadoop-2.8.2/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_151/

 4.2 /hadoop/hadoop-2.8.2/etc/hadoop/yarn-env.sh

[hadoop@hadoop02 hadoop]$ vi /hadoop/hadoop-2.8.2/etc/hadoop/yarn-env.sh
JAVA_HOME=/usr/java/jdk1.8.0_151/

本文由澳门皇冠金沙网站发布于网络操作,转载请注明出处:【皇冠体育投注】6安装部署