Hadoop分布式集群搭建

发表: 2017-05-15 浏览: 1798

Hadoop安装配置

安装linux centos6.7
安装完成后配置网络

vi /etc/sysconfig/network-scripts/ifcfg-eth0

修改OnBoot为yes

指定IP:

修改IP永久生效按以下方法

vi /etc/sysconfig/network-scripts/ifcfg-eth0（eth0，第一块网卡，如果是第二块则为eth1）

按如下修改ip

DEVICE=eth0

IPADDR=172.28.15.221

NETMASK=255.255.255.0

GATEWAY=172.28.15.1

TYPE=Ethernet

UUID=9f8f19f7-16c4-4f84-b673-b25d5b079ca8

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=static

然后

service network restart

IP地址生效，重启系统后不变

如果是临时修改IP重启系统后恢复原始IP则用以下命令

ifconfig IP地址 netmask 子网掩码重启网络 service network restart

下载Hadoop 2.7.2

http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/

Java8:http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpm

Java8:http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz

添加IP, 修改 hosts

vim /etc/hosts

网络机器配置情况:

host:

172.28.15.245 Master01

172.28.15.246 Master02

slave:

172.28.15.247 Slave01

172.28.15.248 Slave02

172.28.15.249 Slave03

拷贝下载后的jdk hadoop安装包到虚拟机上任意目录
在虚拟机任意目录拷贝安装包至 /opt 目录，方便管理
安装 jdk.rpm

rpm -ivh jdk-8u131-linux-x64.rpm

ll /usr/java/default

java8最新rpm安装目录:/usr/java/latest

配置环境变量

vim /etc/profile

移动到最后一行，添加: JAVA_HOME=/usr/java/latest， :wq保存退出

刷新一下profile: source /etc/profile

通过 echo $JAVA_HOME 判断一下是否成功

java -version 也可以验证

安装ssh命令，安装 openssh-clients

yum install openssh-clients

退出SercureCRT，再登陆

配置ssh免密码登陆

ssh-keygen -t rsa

将会/root/.ssh目录生成

id_rsa 是私钥

id_rsa.pub 是公钥

执行命令 cat id_rsa.pub >> authorized_keys文件
执行命令 chmod 644 authorized_keys，然后查看文件属性

修改主机名字

vi /etc/sysconfig/network 修改 HOSTNAME为目标名

重启机器 shutdown -r now
进入/opt目录，解压缩hadoop压缩包 tar zxf hadoop-2.7.2.tar.gz
pwd 显示当前hadoop目录，并复制hadoop路径
vim /etc/profile，末尾添加

JAVA_HOME=/usr/java/latest

JRE_HOME=$JAVA_HOME/jre

HADOOP_HOME=/opt/hadoop-2.7.2

HADOOP_CONF_DIR=/opt/hadoop-2.7.2/etc/hadoop

YARN_CONF_DIR=/opt/hadoop-2.7.2/etc/hadoop

PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

export JAVA_HOME JRE_HOME HADOOP_HOME HADOOP_CONF_DIR YARN_CONF_DIR

如果出现找不到linux命令的情况，执行

export PATH=/bin:/usr/bin:/usr/local/bin

vim ~/.bash_profile, 末尾添加

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

对hadoop进行配置，主要配置文件保存在 $HADOOP_CONF_DIR环境变量中，目录为 /opt/hadoop-2.7.2/etc/hadoop
主要需要配置的文件如下:

core_site.xml

hdfs-site.xml

yarn-site.xml

mapred-site.xml

slaves

core-site.xml

创建目录

mkdir /opt/hadoop-2.7.2/dfs/tmp

mkdir /opt/hadoop-2.7.2/dfs/name

mkdir /opt/hadoop-2.7.2/dfs/data

开始修改core-site.xml

<name>fs.default.name</name>

<value>hdfs://Master01:9000</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/opt/hadoop-2.7.2/dfs/tmp</value>

</property>

<name>fs.trash.interval</name>

</property>

</configuration>

hdfs-site.xml

<name>dfs.namenode.name.dir</name>

<value>/opt/hadoop-2.7.2/dfs/name</value>

</property>

<name>dfs.datanode.name.dir</name>

<value>/opt/hadoop-2.7.2/dfs/data</value>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.webhdfs.enabled</name>

</property>

<name>dfs.permissions.superusergroup</name>

<value>staff</value>

</property>

<name>dfs.permissions.enabled</name>

<value>false</value>

</property>

<name>dfs.namenode.secondary.http-address</name>

<value>Master02:9000</value>

</property>

</configuration>

yarn-site.xml

<name>yarn.resourcemanager.hostname</name>

<value>Master01</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<name>yarn.resourcemanager.address</name>

<value>Master01:18040</value>

</property>

<name>yarn.resourcenamager.scheduler.address</name>

<value>Master01:18030</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>Master01:18025</value>

</property>

<name>yarn.resource.manager.admin.address</name>

<value>Master01:18041</value>

</property>

<name>yarn.resourcemanager.webapp.address</name>

<value>Master01:18088</value>

</property>

<name>yarn.log-aggregation-enable</name>

</property>

<name>yarn.log-aggregation.retain-seconds</name>

</property>

<name>yarn.log-aggregation.retain-check-interval-seconds</name>

</property>

<name>yarn.nodemanager.remote-app-log-dir</name>

</property>

<name>yarn.nodemanager.remote-app-log-dir-suffix</name>

</property>

</configuration>

mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobtracker.http.address</name>

<value>Master01:50030</value>

</property>

<name>mapreduce.jobhisotry.address</name>

<value>Master01:10020</value>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>Master01:19888</value>

</property>

<name>mapreduce.jobhistory.done-dir</name>

<value>/jobhistory/done</value>

</property>

<name>mapreduce.intermediate-done-dir</name>

<value>/jobhisotry/done_intermediate</value>

</property>

<name>mapreduce.job.ubertask.enable</name>

</configuration>

永久关闭防火墙

1 关闭防火墙-----service iptables stop

2 启动防火墙-----service iptables start

3 重启防火墙-----service iptables restart

4 查看防火墙状态--service iptables status

5 永久关闭防火墙--chkconfig iptables off

6 永久关闭后启用--chkconfig iptables on

拷贝虚拟机，创建 Slave01 Slave02。重新打开虚拟机后提示选择“我已复制该虚拟机”
出现 Linux 克隆虚拟机引起的“Device eth0 does not seem to be present, delaying initialization”

如下办法解决:

1. 删除 /etc/sysconfig/network-scripts/ifcfg-eth0 内关于mac地址的行

2. 删除 /etc/udev/rules.d/70-persistent-net.rules

3. shutdown -r now

通过Master01 ssh Slave01 Slave02 创建ssh信任，并永久保存信任关系。依次把 Master01 Slave01 Slave02都做一遍，来创建ssh免密码登陆
/opt/hadoop-2.7.2/sbin/start-all.sh
节点管理地址: http://172.28.15.245:50070
Hadooop管理地址: http://172.28.15.245:18088

安装成功后 Master节点具有如下进程，输入JPS命令

1619 NameNode

2166 NodeManager

1722 DataNode

1900 SecondaryNameNode

2061 ResourceManager

安装成功后 Slave节点具有如下进程，输入JPS命令

1633 NodeManager

1529 DataNode

0 个评论

要回复文章请先登录或注册