- 安装linux centos6.7
- 安装完成后配置网络
vi /etc/sysconfig/network-scripts/ifcfg-eth0
修改OnBoot为yes
指定IP:
修改IP永久生效按以下方法
vi /etc/sysconfig/network-scripts/ifcfg-eth0(eth0,第一块网卡,如果是第二块则为eth1)
按如下修改ip
DEVICE=eth0
IPADDR=172.28.15.221
NETMASK=255.255.255.0
GATEWAY=172.28.15.1
TYPE=Ethernet
UUID=9f8f19f7-16c4-4f84-b673-b25d5b079ca8
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
然后
service network restart
IP地址生效,重启系统后不变
如果是临时修改IP重启系统后恢复原始IP则用以下命令
ifconfig IP地址 netmask 子网掩码重启网络 service network restart
- 下载Hadoop 2.7.2
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/
- 添加IP, 修改 hosts
vim /etc/hosts
- 网络机器配置情况:
host:
172.28.15.245 Master01
172.28.15.246 Master02
slave:
172.28.15.247 Slave01
172.28.15.248 Slave02
172.28.15.249 Slave03
- 拷贝下载后的jdk hadoop安装包 到 虚拟机上任意目录
- 在虚拟机任意目录拷贝安装包至 /opt 目录,方便管理
- 安装 jdk.rpm
rpm -ivh jdk-8u131-linux-x64.rpm
- 定位安装后的jdk目录
ll /usr/java/default
java8最新rpm安装目录:/usr/java/latest
- 配置环境变量
vim /etc/profile
移动到最后一行,添加: JAVA_HOME=/usr/java/latest, :wq保存退出
刷新一下profile: source /etc/profile
通过 echo $JAVA_HOME 判断一下是否成功
java -version 也可以验证
- 安装ssh命令,安装 openssh-clients
yum install openssh-clients
退出SercureCRT,再登陆
- 配置ssh免密码登陆
ssh-keygen -t rsa
将会/root/.ssh目录生成
id_rsa 是私钥
id_rsa.pub 是公钥
- 执行命令 cat id_rsa.pub >> authorized_keys文件
- 执行命令 chmod 644 authorized_keys,然后查看文件属性
- 修改主机名字
vi /etc/sysconfig/network 修改 HOSTNAME为目标名
- 重启机器 shutdown -r now
- 进入/opt目录,解压缩hadoop压缩包 tar zxf hadoop-2.7.2.tar.gz
- pwd 显示当前hadoop目录,并复制hadoop路径
- vim /etc/profile,末尾添加
JAVA_HOME=/usr/java/latest
JRE_HOME=$JAVA_HOME/jre
HADOOP_HOME=/opt/hadoop-2.7.2
HADOOP_CONF_DIR=/opt/hadoop-2.7.2/etc/hadoop
YARN_CONF_DIR=/opt/hadoop-2.7.2/etc/hadoop
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME HADOOP_HOME HADOOP_CONF_DIR YARN_CONF_DIR
- 如果出现找不到linux命令的情况,执行
export PATH=/bin:/usr/bin:/usr/local/bin
- vim ~/.bash_profile, 末尾添加
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
- 对hadoop进行配置,主要配置文件保存在 $HADOOP_CONF_DIR环境变量中,目录为 /opt/hadoop-2.7.2/etc/hadoop
- 主要需要配置的文件如下:
core_site.xml
hdfs-site.xml
yarn-site.xml
mapred-site.xml
slaves
- core-site.xml
创建目录
mkdir /opt/hadoop-2.7.2/dfs/tmp
mkdir /opt/hadoop-2.7.2/dfs/name
mkdir /opt/hadoop-2.7.2/dfs/data
开始修改core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://Master01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-2.7.2/dfs/tmp</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>4320</value>
</property>
</configuration>
- hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop-2.7.2/dfs/name</value>
</property>
<property>
<name>dfs.datanode.name.dir</name>
<value>/opt/hadoop-2.7.2/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions.superusergroup</name>
<value>staff</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>Master02:9000</value>
</property>
</configuration>
- yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>Master01</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>Master01:18040</value>
</property>
<property>
<name>yarn.resourcenamager.scheduler.address</name>
<value>Master01:18030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>Master01:18025</value>
</property>
<property>
<name>yarn.resource.manager.admin.address</name>
<value>Master01:18041</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>Master01:18088</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>86400</value>
</property>
<property>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>86400</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/tmp/logs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
</configuration>
- mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobtracker.http.address</name>
<value>Master01:50030</value>
</property>
<property>
<name>mapreduce.jobhisotry.address</name>
<value>Master01:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>Master01:19888</value>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/jobhistory/done</value>
</property>
<property>
<name>mapreduce.intermediate-done-dir</name>
<value>/jobhisotry/done_intermediate</value>
</property>
<property>
<name>mapreduce.job.ubertask.enable</name>
<value>true</value>
<property>
</configuration>
- 永久关闭防火墙
1 关闭防火墙-----service iptables stop
2 启动防火墙-----service iptables start
3 重启防火墙-----service iptables restart
4 查看防火墙状态--service iptables status
5 永久关闭防火墙--chkconfig iptables off
6 永久关闭后启用--chkconfig iptables on
- 拷贝虚拟机,创建 Slave01 Slave02。重新打开虚拟机后提示选择“我已复制该虚拟机”
- 出现 Linux 克隆虚拟机引起的“Device eth0 does not seem to be present, delaying initialization”
如下办法解决:
1. 删除 /etc/sysconfig/network-scripts/ifcfg-eth0 内关于mac地址的行
2. 删除 /etc/udev/rules.d/70-persistent-net.rules
3. shutdown -r now
- 通过Master01 ssh Slave01 Slave02 创建ssh信任,并永久保存信任关系。依次把 Master01 Slave01 Slave02都做一遍,来创建ssh免密码登陆
- /opt/hadoop-2.7.2/sbin/start-all.sh
- 节点管理地址: http://172.28.15.245:50070
- Hadooop管理地址: http://172.28.15.245:18088
- 安装成功后 Master节点具有如下进程,输入JPS命令
1619 NameNode
2166 NodeManager
1722 DataNode
1900 SecondaryNameNode
2061 ResourceManager
- 安装成功后 Slave节点具有如下进程,输入JPS命令
1633 NodeManager
1529 DataNode