Rhadoop相关记录汇总
一、安装r
参考http://blog.itpub.net/21711990/viewspace-1114418/
1）解压：tar xzf R-3.0.0
2）cd R-3.0.0
3）Mkdir /dwbi/dwbi/R-3.0.0
4）预先安装如下环境插件
yum -y install gcc
yum install glibc-headers
yum install gcc-c++
yum install gcc-gfortran 1
yum -y install bzip2-devel
yum install readline-devel
依赖包：ncurses-devel
yum install libXt-devel
依赖包：
1
5）./configure --prefix=/dwbi/dwbi/R-3.0.0 --enable-R-shlib
注意：如果执行上出现错误1、2、3错误，解决掉错误之后最后执行命令为：
cd到/dwbi/dwbi/R-3.0.0下执行如下：
./configure --prefix=/dwbi/dwbi/R-3.3.3 --enable-R-shlib LDFLAGS="-L/opt/zlib-1.2.11/lib -L/opt/bzip2-1.0.6/lib -L/opt/xz-5.2.3/lib -L/opt/pcre-8.38/lib" CPPFLAGS="-I/opt/zlib-1.2.11/include -I/opt/bzip2-1.0.6/include -I/opt/xz-5.2.3/include -I/opt/pcre-8.38/include"
无错误则ok，继续下一步
6）Make
7）Make install
8）Vi ~/.bash_profile
如下：
PATH=/dwbi/dwbi/R-3.3.3/bin:$PATH:$HOME/bin
Export PATH
8）R
二、安装rHadoop
1）配置rjava环境
R CMD javareconf
2）R
3）依此安装包
Mkdir /dwbi/rhadoop/temp
Mv icudt55l.zip ./temp
Install.packages(“rJava_0.9-8.tar.gz”)
Install.packages(“Rcpp_0.12.10.tar.gz”)
Install.packages(“plyr_1.8.1.tar.gz”)
Install.packages(“magrittr_1.5.tar.gz”)
Install.packages(“/dwbi/rhadoop/stringi_1.0-1.tar.gz”,configure.vars=”ICUDT_DIR=/dwbi/rhadoop/temp/”)
Install.packages(“stringr_1.2.0.tar.gz”)
Install.packages(“reshape2_1.4.2.tar.gz”)
Install.packages(“iterators_1.0.8.tar.gz”)
Install.packages(“itertools_0.1-3.tar.gz”)
Install.packages(“digest_0.6.12.tar.gz”)
Install.packages(“RJSONIO_1.3-0.tar.gz”)
Install.packages(“functional_0.6.tar.gz”)
Install.packages(“bitops_1.0-6.tar.gz”)
Install.packages(“caTools_1.17.1.tar.gz”)
4)安装rhdfs
Export HADOOP_CMD=/usr/lib/hadoop/bin/hadoop
Export HADOOP_STEAMING=/usr/lib/hadoop-mapreduce/hadoop-steaming-2.5.2-transwarp-tdh470.jar
（可以上述环境变量加入文件/edt/profile）
或者在R中设置
Sys.setenv(“HADOOP_CMD”=”/usr/lib/hadoop/bin/hadoop”)
Sys.setenv(“HADOOP_STEAMING”=”/usr/lib/hadoop-mapreduce/hadoop-steaming-2.5.2-transwarp-tdh470.jar”)
Sys.setenv(“JAVA_HOME”=”/usr/java/latest”)
正确则继续
Install.packages(“rhdfs_1.0.8.tar.gz”)
Install.packages(“rmr2_3.3.1.tar.gz”)
测试：
Hdfs.init()
Hdfs.ls(“/user/”)
5、rhive配置
1）R
install.packages("/dwbi/rhadoop/Rserve_0.6-8.tar.gz")
install.packages("/dwbi/rhadoop/RHive_2.0-0.0.tar.gz")
2）配置文件
Mkdir /dwbi/rhadoop/rhive
在新建的rhive目录下创建Rserv.conf文件，写入``remote enable''保存并退出
在Rserve服务器节点执行：
R CMD Rserve --RS-settings
R CMD Rserve --RS-enable-remote
启动Rserve：R CMD Rserve
在各节点执行如下（所有节点启动Rserve）：
R
Library(Rserve)
Rserve()
查看Rserve进程
ps -aux|grep Rserve
netstat -nltp|grep Rserve
修改各个节点以及master 的 /etc/profile 新增环境变量
参考：
http://blog.csdn.net/youngqj/article/details/46819625
三、rstidio安装
1、yum install openss1098e.x86_64
2、yum install --nogpgcheck rstudio-server-0.98.1091-x86_64.rpm
3、启动
rstudio-server start
4、执行rstudio-server verify-installation
5、新建用户用于登陆rstudio
Useradd rhadoop
Password rhadoop 或者passwd rhadoop
新建的用户密码：rhadoop/rhadoop123
注意：输入的用户名密码的权限必须小于100（不要用root权限的用户登录）
---------------------------------
如果执行rstudio-server verify-installation出现如下错误，并且web无法访问：
Unable to find an installation of R on the system; Unable to locate R binary by scanning standard locations
则解决办法：
通过vi创建配置文件
/etc/rstudio/rserver.conf
/etc/rstudio/rsession.conf
并在rserver.conf添加了配置
www-port = 8787
rsession-which-r=/dwbi/dwbi/R-3.0.0/bin/R
重启Rstudio
---
查看服务：
Ps -aux | grep Rserve
Netstat -nlp | grep Rserve
四、R连接hive
配置环境变量：
Sys.setenv(HIVE_HOME="/usr/lib/hive")
Sys.setenv(HADOOP_HOME="/usr/lib/hadoop")
1、首先加载RJDBC包（需要预先安装DBI和RJDBC）
Library(RJDBC)
2、从控制界面下载RJDBC驱动inceptor-driver.jar到/dwbi/rhadoop下
3、加载驱动
f4=list.files("/dwbi/rhadoop",pattern="jar$",full.names=T,recursive=TRUE)
jdbc=JDBC("org.apache.hive.jdbc.HiveDriver",f4)
4、连接hive
conn=dbConnect(jdbc,sprintf('jdbc:hive2://localhost:10000/default','root','nbzd_root123'))
5、sql查询
dbGetQuery(conn,"select * from cdt_bi_basic.acd_file")
各个节点都要装rmr2，rhbase和rhdfs在集群的namenode上安装即可。只安装rmr2也需要安装R
错误记录：
1 ./configure --prefix=/dwbi/dwbi/R-3.3.3 --enable-R-shlib出现错误：
解决：原因是缺少编译环境，执行如下
安装编译环境
yum -y install gcc
yum install glibc-headers
yum install gcc-c++
yum install gcc-gfortran
yum install readline-devel
yum install libXt-devel
错误2：./configure --prefix=/dwbi/dwbi/R-3.3.3 --enable-R-shlib出现错误：checking whether zlib support suffices... Configure: error: zlib library and headers required
解决办法：
yum -y install bzip2-devel
安装后还是显示 error: zlib library and headers required
最后安装最新版本的：
tar xvf zlib-1.2.11.tar.gz
cd zlib-1.2.11
./configure --prefix=/opt/zlib-1.2.11
Make
Make install
（完成zlib-1.2.11安装）
错误3：checking whether bzip2 support suffices... Configure: error: bzip2 library and headers required
解决办法：
tar xvf bziq2-1.0.6.tar.gz
Cd bzip2-1.0.6
make -f Makefile-libbz2_so
Make
Make install PREFIX=/opt/bzip2-1.0.6
（完成bzip2-1.0.6安装）
错误4（执行如下）:
Cd /dwbi/rhadoop/R-3.3.3
./configure --prefix=/dwbi/dwbi/R-3.3.3 --enable-R-shlib LDFLAGS="-L/opt/zlib-1.2.11/lib -L/opt/bzip2-1.0.6/lib -L/opt/bzip2-1.0.6/bin" CPPFLAGS="-I/opt/zlib-1.2.11/include -I/opt/bzip2-1.0.6//include "
出现错误：
Configure:error:”liblzma library and headers are required”
解决办法同上：
（可以先用yum，缺少什么就使用yum search 缺少的名称，再根据相应的包来进行安装）
Yum -y install xz-devel.x86_64
（如果还出现上面问题：提示checking：if lzma version >=1.5.0.3...no，只能自己下载最新办法安装）
Tar xzvf xz-5.2.3.tar.gz
Cd xz-5.2.3
./configure --prefix=/opt/xz-5.2.3
Make -j3
Make install
错误4：PCRE support suffices... Configure: error:pcre>=8.10 library and headers are required
解决：
Tar xzvf pcre-8.38.tar.gz
Cd pcre-8.38
./configure --prefix=/opt/pcre-8.38
Make -j3
Make install
然后再次运行
./configure --prefix=/dwbi/dwbi/R-3.3.3 --enable-R-shlib LDFLAGS="-L/opt/zlib-1.2.11/lib -L/opt/bzip2-1.0.6/lib -L/opt/xz-5.2.3/lib -L/opt/pcre-8.38/lib" CPPFLAGS="-I/opt/zlib-1.2.11/include -I/opt/bzip2-1.0.6/include -I/opt/xz-5.2.3/include -I/opt/pcre-8.38/include"
再次出现错误：checking if PCRE version >= 8.10, < 10.0 and has UTF-8 support...
no checking whether PCRE support suffices...
configure: error: pcre >= 8.10 library and headers are required
解决办法：
./configure --enable-utf8 --prefix=/opt/pcre-8.38
Make -j3
Make install
错误：
tar xzvf curl-7.47.1.tar.gz
cd curl-7.47.1
./configure --prefix=$HOME/packages
make -j3
make install
错误：checking if libcurl is version 7 and >= 7.28.0... yes
checking if libcurl supports https... no
configure: error: libcurl >= 7.28.0 library and headers are required with support for https ##这是一个大坑需要在环境变量中加上curl的bin路径，如果没有报错，那肯定是之前系统默认安装的curl造成的，为了消除潜在风险，是否有报错，请必须执行下面的操作，系统默认安装的curl请保留，防止造成别的系统程序依赖缺失；
错误：
/usr/bin/ld: warning: libpcre.so.1, needed by ../../lib/libR.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: liblzma.so.5, needed by ../../lib/libR.so, not found (try using -rpath or -rpath-link)
解决方法：
/usr/bin/ld: warning: libpcre.so.1, needed by ../../lib/libR.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: liblzma.so.5, needed by ../../lib/libR.so, not found (try using -rpath or -rpath-link)
../../lib/libR.so: undefined reference to `lzma_code@XZ_5.0'
../../lib/libR.so: undefined reference to `lzma_raw_encoder@XZ_5.0'
../../lib/libR.so: undefined reference to `lzma_stream_decoder@XZ_5.0'
../../lib/libR.so: undefined reference to `pcre_free'
../../lib/libR.so: undefined reference to `lzma_lzma_preset@XZ_5.0'
../../lib/libR.so: undefined reference to `lzma_raw_decoder@XZ_5.0'
../../lib/libR.so: undefined reference to `pcre_version'
../../lib/libR.so: undefined reference to `pcre_exec'
../../lib/libR.so: undefined reference to `pcre_config'
../../lib/libR.so: undefined reference to `pcre_fullinfo'
../../lib/libR.so: undefined reference to `pcre_maketables'
../../lib/libR.so: undefined reference to `pcre_compile'
../../lib/libR.so: undefined reference to `lzma_end@XZ_5.0'
../../lib/libR.so: undefined reference to `lzma_alone_decoder@XZ_5.0'
../../lib/libR.so: undefined reference to `lzma_version_string@XZ_5.0'
../../lib/libR.so: undefined reference to `lzma_crc64@XZ_5.0'
../../lib/libR.so: undefined reference to `pcre_study'
../../lib/libR.so: undefined reference to `lzma_stream_encoder@XZ_5.0'
collect2: ld returned 1 exit status
make[3]: *** [R.bin] Error 1
make[3]: Leaving directory `/root/R-3.3.1/src/main'
make[2]: *** [R] Error 2
make[2]: Leaving directory `/root/R-3.3.1/src/main'
make[1]: *** [R] Error 1
make[1]: Leaving directory `/root/R-3.3.1/src'
make: *** [R] Error 1
此处报错是由于没有找到动态库，看第一二行
解决方法：添加动态库
#cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
/opt/pcre-8.38/lib
/opt/xz-5.2.3/lib
然后执行
#ldconfig
再执行：
重新
./configure --prefix=/dwbi/dwbi/R-3.3.3 --enable-R-shlib LDFLAGS="-L/opt/zlib-1.2.11/lib -L/opt/xz-5.2.3/lib -L/opt/pcre-8.38/lib" CPPFLAGS="-I/opt/zlib-1.2.11/include -I/opt/xz-5.2.3/include -I/opt/pcre-8.38/include"
（去掉了-I/opt/bzip2-1.0.6/include和-L/opt/bzip2-1.0.6/lib，因为换了3.2.5，提示这个包版本不行）
执行成功
Make（成功图）
make install（成功图）
完成第一台16，各配置文件内容如下：
1)Linux版本（lsb_release -a）
Centos release 6.5 (Final)
2)R版本（R --version）
R version 3.2.5 platform:x86_64-pc-linux-gnu(64-bit)
3)jdk版本(java -version)
Java version “1.7.0.71”
Java HotSpot(TM) 64-bit Server VM
4)~/.bash_profile 内容：
PATH=/dwbi/dwbi/R-3.2.4/bin:$PATH:$HOME/bin
Export PATH
5)/etc/profile内容
Export PATH=$PATH:$JAVA_HOME/bin:/opt/curl-7.54.0/bin:/dwbi/dwbi/R-3.2.5/bin
6)/etc/ld.so.conf 内容
include ld.so.conf.d/*.conf
/opt/pcre-8.38/lib
/opt/xz-5.2.3/lib
7)最后运行
./configure --prefix=/dwbi/dwbi/R-3.3.3 --enable-R-shlib LDFLAGS="-L/opt/zlib-1.2.11/lib -L/opt/xz-5.2.3/lib -L/opt/pcre-8.38/lib" CPPFLAGS="-I/opt/zlib-1.2.11/include -I/opt/xz-5.2.3/include -I/opt/pcre-8.38/include"
8)Make
9)Make install
Over
参考网址：
http://news.tuxi.com.cn/news/119999990123041/30416034.html
http://blog.sina.com.cn/s/blog_b75539cb0102wn48.html
http://kuxingseng2016.blog.51cto.com/1374617/1846326
http://blog.sina.com.cn/s/blog_b75539cb0102wn48.html
http://kuxingseng2016.blog.51cto.com/1374617/1846326
http://blog.csdn.net/alex_starsky/article/details/52717080
http://kuxingseng2016.blog.51cto.com/1374617/1846326
安装新的jdk，只要不改变如下集群软连接就行
安装stringi时出现问题：
参考：
或者
解决办法：
1、在https://github.com/Rexamine/stringi/
下载icudt55l.zip，放到目录/dwbi/hadoop/temp下
2、运行如下命令：
Install.packages(“/dwbi/rhadoop/stringi_1.0-1.tar.gz”,configure.vars=”ICUDT_DIR=/dwbi/rhadoop/temp/”)
网上评论：
I had the same problem. What I did is similar to the solution proposed by gagolews. Gagolews' solution didn't exactly work for me because I couldn't find a zip copy of stringi to download. I downloaded icudt from http://static.rexamine.com/packages/icudt52l.zip to a folder, say /home/tmp/ then I installed from the net, but providing the local copy of icudt, by typing the following in R:
install.packages("stringi", # this will install from online
configure.vars="ICUDT_DIR=/home/tmp/") #but use the downloaded version of icudt
This worked for me.
错误：运行Install.packages(“rJava_0.9-8.tar.gz”)出现错误
解决办法：
Cp /dwbi/dwbi/R-3.0.0/lib64/R/library/rJava/jri目录下的
JRIEngine.jar
JRI.jar
REngine.jar
三个文件到
${Java_home}/lib下即可解决（/usr/java/latest/lib下）
运行R目录下
错误：configure:error in /dwbi/rhadoop/R-3.0.0
Configure:error: C++ preprocessor “/lib/cpp” fails sanity check
解决办法：
yum install glibc-headers
yum install gcc-c++
错误：--with-x=yes (default) and X11 headers/libs are not available”
解决办法：
yum install libXt-devel
命令：telnet 33.104.103.14 6311
错误：-bash: telnet: command not found
解决办法：
yum list telnet* 列出telnet相关的安装包
yum install telnet-server 安装telnet服务
yum install telnet.* 安装telnet客户端
命令：telnet 33.104.103.16 6311
错误：
telnet 192.168.2.51
Trying 192.168.2.51...
Connected to 192.168.2.51.
Escape character is '^]'.
Connection closed by foreign host.
解决办法：
vi /etc/xinetd.d/telnet
修改disable 为no
然后重新启动：
Service xinetd restart
运行：type -a Rscript
如果没有，则PATH中加入/dwbi/dwbi/R-3.0.0/bin
测试mr：
运行脚本：
library("rmr2")
small.ints = to.dfs(1:10)
mapreduce(input = small.ints, map = function(k, v) cbind(v, v^2))
出现错误：
Caused by: java.io.IOException: Cannot run program “Rscript”: java.io.IOException: error=2, No such file or directory
解决办法：
查看type -a Rscript
在各个节点建立软连接：ln /dwbi/dwbi/R-3.0.0/bin/Rscript /usr/bin/Rscript
参考：
https://github.com/RevolutionAnalytics/RHadoop/issues/87
华青莲，方便自己，成长他人！！！
