hadoop源码编译

为何要自行编译hadoop源码,往往是由于官方提供的hadoop发行版都是基于32位操作系统,在操作hadoop时会发生warn。
 
准备软件:
1)JDK
2)Hadoop源码
3)Maven
4)Protobuf
 
编译小结:
1)JDK安装(略)
2)必要依赖安装
       sudo yum install -y g++ gcc gcc-c++ autoconf automake libtoolcmake zliblg-dev pkg-conf libssl-dev openssh-client make mcurses-devel
3)Protobuf安装
    ./configure --prefix=/usr/local/protoc/
    make
    make install
4)Maven安装(略)
5)vi .bash_profile
   export MAVEN_HOME=/opt/softwares/apache-maven-3.2.5
   PATH=$MAVEN_HOME/bin:/usr/local/protoc/bin:$PATH
6)Hadoop编译
     mvn package –DskipTests –Pdist,native
7)编译成功后,将hadoop-dist/target/hadoop-2.3.0-cdh5.0.2/lib/native下文件拷贝纸$HADOOP_HOME/lib/native下,文件如下:

  libhadoop.a
  libhadooppipes.a
  libhadoop.so -> libhadoop.so.1.0.0
  libhadoop.so.1.0.0
  libhadooputils.a
  libhdfs.a
  libhdfs.so -> libhdfs.so.0.0.0
  libhdfs.so.0.0.0

8)验证

  hadoop checknative
  15/11/06 15:46:41 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
  15/11/06 15:46:41 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
  Native library checking:
  hadoop: true /opt/modules/hadoop-2.3.0-cdh5.0.2/lib/native/libhadoop.so.1.0.0
  zlib: true /lib64/libz.so.1
  snappy: false
  lz4: true revision:99
  bzip2: false

原文地址:https://www.cnblogs.com/riordon/p/4939986.html