1. 先从官方下载源码：源码下载地址：https://hadoop.apache.org/releases.html，下载 3.1.2 版本
2. 解压缩源码：tar xvf hadoop-3.1.2-src.tar.gz
3. 安装依赖（注：由于我本机环境不是从零开始安装的，部分步骤省略掉）：
3.1 安装并配置 java 环境
3.2 安装并配置 cmake 环境， 可以参照：https://blog.csdn.net/baimafujinji/article/details/78588488
3.3 安装并配置 protobuf 环境 ，可以参照：https://www.jianshu.com/p/0a70eef1bd4a
3.4 安装并配置 openssl 环境：brew install openssl
4. 编译生成安装包：mvn package -Pdist,native -DskipTests=true -Dtar
在编码过程中遇到的问题总结：
问题1 ：cmake 函数库变更引起的问题：
问题描述：
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1227:12: error: no matching function for call to 'getgrouplist' [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups); [WARNING] ^~~~~~~~~~~~ [WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument [WARNING] int getgrouplist(const char *, int, int *, int *);
修改 c 文件：
vim hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c 中 getgrouplist函数调用的 groups参数，强转成(int *)
具体如下所示：
1384 int rc = getgrouplist(user, pw->pw_gid, (int *)groups, &ngroups); 1385 if (rc < 0) { 1386 groups = (gid_t *) alloc_and_clear_memory(ngroups, sizeof(gid_t)); 1387 if (groups == NULL) { 1388 fprintf(ERRORFILE, "Failed to allocate buffer for group lookup for user %s. ", user); 1389 exit(OUT_OF_MEMORY); 1390 } 1391 if (getgrouplist(user, pw->pw_gid, (int * )groups, &ngroups) == -1) { 1392 fprintf(ERRORFILE, "Fail to lookup groups for user %s. ", user); 1393 ret = 2; 1394 } 1395 }
问题2:由于 openssl 环境变量引起的
问题描述：
[WARNING] CMake Error at /Applications/CMake.app/Contents/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:137 (message): [WARNING] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the [WARNING] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR) [WARNING] Call Stack (most recent call first): [WARNING] /Applications/CMake.app/Contents/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE) [WARNING] /Applications/CMake.app/Contents/share/cmake-3.15/Modules/FindOpenSSL.cmake:413 (find_package_handle_standard_args) [WARNING] CMakeLists.txt:24 (find_package) [WARNING] [WARNING] [WARNING] -- Configuring incomplete, errors occurred! [WARNING] See also "/Users/xxx/local/hadoop-3.1.2-src/hadoop-tools/hadoop-pipes/target/native/CMakeFiles/CMakeOutput.log".
由于我本机之前装过 openssl，但配置不完全，解决方案如下：
1. 先查看 openssl 的安装目录：
解决方案：
(base) bogon:hadoop-3.1.2-src johnnybai$ brew list openssl /usr/local/Cellar/openssl/1.0.2o_1/.bottle/etc/ (8 files) /usr/local/Cellar/openssl/1.0.2o_1/bin/c_rehash /usr/local/Cellar/openssl/1.0.2o_1/bin/openssl /usr/local/Cellar/openssl/1.0.2o_1/include/openssl/ (74 files) /usr/local/Cellar/openssl/1.0.2o_1/lib/libcrypto.1.0.0.dylib /usr/local/Cellar/openssl/1.0.2o_1/lib/libssl.1.0.0.dylib /usr/local/Cellar/openssl/1.0.2o_1/lib/engines/ (12 files) /usr/local/Cellar/openssl/1.0.2o_1/lib/pkgconfig/ (3 files) /usr/local/Cellar/openssl/1.0.2o_1/lib/ (4 other files) /usr/local/Cellar/openssl/1.0.2o_1/share/man/ (1680 files)
2. 配置 openssl 环境变量
sudo vim /etc/profile 添加如下环境变量： # openssl export OPENSSL_ROOT_DIR="/usr/local/Cellar/openssl/1.0.2o_1" export OPENSSL_INCLUDE_DIR="/usr/local/Cellar/openssl/1.0.2o_1/include"
编译成功之后，可以在 hadoop-3.1.2-src/hadoop-dist/target/ 目录下找到安装部署binary 文件： hadoop-3.1.2.tar.gz
注： 编译步骤，部分参照了 https://github.com/shawfdong/hyades/wiki/Building-Hadoop-from-source 的做法。