mac下编译Hadoop

以前都是win下装个centos虚拟机编译hadoop，最近换了Mac，不用再装虚拟机了，直接编译Hadoop，很爽。但需要配置下hadoop的编译环境。在此记录下。

findbugs安装

下载findbugs-3.0.1.tar.gz，解压tar -zvxf findbugs-3.0.1.tar.gz，配置环境变量。
在.bash_profile中添加FINDBUGS_HOME，并export

1
2
3

FINDBUGS_HOME=/Users/hadoop/soft/findbugs-3.0.1
PATH=$FINDBUGS_HOME/bin
export FINDBUGS_HOME

protobuf安装

安装protobuf之前验证下是否有gcc，命令gcc --version
进入protobuf-2.5.0目录执行./configure --prefix=/Users/hadoop/soft/protobuf-2.5.0
然后安装就可以了，命令make && make install
为了使用protoc命令方便，将bin目录添加到.bash_porfile中
PATH=/Users/hadoop/soft/protobuf-2.5.0/bin

cmake安装

解压cmake-3.0.0，进入目录中，执行./bootstrap --prefix=/Users/hadoop/soft/cmake-3.0.0进行编译
执行make && make install进行
为了使用cmake命令方便，将bin目录添加到.bash_porfile中
PATH=/Users/hadoop/soft/cmake-3.0.0/bin

mac环境中由于已经安装了xcode-select则，不用额外安装zlib-devel

openssl-devel安装

这个在mac上没有这个依赖包，只是装了openssl，但是编译不过去，报错，内容如下：

1
2

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec dir="/Users/hadoop/bigdata/hadoop-2.6.0-src-eclipse/hadoop-tools/hadoop-pipes/target/native" executable="cmake" failonerror="true">... @ 5:141 in /Users/hadoop/bigdata/hadoop-2.6.0-src-eclipse/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

网上大部分都说是缺少openssl-devel依赖包，但苦逼的是mac上没有此依赖包，然后就下载了openssl源码进行自编译安装吧，可是就是编译不过去，下载了各个版本的openssl就是无法编译成功。然后看error信息，貌似是和ant和cmake有关，hadoop现在的版本已经不用ant了呀，而且contos上的编译环境也没有ant呀，难道是cmake的原因？首先想到cmake版本问题，换了个新版也不成功，最后快要绝望时搜到了一篇文章

上面说可以去build-main.xml中查看具体出错的行，用cmake单独执行下，看报什么错

当我执行cmake /Users/hadoop/bigdata/hadoop-2.6.0-src-eclipse/hadoop-tools/hadoop-pipes/src/ -DJVM_ARCH_DATA_MODEL = 64时，错误信息显示出来了，

Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the System variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR), suggesting that not found openssl suggested that the need to add environment variables

然后我就按照文章中介绍的将OPENSSL_INCLUDE_DIR和OPENSSL_ROOT_DIRexport出，再次编译就成功了。

由于我并不知道mac默认的openssl的安装目录，所以我使用brew install openssl重新安装了下，安装成功会提示openssl的安装目录

brew install openssl
==> Downloading https://homebrew.bintray.com/bottles/openssl-1.0.2k.sierra.bottle.tar.gz
Already downloaded: /Users/netease/Library/Caches/Homebrew/openssl-1.0.2k.sierra.bottle.tar.gz
==> Pouring openssl-1.0.2k.sierra.bottle.tar.gz
==> Using the sandbox
==> Caveats
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
  /usr/local/etc/openssl/certs

and run
  /usr/local/opt/openssl/bin/c_rehash

This formula is keg-only, which means it was not symlinked into /usr/local.

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

If you need to have this software first in your PATH run:
  echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.zshrc

For compilers to find this software you may need to set:
    LDFLAGS:  -L/usr/local/opt/openssl/lib
    CPPFLAGS: -I/usr/local/opt/openssl/include
For pkg-config to find this software you may need to set:
    PKG_CONFIG_PATH: /usr/local/opt/openssl/lib/pkgconfig

==> Summary
🍺  /usr/local/Cellar/openssl/1.0.2k: 1,696 files, 12MB


If you need to have this software first in your PATH run:
  echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.zshrc

编译hadoop

所需的依赖组件都安装好之后就可以正确的编译hadoop了，在hadoop_home目录下执行命令
mvn clean package -DskipTests -Pdist,native -Dtar -X

这里还有个小插曲，就是会报个找不类的错，

1	Exception in thread “main” Java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()

解决办法就是在JAVA_HOME，也即/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home路径下新建目录Class，并执行创建软符号链接命令：
sudo ln -s $JAVA_HOME/lib/tools.jar $JAVA_HOME/Classes/classes.jar