Hadoop支持LZO
阅读原文时间:2023年07月08日阅读:1

maven(下载安装,配置环境变量,修改sitting.xml加阿里云镜像)
gcc-c++
zlib-devel
autoconf
automake
libtool
通过yum安装即可

yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
tar -zxvf lzo-2.10.tar.gz
cd lzo-2.10
./configure -prefix=/usr/local/hadoop/lzo/
make
make install

2.1 下载hadoop-lzo的源码

下载地址:https://github.com/twitter/hadoop-lzo/archive/master.zip

2.2 解压之后,修改pom.xml

2.7.2

2.3 声明两个临时环境变量

export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
export LIBRARY_PATH=/usr/local/hadoop/lzo/lib

进入hadoop-lzo-master,执行maven编译命令

mvn package -Dmaven.test.skip=true

2.5 进入target,将hadoop-lzo-0.4.21-SNAPSHOT.jar放到hadoop的classpath下

如${HADOOP_HOME}/share/hadoop/common

2.6 修改core-site.xml增加配置支持LZO压缩

io.compression.codecs org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, org.apache.hadoop.io.compress.SnappyCodec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec io.compression.codec.lzo.class com.hadoop.compression.lzo.LzoCodec

手机扫一扫

移动阅读更方便

阿里云服务器
腾讯云服务器
七牛云服务器

你可能感兴趣的文章