分類  >  資料庫 >

裝配HBase Snappy

tags:    時間:2013-12-23 21:49:18
安裝HBase Snappy

伺服器操作系統:CentOS 6.3

[talkyun@tbe192168147058 snappy-1.1.0]# uname -a

Linux tbe192168147058 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[talkyun@tbe192168147058 snappy-1.1.0]# lsb_release -a

Description:    CentOS release 6.3 (Final)

[talkyun@tbe192168147058 snappy-1.1.0]# gcc --version

bash: gcc: command not found

   

一、安裝Snappy

1.檢查是否已經安裝

      ls /usr/local/lib/libsn*

2.安裝Snappy

      2.1.從https://code.google.com/p/snappy/下載

            https://code.google.com/p/snappy/downloads/list

            curl https://snappy.googlecode.com/files/snappy-1.1.0.tar.gz -o snappy-1.1.0.tar.gz

      2.2.安裝

            [talkyun@tbe192168147058 snappy]$ 

            snappy-1.1.0.tar.gz

            tar xvfz snappy-1.1.0.tar.gz

            cd snappy-1.1.0

            [talkyun@tbe192168147058 snappy-1.1.0]$ sudo yum install gcc

            錯誤一:

[talkyun@tbe192168147014 snappy-1.1.0]$ sudo yum install gcc Loaded plugins: fastestmirror, security Determining fastest mirrors Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os error was 14: PYCURL ERROR 6 - "Couldn't resolve host 'mirrorlist.centos.org'" Error: Cannot find a valid baseurl for repo: base [talkyun@tbe192168147014 snappy-1.1.0]$ nslookup mirrorlist.centos.org ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached  [talkyun@tbe192168147014 snappy-1.1.0]$ cat /etc/resolv.conf

            解決方案:

[talkyun@tbe192168147014 snappy-1.1.0]$ sudo vim /etc/resolv.conf [talkyun@tbe192168147014 snappy-1.1.0]$ cat /etc/resolv.conf  # Generated by NetworkManager nameserver 192.168.200.19

 

            [talkyun@tbe192168147058 snappy-1.1.0]$ sudo yum install autoconf

            [talkyun@tbe192168147058 snappy-1.1.0]$ sudo yum install automake

            [talkyun@tbe192168147058 snappy-1.1.0]$ sudo yum install libtool

            [talkyun@tbe192168147058 snappy-1.1.0]$ ./configure

            [talkyun@tbe192168147058 snappy-1.1.0]$ make

            錯誤二:./libtool: line 1125: g++: command not found

            解決方案:sudo yum -y install gcc-c++

            如果出現錯誤,解決之後,要刪除原來的目錄,重新創建和使用新的。

            [talkyun@tbe192168147058 snappy-1.1.0]$ cd ..

            [talkyun@tbe192168147058 snappy]$ rm -rf snappy-1.1.0

            [talkyun@tbe192168147058 snappy]$ tar xvfz snappy-1.1.0.tar.gz

            [talkyun@tbe192168147058 snappy]$ cd snappy-1.1.0

            [talkyun@tbe192168147058 snappy-1.1.0]$ ./configure

            [talkyun@tbe192168147058 snappy-1.1.0]$ make

            [talkyun@tbe192168147058 snappy-1.1.0]$ sudo make install

3.檢查安裝是否成功

      ls /usr/local/lib/libsn*

 

二、安裝Hadoop Snappy

1.下載hadoop-snappy源碼

      svn checkout http://hadoop-snappy.googlecode.com/svn/trunk/ hadoop-snappy

2.安裝hadoop-snappy

      cd hadoop-snappy

      mvn package

      [bigdata@tbe192168147015 hadoop-snappy]$ /home/bigdata/maven/apache-maven-3.0.4/bin/mvn package

      錯誤一:

     [exec] /bin/sh ./libtool --tag=CC   --mode=link gcc -g -Wall -fPIC -O2 -m64 -g -O2 -version-info 0:1:0 -L/usr/local/lib -o libhadoopsnappy.la -rpath /usr/local/lib src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.lo src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.lo  -ljvm -ldl       [exec] /usr/bin/ld: cannot find -ljvm      [exec] collect2: ld returned 1 exit status      [exec] make: *** [libhadoopsnappy.la] Error 1

       解決方案:sudo ln -s /home/bigdata/apps/jdk/jre/lib/amd64/server/libjvm.so /usr/local/lib/

3.複製構建好的庫文件到相應目錄

      注意:libsnappy.so需要依賴libhadoop.so,libsnappy和libhadoop的裝載是緊耦合,因此需要把libhadoop.so一起複制到相應目錄。

      [bigdata@tbe192168147015 hadoop-snappy]$ cd target/hadoop-snappy-0.0.1-SNAPSHOT-tar/hadoop-snappy-0.0.1-SNAPSHOT/

      [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ cp /tmp/libhadoop.so lib/native/Linux-amd64-64/libhadoop.so

      [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ cp -r lib/ /home/bigdata/apps/hadoop-mr-talkyun/

      [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ cp -r lib/ /home/bigdata/apps/hbase-talkyun/ 

 

      [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ scp -r lib/ tbe192168147010:/home/bigdata/apps/hadoop-mr-talkyun/

      [bigdata@tbe192168147015 hadoop-snappy-0.0.1-SNAPSHOT]$ scp -r lib/ tbe192168147010:/home/bigdata/apps/hbase-talkyun/ 

 

三、重啟HDFS、MapReduce、HBase

 

四、檢查安裝是否成功

1.Testing Compression is enabled

hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/testfile snappy

hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/testfile snappy

2.create table

create 'mytable', {NAME => 'colfam', COMPRESSION => 'snappy'}

 

五、參考資料

1.snappy - A fast compressor/decompressor

2.Spark 0.7 installation on EC2 Can't read Snappy Compressed files

3.hadoop-snappy - A Hadoop library of Snappy compression

4.cloudera中hbase使用Snappy演算法安裝及設置

5.Hadoop HBase 配置 安裝 Snappy 終極教程

6.Unable to load native-hadoop library 和 Snappy native library not loaded 的解決

7.mac os上配置CDH4.1.1版hbase並啟用snappy

 

推薦閱讀文章

Bookmark the permalink ,來源:互聯網