sqoop安装配置

注：sqoop只需要在namenode下安装即可
版本:Sqoop-1.4.7

1. 下载解压

• 上官网下载:sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
• 上传. 解压tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /soft/module
• 重命名mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop

2.修改配置文件

• 进入目录cd /soft/module/sqoop/conf/ 拷贝配置文件cp sqoop-env-template.sh sqoop-env.sh 修改配置文件 vim sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/soft/module/hadoop-2.9.2

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/soft/module/hadoop-2.9.2

#set the path to where bin/hbase is available
export HBASE_HOME=/soft/module/hbase

#Set the path to where bin/hive is available
export HIVE_HOME=/soft/module/hive

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/soft/module/zookeeper
export ZOOKEEPER_HOME=/soft/module/zookeeper

• 拷贝jdbc驱动
  cp /soft/mysql-connector-java-5.1.48.jar /soft/sqoop/bin/
  cp /soft/software/mysql-connector-java-5.1.48.jar /soft/module/sqoop/lib

3.验证及使用

• 验证配置:./bin/sqoop help
  出现以下

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

• 测试Sqoop是否能够成功连接数据库
  bin/sqoop list-databases --connect jdbc:mysql://hadoop1:3306/ --username root --password 123456789
  出现以下

information_schema
metastore
mysql
performance_schema
sys

错误:

• sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
  [SQOOP_HOME]/lib/下缺少mysql驱动包

• 注释掉没有使用的组件(解决启动时的警告)

bin/config-sqoop

# Moved to be a runtime check in sqoop.
#if [ ! -d "${HCAT_HOME}" ]; then
#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
#fi

#if [ ! -d "${ACCUMULO_HOME}" ]; then
#  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
#fi