八 sqoop 安装

1、准备环境

  sqoop-1.99.7-bin-hadoop200(https://mirror.bit.edu.cn/apache/sqoop/)

2、解压

  

tar zxvf  /opt/software/sqoop-1.99.7-bin-hadoop200.tar.gz  -C /opt/module/

3、配置文件

hadoop配置文件路径

cd /opt/module/sqoop-1.99.7/conf

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/

  改为(此为hadoop 路径)

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/module/hadoop-3.2.1/etc/hadoop

   安全验证方式部分的注释

org.apache.sqoop.security.authentication.type=SIMPLE
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler
org.apache.sqoop.security.authentication.anonymous=true

  

配置hadoop代理访问

因为sqoop访问Hadoop的MapReduce使用的是代理的方式,必须在Hadoop中配置所接受的proxy用户和组。找到Hadoop的core-site.xml配置文件

<property>
<name>hadoop.proxyuser.sqoop2.hosts</name>
<value>root</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.groups</name>
<value>root</value>
</property>

  

创建文件夹

mkdir /opt/module/sqoop-1.99.7/logs
mkdir /opt/module/sqoop-1.99.7/data

  

替换 @LOGDIR@ 及 @BASEDIR@

 4、系统环境变量

修改环境变量:vi /etc/profile

#sqoop
export SQOOP_HOME=/opt/module/sqoop-1.99.7
export PATH=$PATH:$SQOOP_HOME/bin

  

刷新配置:source /etc/profile

5、验证 & 启动

  验证:

sqoop2-tool  verify

   

   启动:

sqoop2-server start

  

6、测试

  执行:sqoop2-shell

 建立hdfs的连接

sqoop:000> create link --connector hdfs-connector
Creating link for connector with name hdfs-connector
Please fill following values to create new link object
Name: hdfs-link01 (#连接名字,必填)

HDFS cluster

URI: hdfs://mycluster (#hdfs的地址,必填)
Conf directory: /opt/module/hadoop-3.2.1/etc/hadoop (#hadoop的配置地址,必填)
Additional configs::
There are currently 0 values in the map:
entry#
New link was successfully created with validation status OK and name hdfs-link01

建立mysql的连接

sqoop:000> create link --connector generic-jdbc-connector
Creating link for connector with name generic-jdbc-connector
Please fill following values to create new link object
Name: jdbc-link01 (#link名称,必填)

Database connection

Driver class: com.mysql.jdbc.Driver (#jdbc驱动类,必填)
Connection String: jdbc:mysql://localhost:3306/dbtest1?userUnicode=true&characterEncoding=UTF8& useSSL=false (# jdbc链接url,必填)
Username: root (#数据库的用户,必填)
Password: **** (#数据库密码,必填)
Fetch Size:
Connection Properties:
There are currently 0 values in the map:
entry#

SQL Dialect

Identifier enclose: (#这里有一个空格,不然会报错)
New link was successfully created with validation status OK and name jdbc-link01

  执行:show link

问题集锦

报错:安装sqoop-1.99.7报caused by java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration   ,配置hadoop路径错误

解决:https://blog.csdn.net/lvyuan1234/article/details/60143224

报错:java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

解决:sqoop/tools/lib/ 、/server/lib/  与 hadoop/share/hadoop/common/lib  中guava 包版本不一致导致,删除低版本,保留高版本

问题:java.lang.SecurityException: sealing violation: package org.apache.derby.impl.jdbc.authentication is sealed

原因:derby 版本过低导致

解决:apache-hive-3.1.2/lib 中 derby-10.14.1.0.jar  复制到 sqoop-1.99.7/server/lib 下

https://blog.csdn.net/debimeng/article/details/101162801

报错

原文地址:https://www.cnblogs.com/qk523/p/12468055.html