2017-09-26 ShangHai Company
WebSite: flume.apache.org
Flume is a distributed, reliable and available service for efficiently collecting, aggregating, and moving large amounts of log data.
It has a simple and flexible architecture based on streaming data flows.
It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.
It uses a simple extensible data model that allows for online analytic application.
环境搭建：
从官网下载 -bin version
使用解压缩命令
$ tar -zxvf apache-flume-[version]
进入到 解压后的文件目录下
Create your own properties file based on the working template (or create from scratch)
$ cp conf/flume-conf.propertires.template conf/flume.conf
Configuration
Flume uses a java property file based configuration format .
-f <file> option
官方文档给出了 Source, 可以配置的各种数据源， Channel 多种配置及组合， Sink 的多样性配置。
测试：
The flume-ng executable lets you run a Flume NG agent or an Avro client which is useful for tesing and experiments.
need to specify a command (e.g. agent or avro-client) and a conf directory ( --conf <conf dir>)
all other options are command-specific.
To start the flume server using the flume.conf above:
bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1
重要的配置 Agent (required) 全部必须
| Option | Description |
| --conf-file,-f <file> | Indicates which configuration file you want to run with |
| --name, -n <agetname> | Indicates the name of agent on which we`re running |
Avro schema 二进制 或 JSON 格式为 Schema 配 二进制或 JSON
|
Option |
Description |
| --host, -H <hostname> | Specifies the hostname of the Flume agent |
| --port, -p <port> | Specifies the port on which the Avro source is listening |
| --filename, -F <filename> | Sends each line of <filename> to Flume (optional) |
| --headerFile, -F <file> | Header file containing headers as key/value pairs on each new line |
in a new window type the following:
$ bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F /etc/passwd -Dflume.root.logger=DEBUG,console
各项配置无误的话， Apache Flume running;