flume之监视日志数据抽出到hdfs上示例

今天学习了一个新的轻量级框架&＃xff0c;就是一个的实时日志收集系统&＃xff0c;由cloudera公司开发的框架

下面贴上代码

##对log的完成数据&＃xff0c;进行数据抽取&＃xff0c;并对文件夹下所有文件进行实时监视###define agent
a3.sources &＃61; r3
a3.channels &＃61; c3
a3.sinks &＃61; k3#define sources
a3.sources.r3.type &＃61; spooldir
a3.sources.r3.spoolDir &＃61; /opt/module/cdh/flume-1.5.0-cdh5.3.6/spool_logs
a3.sources.r3.fileSuffix &＃61; .completed
a3.sources.r3.ignorePattern &＃61; ^(.)*\\.tmp$#define channels
a3.channels.c3.type &＃61; file
a3.channels.c3.checkpointDir &＃61; /opt/module/cdh/flume-1.5.0-cdh5.3.6/flume_file/checkpoint
a3.channels.c3.dataDirs &＃61; /opt/module/cdh/flume-1.5.0-cdh5.3.6/flume_file/data#define sinks
a3.sinks.k3.type &＃61; hdfs
a3.sinks.k3.hdfs.useLocalTimeStamp &＃61; true
a3.sinks.k3.hdfs.path &＃61; /user/make/flume/hive_spool_log/%Y-%m-%d
a3.sinks.k3.hdfs.fileType &＃61; DataStream
a3.sinks.k3.hdfs.writeFormat &＃61; Text
a3.sinks.k3.hdfs.batchSize&＃61; 10#bind
a3.sources.r3.channels &＃61; c3
a3.sinks.k3.channel &＃61; c3

上面的具体参数&＃xff0c;以及三大核心的类型&＃xff0c;上官网看&＃xff0c;写得十分详细&＃xff1a;http://flume.apache.org/FlumeUserGuide.html#hdfs-sink

a3.channels.c3.checkpointDir &＃61; /opt/module/cdh/flume-1.5.0-cdh5.3.6/flume_file/checkpoint
a3.channels.c3.dataDirs &＃61; /opt/module/cdh/flume-1.5.0-cdh5.3.6/flume_file/data

这两个参数自己建立的目录

a3.sources.r3.spoolDir &＃61; /opt/module/cdh/flume-1.5.0-cdh5.3.6/spool_logs

这是我们监视的目录

a3.sinks.k3.hdfs.useLocalTimeStamp &＃61; true
a3.sinks.k3.hdfs.path &＃61; /user/make/flume/hive_spool_log/%Y-%m-%d

如果要按照抽取的时间&＃xff0c;自动建立文件夹&＃xff0c;则需要把hdfs.uselocaltimestamp参数设置为true&＃xff0c;自动调用系统当前时间

这是我么运行测试日志之后&＃xff0c;得到的文件夹&＃xff0c;2018-6-19是我们手动修改时间&＃xff0c;形成的文件夹&＃xff0c;所以有两个&＃xff0c;理解得更明显

在来查看&＃xff0c;抽取到hdfs上的数据&＃xff0c;内容跟我们的日志数据一毛一样&＃xff0c;所以就可以对其进行分析&＃xff0c;或者进行日志异常监控&＃xff0c;监控某个值是否异常等等。