基于Flume+Log4j+Kafka的日志采集架构方案(2)

{ "timeMillis" : 1462712870612, "thread" : "main", "level" : "FATAL", "loggerName" : "com.banksteel.log.demo.log4j2.Demo", "message" : "发生了一个可能会影响程序继续运行下去的异常!", "thrown" : { "commonElementCount" : 0, "localizedMessage" : "错误消息啊", "message" : "错误消息啊", "name" : "java.lang.Exception", "extendedStackTrace" : [ { "class" : "com.banksteel.log.demo.log4j2.Demo", "method" : "main", "file" : "Demo.java", "line" : 20, "exact" : true, "location" : "classes/", "version" : "?" }, { "class" : "sun.reflect.NativeMethodAccessorImpl", "method" : "invoke0", "file" : "NativeMethodAccessorImpl.java", "line" : -2, "exact" : false, "location" : "?", "version" : "1.7.0_80" }, { "class" : "sun.reflect.NativeMethodAccessorImpl", "method" : "invoke", "file" : "NativeMethodAccessorImpl.java", "line" : 57, "exact" : false, "location" : "?", "version" : "1.7.0_80" }, { "class" : "sun.reflect.DelegatingMethodAccessorImpl", "method" : "invoke", "file" : "DelegatingMethodAccessorImpl.java", "line" : 43, "exact" : false, "location" : "?", "version" : "1.7.0_80" }, { "class" : "java.lang.reflect.Method", "method" : "invoke", "file" : "Method.java", "line" : 606, "exact" : false, "location" : "?", "version" : "1.7.0_80" }, { "class" : "com.intellij.rt.execution.application.AppMain", "method" : "main", "file" : "AppMain.java", "line" : 144, "exact" : true, "location" : "idea_rt.jar", "version" : "?" } ] }, "endOfBatch" : false, "loggerFqcn" : "org.apache.logging.log4j.spi.AbstractLogger", "source" : { "class" : "com.banksteel.log.demo.log4j2.Demo", "method" : "main", "file" : "Demo.java", "line" : 23 } } 

我们看到,这种格式,无论用什么语言都能轻松解析了。

日志框架的Kafka集成

我们这里只用log4j 1.x 和 log4j 2.x 进行示例。

log4j 1.x 与 Kafka 集成

首先POM.xml的内容如下:

<dependencies> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.17</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-core</artifactId> <version>2.7.4</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.7.4</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-annotations</artifactId> <version>2.7.4</version> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.8.2.1</version> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.11</artifactId> <version>0.8.2.1</version> </dependency> </dependencies> 

注意,我们这里使用的Kafka版本号是0.8.2.1,但是对应0.9.0.1是可以使用的并且0.9.0.1也只能用0.8.2.1才不会发生异常(具体异常可以自己尝试一下)。

而log4j 1.x 本身是没有 JsonLayout 可用的,因此我们需要自己实现一个类,如下所示:

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/34e1564505a6b7a1c6395fcee32dc310.html