运维开发网

JMXtrans + InfluxDB + Grafana实现Kafka性能指标监控

运维开发网 https://www.qedev.com 2020-03-19 09:48 出处:网络 作者:运维开发网整理
架构 一般系统监控通常分为3部分: 数据采集 分析与转换 展现(可视化) 数据采集 对于前端应用,一般需要埋点,对用户的行为进行记录。 如果不埋点,则需要通过Pagespeed、PhantomJS这样的工具去模拟用户行为进行测试。后端的系统通常有自己的性能指标。我们可以通过命令/脚本的方式进行采集。   JMX(Java Management Extensions,即Java管理扩展)是一个为应用

架构

一般系统监控通常分为3部分:

  1. 数据采集

  2. 分析与转换

  3. 展现(可视化)

数据采集

对于前端应用,一般需要埋点,对用户的行为进行记录。 如果不埋点,则需要通过Pagespeed、PhantomJS这样的工具去模拟用户行为进行测试。后端的系统通常有自己的性能指标。我们可以通过命令/脚本的方式进行采集。

  JMX(Java Management Extensions,即Java管理扩展)是一个为应用程序、设备、系统等管理功能的框架,通常可以用来监控和管理Java应用系统。

  Kafka做为一款Java应用,已经定义了丰富的性能指标,(可以参考Kafka监控指标),通过JMX可以轻松对其进行监控。

  

测试 

  首先需要在Kafka上打开JMX

 1.修改${kafka_home}/bin/kafka-server-start.sh , 增加一个JMX_PORT的配置,指定一个端口用于接受外部连接,注意如部署、运行在非root用户下,必须指定1024以上端口

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export JMX_PORT="9999"
fi

2.重启kafka

bin/kafka-server-stop.sh
bin/kafka-server-start.sh config/server.properties &

3.重启后观察可以发现JMX已经启动了

$ps -ef | grep kafka
www       51065      1  0 Nov03 ?        00:08:20 /opt/vdian/java/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/home/www/kafka_2.11-0.10.0.1/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999 -Dkafka.logs.dir=/home/www/kafka_2.11-0.10.0.1/bin/../logs -Dlog4j.configuration=file:bin/../config/log4j.properties -cp :/home/www/kafka_2.11-0.10.0.1/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/argparse4j-0.5.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/connect-api-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/connect-file-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/connect-json-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/connect-runtime-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/guava-18.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/hk2-api-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/hk2-locator-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/hk2-utils-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-annotations-2.6.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-core-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-databind-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javassist-3.18.2-GA.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.annotation-api-1.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.inject-1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.inject-2.4.0-b34.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.servlet-api-3.1.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-client-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-common-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-container-servlet-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-guava-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-media-jaxb-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jersey-server-2.22.2.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/jopt-simple-4.9.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-clients-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-log4j-appender-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-streams-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-streams-examples-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka-tools-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1-sources.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1-test-sources.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/log4j-1.2.17.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/lz4-1.3.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/metrics-core-2.2.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/reflections-0.9.10.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/rocksdbjni-4.8.0.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/scala-library-2.11.8.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/slf4j-api-1.7.21.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/slf4j-log4j12-1.7.21.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/snappy-java-1.1.2.6.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/validation-api-1.1.0.Final.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/zkclient-0.8.jar:/home/www/kafka_2.11-0.10.0.1/bin/../libs/zookeeper-3.4.6.jar kafka.Kafka config/server.properties
 
$ netstat -an | grep 9999
tcp        0      0 :::9999                     :::*                        LISTEN

4.在安装了java的主机上运行jconsole,就会弹出一个控制台,在可以看到MBean中的性能指标

JMXtrans + InfluxDB + Grafana实现Kafka性能指标监控

尾声

Tomcat、JBoss这类的JAVA应用都支持JMX,下面的还用我说么?

0

精彩评论

暂无评论...
验证码 换一张
取 消