运维开发网

使用supervisor 管理logstash

运维开发网 https://www.qedev.com 2021-04-13 20:01 出处:51CTO 作者:ww51pp
使用supervisor管理logstashsupervisor和logstash此处不做介绍,直接进行操作配置环境准备需要创建一个普通用户,如jianxp以下均使用该用户部署服务,设用户目录为/home/jianxp,安装目录为/opt准备部署包jdk/supervisor/logstash上传至服务器,默认在/tools目录下,解压到/opt目录下。JDK安装若已安装jdk1.8.0,则只需要

使用supervisor 管理logstash

supervisor和logstash此处不做介绍,直接进行操作配置

环境准备

需要创建一个普通用户,如jianxp以下均使用该用户部署服务,设用户目录为 /home/jianxp,安装目录为 /opt

准备部署包 jdk/supervisor/logstash 上传至服务器,默认在 /tools 目录下,解压到/opt目录下。

JDK安装

  • 已安装 jdk1.8.0,则只需要建立已有JDK的软连接,可跳过后面的步骤:

    mkdir -p /opt/java
    ln -sf ${JAVA_HOME} /opt/java/jdk
  • 解压安装包:

    tar zxf jdk-8u191-Linux-x64.tar.gz -C /opt
    ln -s /opt/jdk1.8.0_191 /opt/jdk
  • 添加环境变量:

    编辑文件 vi ~/.bash_profile,追加如下环境变量:

    export JAVA_HOME=/opt/jdk
    export PATH=$JAVA_HOME/bin:$PATH
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
  • 执行命令 source ~/.bash_profile 使环境变量生效。

  • 执行命令 java -version,若输出 java version "1.8.0_191",则说明jdk安装成功

Supervisor安装

  • 离线安装supervisor,解压安装包 tar -zxf /tools/supervisor-3.3.4.tar -C /opt/,依次执行下列步骤

    cd /opt/supervisor-3.3.4
    tar -zxf setuptools-32.3.1.tar.gz && cd setuptools-32.3.1
    python setup.py install --user
    cd .. && tar -zxf meld3-1.0.2.tar.gz && cd meld3-1.0.2
    python setup.py install --user
    cd .. && tar -zxf supervisor-3.3.4.tar.gz && cd supervisor-3.3.4
    python setup.py install --user
  • 编辑supervisor配置文件

    mkdir -p /opt/supervisor && cd /opt/supervisor && vi supervisord.conf

    supervisord.conf 文件内容如下

    [unix_http_server]
    file=/opt/supervisor/supervisor.sock
    ; (the path to the socket file)
    chmod=0700
    ; sockef file mode (default 0700)
    
    [supervisord]
    logfile=/opt/supervisor/supervisord.log
    ;  (main log file;default $CWD/supervisord.log)
    pidfile=/opt/supervisor/supervisord.pid
    ; (supervisord pidfile;default supervisord.pid)
    childlogdir=/opt/supervisor
    ; ('AUTO' child log dir, default $TEMP)
    
    ; the below section must remain in the config file for RPC
    ; (supervisorctl/web interface) to work, additional interfaces may be
    ; added by defining them in separate rpcinterface: sections
    
    [rpcinterface:supervisor]
    supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface
    
    [supervisorctl]
    serverurl=unix:///opt/supervisor/supervisor.sock
    ; use a unix:// URL  for a unix socket
    
    ; The [include] section can just contain the "files" setting.  This
    ; setting can list multiple files (separated by whitespace or
    ; newlines).  It can also contain wildcards.  The filenames are
    ; interpreted as relative to this file.  Included files *cannot*
    ; include files themselves.
    
    [include]
    files = /opt/supervisor/conf.d/*.conf
  • 创建配置文件里涉及到的目录

    • 日志目录

    mkdir -p /opt/supervisor && mkdir -p /opt/supervisor

    • 管理进程的配置文件目录:

    mkdir -p /opt/supervisor/conf.d

  • 建立软连接

    mkdir -p ~/.local/etc
    ln -s /opt/supervisor/supervisord.conf ~/.local/etc/
  • 至此,supervisor安装完毕

    • 执行 supervisod 即可启动supervisor
    • 执行 supervisorctl status,无报错则说明启动成功。

    logstash 安装

解压安装包,添加软连接

tar -zxf /tools/logstash-6.8.9-fix.tar.gz -C /opt/
ln -s /opt/logstash-6.8.9 /opt/logstash

配置logstash用supervisor管理(logstash的所有配置均在supervisor管理的配置文件中完成

编辑配置文件,执行 cd /opt/supervisor/conf.d && vi logstash.conf

; logstash config file

[program:logstash]
command=/opt/logstash-6.8.9/bin/logstash 
directory=/opt/logstash-6.8.9/
user=jianxp
numprocs=1
stdout_logfile=/opt/logstash/logs/stderr.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=1
redirect_stderr=true
autostart=true
autorestart=true
startsecs=1
stopwaitsecs=1
killasgroup=true
priority=1

说明
command=/opt/logstash-6.8.9/bin/logstash   这个没有加-f等参数,默认读config/pipelines.yaml文件,此文件也是后续要配置的。

在/opt/logstash-6.8.9/config/conf.d 创建两个config文件,作为logstash的pipeline配置

vim config/conf.d/test.conf
input{
file{
    path =>"/opt/logstash_test/test.log"
    start_position=>"beginning" 
}
}
output { 
file {
    path =>"/opt/logstash_test/output.log"
} 
}

vim config/conf.d/file.conf
input{
file{
    path =>"/opt/logstash_test/test.log"
    start_position=>"beginning"

}
}

output { 
file {
    path =>"/opt/logstash_test/output33.log"
} 
}

上面两个pipeline时比较简单的,以文件输入和和文件输出为例子,两个pipeline都是以文件test.log为共同输入,然后输出到不同的文件中。

 编辑config/pipelines.yaml 文件,添加如下内容

 - pipeline.id: file
   pipeline.workers: 1
   pipeline.batch.size: 1
   path.config: "/opt/logstash-6.8.9/config/conf.d/file.conf"

 - pipeline.id: test
   pipeline.workers: 1
   pipeline.batch.size: 1
   path.config: "/opt/logstash-6.8.9/config/conf.d/test.conf"

上面的logstash的pipeline配置完成后,并且supervisor目录下的logstash.conf也配置完成,既可以使用supervisor启动logstash

supervisorctl update
supervisorctl status

在/opt/logstash-6.8.9/logs 目录下,可以查看logstash输出的日志,通过日志可以看到,Pipelines running count=2,表示启动两个pipeline。

可以开启另外一个shell窗口,向test.log文件中增加内容,log中也会输出Opening file 等信息。

[2021-03-10T16:18:04,953][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"test", :thread=>"#<Thread:0x24bd4c10 run>"}
[2021-03-10T16:18:04,958][INFO ][logstash.runner          ] Logstash shut down.
[2021-03-10T16:18:42,012][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.9"}
[2021-03-10T16:18:47,360][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"file", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50}
[2021-03-10T16:18:47,721][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"test", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50}
[2021-03-10T16:18:47,940][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/opt/logstash-6.8.9/data/plugins/inputs/file/.sincedb_513b5c3d29a45c373b60caa3e09938d7", :path=>["/opt/logstash_test/test.log"]}
[2021-03-10T16:18:47,940][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/opt/logstash-6.8.9/data/plugins/inputs/file/.sincedb_513b5c3d29a45c373b60caa3e09938d7", :path=>["/opt/logstash_test/test.log"]}
[2021-03-10T16:18:47,992][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"file", :thread=>"#<Thread:0x5c0e2cd6 run>"}
[2021-03-10T16:18:47,993][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"test", :thread=>"#<Thread:0x6626055f run>"}
[2021-03-10T16:18:48,074][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2021-03-10T16:18:48,074][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2021-03-10T16:18:48,108][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:file, :test], :non_running_pipelines=>[]}
[2021-03-10T16:18:48,531][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-03-10T16:19:11,774][INFO ][logstash.outputs.file    ] Opening file {:path=>"/opt/logstash_test/output.log"}
[2021-03-10T16:19:11,776][INFO ][logstash.outputs.file    ] Opening file {:path=>"/opt/logstash_test/output33.log"}

查看输出的文件

[jianxp@localhost logstash_test]$ ll
总用量 16
-rw-rw-r--. 1 jianxp jianxp  335 3月  10 15:23 aa
-rw-r--r--. 1 jianxp jianxp 3338 3月  10 16:19 output33.log
-rw-r--r--. 1 jianxp jianxp 3338 3月  10 16:19 output.log
-rw-rw-r--. 1 jianxp jianxp 1340 3月  10 16:19 test.log
0

精彩评论

暂无评论...
验证码 换一张
取 消