ELK-elkstack-使用消息队列

时间:2022-07-26
本文章向大家介绍ELK-elkstack-使用消息队列，主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项，具有一定的参考价值，需要的朋友可以参考一下。

日志通过logstash收集到redis，之后从logstash从redis读取数据存入到ES

1. logstash使用redis测试

通过标准输入到redis中

logstash配置与启动

 1 [yun@mini03 config]$ pwd
 2 /app/logstash/config
 3 [yun@mini03 config]$ cat redis_test.conf 
 4 input{
 5   stdin{}
 6 }
 7 
 8 filter{
 9 }
10 
11 output{
12   redis {
13     data_type => "list"
14     # 生产环境需要规划
15     db => 1
16     host => "mini03"
17     port => 6379
18     key => "redis_test"
19   }
20 }
21 
22 ### 使用yun用户即可
23 [yun@mini03 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_test.conf 
24 …………
25 111
26 222
27 333
28 444
29 123555
30 1234
31 654321zhags

redis查看

 1 [root@mini03 ~]# redis-cli -h mini03 -p 6379
 2 mini03:6379> select 1
 3 OK
 4 mini03:6379[1]> KEYS *  # 生产环境禁止使用该命令
 5 1) "redis_test"
 6 mini03:6379[1]> type redis_test
 7 list
 8 mini03:6379[1]> llen redis_test
 9 (integer) 7
10 mini03:6379[1]> lindex redis_test -1
11 "{"host":"mini03","message":"654321zhags","@timestamp":"2018-08-29T13:58:02.184Z","@version":"1"}"

2. httpd日志收集到redis中

logstash配置与启动

 1 [yun@mini03 config]$ pwd
 2 /app/logstash/config
 3 [yun@mini03 config]$ cat redis_httpd_test.conf 
 4 input{
 5   file{
 6     path => ["/var/log/httpd/access_log"]
 7     type => "httpd-access-log"
 8     start_position => "beginning"
 9   }
10 }
11 
12 filter{
13 }
14 
15 output{
16   redis {
17     data_type => "list"
18     # 生产环境需要规划
19     db => 1
20     host => "mini03"
21     port => 6379
22     key => "apache-access-log"
23   }
24 }
25 
26 #### 使用root用户，涉及权限
27 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/redis_httpd_test.conf  # 使用root用户

使用谷歌、火狐或者IE浏览器访问

redis查看

[root@mini03 ~]# redis-cli -h mini03 -p 6379
mini03:6379> select 1
OK
mini03:6379[1]> KEYS *
1) "apache-access-log"
2) "redis_test"
mini03:6379[1]> llen apache-access-log
(integer) 28
mini03:6379[1]> lindex apache-access-log -1
"{"message":"10.0.0.1 - - [29/Aug/2018:22:08:30 +0800] \"GET /aaabbb/?aaa=bbb HTTP/1.1\" 404 205 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0\"","type":"httpd-access-log","path":"/var/log/httpd/access_log","host":"mini03","@timestamp":"2018-08-29T14:08:31.442Z","@version":"1"}"

3. logstash从redis读取数据标准输出

注意：该logstash在mini02上读取mini03上redis的数据

读取之后先使用grok进行过滤

之后进行标准输出【命令行输出】

logstash配置与启动

[yun@mini02 config]$ pwd
/app/logstash/config
[yun@mini02 config]$ cat redis_stdout.conf 
input{
  redis {
    data_type => "list"
    db => 1
    host => "mini03"
    port => 6379
    key => "apache-access-log"
  }
}

filter{
  grok {
    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
  }
}

output{
  stdout { codec => rubydebug }
}

###### 使用yun用户即可
[yun@mini02 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_stdout.conf
……………………
{
        "request" => "/noindex/css/fonts/Bold/OpenSans-Bold.ttf",
        "message" => "10.0.0.1 - - [30/Aug/2018:17:22:13 +0800] "GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1" 404 238 "http://mini03/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"",
       "@version" => "1",
          "bytes" => "238",
           "auth" => "-",
       "referrer" => ""http://mini03/noindex/css/open-sans.css"",
       "response" => "404",
           "type" => "httpd-access-log",
       "clientip" => "10.0.0.1",
     "@timestamp" => 2018-08-30T09:22:13.950Z,
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
           "host" => "mini03",
          "agent" => ""Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"",
      "timestamp" => "30/Aug/2018:17:22:13 +0800",
    "httpversion" => "1.1"
}
{
        "request" => "/?refresh=1m&orgId=1",
        "message" => "10.0.0.1 - - [30/Aug/2018:17:22:13 +0800] "GET /?refresh=1m&orgId=1 HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"",
       "@version" => "1",
          "bytes" => "4897",
           "auth" => "-",
       "referrer" => ""-"",
       "response" => "403",
           "type" => "httpd-access-log",
       "clientip" => "10.0.0.1",
     "@timestamp" => 2018-08-30T09:22:13.949Z,
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
           "host" => "mini03",
          "agent" => ""Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"",
      "timestamp" => "30/Aug/2018:17:22:13 +0800",
    "httpversion" => "1.1"
}
……………………

4. elkstack-使用redis作为消息队列【汇总】

在mini03的logstash读取httpd的日志，并存储到redis

4.1. mini03的 logstash配置如下：

 1 [yun@mini03 config]$ pwd
 2 /app/logstash/config
 3 [yun@mini03 config]$ cat redis_httpd_test.conf 
 4 input{
 5   file{
 6     path => ["/var/log/httpd/access_log"]
 7     type => "httpd-access-log"
 8     start_position => "beginning"
 9   }
10 }
11 
12 filter{
13 }
14 
15 output{
16   redis {
17     data_type => "list"
18     # 生产环境需要规划
19     db => 1
20     host => "mini03"
21     port => 6379
22     key => "apache-access-log"
23   }
24 }
25 
26 ######## 使用root用户，涉及权限
27 [root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/redis_httpd_test.conf    
28 ………………

在mini02的logstash读取redis信息，并存储在ES

4.2. mini02的logstash配置

 1 [yun@mini02 config]$ pwd
 2 /app/logstash/config
 3 [yun@mini02 config]$ cat redis_es.conf 
 4 input{
 5   redis {
 6     data_type => "list"
 7     db => 1
 8     host => "mini03"
 9     port => 6379
10     key => "apache-access-log"
11   }
12 }
13 
14 filter{
15   grok {
16     match => { "message" => "%{HTTPD_COMBINEDLOG}" }
17   }
18 }
19 
20 output{
21   # es有3台，随便指定一台即可  也可以是多台如 ["127.0.0.1:9200","127.0.0.2:9200"]
22   elasticsearch {
23     hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
24     index => "httpd-access-log-%{+YYYY.MM.dd}"
25   }
26 }
27 
28 ####### 使用yun用户即可
29 [yun@mini02 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/redis_es.conf  
30 ………………

4.3. 浏览器访问httpd

浏览器

1 # 可以通过谷歌、火狐、IE访问
2 http://mini03/    
3 http://mini03/indweg.html

Linux命令行访问

1 [yun@mini02 ~]$ ab -n40 -c 1 http://mini03/
2 [yun@mini02 ~]$ ab -n40 -c 1 http://mini03/wet/bdhw/

4.4. 信息查看

elasticsearch-head查看

kibana查看

随机文章
本站知识点必读
最近更新