ElasticSearch + Logstash + Kibana 日志采集
时间:2022-05-03
本文章向大家介绍ElasticSearch + Logstash + Kibana 日志采集,主要内容包括19.3. TCP/UDP 接收日志并写入 elasticsearch、19.4. 配置 Broker(Redis)、19.4.2. shipper、19.5. Kafka、19.8. FAQ、19.8.2. logstash 无法写入 elasticsearch、基本概念、基础应用、原理机制和需要注意的事项等,并结合实例形式分析了其使用技巧,希望通过本文能帮助到大家理解应用这部分内容。
本文节选自《Netkiller Monitoring 手札》
ElasticSearch + Logstash + Kibana 一键安装
配置 logstash 将本地日志导入到 elasticsearch
input {
file {
type => "syslog"
path => [ "/var/log/maillog", "/var/log/messages", "/var/log/secure" ]
start_position => "beginning"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["127.0.0.1:9200"]
}
}
19.3. TCP/UDP 接收日志并写入 elasticsearch
input {
file {
type => "syslog"
path => [ "/var/log/auth.log", "/var/log/messages", "/var/log/syslog" ]
}
tcp {
port => "5145"
type => "syslog-network"
}
udp {
port => "5145"
type => "syslog-network"
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
}
}
19.4. 配置 Broker(Redis)
19.4.1. indexer
input {/etc/logstash/conf.d/indexer.conf
redis {
host => "127.0.0.1"
port => "6379"
key => "logstash:demo"
data_type => "list"
codec => "json"
type => "logstash-redis-demo"
tags => ["logstashdemo"]
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["127.0.0.1:9200"]
}
}
测试
# redis-cli
127.0.0.1:6379> RPUSH logstash:demo "{"time": "2012-01-01T10:20:00", "message": "logstash demo message"}"
(integer) 1
127.0.0.1:6379> exit
如果执行成功日志如下
# cat /var/log/logstash/logstash-plain.log
[2017-03-22T15:54:36,491][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2017-03-22T15:54:36,496][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-03-22T15:54:36,600][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x20dae6aa URL:http://127.0.0.1:9200/>}
[2017-03-22T15:54:36,601][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-03-22T15:54:36,686][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-03-22T15:54:36,693][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2017-03-22T15:54:36,780][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x2f9efc89 URL://127.0.0.1>]}
[2017-03-22T15:54:36,787][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2017-03-22T15:54:36,792][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@127.0.0.1:6379/0 list:logstash:demo"}
[2017-03-22T15:54:36,793][INFO ][logstash.pipeline ] Pipeline main started
[2017-03-22T15:54:36,838][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-03-22T15:55:10,018][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.
[2017-03-22T15:55:10,024][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
19.4.2. shipper
input {
file {
path => [ "/var/log/nginx/access.log" ]
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{NGINXACCESS}" }
add_field => { "type" => "access" }
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
output {
redis {
host => "127.0.0.1"
port => 6379
data_type => "list"
key => "logstash:demo"
}
}
19.5. Kafka
input {
kafka {
zk_connect => "kafka:2181"
group_id => "logstash"
topic_id => "apache_logs"
consumer_threads => 16
}
}
19.8. FAQ
19.8.1. 查看 Kibana 数据库
# curl 'http://localhost:9200/_search?pretty'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : ".kibana",
"_type" : "config",
"_id" : "5.2.2",
"_score" : 1.0,
"_source" : {
"buildNum" : 14723
}
}
]
}
}
19.8.2. logstash 无法写入 elasticsearch
elasticsearch 的配置不能省略 9200 端口,否则将无法链接elasticsearch
elasticsearch {
hosts => ["127.0.0.1:9200"]
}
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- 二叉树:递归函数究竟什么时候需要返回值,什么时候不要返回值?
- 二叉树:构造二叉树登场!
- RabbitMQ是如何确定消息是否投递到队列中的
- 图文并茂入门一下Git
- python教程 | 最标准的地图调用方式(国家测绘局提供数据)
- Linux的文件和文件系统的管理
- 38.opengl-字体渲染
- 使用Sentinel对Spring MVC接口进行限流
- IDEA Pycharm WebStorm JetBranis全版本 2020年 最新激活方式
- SpringBoot整合MyBatis
- Sublime Text 3解决中文乱码
- pyPI: Python计算热带气旋潜在强度(Potential Intensity, 数据+代码)
- CVE-2019-0808 从空指针解引用到权限提升
- 打卡群刷题总结0926——零钱兑换
- 这样写的代码,都是垃圾......