prometheus-nginxlog-exporter构建Nginx日志监控
时间:2022-07-22
本文章向大家介绍prometheus-nginxlog-exporter构建Nginx日志监控,主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项,具有一定的参考价值,需要的朋友可以参考一下。
lunimous 可以开启 dashborad
rgw 通过 prometheus-nginxlog-exporter 可以解析 Nginx 日志,还有一些正则可以去处理,然后通过 relabel
的配置,让其在指标数据上打上标签。
civetweb 没有在日志记录请求返回的时间,所以考虑用 Nginx 来替换,为了能给 Ceph RGW 加一个请求相关的监控,所以需要去解析日志,在 prometheus-nginxlog-exporter 中,通过配置文件,可以把标签和端口等设置好,然后让 prometheus 加上这个 endpoint 就可以拉出来数据了。
listen {
port = 4040
}
enable_experimental = true
namespace "nginx" {
source = {
files = [
"/var/log/nginx/access.log"
]
}
format = "$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$request_time""
labels {
app = "default"
}
#relabel "request" {
# from = "request"
#}
relabel "bucket" {
from = "request"
split = 2
match "^/general__lingqu/.*" {
replacement = "general__lingqu"
}
}
}
Nginx 日志。
xx.xx.xxx.x - - [26/Jun/2020:01:23:37 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/_SUCCESS HTTP/1.1" 200 0 "0.001"
xx.xx.xxx.x - - [26/Jun/2020:01:23:43 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00000 HTTP/1.1" 200 1165212977 "6.006"
xx.xx.xxx.x - - [26/Jun/2020:01:23:49 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00001 HTTP/1.1" 200 1180678766 "6.130"
xx.xx.xxx.x - - [26/Jun/2020:01:23:55 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00002 HTTP/1.1" 200 1181307932 "6.049"
xx.xx.xxx.x - - [26/Jun/2020:01:24:01 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00003 HTTP/1.1" 200 1207342395 "6.138"
xx.xx.xxx.x - - [26/Jun/2020:01:24:08 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00004 HTTP/1.1" 200 1198727695 "6.080"
xx.xx.xxx.x - - [26/Jun/2020:01:24:14 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00005 HTTP/1.1" 200 1206446826 "6.137"
xx.xx.xxx.x - - [26/Jun/2020:01:24:20 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00006 HTTP/1.1" 200 1204772146 "6.144"
xx.xx.xxx.x - - [26/Jun/2020:01:24:26 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00007 HTTP/1.1" 200 1165911854 "5.985"
xx.xx.xxx.x - - [26/Jun/2020:01:24:32 +0800] "GET /general__lingqu/kp/out/390280/n1590907415729/107916/20200601220441/part-00008 HTTP/1.1" 200 1179420268 "6.077"
prometheus-nginxlog-exporter 采集到的指标。
# HELP nginx_http_response_count_total Amount of processed HTTP requests
# TYPE nginx_http_response_count_total counter
nginx_http_response_count_total{app="default",bucket="",method="GET",status="200"} 1
nginx_http_response_count_total{app="default",bucket="general__lingqu",method="DELETE",status="204"} 2
nginx_http_response_count_total{app="default",bucket="general__lingqu",method="GET",status="200"} 214
nginx_http_response_count_total{app="default",bucket="general__lingqu",method="HEAD",status="200"} 8474
nginx_http_response_count_total{app="default",bucket="general__lingqu",method="PUT",status="200"} 5
# HELP nginx_http_response_size_bytes Total amount of transferred bytes
# TYPE nginx_http_response_size_bytes counter
nginx_http_response_size_bytes{app="default",bucket="",method="GET",status="200"} 338
nginx_http_response_size_bytes{app="default",bucket="general__lingqu",method="DELETE",status="204"} 0
nginx_http_response_size_bytes{app="default",bucket="general__lingqu",method="GET",status="200"} 2.1549611919e+10
nginx_http_response_size_bytes{app="default",bucket="general__lingqu",method="HEAD",status="200"} 0
nginx_http_response_size_bytes{app="default",bucket="general__lingqu",method="PUT",status="200"} 0
# HELP nginx_http_response_time_seconds Time needed by NGINX to handle requests
# TYPE nginx_http_response_time_seconds summary
nginx_http_response_time_seconds{app="default",bucket="",method="GET",status="200",quantile="0.5"} NaN
nginx_http_response_time_seconds{app="default",bucket="",method="GET",status="200",quantile="0.9"} NaN
nginx_http_response_time_seconds{app="default",bucket="",method="GET",status="200",quantile="0.99"} NaN
nginx_http_response_time_seconds_sum{app="default",bucket="",method="GET",status="200"} 0.002
nginx_http_response_time_seconds_count{app="default",bucket="",method="GET",status="200"} 1
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="DELETE",status="204",quantile="0.5"} 0.002
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="DELETE",status="204",quantile="0.9"} 0.009
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="DELETE",status="204",quantile="0.99"} 0.009
nginx_http_response_time_seconds_sum{app="default",bucket="general__lingqu",method="DELETE",status="204"} 0.011
nginx_http_response_time_seconds_count{app="default",bucket="general__lingqu",method="DELETE",status="204"} 2
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="GET",status="200",quantile="0.5"} 0.014
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="GET",status="200",quantile="0.9"} 0.112
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="GET",status="200",quantile="0.99"} 7.037
nginx_http_response_time_seconds_sum{app="default",bucket="general__lingqu",method="GET",status="200"} 115.89000000000006
nginx_http_response_time_seconds_count{app="default",bucket="general__lingqu",method="GET",status="200"} 214
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="HEAD",status="200",quantile="0.5"} 0.002
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="HEAD",status="200",quantile="0.9"} 0.002
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="HEAD",status="200",quantile="0.99"} 0.024
nginx_http_response_time_seconds_sum{app="default",bucket="general__lingqu",method="HEAD",status="200"} 22.99799999999861
nginx_http_response_time_seconds_count{app="default",bucket="general__lingqu",method="HEAD",status="200"} 8474
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="PUT",status="200",quantile="0.5"} NaN
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="PUT",status="200",quantile="0.9"} NaN
nginx_http_response_time_seconds{app="default",bucket="general__lingqu",method="PUT",status="200",quantile="0.99"} NaN
nginx_http_response_time_seconds_sum{app="default",bucket="general__lingqu",method="PUT",status="200"} 0.839
nginx_http_response_time_seconds_count{app="default",bucket="general__lingqu",method="PUT",status="200"} 5
Reference
- https://github.com/martin-helmich/prometheus-nginxlog-exporter
- https://www.martin-helmich.de/en/blog/monitoring-nginx.html
- 【go语言】Goroutines 并发模式(一)
- 【go语言】Goroutines 并发模式(二)
- 从零开始搭建Nginx和Tomcat的web集群环境
- 【翻译】为什么 goroutine 的栈内存无穷大?
- 从PowerVM,KVM到Docker:存储池的配置与调优---第一篇终结(第3子篇)
- goroutine背后的系统知识
- 从PowerVM,KVM到Docker:存储池的配置与调优---第一篇(第2子篇)
- 2017最后一天的学习-TensorFlow
- 从PowerVM,KVM到Docker:存储池的配置与调优---第一篇(第1子篇)
- 36小时,造一个亚马逊无人商店 | 实战教程+代码
- 使用Echarts来实现数据可视化
- 手把手教你用TensorFlow实现看图说话 | 教程+代码
- 自动化平台开发的几点总结
- epoll入门
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- jointplot快速探究两组变量的分布及关系
- linux中alarm函数的实例讲解
- 如何给Linux虚拟机连上WiFi详解
- Linux系统查看CPU、机器型号、内存等信息
- 实战基本的Linux sed命令示例代码
- 在Linux中如何查看可用的网络接口详解
- Linux程序运行时加载动态库失败的解决方法
- Linux/CentOS系统同步网络时间的2种方法详解
- Linux 常用命令之Linux more命令使用方法
- Ubuntu下VIM配置成C++开发编辑器
- Ubuntu环境下使用G++编译CPP文件
- Linux下Tomcat的几种运行方式讲解
- linux中chmod命令用法详解
- Linux强制释放占用端口以及Linux防火墙端口开放方法详解
- CentOS7.5从零安装Python3.6.6的教程详解