ELK学习实验014:Nginx日志JSON格式收集
阅读原文时间:2022年04月02日阅读:1

1 Kibana的显示配置

https://demo.elastic.co/app/kibana#/dashboard/welcome_dashboard

环境先处理干净

安装nginx和httpd-tools

2 使用压测工具产生日志

[root@node4 ~]# ab -n 100 -c 100 http://192.168.132.134/

This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.132.134 (be patient)…..done

Server Software: nginx/1.16.1
Server Hostname: 192.168.132.134
Server Port: 80

Document Path: /
Document Length: 612 bytes

Concurrency Level: 100
Time taken for tests: 0.011 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 84500 bytes
HTML transferred: 61200 bytes
Requests per second: 9071.12 [#/sec] (mean)
Time per request: 11.024 [ms] (mean)
Time per request: 0.110 [ms] (mean, across all concurrent requests)
Transfer rate: 7485.44 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 3 0.5 3 3
Processing: 2 3 1.1 2 5
Waiting: 0 3 1.2 2 5
Total: 4 6 0.7 5 7
WARNING: The median and mean for the total time are not within a normal deviation
These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
50% 5
66% 6
75% 6
80% 7
90% 7
95% 7
98% 7
99% 7
100% 7 (longest request)

[root@node4 ~]# tail -f /usr/local/nginx/logs/access.log

192.168.132.181 - - [18/Jan/2020:21:47:23 -0500] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3"
192.168.132.181 - - [18/Jan/2020:21:47:23 -0500] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3"
192.168.132.181 - - [18/Jan/2020:21:47:23 -0500] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3"

3 安装filebeat

[root@node4 src]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-x86_64.rpm

[root@node4 src]# rpm -ivh filebeat-7.5.1-x86_64.rpm

[root@node4 src]# rpm -qc filebeat

[root@node4 src]# cd /etc/filebeat/

[root@node4 filebeat]# cp filebeat.yml /opt/

[root@node4 filebeat]# grep -Ev "^$|[#;]" filebeat.yml

filebeat.inputs:

  • type: log enabled: true paths:
    • /usr/local/nginx/logs/access.log
      filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
      setup.template.settings:
      index.number_of_shards: 1
      setup.kibana:
      output.elasticsearch:
      hosts: ["192.168.132.131:9200","192.168.132.132:9200","192.168.132.133:9200"]
      processors:
    • add_host_metadata: ~
    • add_cloud_metadata: ~
    • add_docker_metadata: ~
    • add_kubernetes_metadata: ~

[root@node4 filebeat]# systemctl restart filebeat

使用kibana查看

访问:http://192.168.132.131:5601/app/kibana

对于一些基本的操作,再前面的学习中已经演示,参看前面的学习实验

4 配置nginx的日志改为json格式

log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status"}';

添加日志格式

访问后

{"@timestamp":"2020-01-18T23:33:57-05:00","host":"192.168.132.134","clientip":"192.168.132.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.132.134","url":"/index.html","domain":"192.168.132.134","xff":"-","referer":"-","status":"304"}

安装jq工具

[root@node4 filebeat]# yum -y install jq

[root@node4 ~]# tail -f /usr/local/nginx/logs/access.log |jq .

但是读取的依然是整个的message日志

5 filebeat解析成json格式

[root@node4 ~]# grep -Ev "^$|[#;]" /etc/filebeat/filebeat.yml

filebeat.inputs:

  • type: log enabled: true paths:
    • /usr/local/nginx/logs/access.log
      json.key_under_root: true
      json.overwrite_keys: true
      filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
      setup.template.settings:
      index.number_of_shards: 1
      setup.kibana:
      output.elasticsearch:
      hosts: ["192.168.132.131:9200","192.168.132.132:9200","192.168.132.133:9200"]
      processors:
    • add_host_metadata: ~
    • add_cloud_metadata: ~
    • add_docker_metadata: ~
    • add_kubernetes_metadata: ~

[root@node4 ~]# systemctl restart filebeat

得到格式的日志

ES原始数据

{
"_index": "filebeat-7.5.1",
"_type": "_doc",
"_id": "oq03vG8BcnOPLK2r_2wa",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2020-01-19T05:13:02.912Z",
"json": {
"upstreamtime": "-",
"host": "192.168.132.134",
"@timestamp": "2020-01-19T00:13:01-05:00",
"Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36",
"clientip": "192.168.132.1",
"domain": "192.168.132.134",
"http_host": "192.168.132.134",
"status": "304",
"referer": "-",
"xff": "-",
"responsetime": 0,
"url": "/index.html",
"size": 0,
"upstreamhost": "-"
},
"input": {
"type": "log"
},
"ecs": {
"version": "1.1.0"
},
"host": {
"name": "node4",
"hostname": "node4",
"architecture": "x86_64",
"os": {
"platform": "centos",
"version": "7 (Core)",
"family": "redhat",
"name": "CentOS Linux",
"kernel": "3.10.0-957.27.2.el7.x86_64",
"codename": "Core"
},
"id": "a833bbe7e3634c75ab1b427c4049d056",
"containerized": false
},
"agent": {
"ephemeral_id": "bab448ec-726f-48bf-aa62-0b8fe39c2684",
"hostname": "node4",
"id": "3d856c84-4859-4b20-a25f-4b6c1a8a8108",
"version": "7.5.1",
"type": "filebeat"
},
"log": {
"file": {
"path": "/usr/local/nginx/logs/access.log"
},
"offset": 15601
}
}
}

Kibana显示

使用json格式收集nginx的日志介绍到这里

手机扫一扫

移动阅读更方便

阿里云服务器
腾讯云服务器
七牛云服务器