请问有没有离线的 nginx 日志分析的工具

[LxnChan]

由于我的网站访问人数越来越多，nginx 的日志大小也在指数级增长，想问一下大家有没有 nginx 日志分析的工具，即我将 nginx 日志复制出来，然后通过该工具进行数据的相关分析（错误类型、各地区 IP 访问量等）

[hasdream]

goaccess

[julyclyde]

如果真的是指数级那估计是被攻击了
应该线性增长才对啊

[seers]

一般都是用 prometheus

[bjzhush]

不用这么麻烦，直接 awk sort 几个组合一下，按 IP 排序访问数量就知道了

[tonlmy]

logstalgia

[LxnChan]

@hasdream @seers @tonlmy 谢谢，稍后我会逐个尝试。
@julyclyde 就是因为 nginx 日志异常增长且量大没法手动分析了才打算找的分析工具🤣，我现在也不是很清楚是被打了还是真的有那么高的访问量

[PolarBears]

默认格式就 goaccess 吧,如果有条件的话还是建议配置 nginx 日志格式为 json 格式然后丢 elasticsearch 上分析吧

[tiga99]

如果资源足够，可以将日志写到 es ，用 grafana 做展示；参考博客： https://www.xiaoleizhang.com/index.php/archives/120/

如果资源不足或者不想这么复杂，可以将 nginx 重新编译一下，加上 nginx-module-vts 和 geoip2 模块，大概效果如下：
```text
# HELP nginx_vts_filter_bytes_total The request/response bytes
# TYPE nginx_vts_filter_bytes_total counter
# HELP nginx_vts_filter_requests_total The requests counter
# TYPE nginx_vts_filter_requests_total counter
# HELP nginx_vts_filter_request_seconds_total The request processing time in seconds counter
# TYPE nginx_vts_filter_request_seconds_total counter
# HELP nginx_vts_filter_request_seconds The average of request processing times in seconds
# TYPE nginx_vts_filter_request_seconds gauge
# HELP nginx_vts_filter_request_duration_seconds The histogram of request processing time
# TYPE nginx_vts_filter_request_duration_seconds histogram
# HELP nginx_vts_filter_cache_total The requests cache counter
# TYPE nginx_vts_filter_cache_total counter
nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="unknow",direction="in"} 7332314
nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="unknow",direction="out"} 504487933
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="1xx"} 0
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="2xx"} 25577
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="3xx"} 6
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="4xx"} 4
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="5xx"} 0
nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="unknow"} 0.000
nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="unknow"} 0.000
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="miss"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="bypass"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="expired"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="stale"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="updating"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="revalidated"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="hit"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="scarce"} 0
nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="四川省",direction="in"} 3647380
nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="四川省",direction="out"} 5444493
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="1xx"} 0
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="2xx"} 12939
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="3xx"} 0
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="4xx"} 6441
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="5xx"} 0
nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="四川省"} 0.000
nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="四川省"} 0.000
nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.100"} 19380
nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.200"} 19380
nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.500"} 19380
nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="1.000"} 19380
nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="3.000"} 19380
nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="5.000"} 19380
nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="10.000"} 19380
nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="+Inf"} 19380
nginx_vts_filter_request_duration_seconds_sum{filter="a.example.com",filter_name="四川省"} 0.000
nginx_vts_filter_request_duration_seconds_count{filter="a.example.com",filter_name="四川省"} 19380
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="miss"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="bypass"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="expired"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="stale"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="updating"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="revalidated"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="hit"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="scarce"} 0
nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="overseas",direction="in"} 2160
nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="overseas",direction="out"} 4680
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="1xx"} 0
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="2xx"} 18
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="3xx"} 0
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="4xx"} 0
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="5xx"} 0
nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="overseas"} 0.000
nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="overseas"} 0.000
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="miss"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="bypass"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="expired"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="stale"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="updating"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="revalidated"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="hit"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="scarce"} 0
nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="江西",direction="in"} 14156529
nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="江西",direction="out"} 19690820
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="1xx"} 0
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="2xx"} 74493
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="3xx"} 0
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="4xx"} 1000
nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="5xx"} 0
nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="江西"} 0.000
nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="江西"} 0.000
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="miss"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="bypass"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="expired"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="stale"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="updating"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="revalidated"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="hit"} 0
nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="scarce"} 0
```

[davidyin]

Awstat

[eroko]

我们用的是 loki

[LxnChan]

@tiga99 这个我看了一下，资源消耗好像真的有点大🤣，不过还是谢谢了

[LxnChan]

@davidyin @eroko 谢谢，稍后我会逐个尝试。

[changdig]

我是简单写了个 py 脚本去处理 nginx 日志然后写库或者写 csv 文件

[killva4624]

elk 或者 loki ，把日志结构化入库分析。