分类 工作 下的文章

RAID

RAID(Redundant Array of Independent Disk 独立冗余磁盘阵列)技术

RAID0

RAID1

RAID5

RAID10

goaccess访问日志分析(apache/nginx)

官方man page

分析的日志格式

COMBINED     | Combined Log Format
VCOMBINED    | Combined Log Format with Virtual Host
COMMON       | Common Log Format
VCOMMON      | Common Log Format with Virtual Host
W3C          | W3C Extended Log File Format
SQUID        | Native Squid Log Format
CLOUDFRONT   | Amazon CloudFront Web Distribution
CLOUDSTORAGE | Google Cloud Storage
AWSELB       | Amazon Elastic Load Balancing
AWSS3        | Amazon Simple Storage Service (S3)

终端分析

  • goaccess使用首先要就是配置LOG/DATE/TIME FORMAT这三个参数, 通过--time-format, --log-format, --date-format选项配置
// nginx(common格式)
goaccess -f access.log --log-format="%h %^[%d:%t %^] \"%r\" %s %b \"%R\" \"%u\"" --time-format="%T" --date-format="%d/%b/%Y"

// apache(log format使用common格式, 可以到httpd.conf中查看)
goaccess -f access_apache.log --log-format="%h %^[%d:%t %^] \"%r\" %s %b" --time-format="%T" --date-format="%d/%b/%Y"

// 如果确认日志是何种格式的话(比如COMMON), 也可以这样写:
goaccess -f access_apache.log --time-format="COMMON" --date-format="COMMON" --log-format="COMMON"

// 读取配置文件
goaccess -p "配置文件" -f access.log
  • 配置文件格式列子:
// ~/.goaccessrc
time-format COMMON
date-format COMMON
log-format COMMON

导出文件

goaccess -f access.log -p "配置文件" -o [report.html|report.json|report.csv]

多个压缩的日志文件分析

zcat -f access.log* | goaccess

其他技巧

// 按日期查找时间段
sed -n "/14\/Jul\/2015:00:00:00/,/15\/Jul\/2015:15:00:00/ p" access.log > time_access.log

// 查找504错误的页面和数量
awk '($9 ~ /504/)' time_access.log | awk '{print $7}' | sort | uniq -c | sort -rn > 504.log

// 查找访问最多的20个IP及访问次数
awk '{print $1}' time_access.log | sort | uniq -c | sort -n -https://www.zhorz.pw/admin/write-post.php?cid=81#wmd-previewk 1 -r | head -n 20 > top.log

对于log-format中的字符表示什么含义,官方的man page有详细的解析

%x A date and time field matching the time-format and date-format variables. This is used when a timestamp is given instead of the date and time being in two separate variables.
%t time field matching the time-format variable.
%d date field matching the date-format variable.
%v The server name according to the canonical name setting (Server Blocks or Virtual Host).
%e This is the userid of the person requesting the document as determined by HTTP authentication.
%h host (the client IP address, either IPv4 or IPv6)
%r The request line from the client. This requires specific delimiters around the request (as single quotes, double quotes, or anything else) to be parsable. If not, we have to use a combination of special format specifiers as %m %U %H.
%q The query string.
%m The request method.
%U The URL path requested.
%H The request protocol.
%s The status code that the server sends back to the client.
%b The size of the object returned to the client.
%R The "Referer" HTTP request header.
%u The user-agent HTTP request header.
%D The time taken to serve the request, in microseconds.
%T The time taken to serve the request, in seconds with milliseconds resolution.
%L The time taken to serve the request, in milliseconds as a decimal number.
%^ Ignore this field.
%~ Move forward through the log string until a non-space (!isspace) char is found.
|  Vertical pipe or bar is used for either the character prior to the vertical pipe or followed by the vertical pipe.

mysql小技巧

  1. 灵活使用 sum + case
  2. 灵活使用 select 语句中的子查询

curl返回错误

错误描述:

GnuTLS recv error (-9): A TLS packet with unexpected length was received. 

如何解决?
暂无头绪...

git提交中断

切换分支的时候一直提示错误:

ignoring broken ref refs remotes origin head

暂时不清楚如何解决, 只能重新checkout一份