测试必会之 Linux 三剑客之 grep

grep (global search regular expression(RE) and print out the line,全面搜索正则表达式并把行打印出来)是一种强大的文本搜索工具，它能使用正则表达式搜索文本，并把匹配的行打印出来

可用于数据查找定位

先列举出测试工作常用的grep命令和意义：

grep pattern file
grep -i pattern file 忽略大小写
grep -v pattern file 不显示匹配行
grep -o pattern file 只把每个匹配的内容独立的行显示
grep -E pattern file 使用拓展正则表达式,相当于egrep
#注意：grep 'a[0-9]{10}' 等同于 grep -E 'a[0-9]{10}' 
grep  -B, --before-context=NUM  打印以文本起始的NUM 行
grep  -A, --after-context=NUM   打印以文本结尾的NUM 行
grep  -C, --context=NUM         打印输出文本NUM 行
grep pattern -r dir/ 递归搜索
grep -m1 匹配匹配中的第一个
grep -n 顺便输出行号
grep -P 可以让grep使用perl的正则表达式语法

BRE基本正则表达式：

^开头 $结尾
[a-z] [0-9]区间，如果开头带有^表示不能匹配区间内的元素
*0个或多个
.表示任意字符

ERE扩展正则表达式

ERE扩展表达式是基本正则表达式（BRE）基础上的扩展

？非贪婪匹配
+ 一个或多个
() 分组
{} 范围约束
| 匹配多个表达式的任意一个

话不多说，先举例

假设有如下文件，名称为nginx.log

40.77.192.252 - - [05/Dec/2018:00:08:51 +0000] "GET /uploads/photo/2017/ed39a6f0-8111-48bc-88e0-604939b1dca1.png!large HTTP/1.1" 200 10065 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b" 0.000 0.000 .
220.181.108.187 - - [05/Dec/2018:00:09:04 +0000] "GET /topics/15275?locale=en HTTP/1.1" 200 28527 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 0.090 0.090 .
116.25.42.125 - - [05/Dec/2018:00:09:17 +0000] "GET /cable HTTP/1.1" 101 1473 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3622.0 Safari/537.36" 108.100 108.100 .
123.127.112.18 - - [05/Dec/2018:00:09:18 +0000] "GET /cable HTTP/1.1" 101 1017 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" 70.577 70.577 .
139.180.132.174 - - [05/Dec/2018:00:09:20 +0000] "GET /bbs.zip HTTP/1.1" 404 1264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.011 0.011 .
139.180.132.174 - - [05/Dec/2018:00:09:12 +0000] "GET /__zep__/js.zip HTTP/1.1" 500 2183 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.018 0.018 .
141.8.142.131 - - [05/Dec/2018:00:09:12 +0000] "GET /topics/14442 HTTP/1.1" 200 21980 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 0.195 0.195 .
220.181.108.181 - - [05/Dec/2018:00:09:13 +0000] "GET /syyair/following?locale=zh-CN HTTP/1.1" 200 13355 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 0.035 0.035 .
139.180.132.174 - - [05/Dec/2018:00:09:13 +0000] "GET /admin.rar HTTP/1.1" 401 50 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.006 0.006 .
40.77.167.1 - - [05/Dec/2018:00:09:17 +0000] "GET /topics/2547/replies/17243/reply_suggest HTTP/1.1" 404 3530 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 0.014 0.014 .
139.180.132.174 - - [05/Dec/2018:00:09:17 +0000] "GET /admin.tar.gz HTTP/1.1" 404 1264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.052 0.052 .

找出500错误时候的上下文

chenshifengdeMacBook-Pro:~ chenshifeng$ grep -A 2 ' 500 ' nginx.log    #显示500错误及其后2行
139.180.132.174 - - [05/Dec/2018:00:09:12 +0000] "GET /__zep__/js.zip HTTP/1.1" 500 2183 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.018 0.018 .
141.8.142.131 - - [05/Dec/2018:00:09:12 +0000] "GET /topics/14442 HTTP/1.1" 200 21980 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 0.195 0.195 .
220.181.108.181 - - [05/Dec/2018:00:09:13 +0000] "GET /syyair/following?locale=zh-CN HTTP/1.1" 200 13355 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 0.035 0.035 .
chenshifengdeMacBook-Pro:~ chenshifeng$ grep -B 2 ' 500 ' nginx.log    #显示500错误及其前2行
123.127.112.18 - - [05/Dec/2018:00:09:18 +0000] "GET /cable HTTP/1.1" 101 1017 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" 70.577 70.577 .
139.180.132.174 - - [05/Dec/2018:00:09:20 +0000] "GET /bbs.zip HTTP/1.1" 404 1264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.011 0.011 .
139.180.132.174 - - [05/Dec/2018:00:09:12 +0000] "GET /__zep__/js.zip HTTP/1.1" 500 2183 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.018 0.018 .
chenshifengdeMacBook-Pro:~ chenshifeng$ grep -C 2 ' 500 ' nginx.log    #显示500错误及其前后2行
123.127.112.18 - - [05/Dec/2018:00:09:18 +0000] "GET /cable HTTP/1.1" 101 1017 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" 70.577 70.577 .
139.180.132.174 - - [05/Dec/2018:00:09:20 +0000] "GET /bbs.zip HTTP/1.1" 404 1264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.011 0.011 .
139.180.132.174 - - [05/Dec/2018:00:09:12 +0000] "GET /__zep__/js.zip HTTP/1.1" 500 2183 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.018 0.018 .
141.8.142.131 - - [05/Dec/2018:00:09:12 +0000] "GET /topics/14442 HTTP/1.1" 200 21980 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 0.195 0.195 .
220.181.108.181 - - [05/Dec/2018:00:09:13 +0000] "GET /syyair/following?locale=zh-CN HTTP/1.1" 200 13355 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 0.035 0.035 .

1、先访问博客主页，利用 grep href 过滤出所有包含 url 的内容。命令：

chenshifengdeMacBook-Pro:~ chenshifeng$ curl -s https://www.cnblogs.com/feng0815/ | grep -o "https:[^"]*"
https://www.cnblogs.com/feng0815/rss
https://www.cnblogs.com/feng0815/rsd.xml
https://www.cnblogs.com/feng0815/wlwmanifest.xml
https://common.cnblogs.com/scripts/jquery-2.2.0.min.js
https://mathjax.cnblogs.com/2_7_5/MathJax.js?config=TeX-AMS-MML_HTMLorMML&amp;v=20200504
https://www.cnblogs.com/
https://news.cnblogs.com/
https://q.cnblogs.com/
https://brands.cnblogs.com/
https://ing.cnblogs.com/
https://edu.cnblogs.com/
https://zzk.cnblogs.com/s
https://passport.cnblogs.com/GetBlogApplyStatus.aspx
https://msg.cnblogs.com/
https://home.cnblogs.com/
https://passport.cnblogs.com/GetBlogApplyStatus.aspx
https://home.cnblogs.com/
https://account.cnblogs.com/settings/account
https://account.cnblogs.com/signup/
......

再用grep命令匹配"200 OK"作为判断条件，筛选出成功的url并打印，然后将失败的 url 加上 ERR 标记也一起打印出来。

chenshifengdeMacBook-Pro:~ chenshifeng$ curl -s https://www.cnblogs.com/feng0815/ | grep -o "https:[^"]*"| while read line;do curl -s -I $line | grep ' 200 ' && echo $line || echo ERR $line;done
ERR https://www.cnblogs.com/feng0815/rss
HTTP/1.1 200 OK
https://www.cnblogs.com/feng0815/rsd.xml
HTTP/1.1 200 OK
https://www.cnblogs.com/feng0815/wlwmanifest.xml
HTTP/1.1 200 OK
https://common.cnblogs.com/scripts/jquery-2.2.0.min.js
HTTP/1.1 200 OK
https://mathjax.cnblogs.com/2_7_5/MathJax.js?config=TeX-AMS-MML_HTMLorMML&amp;v=20200504
HTTP/1.1 200 OK
https://www.cnblogs.com/
HTTP/1.1 200 OK
https://news.cnblogs.com/
HTTP/1.1 200 OK
https://q.cnblogs.com/
HTTP/1.1 200 OK
https://brands.cnblogs.com/
HTTP/1.1 200 OK
https://ing.cnblogs.com/
HTTP/1.1 200 OK
https://edu.cnblogs.com/
HTTP/1.1 200 OK
https://zzk.cnblogs.com/s
ERR https://passport.cnblogs.com/GetBlogApplyStatus.aspx
ERR https://msg.cnblogs.com/
HTTP/1.1 200 OK
https://home.cnblogs.com/
ERR https://passport.cnblogs.com/GetBlogApplyStatus.aspx
......