HTTP metadata数据

信息元位置

信息元名称

信息元ID

信息元描述

1

MetadataVersion

5000

Metadata版本号

当前版本号为1.0

2

MetadataID

1019

MetadataID

3

sourceIPv4Address or

sourceIPv6Address

8 or

27

源IP(IPv4或者IPv6)

4

destinationIPv4Address or

destinationIPv6Address

12 or

28

目的IP(IPv4或者IPv6)

5

sourceTransportPort

7

源端口

6

destinationTransportPort

11

目的端口

7

protocolIdentifier

4

IP协议:TCP:6,UDP:17,ICMP:1

8

octetDeltaCount

1

一条流的上行字节数(C2S)

9

packetDeltaCount

2

一条流的上行报文数(C2S)

10

postOctetDeltaCount

23

一条流下行报文字节数(S2C)

11

postPacketDeltaCount

24

一条流下行报文数(S2C)

12

flowStartSeconds

150

流第一个包的绝对时间戳(秒)

13

flowEndSecond

151

流最后一个包的绝对时间戳(秒)

26

applicationProtocolName

2000

应用协议名称:HTTP、HTTPS、SSL等

定义参见《知识库-类别应用协议ID》

27

applicationName

96

一个应用的名称:

28

applicationCategoryName

372

应用大类

29

applicationSubCategoryName

373

应用小类

30

httpMethod

2001

操作类型:get、post等

31

httpRequestHostName

2003

请求中的Host name

32

httpResponseCode

2004

应答状态码,200、404等

33

httpContentType

2007

实体头用于向接收方指示实体的介质类型,指定HEAD方法送到接收方的实体介质类型,或GET方法发送的请求介质类型Content-Range实体头

34

httpRefer

2009

协议中的Referer域

35

httpRequestUserAgent

2010

协议中的User-Agent域,用户的浏览器类型,发起http请求的应用名称、版本,这个是从请求信息中提取的

36

httpRequestURL

2055

http请求URL

37

httpRequestLabelNum

2057

http请求的标签数目

38

httpReplyLabelNum

2058

http应答的标签数目

39

httpRequestVersion

2059

http请求的版本号

40

httpReplyVersion

2060

http应答的版本号

41

fileName

2047

HTTP上传或者下载的文件的文件名;

或者邮件(STMP/POP3/IMAP4)附件文件名

42

fileEncrypt

2048

HTTP上传、下载文件或者邮件(STMP/POP3/IMAP4)附件的文件类型,是否加密0X00:未加密

0X01:加密

43

fileType

2049

HTTP上传下载文件或者邮件(STMP/POP3/IMAP4)附件文件的文件格式,支持的文件格式包括:

常见可执行文件类型(exe、msi、rpm、ocx、com、a、so、out、elf、dll、sys) 2、常见文档类文件类型(doc、ppt、xls、dot、pot、xlt、pps、docx、docm、dotx、dotm、pptx、pptm、potx、potm、ppsx、ppsm、xlsx、xlsm、xltx、xltm、pdf、vsd、mpp、ods、odt、odp、eml、uof) 3、常见压缩类文件类型(rar、tar、zip、gzip、cab、bz2) 4、常见视频类文件类型(mdi、mov、mpeg、mpg、avi、rmvb、asf、swf、mp3、mp4、wmv、wma、midi) 5、常见图片类文件类型(png、tif、wmf、bmp、gif、jpeg、jpg) 7、HTML类型(chm) 9、其他类型(torrent、crt) 10、支持:enc-doc、enc-xls、enc-ppt、enc-office2007、enc-zip、Zcompressed、enc-rar、lha、bat、cmd、flv、hta、iso、pgp、pif、pl、reg、rtf、sh、dsm、dwg、edif、lnk、lzh、mkv、tdb、wsf、apk、mdb、ico、inf、jar、psd、accdb、cat、nsf、odg、odb、res、sln、vcproj、ilk、pdb、obj、cap、ttf、fon、enc_pdf、7zip、pcap、odt

44

fileSize

2050

HTTP上传下载文件或者邮件(STMP/POP3/IMAP4)附件文件的大小(bytes)

45

fileMd5

2100

HTTP上传下载文件或者邮件(STMP/POP3/IMAP4)附件文件的md5

66

SrcIPUser

字段含义:SrcIP对应用户信息

数据类型:String

字段最大长度:256

67

DestIPUser

字段含义:DestIP对应用户信息

数据类型:String

字段最大长度:256

68

SrcGeographyLocationCountryOrRegion

字段含义:源IP所在国家或地区

数据类型:String

字段最大长度:128

69

SrcGeographyLocationCity

字段含义:源IP所在城市

数据类型:String

字段最大长度:128

70

SrcGeographyLocationLongitude

字段含义:源IP所在经度

数据类型:String

字段最大长度:128

71

SrcGeographyLocationLatitude

字段含义:源IP所在纬度

数据类型:String

字段最大长度:128

72

DestGeographyLocationCountryOrRegion

字段含义:目的IP所在国家或地区

数据类型:String

字段最大长度:128

73

DestGeographyLocationCity

字段含义:目的IP所在城市

数据类型:String

字段最大长度:128

74

DestGeographyLocationLongitude

字段含义:目的IP对应经度

数据类型:String

字段最大长度:128

75

DestGeographyLocationLatitude

字段含义:目的IP对应纬度

数据类型:String

字段最大长度:128

82

requestLength

2061

requestLength

表示http请求长度

83

replyLength

2062

replyLength

表示http应答长度

84

httpCookie

2063

httpCookie

表示Cookie内容

85

httpLabelList

2064

httpLabelList

表示http请求标签清单,各标签定义之间使用;号间隔

86

httpSelfDefineLabel

2065

httpSelfDefineLabel

表示http请求自定义标签

102

httpRequesHead

2066

http请求头,Base64编码,编码后长度最大1K字节

103

httpRequestBody

2067

http请求体,Base64编码,编码后长度最大1K字节

104

httpReplyHead

2068

http响应头,Base64编码,编码后长度最大1K字节

105

httpReplyBody

2069

http响应体,Base64编码,编码后长度最大512字节

附1:HTTP请求标签编码,在Metadata中用编号代替对应的标签

编号

名称

示例值

字段类型

字段用途备注说明

1

Accept

"text/html, image/*"

请求头字段

告诉WEB服务器自己接受什么介质类型,*/* 表示任何类型,type/* 表示该类型下的所有子类型,type/sub-type。

2

Accept-Charset

"iso8859-5"

请求头字段

浏览器申明自己接收的字符集

。。。。

GET示例数据:

6^c55e5b337860185e01000000^162.105.162.78^61.158.132.76^51597^80^6^874^3^141^3^1530099808^1530099808^^^^^^^^^^^^^HTTP^Thunder^General_Internet^FileShare_P2P^GET^vod076.t37.lixian.vip.xunlei.com^403^text/plain; charset=utf-8^http://dynamic.lixian.vip.xunlei.com/user_task?userid=332577465&st=0&p=3^Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729)^http://vod076.t37.lixian.vip.xunlei.com/download?fid=Ika3UDwJJcr8KQQR3bOBGMkA8bUTkgEAAAAAAH9DTq62A8iCBKQJReBkHSFGK006&mid=666&threshold=150&tid=8F6A00F2D2D8AD9F74D499C542060CC3&srcid=4&verno=1&g=7F434EAEB603C88204A40945E0641D21462B4D3A&scn=c11&i=5CAA3D48C2972069FD440F7980A299C1FC818D3D&t=6&ui=332577465&ti=496130517894721&s=102931&m=0&n=015304CA3869660000&ih=5CAA3D48C2972069FD440F7980A299C1FC818D3D&fi=19&pi=496130517698049&ff=0&co=CD45256D12A08460B3458A2D7F63FAED&cm=1&pk=lixian&ak=1:1:6:4&e=1450442190&ms=10485760&c&k=1&ts=1530100575^6^4^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530100600648^pku162^^^^^^^^CN^Zhengzhou^113.5325^34.683594^^^^^2018-06-27T11:56:40.648^^874^141^^7;9;26;39;40;51^^TCP^^^^^^dc:fe:18:34:36:13^61.158.132.76^^^^^^^^Q2FjaGUtQ29udHJvbDogbm8tY2FjaGUNCkNvbm5lY3Rpb246IGtlZXAtYWxpdmUNCkhvc3Q6IHZvZDA3Ni50MzcubGl4aWFuLnZpcC54dW5sZWkuY29tDQpSYW5nZTogYnl0ZXM9MC0xMDI5MzANClJlZmVyZXI6IGh0dHA6Ly9keW5hbWljLmxpeGlhbi52aXAueHVubGVpLmNvbS91c2VyX3Rhc2s/dXNlcmlkPTMzMjU3NzQ2NSZzdD0wJnA9Mw0KVXNlci1BZ2VudDogTW96aWxsYS80LjAgKGNvbXBhdGlibGU7IE1TSUUgNy4wOyBXaW5kb3dzIE5UIDYuMjsgV09XNjQ7IFRyaWRlbnQvNy4wOyAuTkVUNC4wRTsgLk5FVDQuMEM7IC5ORVQgQ0xSIDMuNS4zMDcyOTsgLk5FVCBDTFIgMi4wLjUwNzI3OyAuTkVUIENMUiAzLjAuMzA3MjkpDQoNCg==^^SGRzZXJ2ZXItRXJyb3I6IA0KRGF0ZTogV2VkLCAyNyBKdW4gMjAxOCAxMTo1Njo1OCBHTVQNCkNvbnRlbnQtTGVuZ3RoOiAwDQpDb250ZW50LVR5cGU6IHRleHQvcGxhaW47IGNoYXJzZXQ9dXRmLTgNCg0K^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

POST示例数据:

5^c8a85b337bf6114301000000^10.2.211.234^14.17.41.197^35890^8080^6^941^6^443^4^1530100726^1530100726^^^^^^^^^^^^^HTTP^HTTP^Network^Infrastructure^POST^14.17.41.197:8080^200^application/multipart-formdata^^Mozilla/5.0 (Linux; U; Android 8.0.0; zh-cn; MI 6 Build/OPR1.170623.027) AppleWebKit/533.1 (KHTML, like Gecko) Mobile Safari/533.1^http://14.17.41.197:8080/?tk=0915289f9055fa5778b36a8c6193e411f6b65c3b72bb76ec4d1bfc7f599e9269323f6e7f03b881db21133b1bf2ae5bc5&iv=ba88a89747d8e7af&encrypt=17^10^8^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530100600575^总区域^^^^^^^^CN^Guangzhou^113.25^23.1167^^^^^2018-06-27T11:56:40.575^^941^219^^Q-UA2;Q-GUID;QQ-S-ZIP;1;51;9;3;21;14;26^Q-UA2: QV=3&PL=ADR&PR=TRD&PP=com.shanbay.words&PPVN=8.0.302&TBSVC=43500&CO=BK&COVC=043613&PB=GE&VE=GA&DE=PHONE&CHID=0&LCID=9422&MO= MI6 &RL=1080*1920&OS=8.0.0&API=26;_Q-GUID: 6A53E0FFE1E66FE944D2D577AC72A30E0AA8B223C1F04164DCB59DA5CD97D0F8;_QQ-S-ZIP: gzip^TCP^^^^^^ec:d0:9f:dd:d6:7b^14.17.41.197^^^^^^^^US1VQTI6IFFWPTMmUEw9QURSJlBSPVRSRCZQUD1jb20uc2hhbmJheS53b3JkcyZQUFZOPTguMC4zMDImVEJTVkM9NDM1MDAmQ089QksmQ09WQz0wNDM2MTMmUEI9R0UmVkU9R0EmREU9UEhPTkUmQ0hJRD0wJkxDSUQ9OTQyMiZNTz0gTUk2ICZSTD0xMDgwKjE5MjAmT1M9OC4wLjAmQVBJPTI2DQpRLUdVSUQ6IDZBNTNFMEZGRTFFNjZGRTk0NEQyRDU3N0FDNzJBMzBFMEFBOEIyMjNDMUYwNDE2NERDQjU5REE1Q0Q5N0QwRjgNClFRLVMtWklQOiBnemlwDQpBY2NlcHQ6ICovKg0KVXNlci1BZ2VudDogTW96aWxsYS81LjAgKExpbnV4OyBVOyBBbmRyb2lkIDguMC4wOyB6aC1jbjsgTUkgNiBCdWlsZC9PUFIxLjE3MDYyMy4wMjcpIEFwcGxlV2ViS2l0LzUzMy4xIChLSFRNTCwgbGlrZSBHZWNrbykgTW9iaWxlIFNhZmFyaS81MzMuMQ0KQ29ubmVjdGlvbjogQ2xvc2UNCkFjY2VwdC1FbmNvZGluZzogZ3ppcA0KQ29udGVudC1UeXBlOiBhcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQNCkNvbnRlbnQtTGVuZ3RoOiAyNDANCkhvc3Q6IDE0LjE3LjQxLjE5Nzo4MDgwDQoNCg==^2olt5FIKWdK4t07bNKITmTRJP6pja04WOz4Gi2suxjQOigfz+sUJ33aT4/qHy8z1GLQ5a5WNp12HmJMrjWsQIMNLEMb/ENldvBKM5I+I1HPlXxYwZUB+B/AhKqCrj8RQTKZbYMZIi47B0mJtRYf2msK3S3S3y2zvdjB75KjyIjWJRUq3Q0KfOV/kyf1WubjpOC8HOdI0kHEw/Oph90TT8Yxsmv45i7wdWS9rE1XYy5HRcflnmHt5tdHCJymI+WsJl4tg6PFuzNWEekPpec08CJlhjuBJCq4a+voSWGOeeU7tWGuX7SJ3gUE3QwaMtSdc^Q2FjaGUtQ29udHJvbDogbm8tY2FjaGUNCkNvbm5lY3Rpb246IGNsb3NlDQpDb250ZW50LUxlbmd0aDogMjI0DQpDb250ZW50LVR5cGU6IGFwcGxpY2F0aW9uL211bHRpcGFydC1mb3JtZGF0YQ0KRGF0ZTogV2VkLCAyNyBKdW4gMjAxOCAxMTo1Njo1OSBHTVQNClFRLVMtRW5jcnlwdDogMTcNClFRLVMtWklQOiBnemlwDQpTZXJ2ZXI6IFFCU2VydmVyDQoNCg==^VoW5Cb18ouvhUQI+tJW58ZbiVW58uQEtVf5b4bzpSLA7REqIi0SoWzgYsIOeGInGgNj78tQCJQjfc6cORO+gN6gpMaC9cqzjhQzL4iEor8YQq4S3KvyQ6OlXaQuvatGbr92M3jzKrjutpT6M4RSoZRq8OQJsFScqRHl1SJNoA2xRIMZIHE2S0ErgtMoT1hoKEO3iwhu41nrAHHx9gou9FPjr2MeKK8ScuNvfENI6m5jPW1+ABFpiFXffnfIuKqRNCr3CMe1sun9jnOtrqrr0/CX33Uepb4/jo5jWPuVO2e0=^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ddos攻击时候,脚本:

#!/bin/bash
x=1
while [ $x -ge 0 ]
do
  echo "Welcome $x times"
  wget http://www.pku.edu.cn/campuslife/index.htm
  x=$(( $x + 1 ))
done

采集到的数据:

6^80725b35edb0184f01000000^61.148.244.168^162.105.131.196^7203^80^6^320^4^1368^4^1530260912^1530260916^^^^^^^^^^^^^HTTP^Wget^General_Internet^File_Sharing^GET^www.pku.edu.cn^200^text/html; charset=UTF-8^^Wget/1.19.2 (darwin16.7.0)^http://www.pku.edu.cn/campuslife/index.htm^5^9^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530261627439^^重要服务器^^^CN^Beijing^116.388306^39.928894^^^^^^^^^2018-06-29T08:40:27.439^^160^310^^51;1;3;26;9^^TCP^^^^^^61.148.244.168^162.105.131.196^^^^^^^^VXNlci1BZ2VudDogV2dldC8xLjE5LjIgKGRhcndpbjE2LjcuMCkNCkFjY2VwdDogKi8qDQpBY2NlcHQtRW5jb2Rpbmc6IGd6aXANCkhvc3Q6IHd3dy5wa3UuZWR1LmNuDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQoNCg==^^RGF0ZTogRnJpLCAyOSBKdW4gMjAxOCAwODowNDo0OSBHTVQNClNlcnZlcjogQXBhY2hlLzIuMi4xNSAoQ2VudE9TKQ0KTGFzdC1Nb2RpZmllZDogVGh1LCAyNCBNYXkgMjAxOCAwMjowMToyMyBHTVQNCkVUYWc6ICI0MTExMy03M2VkLTU2Y2VhMDc0OTdhODAiDQpBY2NlcHQtUmFuZ2VzOiBieXRlcw0KQ29udGVudC1MZW5ndGg6IDI5Njc3DQpLZWVwLUFsaXZlOiB0aW1lb3V0PTE1LCBtYXg9MTAwDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD1VVEYtOA0KDQo=^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

6^e1c75b35edf2184102000000^61.148.244.168^162.105.131.196^9200^80^6^^^^^1530260978^1530260982^^^^^^^^^^^^^HTTP^HTTP^^^GET^^^^^^http://www.pku.edu.cn/campuslife/index.htm^^^^^index.htm^0^HTML^28508^faf301c63c5d5a7419f05053d990ad14^^^^^^^^^^^^^^^^^^1530261692961^^重要服务器^^^CN^Beijing^116.388306^39.928894^^^^^^^1^^2018-06-29T08:41:32.961^^^^^^^TCP^^^^^^61.148.244.168^162.105.131.196^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

使用hulk,命令:python2 hulk.py http://www.pku.edu.cn/research/index.htm safe

采集到的数据:

6^2dea5b35ea4e185501000000^61.148.244.168^162.105.131.196^29768^80^6^337^4^1368^3^1530260046^1530260051^^^^^^^^^^^^^HTTP^AOL_Video^Entertainment^Online_Media^GET^www.pku.edu.cn^200^text/html; charset=UTF-8^http://engadget.search.aol.com/search?q=KWBBTBLTE^Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)^http://www.pku.edu.cn/research/index.htm?PYWWL=ETUIC^8^8^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530260762731^^重要服务器^^^CN^Beijing^116.388306^39.928894^^^^^^^^^2018-06-29T08:26:02.731^^337^272^^3;26;56;51;2;9;40;7^^TCP^^^^^^61.148.244.168^162.105.131.196^^^^^^^^QWNjZXB0LUVuY29kaW5nOiBpZGVudGl0eQ0KSG9zdDogd3d3LnBrdS5lZHUuY24NCktlZXAtQWxpdmU6IDExNg0KVXNlci1BZ2VudDogTW96aWxsYS81LjAgKFdpbmRvd3M7IFU7IE1TSUUgNy4wOyBXaW5kb3dzIE5UIDYuMDsgZW4tVVMpDQpBY2NlcHQtQ2hhcnNldDogSVNPLTg4NTktMSx1dGYtODtxPTAuNywqO3E9MC43DQpDb25uZWN0aW9uOiBjbG9zZQ0KUmVmZXJlcjogaHR0cDovL2VuZ2FkZ2V0LnNlYXJjaC5hb2wuY29tL3NlYXJjaD9xPUtXQkJUQkxURQ0KQ2FjaGUtQ29udHJvbDogbm8tY2FjaGUNCg0K^^RGF0ZTogRnJpLCAyOSBKdW4gMjAxOCAwNzo1MDoyNSBHTVQNClNlcnZlcjogQXBhY2hlLzIuMi4xNSAoQ2VudE9TKQ0KTGFzdC1Nb2RpZmllZDogVGh1LCAyNCBNYXkgMjAxOCAwMjowMToyNSBHTVQNCkVUYWc6ICI0MzM4Mi03NzI1LTU2Y2VhMDc1Y2E0YzAiDQpBY2NlcHQtUmFuZ2VzOiBieXRlcw0KQ29udGVudC1MZW5ndGg6IDMwNTAxDQpDb25uZWN0aW9uOiBjbG9zZQ0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9VVRGLTgNCg0K^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

6^e6e95b35ea53187601000000^61.148.244.168^162.105.131.196^29791^80^6^378^5^1368^4^1530260051^1530260058^^^^^^^^^^^^^HTTP^HTTP^Network^Infrastructure^GET^www.pku.edu.cn^200^text/html; charset=UTF-8^http://www.pku.edu.cn/KWMPOXSOSZ^Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.1 (KHTML, like Gecko) Chrome/4.0.219.6 Safari/532.1^http://www.pku.edu.cn/research/index.htm?JJMGHXS=FVP^8^8^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530260770023^^重要服务器^^^CN^Beijing^116.388306^39.928894^^^^^^^^^2018-06-29T08:26:10.023^^378^272^^3;26;56;51;2;9;40;7^^TCP^^^^^^61.148.244.168^162.105.131.196^^^^^^^^QWNjZXB0LUVuY29kaW5nOiBpZGVudGl0eQ0KSG9zdDogd3d3LnBrdS5lZHUuY24NCktlZXAtQWxpdmU6IDExNQ0KVXNlci1BZ2VudDogTW96aWxsYS81LjAgKFdpbmRvd3M7IFU7IFdpbmRvd3MgTlQgNS4xOyBlbi1VUykgQXBwbGVXZWJLaXQvNTMyLjEgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvNC4wLjIxOS42IFNhZmFyaS81MzIuMQ0KQWNjZXB0LUNoYXJzZXQ6IElTTy04ODU5LTEsdXRmLTg7cT0wLjcsKjtxPTAuNw0KQ29ubmVjdGlvbjogY2xvc2UNClJlZmVyZXI6IGh0dHA6Ly93d3cucGt1LmVkdS5jbi9LV01QT1hTT1NaDQpDYWNoZS1Db250cm9sOiBuby1jYWNoZQ0KDQo=^^RGF0ZTogRnJpLCAyOSBKdW4gMjAxOCAwNzo1MDozMiBHTVQNClNlcnZlcjogQXBhY2hlLzIuMi4xNSAoQ2VudE9TKQ0KTGFzdC1Nb2RpZmllZDogVGh1LCAyNCBNYXkgMjAxOCAwMjowMToyNSBHTVQNCkVUYWc6ICI0MzM4Mi03NzI1LTU2Y2VhMDc1Y2E0YzAiDQpBY2NlcHQtUmFuZ2VzOiBieXRlcw0KQ29udGVudC1MZW5ndGg6IDMwNTAxDQpDb25uZWN0aW9uOiBjbG9zZQ0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9VVRGLTgNCg0K^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

原文地址:https://www.cnblogs.com/bonelee/p/9238512.html