Understanding on 'Error to Origin (50x)' , 'Internal CDN Error (50x)' and 'External Error (50x)' in Chartron

Overview

This document explains about definition of these values on OUI Chartron.

Definition of Error Types

Error to Origin (50x) Hits/Sec
- The hits per seconds of response 504. As RFC2616, "504" means "Gateway Timeout". But CS use this value for logging "Error to Origin".
- The situation to return response 504 : If PX-503 response header starts with "O ", then log as 504.
Internal CDN Error (50x) Hits/Sec
- Definition : The hits per seconds of response 505. As RFC2616, "505" means "HTTP Version Not Supported". But CS use this value for logging "Internal CDN Error".
- The situation to return response 505 : If PX-503 response header not starts with "O " , then log as 505.
External Error (50x) Hits/Sec
- Definition : The hits per seconds of response 50x received from origin.
- If origin returns 504 or 505, CS logs as 503. (i.e. External Error = all of 5xx except 504,505)
Internal timeout (202)
- Definition : The hits per seconds of response 202. The situation that a request has accepted, but internal timeout occurs. As RFC2616, "202" means "Accepted". CS use similary this meaning.
  - If a real 202 response come from origin, perhit log has other reponse headers like "Server" or "Date", etc. But 202 response without other headers, the situation is internal timeout.
  - While CS retries to other nodes, CS will disconnect client connection and log as 202. In this case, request client cannot receives any http response code nor headers.

Test environment

Edge node : n5 (10.40.196.201)
Shield node : n7 (10.40.196.203), n4(10.40.196.204)
Origin : 10.40.198.222 (CustomHttpd)
Attached file : very_small.mp4 (325117 bytes)
Reproduce conditions
- User -> CS edge
- CS edge -> master or shield
- CS shield -> origin

Error Cases by connection state

(CS Last level shield - origin)
(CS node - CS node)
Client - CS edge node

http.log messages

Edit Document

Error type	http.log message	Description
Error To Origin	"read error"	When using gzip compression, error occured on reading data to gzip buffer.
Error To Origin	"too many downloads"	The count of active connection to origin is over limit. (site's max_origin_downloads)
Error To Origin	"PAD is marked dead"	When site is not dead, total origin connection per site is over 100 and origin timeout per site is over 40% of total origin request per site, the site is marked dead. This check is working every 20 second. And origin connection counters are also reset every 20 seconds. When site is dead, total origin connection per site is over 0 and origin timeout per site is under 20% of total origin request per site, this condition has been met every 20 seconds 3 times in raw (during 60 seconds), the site is marked undead. When the requested cache object does not exist and the site is dead, if num of total upstream is over than 100, CS respond 503 with "Px-503: O PAD is marked dead".
Error To Origin	"preceding upstream connection has not yet finished"	When waiting prceding upstream connection at using NTLM feature (v8.2.1~)
Error To Origin	"Too many upstream connections, aborting download for $(URL)"	Upstream connection is over limit.
Internal CDN Error	"file too large (chunked-transfer-encoding on a chunking pad)"	(~ v8.0.x) Redirect to origin when data size is larger than 200MB and use_sub_file is false and use transfer-encoding is chunked.
Internal CDN Error	"bad file (disk)"	The data from disk is bad.
Internal CDN Error	"subfile mismatch"	When using sub_file (chunking) , subfile's total content length is mismatched from first chunk's.
Internal CDN Error	"short read"	When Disk Cache cannot read data in right length from Disk.
Internal CDN Error	"Init Task Failed"	Various reason for fail MyStack.initStack()
Internal CDN Error	"Selecting Master"	Exception occurs on selectMaster()
Internal CDN Error	"null"	UpstreamConnectionHandler.doConnect()
Conditional	"unknown host"	upstream download fail : UnknownHostException occurred
Conditional	"socket timeout"	upstream download fail : SocketTimeoutException occurred
Conditional	"io exception"	upstream download fail : IOException occurred
Conditional	"unknown exception"	upstream download fail : Exception occurred
Conditional	"Host lookup failed for $(host)"	upstream download fail : DNS lookup failed
Conditional	"Stalled download" + "Killed by connection manager after XXX"	Upstream timeout
Conditional	"Connect Failed"	Upstream connect fail
Conditional	"Read buffer filled but cannot parse"	Read buffer done, but still cannot parse response header.
Conditional	"reading (connection was closed prematurely)"	Receiving socket close packet from upstream.
Conditional	"reading response: connection was closed prematurely"	connection was closed by upstream.
Conditional	"The client side connection was already closed!"	connection was closed by client.
Conditional	"Downstream connection closed while upstream transfer chunked channel setting."	Downstream connection closed while upstream transfer chunked channel setting.
Conditional	"Connection closed while writing request"	upstream connection was closed while writing request to upstream
Conditional	"Connection closed while writing request body"	upstream connection was closed while writing request body to upstream
Conditional	"Connection closed while writing request body(chunked) : XX bytes sent"	Error occured on ending stream Transfer-Encoding : chunked
Conditional	"Invalid selection key"	Java nio selection key is invalid on handling upstream connections.
Conditional	"gotMoreContentDataThreaded"	Exception occured in UpstreamConnection thread

"Conditional" : When upstream fails, CS retries another upstream server that is called "failover" logic. Following failover logic, If the last selected server is origin, Error type decides "Error To Origin", other "Internal CDN Error"