When UE4 implement the toon material, it has its own edge test arithmetic, that is use UE4's depth texture and offset the UV to judge the current pixel whether is belong to the edge or not.
And before the lines are actually rendered, a second SceneTexture is calculated from Ambient Occlusion, This is used to mask out the outline.The purpose to prevent internal lines that appear on some wireframe edges.
I think, if four pixel(assume 4 times uv offset)'s depth are very different, current pixel should be at edge, and we also can calculate the "mathematical expectation" of these pixel to judge the value;
more infomation:
https://docs.unrealengine.com/lastest/INT/Resources/Showcases/Stylized/index.html