i tested mental ray, its memory usage linearly increased during rendering while displacement was on...so i guess mental ray simply tessellates all triangles and then displaces them along vertex normals, finally sends those generated triangles into regular bsp traverse and construction routines...
i got through the ray-tracing part of Pixie, it bounds moving triangles and build hiereachy then interpolates them during intersecting on-demand, very simple but seems rather better than my approach which gets use of push/pop triangles to modify bsp structures numerous times causing inefficiency...
so i decide to rewrite my renderer to do that, make it really fast...