Variance Stabilizing Transformation
It is obvious that the variance of counts is generally greater than their mean, especially for genes expressed at a higher level.
There are two mathematically equivalent formulations of the NB distribution.
In its traditional form, which I will mention only for the sake of completion, the NB distribution estimates the probability of having a number of failures until a specified number of successes occur.
The second definition sounds more intimidating but is much more useful. The NB distribution can be defined as a Poisson-Gamma mixture distribution.