<% %> is simply short for <script runat="server"> </script> aka code render blocks.
<%# %> are binding expressions (plus the above).
<%= %> is the above + a Response.Write().
<%: %> is the above + a Response.Write() wrapped in Html.Encode (new in .NET 4.0).
<%$ %> is an ASP.NET expression, used to bind configuration or resource file data during runtime.