MaxInstances This says....how many instances of same version of browser can run over the Remote System.
For example, i have a FF12,IE and i declared the command as follows
-browser browserName=firefox,version=12,maxInstances=5,platform=LINUX
-browser browserName=InternetExplorer,version=9.0,maxInstances=5,platform=LINUX
So i can run 5 instances of Firefox 12 and as well as 5 instances of IE9 at the same time in remote machine. So total user can run 10 instances of different browsers (FF12 & IE9) in parallel.
MaxSession This says....how many browsers (Any Browser and any version) can run in parallel at a time in the remote system. So this overrides the Max Instances settings and can restrict the number of browser instances that can run in parallel.
For above example, when maxSession=1 forces that you never have more than 1 browser running.
With maxSession=2 you can have 2 Firefox tests at the same time, or 1 Internet Explorer and 1 Firefox test).
Irrespective of what MaxInstances you have defined.
For more clear info do visit - https://seleniumhq.github.io/docs/grid.html