The button you shoot with, if you hold it down when you shoot it won't allow you to fast switch. If you just click it once then you can.
It depends on the weapon you switch to.
After a shotgun, if you fast-switch to Fist (with LMB pressed), it does switch.
But it wont work if you switch to Colt or Ingram.
That's the reason some guys (Hanney, SpiralRock...) dont carry any pistols or melee weapons.