If I'm going wading and it is cold out, I wear my waders. I wear my waders well into the warm weather so if it is warm out, I'll put them on when I get there. If we move to another spot, depending on how far away it is, I may take them off, peel down to my waist or leave them on. I sometimes wear a pfd but not wading. As a rule, I won't wade deeper than belly button deep on me. I always use the belt that came with my waders (Simms) and I always wear my kill switch.
When I took water survival back when I worked offshore, we had to put on coveralls and then jump in the pool. They filled up with and held water. We had to take them off, drain the water, and then catch air in them to form a makeshift life vest. Wasn't very hard to do as long as you didn't panic.
Now that I'm thinking about it, that would be a great class for any boater to take.
It's a myth that they will pull you down and drown you. However, when you try to get out of the water and back onto a boat or on a dock any water waders hold is water you have to lift up at about 8.6 pounds per gallon. It can be a little cumbersome.
I don't put mine on until I get to my first destination to wade. I will not keep them on going from first place to second place to wade if getting on a plane. If wading mud using my 'cinder block' Ocean Tek lace up Simms boots, they will only be on my feet when the boat is anchored in waist deep water. Regardless of waders or not, I'm pretty sure those boots would drown me if I fell out in water over my head.
I think the biggest thing is going to be the shock and lack of practice. The swimming stroke that is used is super important as well. The problem is that the cold will shock us and cause us to make very small quick movements, probably trying to dog paddle or just keep head above water. If you make large movements with the legs and arms and move more water then it's easy to stay up and move sideways. Don't just try to tread water, actually make big movements with arms and legs in such a way as to move sideways, just like moving sideways to save ourselves in the surf.