There's no hard and fast rule, but to prevent conflicts with swimmers, divers, and watercraft (including jetskis, surfers, and kits boarders), one should maintain a separation of 150 feet.
Here's the Texas diver law, meaning it is unlawful to:
Operate within 50 feet of a "Diver Down Flag" or operate a boat within 150 feet of a "Diver Down Flag" except at Headway/Steerage Speed.
There's a reason why we need such separation zones - people get hurt or killed when they get too close. Be careful out there, and if you see a potlicker coming ...