Another of my favourite places to swim while in New Zealand was in Resolution Bay on Queen Charlotte Sound in the South Island. We spent a couple of days walking two sections of the Queen Charlotte Track from Ship Cove to Endeavour Inlet (a mere 15km of the whole 71km track). It's a great walk through thick coastal forest and along ridge lines with incredible views of the Sounds.
We stayed a couple of nights at Resolution Bay in a wonderful cabin. They had their own wharf from where you could jump or dive into cool, clear, blue water.
I couldn't stay out of the water and went for several swims both in the sun and, later, in the rain. The saltwater was cool (the locals reckonned 16 or 17 degrees C, which felt about right), but, it was so wonderful, I stayed in for as long as I could. The cabin had a hot shower which was really welcome after I came out shivering uncontrollably.
It was lots of fun acting like a kid and throwing myself off the wharf, but it was also great to swim out to deep water. It was so still and clear that I could see the bottom, but couldn't possibly duck dive to the bottom (I kept trying).
It was also lots of fun swimming across the bay to a couple of the pebbly beaches, sitting in the sun, then swimming back to the wharf.
We took some sea kayaks out to explore the bay and some of the coves around this part of the Sounds, and were constantly amazed at how clear and how blue the water was.