Hit the pool!
Whatever the weather might be, going to the pool is a big part of Icelandic culture. Even the tiniest towns in Iceland have a swimming pool, where locals swim, play, relax and catch up on the latest gossip.
The pools in Iceland are supplied with the same geothermally heated water that runs through our taps, which means that you're always assured a warm dip, even though it might be snowing outside. Most pools also feature "heitir pottar" or hot tubs of varying temperatures in which to relax. Watch out though as many of them are extremely hot, but you'll find the temperature marked next to each one.
Another thing to remember when going swimming in Iceland is that the pools are very clean and people are required to shower with soap, naked, before going into the pool.
Here's a great website in English detailing every single pool in Reykjavik and in the whole of Iceland.