Randall Munroe, who draws the geekerrific xkcd webcomic, has created a really good chart showing relative radiation doses absorbed by humans doing various activities.
I've put a piece of it here, the section with the lowest doses. I like this! A lot of folks don't understand what radiation is -- light is radiation, for example -- or that just by existing on the surface of our planet you absorb a certain amount all the time: from the ground, from space, from things you eat. Wikipedia actually has an excellent rundown of what radiation is, and the critical distinction between ionizing and non-ionizing radiation (there's also electromagnetic versus subatomic particle radiation, but that's less of a concern here).
In the chart, Russel deals with doses from ionizing radiation. This is the kind that can cause damage... but only in sufficiently high doses. For example, bananas are a natural source of gamma rays due to the decay of an isotope of potassium (40K). It's a pretty weak source -- a few years back I had access to a gamma-ray detector and we could barely detect a banana's emission -- and it doesn't affect you in any real way. Potassium iodide is a common salt that's also a gamma-ray emitter, but again you'd need a lot of it for it to be dangerous... and if you ate that much you'd have worse issues!
The average amount of radiation you absorb in a year is about 3 - 4 milliSieverts, depending on where you live. At higher elevations -- like, say, Boulder, where I live -- cosmic radiation puts you on the higher end of that scale. I'll note that cancer risk is not really higher living up here than at sea level (lung cancer rates are lower than average here, probably due to the healthy lifestyles most people follow in Boulder, but skin cancer rates are slightly higher than average, probably due to a combination of people being outside more than average together with the thinner air blocking less UV).
In general, you can actually absorb a much higher than usual radiation dose (up to a point, of course) without ill effects, since your body can heal some amount of damage (just like it heals from a cut). Too many such doses too close together, or too big a dose all at once, can do too much tissue damage and be fatal (I guess, again, like a cut). For example, I like to point out that the Apollo astronauts got roughly a year's worth of radiation absorption in their tissue while voyaging to the Moon and back, but didn't suffer any ill effects.
Obviously, this is a complicated issue, but the xkcd chart looks like a pretty good way to eyeball where things fall on a scale of "nothing to worry about" to "AIEEEEPANICPANICPANIC".