In finishing up a previous post I was going to link to the Wikipedia article about dirty bombs. I ended up not doing it because apparently they're just not that dangerous. According to a number of sources the average radiation dose would be on the order of one-to-two years natural radiation exposure, which is to say not a problem at all.
So I guess if you hear about a dirty bomb going off, you need to worry more about being trampled by the crowd than dying of radiation poisoning or cancer.
Now you know... and knowing is half the battle :)