Maybe it's more like scars. Get a bunch of men & beer together, and they (the men) will start talking about their scars. The thumb they split with a wood chisel, the stitches where the chainsaw kicked back and split their scalp, the band of scar tissue where they nearly cut off a thumb while reaching up inside an elk's chest cavity while wielding a sharp knife in the other hand... Antlers, like scars, are a material reminder of memorable events.
Certainly, many antlers just end up a debris in the corner of the basement. At my house, most antlers do not get saved anymore.