Eric,
You didn't specify how you're modifying dates/times, but based on your description it sounds like you're using the Adjust Capture Dates and Times dialog. All dates/times are adjusted relative to the set of values in the top half of the dialog (the Adjust Relative) section, even if you're manipulating the lower section controls (Adjust Absolute). The two interfaces are locked together. When you modify one, the other changes. If you've got a folder containing images that were shot over years, you're going to need to adjust them separately or the years will change dramatically.
All changes to dates are relative, not absolute. You can't change the dates of images in different years with a single relative year delta.
-Kirk