Joe,

You're not doing anything wrong. Most banjos' necks are thin enough and flexible enough that the natural "bow" in the neck (a very slight concave bend resulting from the collective tension of the 5 strings pulling on the two ends of the neck, something like an archery bow) is influenced by strings being loosened or tightened a fair amount.

If you loosen both the 2nd and 3rd string for D tuning, the neck ends up straightening a bit, its natural tendency, and that pulls the other strings sharp. That would happen whether or not you use D tuners. It's even possible that after tuning the 2nd and 3rd as exactly as you can, then having to retune the other strings can then make the 2nd and 3rd a bit out of tune and needing a slight touch-up.

Having the banjo as in-tune as you can get it really does help the music sound better, so taking a lot of care with the tuning is a good idea.

A thicker neck on a banjo, or using light gauge strings reduces the effect described. And tuning generally is helped by a bit of lubrication (such as graphite powder) in the slots of the nut, to let strings slide freely and not get hung up by friction in the slot, when a string’s tuning is changed.

Hope that's useful.