No if both primers are quality primers with equal enamel holdout. Yes, if one primer is cheap and lacks enamel holdout equal to the other.

What primers are you considering?

I love learning new terms:

Enamel holdout can be measured as a difference in observed gloss of the topcoat over a primer relative to the gloss of the same topcoat over a smooth sealed surface.

My favorite one this month is paint holiday, which is nothing like what it seems.

Both are Kilz. I'm a little concerned because I bought a Martha Stewart washable kind (I think semi-gloss).

The Kilz are equal performers.

The concern with losing gloss is usually attributed to painting over flat paint. Dead flat paint absorbs some of the sheen from the topcoat, causing an uneven appearance when dry.

Primers can vary, so there is always a risk when using two types.