If it looks more like the doors in your home or an entrance to somewhere rather than a cabinet I think keeping it the same color of your doors and trim would be best. However it helps to see the space. Have any pictures to share?

I painted my pantry door and trim the same white as my cabs. But...I painted my whole house full of new interior doors and all new interior trim so maybe that doesn't count.

Somewhere, probably Houzz, I saw a pretty kitchen with painted cabs and painted pantry door trim with a wood door. It struck me because I would have thought it wrong before I saw it. It looked pleasing.

I too think if it looks like an interior door, same as the other interior doors (white? or contrasting?) but if it looks like a cabinet, same as the cabinets.

I googled "painted pantry door" and there certainly are a lot of pretty examples out there! Here is one with painted cabinets:

Sorry, no pictures at the moment. This is a tiny pantry so the door is too narrow to be a regular door. But when I try to picture it painted a color with the white trim, that seems awkward. Thanks for the yellow pantry door picture!