three apples - mortar search
Not sure if you are still looking at mortars but we are winding up and had the hearth put on our outdoor fieldstone fireplace. They used gray mortar. It looks awful. They are going to take it out and replace the mortar, but in this discussion my builder suggested light buff mortar. I had never heard of it but it seems like what you are looking for - darker than ivory buff and lighter than buff.
Hope this helps.