The taper of pants also can cause problems

For flared pants you have to ease the excess material at the top so it's evenly distributed between the seams

For tapered pants you often have to open the side seam a bit to get enough slack to hem them.

Lazygardens is correct. Most likely it has to do with the taper in the legs. If the legs are tapered, try to make the hem allowance narrower as the wider the hem allowance, the greater the discrepancy you will have to deal with.

Depending on the material, you may find it easier to ease the hem in if you use a seam binding which can have some give in it....

Hemming without puckers for flared and tapered pants may be helpful for you.

Below contains a nice video on hand and machine hemming.

Here is a link that might be useful: How to sew a hem