The only difference is the type of base for the lamp. The screw-in type CFLs can go into an incandescent fixture, which can run a large variety of lights including halogen and even LED, whereas the pin-type fluorescent fixtures accept only pin-type CFLs. If you are in California, only the pin-type count towards Title 24 compliance, since someone can always change the bulbs to old-fashioned incandescents. But you can achieve the same energy savings either way. The size of the bulb would depend on how big of an area you would want each bulb to illuminate. Most lighting showrooms or on-line stores should be able to talk you through what you need.

I've read, and it makes sense to me, that dedicated fluorescent cans have an advantage in that they can have reflectors specifically designed to work with fluorescent bulbs whereas screw-in CFLs used in fixtures intended for incandescent bulbs may waste some of their output. I don't have any numbers to back that up, though.

If you need to be able to dim them, I think it's still much cheaper to buy screw-in dimmable CFLs than to buy dedicated fluorescent fixtures with dimmable ballasts.

I think I like the flexibility described by remodeler_matt in being able to use a screw-in can start with dimmable CFL bulbs and if I need to change to another type. Maybe even LED when that becomes a bit more standardized.

The next question is if I'm using 5 or 6" cans, what bulb would I use? Something like a R40- Dimmable - Reflector Compact Fluorescent Flood Light?

5" will let you use the R30 size. R40s, I believe, would require 6" cans.

Screw-in CFs have another downside for those who have something of a "green" attitude, and that's the amount of additional waste generated by replacing the ballast when the lamp only fails. With dedicated CF cans, when the lamp fails, that's all you replace.

OTOH, the flexibility of screw-in CFs is hard to resist.

I wouldn't use cans for general lighting (I think they're about the worst fixtures for room lighting). However, if I did, I have to admit I'd be tempted to use incandescent cans with screw-in CFs.

I've read that some CFL reflector bulbs may not work with certain recessed cans. Is that based upon quality of the can or just random pairing?
Is there any brands or models that seem to work best?

Mostly it's that the darn CFL ballast "bulge" won't fit - the reflectors get in the way. This isn't specific to the reflector bulbs; regular CFLs won't fit either.

You can get "extenders" that just have a regular male Edison base on one end and a female Edison base on the other.. those will bump the light out an inch or two lower and sometimes make it fit.