Try an actual electrical supply house.

Use two-gang boxes. Or install the boxes horizontally. Running conductors in the sides of single gang boxes leaves no room for the device, such as a receptacle. You will not find a single gang box metal or non-metallic, with knockouts on the side of the narrow dimension.

I was actually planning on having boxes containing two duplex outlets each. In other words each box would have 4 places to plug something in. Do they make double gang PVC conduit compatible boxes with holes on the side?

Like bus said, just turn the boxes 180 degrees. A two-gang box is square.

Sorry, meant 90 degrees.

"You will not find a single gang box metal or non-metallic, with knockouts on the side of the narrow dimension."

I still have plenty of single gang expandable boxes with knockouts on all four sides.

The built in cable clamps are on the smaller dimension (top and bottom typically) while an outside cable clamp must be used on the knockouts on the long sides (left and right typically).

I confess to not having seen gangable boxes with knockouts on the sides.

The knockouts are towards the back on each of the 'expandable' sides.