I initially learned to drive on the left in Ireland, but have spent the last 20 years driving on the right in the States. I personally never had any difficulty switching from one side to the other—except when extremely jetlagged—but I know several people who find it enormously stressful. I find it easy enough to orient myself so that the lane divider is at the correct position.
When I was a kid, my father often brought his Irish car over to mainland Europe on the car ferries. A right-hand drive car driving on the right is doubly tricky. It didn’t seem to bother him too much, but I don’t think my mother ever tried it. I’ve never had the opportunity to try it myself.