Brinks, I dont know my man. Nothing really stands out for me in your answers except the broken elbow, which wouldnt cause the contralateral side to develop symptoms.
It may be some arthritic anomaly.. perhaps hereditary.
Only other thing I can think of is that your joints, elbows specifically, could be hypermobile, and have consequently developed arthritis.
My advice would be to walk into a PT clinic and get a free consultation. Get them to look at your elbows and see what they recommend.