Does walking cause upper body buildup?

I walk for my health/weight loss/relaxation, and I have noticed my shoulders seem pumped up more..Could this be because of the way I hold my arms(swinging by my side)..I dont want big shoulders at all...Is there a right way maybe to hold your arms?