I posted a picture in the Rem. pattering section, this morning. Using 21" barrel, .665 Sumtoy choke, and LB's #6 in 3". 225 in the 10", and had torn and missing petals, while sighting it in. So IMO I don't think that is causing you a problem, but suggest going to #5 or #6 shot.
The only time I have ever used #4 shot is in the Remington Duplex loads of 4X6 loads. Still have a box or two of them.