I've run an M2 compact as my primary turkey gun the last 5 seasons with no complaints.
My POA/POI is dead on with all three of my M2s (M2 20 compact, M2 20 slugger, 20 and M2 12).
That being said, I know plenty of folks who do not have straight shooting Benellis. The issue with Benellis when POA/POI issues exist is almost always a misaligned magazine tube. The competitive three gun shooters discovered this. Essentially, when you set a barrel down over the mag tube and screw the cap on, the cap tightens the barrel against the mag tube and a misaligned one will pull the barrel in the direction of that misalignment. It's pretty easy to fix if you gently bend the mag tube (NOT the barrel) in the direction of the pattern.
Just like the OP, I have read many opinions on the M2 but I've got close to a dozen turkey guns and it is by far the lightest, smoothest functioning, sleekest gun I own.
I do plan on purchasing an Affinity at some point because so many of my closest friends have had positive experiences with them. But I've held them and cycled their actions; the M2 is just in a different category of firearm, IMO.