I modify my vest every season to some extent. There's really nothing crazy about it, guys have been doing it for a long time.
When I bought my original Ol Tom vest it had magnetic closures which I cut out and replaced with buttons because they are more secure. I also sewed a dedicated longbox pocket into the side and a trumpet tube holder so I can carry a backup trumpet besides the one around my neck.
I also fashioned a slim fixed blade knife to the front vest strap and created a slot to hold my compact binoculars. I'm still working on getting the striker pockets just right so I won't lose anymore strikers!
These are just a few of the modifications I've made over the course of owning the vest and I got to tell you I like it much better now then when I bought it.
So I think you should definitely customize your vest and make it work best for you. If you're anything like me, you'll never find the perfect vest right off the rack.