There may be an element of safety at play here where the gobbler feels safer from predators being uphill or going uphill. When he is looking down on you, he can spot that hen so much farther away and when he gets to a point and does not see the hen he knows something is not right. He is likely to hang-up and eventually leave. I do not use decoys so this may not apply if you use them, I would not know.
Not saying you cannot call them in downhill, sometimes you can. But I would much rather be on same level or above.