Matt Stevens tried to force Nicky Cadden's corner over the line with 20 minutes to play but Cobblers goalkeeper Liam Roberts kept him at bay.
However, despite the ball not being close to crossing the line, the linesman raised his flag and after a long discussion with the referee, the goal was given.
That put Rovers on course for victory until Mitch Pinnock's stoppage-time leveller.
"We have a camera behind the goal and it's very clear that the ball is a yard from the line," said Brady. "It's the wrong decision, it's as simple as that.
"I can't say much more but to give it some context, the linesman is on the opposite side to where the ball has gone in and he's guessed. It's just a guess and it's the wrong decision.
"The ball doesn't go anywhere near the line. That was highly frustrating and highly disappointing but we showed immense character to come back from a decision like that and keep going and score in the last minute.
"I was hoping to speak to the officials but they don't want to speak to me at the moment so maybe they're embarrassed about the decision. Maybe it shows a level of guilt!"