Sam Hoskins' excellent goal hauled Town level at half-time following Harry McKirdy's opener.
Jon Guthrie put Cobblers into a brief lead early in the second-half but, crucially, Swindon levelled within 60 seconds and then went on to run away with victory.
McKirdy scored three more and Tyreece Simpson was also on target in an ultimately emphatic victory.
"It was tough to take," said Brady. "I feel like we had a strange 20 minutes that they really capitalised on.
"On four occasions there was some uncharacteristic defending from us inside the penalty box and each time it ricocheted to them and they put it in the net. It fell to them very kindly but we didn't clear our lines properly.
"We went 2-1 up and then to concede straightaway is a body blow and then we concede a couple more times and it was tough for us to take.
"I felt if we stayed 2-1 ahead, we could have worked our subs well enough to see it out but obviously, they score straight after and that's really disappointing.
"The way we stepped on in the last 15 minutes of the first-half was excellent and we were really in the ascendency of the game and then came out after half-time and scored.
"But we knew at some point it would hit us because we haven't played in three weeks. Swindon were probably better off for having the game three or four days earlier because they have their distances and their timings back a bit more."