Sutton, of League Two, took the lead thanks to Donovan Wilson's deft finish on the half-hour mark, but that was cancelled out before the break by a superb curling effort from Rotherham's Ben Wiles.
The Amber army restored their lead early on in the second-half through captain Craig Eastmond, a goal which looked to be sufficient to earn then victory right up until Jordi Osei-Tutu's dramatic equaliser deep into stoppage time.
Chiedozie Ogbene then gave Rotherham the lead for the first time in the contest early into extra time, and the contest was sealed by Michael Ihiekwe's header in the second half of the extended period — a goal which sent the travelling Millers into raptures as their side lifted the silverware.
Story of the game
What an occasion it was for these two teams, with Rotherham aiming to secure the first piece of a historic league and cup double and Sutton showing up at Wembley for the first time since 1981, hoping to make their inaugural EFL Trophy campaign a successful one.
Top of League One versus outsiders for promotion from League Two, the Millers were clear favourites here, and the first chance of the encounter fell their way when Richard Wood rose highest to meet Dan Barlaser's in-swinging free-kick, but the skipper failed to direct his effort on target from a good position.
But Sutton got here on merit, by performing as more than the sum of their parts, and they were doing just that in the opening stages here.
David Ajiboye in particular was proving a constant threat with his pace and dribbling ability, and with just under 15 minutes gone he used those to full effect to set Will Randall through on goal. The midfielder did brilliantly to skip past Joe Mattock and cut onto his favoured right foot, however his eventual effort nestled in the side netting, much to the relief of goalkeeper Viktor Johansson.
And sure enough, against the odds, it was Sutton who took the lead. Ajiboye's cross was only-half cleared by the Rotherham defence and the ball dropped for Randall on the edge of the area. He controlled the ball before laying it off for Wilson, who shifted it onto his left foot before sending an accurate drive into the far bottom corner.
The travelling Amber army were sent into sheer raptures as their team took a deserved lead. It was an unwanted early setback for Paul Warne's Millers but a side of their ability always had the potential to turn it around. All in all, we now had a fantastic contest on our hands with just over 30 minutes gone.
Rotherham worked a good opportunity to get themselves level when Michael Smith's miscued header dropped for Ogbene in the box. The winger controlled and opened his body up, but should've done better as he blazed a shot way high and wide of the target.
On the cusp of half-time, the Millers restored parity courtesy of a stupendous effort from Wiles. The midfielder picked up a loose ball 20 yards out, dummied past Harry Beautyman's weak challenge and then curled a pinpoint effort in off the underside of the crossbar, leaving Dean Bouzanis no chance at a save.
Nearing the end of a difficult first-half, a moment of magic like that was just what Rotherham had needed, and they weren't far off again when Oliver Rathbone cut inside and drilled one towards the near post, but the Sutton goalkeeper was equal to that one as the scores remained level heading into the break.
The first chance of the second period fell Rotherham's way as a neat flick from Michael Smith sent Wiles through on goal, but the midfielder failed to beat Bouzanis from a tight angle. Moments later, the Millers would come to regret that missed opportunity.
When Sutton regained their lead just two minutes after the restart, it was the name of skipper Craig Eastmond which was sent up in lights after terrific work once again from Ajiboye. The winger sped his way down the right flank before cutting it across for Eastmond, who was afforded two bites at the cherry to prod the ball home right in front of the Sutton supporters.
Rotherham came close to an equaliser when Ogbene's cross perfectly teed up Smith for aa header from the centre of the box, but the target man didn't quite get his jump right, making contact fro under the ball and thus sending it over the target.
Just moments later, Wiles thought he'd secured his brace when his low effort deflected off the boot of Omar Bugiel and appeared to be heading for the bottom corner, however Bouzanis did remarkably well to get down and clutch the ball in his grasp with Smith following in for the rebound.
Sutton substitute Isaac Olaofe early made an instant impact when he darted effortlessly past Wood and into the box, but the forward tightened the angle too much and eventually sent his effort into the side netting in another let-off for the Millers who had been poor for most of the second-half.
Rotherham needed to get into gear, and quickly with less than 20 minutes of regulation time remaining. They were already beginning to resort to pretty desperate long-ball tactics, and it nearly paid off on one instance when Wes Harding's cross met the head of Freddie Ladapo in the box, however the forward's flicked header resulted in a comfortable save for Bouzanis.
The Australian goalkeeper was called into action once again when Ladapo was sent through on goal by an eye-of-the-needle pass from Barlaser, but again he failed to find the net with Bouzanis rushing out at speed. Those two saves meant Sutton were now less than ten minutes away from glory.
Any early celebrations were premature, however. Bouzanis had been impenetrable all half but he was finally eaten when a cross-field pass fell kindly for Osei-Tutu; the wing-back controlled the ball well before directing an arcing left-footed effort right into the upper echelon of the far top corner.
Deep, deep into stoppage time — in the 96th minute, to be exact — Rotherham had somehow found a leveller, and now the contest headed into extra time.
Early into the extended period, Rotherham earned a free-kick in a dangerous position and substitute Shane Ferguson stepped up to the mark. The shot from the Northern Ireland international might have nestled in the net had it not deflected off the wall.
Sutton then worked an excellent opportunity to restore their advantage once more when Joe Kizzi's pull-back found Beautyman on the edge of area, however the midfielder failed to direct his half-volley on target, much to the relief of the Rotherham defence who had afforded him far too much time and space.
Within a matter of moments, the Londoners would come to pay for their profligacy. From the resultant goal-kick, Ladapo picked up Smith's knock-down before laying the ball off for Ogbene, who charged towards the box before unleashing a vicious effort from 25 yards to give Rotherham the lead for the first time in this match.
And the Millers came agonisingly close to doubling their advantage from a set-piece situation when, first, Ihiekwe's header across goal from Jamie Lindsay's delivery hit the post, before the ball was recycled to Ladapo who drew a fine close-range save from Bouzanis.
Sutton had lost their momentum but they weren't out of this contest just yet. Kizzi's lofted throw-in set up a wrestling match between Olaofe and Wood from which the former eventually emerged victorious, wriggling free and working space for a shit which he blazed far over the bar — the forward should have done much better.
It wasn't long before Rotherham's victory was secured beyond doubt. An excellent corner from Ferguson caused havoc among the Sutton defence and Ihiekwe rose highest in the box to head across goal and secure the silverware for Warne's Millers.
The Rotherham faithful headed home ecstatic but Sutton too could be proud — three divisions separated the two sides last season but they gave a team in the league above them a tough match and came within seconds of victory in a chaotic but thoroughly entertaining cup final.
Rotherham United: Johansson; Ihiekwe, Wood [c], Mattock (Ferguson 80'); Miller (Osei-Tutu 80'), Rathbone (Harding 60'), Barlaser (Lindsay 90'), Wiles, Ogbene; Kayode (Ladapo 60'), Smith.
Unused subs: Chapman, Edmonds-Green.
Sutton United: Bouzanis; Kizzi, Goodliffe (Rowe 73'), John, Milsom (Wyatt 82'); Ajiboye, Eastmond [c] (Davis 82'), Beautyman, Randall; Wilson (Olaofe 66'), Bugiel (Bennett 82').
Unused subs: Nelson, Boldewijn.
Referee: Seb Stockbridge
Sutton, meanwhile, return to their home at Gander Green Lane when fellow Londoners Leyton Orient visit on the same day.