The defender drove home from distance in the dying stages as Leicester overturned the home side's advantage from Ryan Bertrand's spectacular opener. Demarai Gray notched the equaliser for the Foxes before Pierre-Emile Højbjerg saw red for two bookable offences.
Puel's men now rise up to seventh in the Premier League following back-to-back wins whilst the Saints are yet to taste victory.
Southampton boss Mark Hughes made four changes to the team that was defeated by Everton last time out. Jannik Vestergaard returned to the line-up in place of Jack Stephens, whilst Mohamed Elyounoussi, Pierre-Emile Højbjerg and Shane Long were all given starts.
Meanwhile, the Foxes were without the suspended Jamie Vardy following his red card in the 2-0 victory over Wolverhampton Wanderers last weekend. Kelechi Iheanacho was drafted into the team in the striker's absence. Puel made a further two changes as Daniel Amartey and Wes Morgan replaced Marc Albrighton and Jonny Evans.
Speedy start from the Saints but all square at the break
Saints started at a quick pace and thought they had got an early lead as Nathan Redmond volleyed home but the offside flag was up.
Kasper Schmeichel then had to stay alert to stop Elyounoussi’s powerful effort. The towering Vestergaard also got free from a corner but he could only nod his header wide of Leicester’s goal.
The Foxes' first opportunity did not materialise until minutes before the half-time whistle. Demarai Gray trotted forwards and combined excellently with Ben Chilwell but the former nudged wide.
Excellent defending from Harry Maguire was crucial as he denied Danny Ings before a stunning save from Schmeichel frustrated the forward even further as he spun and shot. Consequently, both sides went into the break level.
Bertrand blockbuster livens up second-half
Similarly to the first-half, Southampton resumed at an extremely quick pace. Ings was released by Nathan Redmond but again could not get past Schmeichel.
Hughes’ men did have to wait much longer to grab their opener though. Hojbjerg’s cross trundled all the way to Bertrand who set his sights from the edge of the area and crashed the ball into the top corner.
Despite their dominance, the hosts were only ahead for a couple of minutes. Gray capitalised on poor defending to take down Cedric’s headed clearance in the penalty area. He then clinically buried the ball into the bottom corner to equalise for the Foxes.
Højbjerg sees red as Maguire steals win
The drama was not finished there. After picking up a booking in the first period, Hojbjerg was given his marching orders. The Dane thundered into the Leicester penalty area and tumbled to the floor. Instead of pointing to the spot, Jon Moss gave him another yellow card for simulation and sent him off.
Leicester looked to capitalise and Ben Chilwell almost turned the game on its head but he could not beat Alex McCarthy with a low effort. The hosts then breathed a huge sigh of relief as the ball took a touch off of Wesley Hoedt but it dropped just wide of the Saints’ goal.
With the Foxes looking continually frustrated, Lady Luck shone in their favour. Harry Maguire found space on the edge of the area and produced a speculative effort that nestled into the bottom corner via a deflection, sending the away contingent into raptures and most of the Southampton fans out of the exit door.
Claude Puel's men now turn their attentions to their Carabao Cup second round clash with Fleetwood Town on Tuesday evening. Southampton will look to notch their first victory of the season against Brighton and Hove Albion in the cup.