Marcus Rashford and Anthony Martial combined twice and took their goal tally to 21 for the season - before Michael Obafemi salvaged a point for the Saints in the dying moments.
United in previous years have struggled to profit off of teams slipping up - and they did the same tonight. The game was branded a "must-win" - which they failed to do. They may now have failed to solidify themselves in the top-four as they remain fifth in the table.
Story of the game
Ralph Hassenhuttl's men made an excellent start to the game and punished United for their lousy approach. Armstrong gave his side the lead in the 12th minute after Paul Pogba attempted to turn on the edge of his own box.
Redmond pounced upon the error and picked out Armstrong who denied David De Gea a clean sheet on the Spaniard's 400th appearance for United.
Rashford thought he had the immediate response when he had the ball in the back of the net, however, he was offside and the goal was ruled out.
The English striker did then get the equaliser when he linked up with Martial and thumped it past Alex McCarthy. Martial showed immense strength as he was being brought to the ground - and managed to flick it out to Rashford who did the rest.
Martial then smashed it past McCarthy to put the Red Devils in the lead just before the drinks break. Both Martial and Rashford got their 21st goals of the season.
Oriel Romeu should have seen red before half-time as he went through the back of Mason Greenwood after the ball had gone. Greenwood refrained from retaliation - the VAR checked the incident and remained a bystander.
The best opportunity of the second-half saw Rashford nutmeg Kyle Walker-Peters and play a beautiful one-two with Martial. Rashford was winding up for the shot as Ryan Bertrand beat him to the ball and forced it out for just the corner.
Martial could and should have had a hat-trick when he was in on goal but sent his shot flying over the bar. Greenwood was open, but Martial had no intentions of playing the pass.
Saints never gave up, a 96th-minute corner presented an excellent opportunity to salvage a point and Hassenhuttl's men did exactly that. Substitute, Obafemi poked it past De Gea at the back post and have now taken four points off Solskjaer this season.
A result which has implications for many teams
United now remain in fifth place and are level on points with Leicester who had a 14 point lead in mid-December. A Champions League place is now back in the hands of Brendan Rodgers' side. As for Frank Lampard, his side still have to face Liverpool and Wolves on the final day of the season - and they cannot afford to slip up as they only have a one-point lead.
United travel to Crystal Palace on Thursday night and will look to bounce back with a victory to boost their chances of Champions League football. Solskjaer and his men can still snatch a spot inside the top four if they win all their remaining matches - which would include a victory at the King Power on the final day of the season.
Solskjaer's men then have two nights to rest before a huge FA Cup semi-final against Lampard's, Chelsea. This United side is desperate for silverware and have two great opportunities - the other being in the Europa League.