Blues had raced into the lead inside 40 minutes, after headers from Krystian Pedersen and Gary Gardner put them 2-0 up in this Midlands derby.
However, an instant reply from Coventry and Sheaf from Gustavo Hamer's delivery got them back into it.
The scores were levelled on the stroke of half time, as Sheaf scored his second of the day and the first two of his Sky Blues career to make it 2-2.
Coventry turned the game around completely with 20 minutes to go, after Hamer's corner was powered in by Rose.
And the points were sealed in the final minutes, when O'Hare tapped home for the second week in a row to put the game beyond doubt.
Story of the match
A fairly subdued opening to proceedings at St. Andrew's was shattered when Juninho Bacuna's corner was met by Pedersen, who headed Blues into the lead.
The game remained fairly calm, despite the early goal. The only time either side went remotely close again was when Hamer dropped a vicious corner kick right on top of Zach Jeacock's crossbar.
On 30 minutes, Blues could have doubled their advantage too.
A long ball forwards was picked up by Scott Hogan on the right hand side of the area but he should have done better with his shot, which went high and wide.
The game came to life in the final minutes of the first half, though.
First, Birmingham doubled their lead thanks to a Marc Roberts long throw which was met by Blues captain Gardner to double their lead.
But their two-goal advantage was not to last long, as the Sky Blues dragged themselves back into the game.
Hamer's set-piece from the right-hand side was headed back across goal by Michael Rose for Sheaf to tap in and get Coventry back into a game they had been well out of.
Blues' lead would soon be wiped out altogether.
Hamer's set pieces again caused problems for Birmingham, as his deep ball to the back post was met by Sheaf, who scored his second of the day, having not scored previously for Coventry.
Moments into the second half the Sky Blue turnaround could have been complete.
Jake Bidwell's cross was met by Jamie Allen, but he could not direct his headed effort towards goal.
That missed chance was not to matter, though.
The Sky Blues continued to push throughout the second half and completed their turnaround when Hamer's flat corner was powered in by Rose.
Luckily for the Sky Blues, O'Hare was on hand to fire in for his fifth goal of the season, and to keep Coventry's play-off hopes well and truly alive heading into the final few weeks.
Sky Blues march on
They face second-placed Bournemouth next though on Easter Monday, so whilst the play-offs are far from a guarantee for City, their dreams are still very much alive.
Hamer the key for City again
Gustavo Hamer just keeps getting better for Coventry.
Last week at Fulham, he played a hand in two of the three goals in a shock 3-1 win and here he played a hand in three of the four.
At 2-0 down, it looked bleak for the Sky Blues, but as long as Hamer was on the pitch, they always had a chance.
His delivery from a free-kick out on the right side led to the Sheaf tapping in the first goal, before he directly assisted the second and the third goals for Coventry as they turned the game on its head.
Firstly, he whipped a corner into the back post which was nodded in by Sheaf again to level the game.
And then Hamer delivered a flat ball from the corner which invited Rose to attack it and power in Coventry's third of the game.
Birmingham City: Jeacock (GK), Gordon (Richards 76'), Roberts, Pedersen, Colin, Woods, Gardner, Hernandez, Bacuna (Deeney 75'), Taylor, Hogan (Jutkiewicz 87')
Birmingham unused substitutes: Bela, Graham, Sunjic, Trueman
Coventry City: Moore (GK), Dabo (Kane 62'), Rose, Hyam, Bidwell, Maatsen (Waghorn 87'), Hamer (McFadzean 82'), Sheaf, Allen, O'Hare, Gyokeres
Coventry unused substitutes: Kelly, Jones, Wilson, Tavares