They took a first half lead through Jairo, but were pegged back by Arjen Robben's strike.
When it looked like the game was headed for a draw, Jhon Cordoba popped up with a dramatic late winner.
The game, unsurprisingly, began with Bayern on top. The return of Frank Ribéry was greeted with glee from the Bavarian faithful, and it was the Frenchman who created the first real chance of the game. He burst down the line only to see his crossed cleared out to Arturo Vidal, who had a powerful volley brilliantly kept out by Loris Karius.
Karius was called into action moments later, this time scrambling across his goal to deny Juan Bernat's close range effort. His defence somehow managed to shut out Robert Lewandowski, in what was a real test of their will. Mainz had set their stall out, and would be suitably rewarded for their defensive solidity midway through the half.
A superb run down the right by Giulio Donati started the move, as his cross went all the way to the back post and Jairo. The Spaniard, who had ghosted in unmarked, fired hard and high past Manuel Neuer to stun the Allianz Arena.
Arjen Robben had the final chance of the half, though his trademark move inside that allowed him the space to shoot saw him guide a shot just past the post. Mainz's compact defence was crowding Bayern out, much to their frustration.
Cordoba secures shock win
The opening stages of the second half were nothing like the first, as the opening third produced just two chances of note. Yunus Malli missed the chance to give the Karneval club a two-goal lead, sending his shot well over the bar from a good position. At the other end, Mehdi Benatia wasted a great headed opportunity for Bayern.
Benatia was not left disappointed for long, though, as Robben produced a superb strike to restore parity. Vidal gave the Dutchman the ball, before he picked his spot from 25 yards out and arrowed a shot into the bottom corner to leave Karius helpless.
There was plenty of Bayern pressure late on, but that was soon rendered useless when Cordoba popped up with a late winner. Julian Baumgartlinger picked out the Colombian forward, who beat Neuer at his near post to send the traveling support wild. Mainz manage to navigate the final few plays to secure a famous three points.