Sebastian Schuppan's free-kick gave Bielefeld the lead, but a Wolfgang Hesl error allowed Stephan Fürstner to level. However, Fabian Klos made it 2-1 before the break.
Christopher Nöthe added a third, only to see Steven Skrzybski and Collin Quaner restore parity for Union once again.
They even went ahead through Skrzybski's second of the game late on, yet a David Ulm strike secured a share of the spoils for Bielefeld.
Dull opening quickly forgotten about
The game was a slow burner to begin with, as both teams tried and failed to make any early inroads. Union were fairly limp in attack while the home side came close through Julian Börner's header, and Schuppan's delivery again caused problems when his cross looped narrowly over the bar.
That was all the first 30 minutes could provide, but the final hour was more than worth waiting for. The first of eight goals in 60 minutes came after Schuppan was stood over the ball, 20-odd yards from goal. Slightly to the right, he curled a lovely free-kick up and over the Union wall as Jakob Busk was left to watch it find the top corner.
The Berliners issued an immediate response when Damir Kreilach's set-piece curled just wide of Hesl's post, though they would get level from Kreilach's midfield partner. Some high pressing from the capital club saw Union win the ball back and eventually find Fürstner, free on the edge of the area to rifle through a sea of bodies and Hesl's legs.
Any joy they had was shot-lived and Bielefeld went back in front five minutes later. A superb piece of play from Christoph Hemlein say the winger burst down the right and put in a superb ball to the back post, where Klos was waiting to tap in from mere yards out. Michael Görlitz nearly made it three before the break, but Busk made a brilliant save.
Ulm earns a point after Union score three unanswered goals
Despite the flurry of action at the end of the first half, the restart provided few clear cut chances. That changed when Hemlein went on another mazy run down the wing and found Nöthe on the edge of the area. He volleyed into the pitch andd it seemed an easy catch for Busk, but instead he merely jumped and let the ball go in to make it 3-1.
From there, Union were spurred on like never before and pulled one back three minutes later. Felix Kroos had the presence of mind to slip in Skrzybski with a quick free-kick on the edge of the area, and the forward sent a shot arrowing into the far bottom corner to rejuvenate the visitors.
Where there was one, there was two and Quaner made it a level game once more. A lovely move saw Skrzybski get in behind the defence and cut-back towards the tall attacker, though it was slightly behind him. That mattered little has he swiveled and cracked the ball off the underside of the bar, just crossing the line.
The game went end to end after that and, true to form, that's how the two final goals were scored. Skrzybski gave Union the lead for the first time in the game after he raced cleared, looped the ball over Hesl and headed into an empty goal. However, Bielefeld had the last laugh as Ulm's sweet strike earned a point in an enthralling encounter.