The Hammers almost had the lead inside two minutes when Cheikhou Kouyate's was inches away from scoring but Leicester took over from there with chances for Robert Huth and Danny Drinkwater before Jamie Vardy struck home in the 18th minute.
Andy Carroll, Huth and Shinji Okazaki all had chances to score at the beginning of the second period before Vardy was given his marching orders ten minutes after the restart. Winston Reid had a good opportunity but it was Carroll and Aaron Cresswell's combination which looked to have taken all three points but Ulloa's spot kick shared the spoils.
Hearts in mouths
This clash at the King Power Stadium brought together two of the surprise packages of the season and it was visitors who came within inches of taking the lead after two minutes.
Dimitri Payet's set piece ability was clear to see once again as his free-kick into the area found Kouyate. Lady luck was very much on the side of the home side as a touch from Kasper Schmeichel saw the effort clatter one post trickle along the goal line off the opposite post and into the arms of the Dane.
Claudio Ranieri's side took over from there as they had two half-chances to take the lead. The first came from Huth as he rose highest to Marc Albrighton's free-kick but the header was just wide of Adrian's post.
The second came off the back of a great run from Riyad Mahrez who eventually teed it up for Drinkwater but the recent England international was way off target with the effort.
Vardy strikes again
Vardy returned to the goals last weekend in the 2-0 victory over Sunderland and he was it again just before the 20-minute mark.
It all came from the keeper as Schmeichel's magnificent throw to Mahrez who laid it on for N'Golo Kante. The Frenchman hit a great through ball to Vardy and he showed great composure to thunder it across Adrian and into the far corner.
Bilic brought on Carroll as a half-time substitute to make an impact and almost did when he dived in to get his head on Payet's looking to build on his hat-trick last week but it was wide of the mark.
It was then the turn of Huth once again as Christian Fuch's ball looked perfect for the German but it proved just too high for the defender as it sailed wide.
Hero to Zero
Vardy was inches away from, getting on the end of Okazaki's effort across the face of goal but that proved to be the England internationals final contribution.
Vardy went through on goal with Angelo Ogbonna and the striker seemingly brought his leg across the defender to try and draw the foul and referee Jonathan Moss had no hesitation to hand him a second yellow.
After the dismissal, the Hammers went searching for a way back into the game and almost had it just after the hour mark in spectacular.
Another Payet set piece found Reid this time and despite pressure from Wes Morgan, he looked to spectacularly turn it home with a back-heeled effort but it couldn't skim the post on its way out.
Leicester looked to be heading for another 1-0 victory as the match entered the final ten minutes but a late surge from Bilic's side looked to have snatched the three points away from the Foxes.
The home side, in particular Morgan had been walking a fine line in terms of their defending and his foul on Reid from Payet's corner proved to be the straw that broke the camels back as Moss pointed to the spot.
It was the first penalty away from Boleyn Ground for the Hammers in five and a half years and Carroll made no mistake as he struck home into the bottom corner.
With four minutes to play, West Ham looked to have snatched all three points in spectacular style. Michail Antonio's cross into the area was headed into the path of his defensive partner Cresswell and left-back did well to take it down on chest and did even better when he struck it beyond Schmeichel and into the top corner to send the King Power into a stunned silence.
There were calls for a penalty from the home side as Ogbonna seemingly had his arm around the throat of Huth but nothing was given but he right at the death, they managed to snatch a point.
Moss seemed to make up for his earlier decision as Carroll softly bumped Jeffrey Schlupp which saw him point to the spot and Ulloa sealed the draw as he coolly slotted past Adrian.