Harvey Barnes opened the scoring for the Foxes after 13 minutes but Leeds were level within two minutes through Stuart Dallas.
The second half forced Leeds to weather a Leicester storm as Brendan Rodger's men came out strong but two clinical attacks from the Whites saw them win their first game against a top-six side this season.
• How to win away from home
Like in the reverse fixture, Leicester took the lead through the in-form Barnes fairly early on.
At this stage, many people would have written off Leeds and expected the flood gates to open but in fact, the exact opposite happened.
Bielsa's men did not dwell on conceding and just two minutes later, Dallas leveled proceedings with a lovely finish across goalkeeper Kasper Schmeichel.
Leeds were straight back in it with a typical Bielsa goal. Full-back, Luke Ayling drove down the right flank before playing it into Patrick Bamford who had the presence of mind to see Dallas' run, finding him with a perfectly weighted pass and the Northern Irishman did the rest.
For the rest of the first half, Leeds were certainly the better team. Apart from a well-worked Leicester move that saw Jamie Vardy's replacement, Ayoze Perez find the net from an offside position, the Foxes struggled to create anything clear cut.
Leeds themselves had a goal disallowed for offside when Mateusz Klich tapped home from close range before Schmeichel pulled off a world-class save to deny a Bamford header.
Schmeichel was one of the main reasons Leicester were still in the game at the half-way point. He was again forced into action when wingers Jack Harrison and Raphinha struck goalbound efforts.
A tactical change from Rodgers at half time saw the home side opt for a three at the back system, which allowed them to pile much more pressure on Leeds.
For the first time this season, Leeds were beaten on possession and that is certainly something that has been a rarity under Bielsa for Leeds.
Leicester controlled the second half and looked much more of a threat on the Whites' goal. When teams have piled pressure on Leeds this season, they have usually buckled, but not yesterday. Center-backs, Pascal Struijk and Liam Cooper seemed to block, tackle and clear the majority of Leicester's danger.
During the extended period that the Rodgers' men were on top, it was vital that Leeds defended resolutely, held firm and for the first time Bielsa's reign surrender possession.
Some important saves from Illan Meslier, combined with a stroke of good fortune saw Leeds stay in the game at 1-1, up until the 70th minute.
As the second half went on, Leeds began to press higher up, forcing some Leicester mistakes in their own half and their pressure resulted in a goal.
Struijk latched on to a stray pass, before playing it into the lively Raphinha who found Bamford with a first time ball and Bamford expertly fired his left-footed shot into the top right corner, giving Schmeichel no chance.
Leeds now had something to hold on to and again had to defend in numbers as Leicester tried to equalise. The closest they game was when Caglar Soyuncu flicked on a header from a corner but a stretching Jonny Evans just couldn't direct the ball goalwards.
In the last ten minutes, it was all Leicester.
Then, Leeds completed the perfect away performance. After defending heroically all game, they counter-attacked after a Leicester free-kick. Bamford found himself one-on-one with Schmeichel but he laid it into the onrushing Harrison who tapped it into an empty net to make it 3-1.
In the Premier League, it is so important to defend well. The way Leeds attack, always gives them a chance when they are still in the game and at 1-1, the game looked to be going in only one direction.
But sit in, weather the storm and emphatically hit your opposition on the break, epitomised just how to win away from home.
A near on perfect performance from the Yorkshiremen.
• Bamford at his best
Before yesterday's affair, Bamford had not hit the net in the Premier League for four games.
Three pretty lackluster performances against Tottenham Hotspur, Brighton & Hove Albion, and Newcastle United for not just Bamford but the whole team meant that the striker's impact on recent games had been fairly minimal.
Bamford was last on target against Burnely when his early penalty secured a 1-0 win for the Whites.
Coming into the Premier League season, Bamford had many critics, with some of them even being Leeds fans. Since Bielsa's arrival, he has shown complete belief in Bamford and his ability. He has been a key figure in the way Leeds play and does a lot more than just score goals.
Yesterday he was certainly at his best like he was earlier on in. the season against Aston Villa. His movement was excellent. He held the ball up, brought others into play and forced the Leicester defenders into making a decision as to whether they should go tight and try to win the ball or standoff and then try and tackle him.
His first touch was textbook and his weight of pass on a number of occasions including both of his assists was magnificent.
When Bamford plays like he did yesterday, he is a real handful and whilst he has been good for the majority of the season, maintaining consistency will now be key for him and Leeds in their surge up the table.
Bamford had two shots yesterday, the first being the header that Schmeichel superbly saved and then his goal. Clinical.
To give a mention to his goal, his awareness of letting the ball run onto his favored left foot is something that may go unnoticed. Then there's the finish, nobody is saving that one; off the underside of the crossbar and straight into the top corner.
His assist for Harrison portrayed his unselfish side and how much of a team player he is. An excellent outing from Bielsa's frontman.
• Injury Woes
After touching on mainly positives from yesterday's game, there were some negatives regarding injuries. Both Rodrigo Moreno and Raphinha came off injured, with Rodrigo's looking the more serious.
Bamford was seen holding his leg at the end but Leeds will hope that their attacking threats are all not too seriously hurt.
Diego Llorente and Robin Koch are both still sidelined, meaning all of Leeds' summer signings could miss out on their next fixture.
Leeds ended yesterday's game with 11 players who were all present in last season's Championship squad and whilst they have done a good job of dealing with their absentee's so far this season, everyone involved with Leeds will be hoping their injured men are back sooner rather than later.
Next up for Leeds is a home tie against Everton on Wednesday, where they will be hoping to build on what was an outstanding result against a strong Leicester City.