The win, which in the end was comfortable, started in the first-half when the Blues found themselves on the back foot and wading off pressure from the bitey Terriers.
Through the afternoon, they chanted - in no uncertain terms - about the displeasure of their current manager which sparked a number of glances towards the away stand from Allardyce.
On the pitch however and against the run of play, the visitors took the lead just before half-time through Tosun.
Theo Walcott picked the ball up midway inside his own half but drove forward and beat two defenders on his way inside the Terriers half.
His forward pass to Tosun was scuffed but found its way into the path of the Turkish forward.
From a few yards outside the 18-yard box, Tosun arrowed a low effort across Jonas Lossl’s net and saw his effort nest low to the Danish goalkeeper's right as he wheeled away in celebration.
Gueye doubled the Blues’ lead late into the second-half as they benefitted from some poor Huddersfield defending.
Oumar Niasse drove down the right-hand side and whipped a ball into the box but missed everyone.
Leighton Baines recovered and laid a pass off to a waiting Gueye who fired his effort from the edge of the box past Lossl, wrapping up the three points for the visiting Blues.
Hosts dominate but trailed
Before finding themselves behind, David Wagner’s side were in the ascendancy at home and looked like they were going to snatch an early lead.
Colin Quaner had an effort cut out by Baines before Quaner linked up Aaron Mooy to flash a dangerous cross across the Everton box.
It was the wide areas where the hosts found joy throughout the first period.
Plenty of the Terriers pressure came down the right-hand with Alex Pritchard and Florent Hadergjonaj - looking to take advantage of the banged up Baines who had little support from Walcott in front of him.
The hosts then had a shout for a penalty just before the half-hour mark when Rajiv Van La Parra went over just inside the box.
Replays showed that the winger ultimately tripped over his own foot and not due to the pressure from the trailing Nikola Vlasic.
Moments later, Van La Parra curled an effort over the crossbar following a clever cutback from Pritchard in an effort that probably should have given the hosts the lead.
They found themselves trailing a few minutes after Tosun finished off Walcott’s counter-attacking effort.
Blues better in the second period
The Blues nearly doubled their lead almost instantly after the half-time break as Walcott was once again their key attacking outlet.
He worked his way into the Huddersfield box before cutting back to an unmarked Gueye. The midfielder laid a pass into the path of Vlasic but the Croatian, who tried to wrap his foot around the ball, could only power his effort into the waiting arms of Lossl.
Everton had a number of chances to add to their lead but were unable to beat Lossl.
Vlasic linked with Seamus Coleman on the overlap just before the hour mark but the Irishmen’s powered low effort was stopped well by the Danish shot-stopper.
Wagner’s side introduced Phillip Billing for Quaner on the hour in an effort to open Everton up and level proceedings.
Similarly to the first-half, the Terriers found joy out wide as Van La Parra looked to turn provider for Pritchard in an effort that switched their roles to the first-half chance.
However, where the winger skied his chance, the attacking midfielder was unable to make contact with his and saw Everton clear their lines.
The Blues wrapped up the win thanks to Gueye's fierce finish from range leaving the hosts looking over their shoulder as their fellow strugglers picked up points elsewhere.