Scarlett’s strike just before half-time came against the run of play, with Ipswich enjoying a lot of possession and chances in the opening 40 minutes.
But, once the goal went in, the hosts looked a completely different side, and the former winners grabbed a second deep into stoppage time to book their place in Thursday’s draw.
Unsurprisingly, Ipswich boss Kieran McKenna made seven changes to the side that won 2-0 at Exeter City on Saturday. Captain Sam Morsy and former Portsmouth man Marcus Harness were two of the four to keep their places.
Portsmouth manager Danny Cowley also made seven changes to his side. Tottenham Hotspur loanee Scarlett kept his place while Ronan Curtis and former Ipswich midfielder Ryan Tunnicliffe were among those to come in.
Story of the match
Ipswich began the game with good intent and had a chance to go in front inside two minutes. Morsy clipped a ball from the edge of the area towards the returning Leigh at the back post, but the Jamaican’s low header was well kept out by Josh Oluwayemi.
Just a minute later, though, Portsmouth had a chance of their own when a low cross from Reeco Hackett was parried by Vaclav Hladky, but just away from Scarlett and the hosts cleared their lines.
Owen Dale looked to threaten for the visitors on two occasions. First, he came inside from the right before firing wide and soon was forced out of danger by Richard Keogh after dispossessing George Edmundson on halfway.
On 20 minutes, Harness picked up a knock to the knee and was soon forced off as the Tractor Boys’ injury list continues to mount up.
The hosts continued to have more of the chances and soon let a big opportunity go begging. Leigh’s expert cross found an unmarked Camara, but his low header bounced wide of the target, leaving the former Plymouth Argyle man with his head in his hands.
The source of Town’s next chance was the highly rated teenager Cameron Humphreys. His over-the-top ball for Jackson was recycled for Siziba, who curled an effort wide of the far post as he searched for his first senior goal.
Dale stung the palms of Hladky after racing away into space and evading the sliding challenge of Edmundson, before Pompey did take the lead a few minutes later.
Edmundson again lost possession to Dale before recovering, but the loose ball fell to Scarlett to fire low into the bottom corner, aided by the post, for the teenager’s fifth goal of the campaign.
Ipswich, who had been in control for much of the first half, almost shot themselves in the foot again in stoppage time as Pompey ended the half strongest. Hackett burst through and fed Dale, whose fierce shot was saved by Hladky.
Dale looked to start the second half how he ended the first. With his back to goal, a neat spin saw the Blackpool loanee get a left-footed shot away that was held by Hladky.
The hosts were appearing to lack cohesion in the early stages since the restart, Camara opting to turn down a good shooting opportunity to instead attempt a past that was ultimately intercepted. The crowd made their frustrations known.
Their anger could have been worsened minutes later when Hackett’s cross was headed onto the roof of the net by Pompey’s Suffolk-born captain Michael Morrison.
Kieron Freeman soon was required for some no-nonsense defending as he hacked away a dangerous ball after Keogh had recycled an earlier set-piece.
Scarlett tried his luck from the halfway line but Hladky recovered despite the effort appearing to be off target in any case, before Wes Burns saw a low cross evade Jackson at the other end moments later.
Koroma powerfully struck into the side-netting late on and Ipswich continued to fire crosses into the box in the final stages in a desperate attempt to send the game to penalty kicks.
Burns went down inside the box in stoppage time under the challenge of Denver Hume, but referee Tom Nield was not interested.
Koroma fed Swanson in the final seconds to slot home on the counter-attack and it was the Hampshire side who march on in the competition to knock the Tractor Boys out at this stage for the second straight season.
Player of the Match
The Player of the Match is Portsmouth’s Owen Dale.
The winger came into life in the latter stages of the first half. He set up the opener and could have grabbed a couple of goals himself if not for Ipswich goalkeeper Hladky.
Completed the full 90 minutes and played a big part in Pompey booking their place in the next round of the competition.
Ipswich Town (3-4-2-1): Hladky; Edmundson, Woolfenden (Burgess 62), Keogh; Leigh (Burns 62), Humphreys, Morsy (c), Vincent-Young; Harness (Siziba 24), Camara (Armin 75); Jackson. Subs: Coleman, Ladapo, Agbaje.
Portsmouth (4-4-2): Oluwayemi; Hume, Ogilvie (Robertson 67), Morrison (c), Freeman (Swanson 67); Curtis, Mingi (Raggett 67), Tunnicliffe, Dale; Scarlett (Koroma 78), Hackett (Bishop 67). Subs: Griffiths, Gifford.
Booked: Morrison, Dale, Koroma.
Referee: Tom Nield (West Yorkshire).
Attendance: 5,686 (245 away).