First-half goals from Sadio Mane and Philippe Coutinho established a comfortable lead for Klopp's men at Anfield, but they failed to build on it as they squandered a glut of second-half chances.
Gareth McAuley halved the deficit with less than 10 minutes remaining, firing in at close-range from a corner as the Reds' most glaring weakness reared its ugly head.
It meant that Liverpool have conceded 17 set-piece goals under Klopp - more so than any other Premier League team in that time - and set up a tense finale.
They ultimately held on to go second on the same points as Arsenal, and the manager says he has no concerns over his side's defending, saying that it is generally improving as they continue to cut down the number of chances they afford their opposition.
"In dreamland, you always win four or five-nil and then the crowd can go [home] a little earlier and do something a bit more serious," Klopp told journalists at his post-match press conference.
However he said that the nature of the game meant it was "one of the best atmospheres" they have seen this season "because it was exciting until the end" with "the chances we created" as well as "the chances we didn't use."
The German suggested that in the aftermath it felt "better" that Liverpool "only won 2-1 rather than three or four-four nil" which he said "would have been possible."
No "big" problems at the back, says Reds boss
On whether his side have issues defensively, Klopp continued: "No. Obviously not having a clean sheet is part of our games but it's not a big problem. I think defending, first of all, is not to let them [the opponents] create a lot of chances and I cannot remember a lot of chances in the last few games."
He suggested that the goals Liverpool have conceded have come from "a lucky cross or something or a set-piece", dismissing it as "part of the game" rather than down to their own errors as he also insisted that his side "are improving in this."
Klopp felt that McAuley's late goal "was kind of lucky" because "there were seven players" inside the area and despite the "brilliant" corner delivery, a few defenders were "falling down" and the Baggies "were quicker in mind and scored the goal."
"They were one time quicker in mind in the game than we were and scored a goal from it," Klopp said, explaining "that's how football is" as he admitted that they would have had "to accept it" if West Brom scored a second goal to draw level.
But the Reds boss added that had Liverpool drawn 2-2 then it "wouldn't have changed my opinion of the game" because "playing football like this is not usual."
He praised his team for "staying patient, being lively, staying concentrated" and "being aggressive without making fouls" because they knew that "each free-kick is a real threat" even inside West Brom's own half.
Klopp also declared that Liverpool "did brilliantly" up against what was "a real challenge" against Tony Pulis' men as they stretched their unbeaten run to nine in all competitions.