After Josef Martinez gave Atlanta a 75th-minute lead, New York came up with two goals in the final eight minutes of action, Lewis Morgan leveling with a penalty and Serge Ngoma winning it in the 89th minute.
The goal by Ngoma made him the youngest player to score in franchise history at just 16. His strike was the third-latest by a player 16 years or younger in MLS history.
Story of the match
Atlanta threatened in the 27th minute as Franco Ibarra fed the vertical run of Ronald Cisneros with a ball over the top. His close-range shot off the bounce sailed over the net.
Five minutes later, Matheus Rossetto found Luiz Araujo, who recorded the first shot on goal of the night, but New York goalkeeper Carlos Coronel was on hand to deny him.
Later in the half, Araujo should have opened the scoring after a sensational pass from Aiden McFadden found him, but his first touch was poor and he shot wide.
On 53 minutes, some neat buildup play by Araujo and Rossetto resulted in Cisneros shooting from just outside of the box, Coronel saving his effort.
Finally, in the minute, the Five Stripes scored their first-ever goal at Red Bull Arena. Araujo sent a through ball to Caleb Wiley, who chipped it to the far post.
McFadden headed inside the six-yard box where Emerson Hyndman tapped a shot that was blocked, but Martinez was on hand to finish off of the rebound.
The Red Bulls would equalize when Wiley was called for a foul in the box on a hard tackle and Morgan was able to convert from the spot to set up a grandstand finish.
A mistake by the visitors led to New York's winner a minute from time as a takeaway eventually led to Ngoma squeezing a low-driven ball past Rios Novo.
Man of the match: Serge Ngoma
Who else would it be? He not only made history, but his goal handed New York another victory and brought them joint top of the Eastern Conference. Not a bad way to mark your maiden goal.