"It's not meant for me. For me the national team is over. I've done all I can, it hurts not to be a champion," the 29-year-old said after defeat by Chile at the Copa America.
With Barcelona, Messi has won eight La Liga titles and four Champions Leagues.
But his only major international honour is 2008 Olympic gold.
Argentina were beaten 1-0 in the 2014 World Cup final by Germany before two Copa America final defeats by Chile on penalties. Messi was also on the losing side against Brazil in the 2007 Copa America final.
"It's been four finals, I tried. It was the thing I wanted the most, but I couldn't get it, so I think it's over," he added.
"I think this is best for everyone. First of all for me, and then for everyone. I think there's a lot of people who want this, who obviously are not satisfied, as we are not satisfied reaching a final and not winning it.
"It's very hard, but the decision is taken. Now I will not try more and there will be no going back."
Messi had scored five times at the tournament, including a superb free-kick in the semi-final against hosts the USA to become his country's record scorer with 55 goals.
Argentina and Manchester United goalkeeper Sergio Romero said he hoped Messi would "reflect" on his decision and reconsider, adding: "I think he spoke while he was heated, because a beautiful opportunity escaped us. I can't imagine a national team without Messi."
Manchester City forward Sergio Aguero described the mood in the Argentina dressing room as "the worst I've ever been in", and was quoted by South American media as saying: "There are several players who are evaluating not continuing with the national team."
After Sunday's match finished 0-0 in 120 minutes, Chile won 4-2 on penalties.
Messi's miss was Argentina's first attempt in the shootout and it ballooned over the bar after Romero had denied Chile's Arturo Vidal.
Lucas Biglia also missed from 12 yards, with Chilean substitute Francisco Silva scoring the decisive spot-kick in the final in East Rutherford, New Jersey.
Messi made his debut for Argentina in 2005, going on to make 113 appearances.
For Barca he has scored 453 times in 531 games, including a La Liga record 312 goals. He has won the Ballon d'Or award for the world's best player five times.
In 2014 he was named as the best player at the World Cup, with then-Fifa president Sepp Blatter saying he was "a little bit surprised" and Argentina legend Diego Maradona calling the decision "unfair".
Argentina are already six games into their qualification campaign for the 2018 World Cup. They are third in the table, two points behind leaders Uruguay.
The top four teams progress automatically to the finals in Russia, while the team finishing fifth qualifies for a two-legged play-off against the best team from Oceania.