Reed Elsevier's Food and Chemical Toxicology (FCT) journal, which published the study by the French researcher Gilles-Eric Seralini in September 2012, said the retraction was because the study's small sample size meant no definitive conclusions could be reached.
"This retraction comes after a thorough and time-consuming analysis of the published article and the data it reports, along with an investigation into the peer-review behind the article," the journal said in a statement.
"Ultimately, the results presented -- while not incorrect -- are inconclusive, and therefore do not reach the threshold of publication for Food and Chemical Toxicology," the journal stated.
At the time of its original publication, hundreds of scientists across the world questioned Seralini's research, which said rats fed Monsanto's GM corn had suffered tumors and multiple organ failure.
The European Food Safety Authority (EFSA) issued a statement in November 2012 saying the study by Seralini, who was based at France's University of Caen, had serious defects in design and methodology and did not meet acceptable scientific standards.
Within weeks of its appearance in the peer-reviewed journal, more than 700 scientists had signed an online petition calling on Seralini to release all the data from his research.
In its retraction statement, the FCT said that, in light of these concerns, it too had asked to view the raw data.
Seralini "agreed and supplied all material that was requested by the editor-in-chief," it said.
The journal said that, while it had received many letters expressing concerns about the validity of the findings, the proper use of animals and even allegations of fraud, its own investigation found "no evidence of fraud or intentional misrepresentation of the data".
"However, there is a legitimate cause for concern regarding both the number of animals in each study group and the particular strain selected," it said.
Seralini, who works in Caen with a group called CRIIGEN, the Committee for Research and Independent Information on Genetic Engineering, said the journal's criticisms of his work were "unacceptable".
"Were FCT to persist in its decision to retract our study, CRIIGEN would attack with lawyers, including in the U.S., to require financial compensation for the huge damage to our group," he said in a statement.
Other scientists, however, welcomed the journal's decision, although some said it had come too late.
"The major flaws in this paper make its retraction the right thing to do," said Cathie Martin, a professor at John Innes Centre. "The strain of rats used is highly susceptible to tumors after 18 months with or without GMO (genetically modified organisms) in their diets."
David Spiegelhalter, a professor of the Public Understanding of Risk at the University of Cambridge, said it was "clear from even a superficial reading that this paper was not fit for publication". In this instance, he said, the peer review process had not worked properly.
"But at least this has now been remedied and the journal has recognized that no conclusions can be drawn from this study, so I suppose it is better late than never," he said.