"The earliest known version of the story in its "turtle" form
   appeared in 1854, in a transcript of remarks by preacher Joseph
   Frederick Berg addressed to Joseph Barker: My opponent's
   reasoning reminds me of the heathen, who, being asked on what
   the world stood, replied, "On a tortoise." But on what does the
   tortoise stand? "On another tortoise." With Mr. Barker, too,
   there are tortoises all the way down. (Vehement and vociferous
   applause.) * "Second Evening: Remarks of Rev. Dr. Berg"   It
   originally came from a preacher making fun of the heathen. But
   like all stories that are polemical, it changes form and it got
   flipped around to support the oppositions arguments.