NOT FRUITCAKES
While referring to someone as a fruitcake, it really means that they are crazy or that their behavior is very strange.
When the "migrant caravan" of the 19th century consisted of the indigenous inhabitants of this country - as they suffered forced displacement and relocation - forced to march westward after the U.S. government coerced treaties and used the U.S. Army against those resisting - many were treated brutally.