It is possible that the Indo-Aryans at that time did not undergo a wholesale replacement of the earlier populations, this is the case in many migrations, where the societal superstructure undergoes transition to the hands of a different ethnicity, but most people's ethnicity doesn't change much. This is also something that, if I recall correctly, is the case with the Saxons in Britain, where we are finding now due to genetic research that for the most part, the earlier Britons likely were not replaced, but adapted to the culture and language of the invaders - as contrasted with much of North East Britain, where the Norwegian and Danish Vikings for the most part transplanted the entire population. However, looking at the genetic research, apparently things point to the wholesale integration of Indo-Aryans around 3500 YPB (http://genepath.med.harvard.edu/~reich/ ... _India.pdf
I wouldn't say it's absolutely certain that the Buddha was a Dravidian or "Ancestral South Indian", yes the 'sangha' is called black, but the Buddha is individually usually referred to as golden. This might suggest that he was closer in relation to some of the Tibet-Burman hill tribesmen of Nepal, who are yellow/golden skinned. Though certainly the caste rejectionism of Buddhism probably played a bigger role than the texts suggest, and likely attracted a larger number of outcastes, black and yellow, than would be suggested by the thesis that the Sakyas were Indo-Aryans and that the Buddha 'reformed' 'Hinduism'. Regardless, I think it most likely that he was, as much of the Sangha probably was, a mix of Indigenous and Indo-Aryan ethnicities - I don't buy into the idea that the presence of Brahmanical elements (e.g. claiming to be descended from the solar dynasty) in the Sutras is a conspiracy of later Brahman-caste compilers. Everything seems to point to the idea that these groups mixed very early on, and developed more racist ideologies later - otherwise the genetic data wouldn't show that most Indo-Aryans are mixed with Ancestral South Indians, or that there are no pure Ancestral South Indians at all any more, except on the Andaman Islands. I think the idea of thinking of the Buddha, Sangha, or anyone, as one race or another, is probably missing the point about the early history of these peoples - not only did they mix, but there's pretty much no one who is purely one race or the other anymore - the Indian ethnicities are mixes of all sorts of things, outcastes are Indo-Aryans, just as Brahmins are Dravidians. This is also one of the main argument against the Mimamsaka view (specifically that of Kumarilla) of Caste - clearly the castes didn't remain pure since the time of Manu, since Brahmins sometimes also have black skin, large lips and curly hair, and Sudras sometimes have olive skin and aquiline noses.
On that note, it might be useful to look at what is actually said about caste, and it's somewhat strange that we find very little caste rejection after the Nikaya/Agama literature, i.e. nothing in Abhidharmic writings or early philosophical writings. It's only in the 6th Century CE that Dharmakirti engages in an argument against caste - and strangely, he doesn't refer to any of the arguments in Nikaya/Agama literature, though later writers do. In Eltschinger's "Caste and Buddhist Philosophy
," he looks at these arguments, including those of Aryadeva, Vasubandhu, Dharmapala, and Candrakirti. It's possible that by the 6th Century, Buddhists, realising they were seriously on the decline, had to appeal to anti-caste sentiments to attract outcastes to the order and maintain their numbers. But fundamentally, without the support of a king (of which the Palas provided the final, but unsuccessful, aid), the Sangha couldn't maintain itself - at least in the traditional monastic setting, which is one major factor in the upsurge at that time in lay/non-celibate monasticism.