Indeed, the term was exclusively used for Romance-speakers, which points to some variant of Latin remaining in use amongst Britons for quite some time. Interestingly, the Saxons also had a second term for Britons, Cumber, which appears not exclusively in the North but also in the border regions to the earliest Saxon settlements. I've read some theories that this refers to non-romanized Britons from beyond the wall who were military settlers (most writings point to the provincial Romano-British having little to no military tradition) in disputed regions and who later took over when the older order collapsed either due to Saxon conquest or to instability created by the influx of refugees from the east.
There's an article by Schrijver which sets out a convincing case for large scale use of British Latin in the Southeastern corner of Britain at the time of the Saxon advent. He suggests that contact between Saxons and Romano-Britons would have been conducted in Vulgar Latin, which would already be a second language for many Anglo-Saxons. This would explain the low level of Celtic borrowing in Old English.
Interestingly, he also points at a large scale adoption of Brittonic by Latin speakers leading to phonological and lexical borrowing from Latin in Brittonic, which spurred massive grammatical changes. Your theory about Cumbers v Wealhas in early AS England would fit with that, though the key division was probably not the wall, but Britannia Prima/Britannia Segunda. The Welsh word for England "Lloegr" was already in use
before the AS arrival, so there was obviously some point of difference between the Cumbers and the people living in part of England.
It's also hard because we don't know the actual make up of Britain at the time. There are suggestions that the south east was seeing Germanic populations long before the invasions and that the romano-british were only skin deep. Couple that with the difficulty of splitting out Celtic and Saxon culture definitively and mapping that to actual ethnicity...
I'd be very careful, btw, of falling into the trap of reading too much into genetics. Genetics and languages don't go together, and we have no idea of what the genetics of Britain were at
any point in history. All we can say for certain is that until 500AD every single recorded indigenous individual and placename in Britain is Celtic or pre-Celtic. There is not a single indication of Germanic presence in the British Isles prior to the arrival of small numbers of German Roman auxiliaries in the second century, who were not likely to be permanent residents. A certain geneticist published a very silly book on this topic, which unfortunately gained a lot of popular attention and popularised the notion of an early Germanic settlement of eastern Britain, based solely on modern genetic patterns. This was largely ignored by historians, because it was literally too stupid for words, unfortunately the press loved it and it sold a bundle. Academics really ought to try and respond to this kind of thing, but I suppose there is no news in "things largely as we thought because previous generations of scholars (and Julius Caesar) were not total idiots".