Kaffir is from Arabic, and means "unbeliever" or "infidel", ie. someone who is not Muslim.
Lemons/limes probably originated in south east Asia,(maybe in North India) but are known to have been cultivated in Persia (now Iran) in the 7th century BC. The Arabic (or perhaps, Persian) word is limun, so "lemon" and "lime"are, most probably, English transliterations, like the word kedgeree, transliterated from kitchari, in Sanskrit.
Lemons and limes have a fairly smooth skin, whereas the Thai lime is knobbly . In Thai, it's called makrut.
I can only imagine that, when these knobbly limes came to the middle east (and perhaps further west), they might well have been considered "infidels", because they weren't smooth, like the local variety.