See
here and
here.
Possible reasons:
1) Japanese literacy was associated with Chinese (Japanese literally had no writing system before the introduction of Chinese characters AKA kanji.), and math is also associated with literacy.
2) Kun'yomi pronunciations of numbers tend to be longer than their on'yomi counterparts. Kun'yomi abbreviations (ひ、ふ、み) are also less distinctive and easily confused for other sounds.
Exceptions and possible reasons:
1) 零 is a relatively new word when compared to the other numerals. (ancient Chinese mathematicians would use "空","口", and "無入" instead.) Zero also sounds rather distinctive, especially when compared to "Rei". (which is homophonic with a lot of common Japanese words.)
2) 四, when pronounced シ, is homophonic with 死, or death.
This is considered to be a bad omen in many cultures in the Sinosphere.
3) 七, when pronounced シチ, sounds rather similar to イチ, and hence can be confused for 一.
4) 九 is homophonic with 苦 when pronounced ク, but 九 has an alternate on'yomi pronunciation, キュウ. ここのつ is also rather long, and possibly doesn't sound as distinctive as its on'yomi counterparts.
5) When numerals are used as
ordinal numbers (When counting the order of something as opposed to its numbers. e.g. First instead of One.), the kun'yomi pronunciation is slightly more common. That said, ordinal numbers tend to be attached to 第 (第一, for example.), in which case they would generally have an on'yomi pronunciation instead.