I think some mathematical concepts are difficult to learn because they're taught poorly. Archaic language. It's not a natural language. I see mathematical constructs as a subset of natural language constructs. There's a lot to ignore that are are normal in natural language. Within the brain, afaik, mathematics beyond numerousity (I think that's the term) is linguistic. WE instantly assess - mechanically - 3 objects at once, maybe 4. But beyond that, it's a language with its own grammar.