It depens what you end up working on, whether it is pure maths or maths related to other stuff.
For me, the most struggling is encountering with people who like only maths and with people who like only languages, because they both think that my kind of work (combining both) is strange and that there is too much languages (for math-lovers) or too much maths (for language-lovers).
Harold commented on :
It is not always an easy task to explain
to family/friends what I do and why I do it – examples: why I still work in College, and how scientific edition works. Another struggle is global competition, that is the worldwide fight for recognition, funding, and jobs.