Nowadays, according to stats about 70-80 % of your traffic comes from mobile.
So yes, mobile should be your first priority.
Check what performance testing tools are saying (gt metrix, pingdom, lighthouse…)
Normally they provide results both for desktop and mobile an give some guidelines to optimize and improve your results.
Mobile results are always worse :
First you have a smaller processor running. Next your connection is not quite often that good. You can simulate a slow connection via chrome dev tools. It’s quite eye-opening!
Optimize your pictures, via tinypng.com.
Avoid using page builders (elementor and Divi are the worst). Bloated code.
Consider using some performance plugins.
I personally use wp-rocket together with “asset cleanup”. It’s a good combo.
If your audience is worldwide, a CDN is a good investment. Otherwise a waste of time. Always favor a hosting next to your audience.
Cheers,