It depends on whether your irritation comes from contact with pigments, or because particles get into your eyes. If the former, a "buffer" of mineral makeup might help, rather than a primer, which is really intended to help eyeshadow adhere. You'll find that the application will go on softer, but powder on powder might work as a better buffer. If the latter, cream shadows like Shiseido Hydro whatevers.
It is true that shimmers are generally more abrasive, whereas loose minerals are much softer. Jane Iredale makes mineral eyeshadows, I can only assume they're very gentle, though I can't speak from personal experience, but the line is targeted for super sensitive skin.
Black hair, light golden-undertoned skin, dark brown eyes. Asian.