You've been beaten to it in at least a couple of settings. From memory...In one world, people who can't work magic are known as Sleepers, and they make up the vast majority of the population. As in "99.9999 percent and you could maybe throw some extra nines on." For that place, reality is based on a sort of consensus belief, and there are a few who believe magic is real and they can do it. In practice, it's more complicated -- but that's the core: believe and do. However, Sleepers don't believe in magic -- which is what makes magic hard to work in the first place: the consensus reality resists being changed. In particular, if you work spells with visible effects around them, you are begging for the magic to fail and backlash spectacularly on you: a single Sleeper can serve as a channel for several billion who collectively decided what you're doing is impossible. And then you get punished for it. (The will of the many is strong. And stupid.) They generally can't shut down magic just by being near it, but they can make it very difficult to accomplish and much more likely to take you out if you can't make it work anyway.
(By the way, in that world, highly-advanced science has the same problem as magic: if the population isn't ready to accept it and you can't make them believe it'll work, it doesn't. The prototype works fine in the lab, but take it out in the field...)
In another, we have Mundanes. Mundanes are really narrow-minded. They've shoehorned reality into a tiny box which consists of Everything I Personally Understand, and they will not let anything new into that box. Ever. And the more closed-off a Mundane is, the more they inflict their tiny version of reality on everything around them. It's temporary -- the field moves with the Mundane and leaves when they do -- but they wreak havoc on anything supernatural. Spells stop working. Psionics shut down. Vampires find themselves turned into normal humans wearing dollar store novelty teeth. Werewolves become guys in furry costumes with visible zippers. Aliens get rubber suits. The Mundane will not allow anything he doesn't understand to exist -- and so remakes it into something he can barely file under 'silly, pointless, goodbye'. No word on how they generally vote.
Sleepers are protected: after all, there's a tiny chance some of them might come to their senses and join the ranks of the spellcasters. Mundanes are loathed -- unless something supernatural is chasing you and your only chance is to make it run through the field.
Either way, they make for pretty boring protagonists.