Resists, % to hit, Spell Penetration and what it means in PvP...
Sources:

http://forums.worldofwarcraft.com/th...7&tmp=1#post37

http://forums.worldofwarcraft.com/th...p=1#post952187
Basics:
A level 60 player vs another level 60 player has a *base* 96% chance to hit with spells. You can reduce the chance to resist (some people refer to this mechanic as missing) based on level difference by using +% hit gear or talents (eg. Arcane Focus, Elemental Precision, Shadow Focus, Suppression etc).
As far as resist gear goes, 300 resist is the effective cap for equal level players, giving a 75% chance to resist over time. 200 resist would be 50% and 100 would be 25%.
Example:
Talents such as Priest's Umbreakable Will increase the chance to resist (in this scenario silence and stun effects) based on level by 15% in this case. This would result in a 81% chance for a Mage to land a Counterspell on a Priest specced this way (in PvP the vast majority will be). Assuming 0 arcane resists on the Priest, a Mage can spec into Arcane Focus for an extra +10% chance to hit bringing that chance to hit figure up to a more manageable 91% chance to hit.
Counterspell cannot be partially resisted (it is in effect a binary spell) so in this case the calculation would look as follows:
0.96 (base chance to hit before talents and gear, aka 96%)
+0.10 (extra chance to hit from Arcane Focus, aka 10%)
0.15 (extra chance to resist from Unbreakable Will, aka 15%)
______
0.91 (chance to hit, aka 91%)
0.91 * 1 (assuming no extra arcane resists on the priest) = 91% chance to hit
Lets say the priest has stacked arcane resist to 100. This in effect gives a further 25% chance to resist arcane spells, which Counterspell is one of. The calculation would look as follows:
0.91 * 0.75 (being 75% chance to hit taking into account arcane resist) = 0.6825 (aka 68.3% chance to hit)
Now a mage can quite easily achieve 70 resists (or spell penetration) bringing that 100 arcane resist down to 30.
At 30 resist thats roughly a 7% chance to resist over time. The calculation would look as follows:
0.91 * 0.93 (chance to hit taking into account net arcane resist) = 0.8463 (aka 84.6% chance to hit)
At the end of all the calculations the chance to hit cannot exceed 99%, that is the hard cap. There will *always* be at least a 1% chance to resist.
In the example above you can see the effect of +hit talents and what are in effect hit talents, as well as how gear resistance and spell penetration comes into play.
Nonbinary spells:
Lets look at fireball (this should work for any nonbinary spell such as mindblast or shadowbolt as well).
Level 60 vs level 60:
0.96 (base chance to hit, aka 96%)
Lets assume the mage has 3/3 Elemental Precision (+6% hit)
0.96 + 0.06 = 1.02 (aka 102%)
Since resists are hard capped at 99% that 3% extra is wasted in this case. After the level based calculation is done, +hit doesnt come into it again (unlike in binary spells). The next calculation is based on resist gear. Lets assume the target has 100 fire resist, equating to 25% resistance to fire over time. Fireball can be partially resisted so a calculation is done at this stage to determine how much of the damage is resisted. This calculation can be seen in detail here:
http://www.worldofwarcraft.com/info/...sistances.html.
Obviously by wearing spell penetration gear you can lower the effective resistance of your target and that directly effects the secondary resist gear based calculation.
Binary spells:
Lets look at frostbolt (this should work for any binary spell such as mindflay, counterspell or silence).
Level 60 vs level 60:
0.96 (base chance to hit, aka 96%)
Again, let us assume 3/3 Elemental Precision (+6% hit)
0.96 + 0.06 = 1.02 (aka 102%)
The same so far, however, being a binary spell a second calculation is made at this point using a combination of the chance to hit *and* resist gear combined. Lets say that the target has 50 frost resist. That equates to a 12.5% chance to resist damage over time, in the case of binary spells there are no partial resists so it is all or nothing.
1.02 * 0.875 (aka 87.5% chance to hit, 12.5% chance to resist) = 0.8925 (aka 89.3% chance to hit)
In this case due to the binary calculation the extra +hit % is being used and, is in effect, acting as spell penetration.
Let us now assume the Mage has 20 resist (spell penetration):
50 resist = 12.5% chance to resist
20 resist
__________
30 resist = 7% (ish) chance to resist
1.02 * 0.93 (aka 93% chance to hit, 7% chance to resist) = 0.9486 (aka 94.9% chance to hit)
Rogues:
In a slight twist to the formula, Rogues as of 1.12 will have access to a talent to increase their chance to resist spells (and ranged attacks) by up to 4% called Heightened Sense (HS from now on). Against a mage without Elemental Precision (EP from now on) it would look something like this:
0.96  0.04 (HS) = 0.92 (92% chance to hit, 8% chance to miss)
Lets also assume the rogue has stacked frost resist gear to the tune of 100, for a 25% chance to resist and the Mage has no resist gear/talents.
0.92 * 0.75 = 0.69 (69% chance to hit, 31% chance to resist)
Against a mage with EP:
0.96 + 0.06 (EP)  0.04 (HS) = 0.98 (aka 98% chance to hit)
Again assuming the Rogue has 100 frost resist, but this time the Mage has 50 resist spell penetration gear.
0.98 * 0.875 (50 effective resist = 12.5% chance to resist) = 0.8575 (85.6% to hit, 14.4% chance to resist)
What does it all mean?
Well not a lot tbh, the basic rules are as follows:
If nuking with fire as your primary source of damage, anything over +3% hit from gear or talents is wasted in PvP. You are better off stacking resist gear once you hit 99% chance to hit, in addition to +dmg and +crit. Unfortunately your control spells like nova and cone of cold are binary, so if you purposely phase out +hit gear they will suffer in comparison to a Mage with plenty of +hit.
If you are nuking with frost, +hit and resist gear/talents is what you need as both in effect act as spell penetration gear.
With 1.12 on its way, Rogues may well be speccing to adapt to the "frost mage problem" and with a nice amount of frost resist and the Heightened Senses talent, getting around a 20% chance to resist frost nova is not going to be difficult, certainly against fire Mages without Elemental Precision and minimal resist gear.
Hope this helps someone, and if you have any corrections to the formulas used please mention it.