|I can only assume this made sense to somebody...|
Back in early December I wrote a blog discussing some of the considerations that bore thinking about when applying a new skin onto an existing game system (The New Skin). This blog is the flip-side of the coin, making actual mechanical changes to a game to alter the genre, feel, or tone.
In my blog about re-skinning games I basically tackled the idea that sometimes you want to play in one setting, but use the mechanics from another setting, mechanics that you prefer or feel will be better for the game and setting. Assuming you aim to do the least work possible and change nothing but what things are called, I discussed how choosing mechanics for the right tone, look, feel, and gameplay style you want all influence how the final game will come together.
Of course reskinning things isn't always so easy, and many times it requires some adjustment of mechanics. It's my opinion that once you start to tinker with mechanics you exit the realm of re-skinning and start to enter the realm of hacking.
Let's take the example of using fantasy game's mechanics for use with a science fiction setting & genre. There are a multitude of challenges involved to take a game between genres. Some archetypes will transfer over without issue; warriors types probably won't require heavy mechanical rework, aside from adding in abilities for use with guns, but making a cleric or wizard feel like a psychic or somebody who uses nanotechnlogy (or something else exotic) will.
Those others will require a great deal of work; magic is a staple in the fantasy genre and fantasy games often have robust magical systems for those kinds of characters. In a science fiction game those magic systems may require a great deal of work to fit into the genre. Assuming that your setting doesn't afford you the use of psionics, nanotechnology, or mass-to-energy conversion, magic may even need to be dropped entirely. This is where hacking starts to happen.
Generally speaking I think that my two salient points from the discussion on re-skinning also apply when hacking a game:
Game mechanics have a certain "feel" to them.
Finding the correct feel for a game comes down to two aspects: genre and tone.
Unless you really plan to dig deep and change a lot of the core mechanics (in which case maybe you have left hacking and entered the realm of game design), you will probably be making minor or moderate changes that alter the way the game feels to better match your designed endpoint; that is, to better match the feel, genre, and tone.
Taking a fairly stock fantasy system and tweaking it to feel more like a dark fantasy game (a tonal shift and possibly a genre one depending) may mean something as simple as adding a check to spell casting. Success on the check casts the spell as normal, failure may mean corruption. It could also entail adding some aspect of psychological damage. Neither is a major alteration or addition to the rules but they change the feel of the mechanics to closer match "dark fantasy."
More moderate changes may mean adding or removing an entire mechanic. It may also require some additional modifications to impacted secondary factors. As an example: making D&D, which is fairly high fantasy, work for a low fantasy setting may involve simply removing most spellcasting. This has a pretty significant effect on how the game will play, with some classes and creatures changing how they play in major ways. A wizard without spellcasting, limited only to rituals and any items they can enchant, may suddenly become unplayable without modification to attack ability and health.
The trick with these kinds of changes is ensuring that a) you get what you want and b) you don't damage things with your inclusions or exclusions. Adding corruption to arcane spellcasters isn't a huge change, but removing all spell casting to better emulate a low fantasy feel will have potential aftereffects that you may or may not see ahead of time. A good way to try and spot that during the planning stages is to ask yourself "And then what?" When you answer that question try to think about what that change will impact. Alternately you can look at this from the other side by asking "Why?" Regardless of which direction you take ask that question at least three to five times to really dig to the root of what you are doing.
I want to make a low fantasy version of D&D so I'm going to remove all spellcasting. Why? Because in low fantasy wizards seldom wield flashy direct magic effects, their powers are more subtle and often more arcane. Why? The rarity or difficulty of magic means that wizards want to cultivate an air of mystery to help compensate for the fact that their magic is difficult and subtle. Why? In the low fantasy genre power is harder to gain, and more tricky to hold on to, a wizard who makes too much of an impression will gain rivals and enemies.
By answering the questions you help yourself understand what you are trying to get at. You may also see that removing spellcasting won't solve the problem. What you really want is a way to make spells rarer and less frequently cast, but also have magic still retain the power and utility that makes it worth pursuing. As a result maybe you will decide to allow only certain spell schools, and reduce the number of spells granted (or mana to cast spells from), but conversely increase the power of those spells to compensate. Suddenly a wizard who can cast dozens of magic missiles per day can only do so twice, but they operate at a level of power higher than normal for the system. The wizard now has to shepherd his magic, but when he looses it, the result is impressive enough to make up for the rarity.
Lastly consider what you are changing and if it will fit with the game. That picture at the top is humorous because it's ludicrous. The monster truck underbody with the mini-van body is a clearly mismatched pairing. Consider if what you are adding (usually, but sometimes removing, or modifying) will mesh well with the remainder of the game mechanics to bring things closer to your desired game feel.
In the end I want to emphasize that you shouldn't be afraid to tinker with your games to suit your needs, but you should take care and consider the ramifications as you change, add, or remove things. Likewise, if you see that a change does not work in play as you hoped or expected remember that you can change things. Game go through playtesting, and nothing is perfect in the first try. Hacking a game system is little different from creating one from the group up except in a sense of scale. Keep in mind your desired goal of tone, genre, and feel, and consider the intended and unintended consequences of your changes on other aspects of the game system.