One of the most integral parts of any fiction story is the "Magic System." It doesn't necessarily apply to actual magic and for that reason or more, the term in itself may seem somewhat elusive, so let's define it. In writing, a magic system is a set of abilities, skills, or technology that a character, or world has, and a guide to how said system is defined. Magic Systems can play a strong narrative role in your story and is one of the core world building concepts that you should be applying to your writing. Let's get into.
1. Soft Magic
This is when you play with magic, or skills without much defining its capabilities, whether in a world, or a character. Sometimes the writer knows the exact rules of the system and are purposefully keeping this information hidden from the reader, and other times the writer is moving "by the seat of his/her pants" with it. This could be a dangerous practice if the writer isn't able to keep the reader's belief suspended while wading through this "anything can happen" complex. It is highly susceptible to the problem of the Deus Ex Machina if no strategy is set up to prevent that. Soft magic is best when it is sparingly used and/or isn't the solution to major story conflicts. It can add a stunning sense of mystery, wonder, and awe if used correctly. It's what gives us our Gandalfs and Lions, Witches, and Wardrobes.
2. Hard Magic
This is where you play with RULES. Rules are the key. As the writer you have hard rules for how this magic works. What it can do, what it can't do. What does it require? When does it give out? What hard stops are there to the upper and lower extremes? And then, as Sanderson's first law states, you decide how well the reader understands this magic. This can be a powerful storytelling tool, or it can destroy your story by taking readers out if it with foul use or poor definition. The better you define rules in a hard-magic system, the better it is when you use this magic while sticking to said rules, to resolve conflict.
One of the best examples of a hard-magic system is the one used in Avatar: The Last Airbender. In this animated show's world, some individuals have the ability to control air, water, earth, and fire. The element must be present, and they can only control that specific element (save for the Avatar who can learn to use all four). These rules make for an interesting story in which characters use their particular abilities, and the watcher pretty much understands the capabilities.
Hard magic also applies well with the use of advanced technology. Because of the technical nature of technology, it may be beneficial to provide some solid rules for it that can make it more believable to the reader. Not to mention some real-life science and math can play a role in helping to explain the more fantastic elements of one of these magic systems, even if it's being stretched to some extent. In Pacific Rim, the magic system involves giant robots. In order for these robots to function, it requires two pilots who are able to sync minds. This is an excellent system, and with no actual magic involved. The point here is to make sure your reader understands it. If they understand they will enjoy it when used within the parameters of the set rules.
4. Be Clever
This is the strong point of a hard-magic system. Because rules are in place, your characters can't do just anything. The X-man Wolverine can't fly, and Storm doesn't have knives that pop out of her hands. They can only use what they have to solve problems within a set of previously defined rules. As a writer it’s your job to apply these rules to advance story and sometimes even resolve conflicts. Remember, sometimes your magic systems aren't magical at all. Sometimes you define a character based on immense skill as opposed to power.
In the web comic Kamikaze, Markesha doesn't have any super powers, but her "magic system" is defined by her ability to free run (parkour) for extended periods of time with little food, and in an environment where any physical exertion is severely compounded. While it would be silly to have her resolve conflict by winning a shootout against an opponent who is a skilled marksman, it is far more interesting and fitting for her to use her skills to resolve whatever dilemma she's in. Find out how to make your story work with what you have instead of inventing some Deus ex Machina that will disappoint your audience.
Giving your magic systems limitations can help to make them more interesting in the same way that rules can. It can also force you to think of more creative ways to use them in your stories. For example, in the Image comic Skyward, everyone can fly... because there is no gravity... which means they really only have so much control over their flight. These limitations keep characters in yours story from being all powerful gods who can solve any problem without issue. Or even if you are writing a story with gods, you still probably want to come up with a good way to be interesting within the context of your story and magic system. Remember to always keep the stakes there. A story without stakes falls flat quickly.
Cost almost goes hand in hand with limitations, but it doesn't have to. What you're looking for here is a payment that must be made for using amazing power. Sometimes this may be a prerequisite. In Avatar: The Last Airbender, cost is associated with being born with the genetics (the right kind of chi) to control (or bend) a particular element. You also need training and exertion to use such powers and one may tire out. An even better definition of cost is in the anime series Fullmetal Alchemist: Brotherhood. Here, the magic system is alchemy and it uses the law of equivalent exchange, which means in addition to training, studying, and skill one must also have the correct ingredients in order to transmute something into something else. As you may find if you watch the series, the result of using alchemy incorrectly has dire consequences.
7. Less is more
The goal in your story isn't necessarily to define several different magic systems (or a magic system with too many parts) but rather to spell out one or very few very well defined magic systems. The better written it is, the more your readers will appreciate them. As Brandon Sanderson says "Go deep, not wide." Focus on creating the system the rules, the culture around it and on a larger scale, even how it shapes the society of your world.
I just mentioned it above, but I will again here because it's so important. Magic systems are one of the key aspects of building your world. In Naruto there are a system of ninjas and ninjutsus. In Game of Thrones there are dragons and ice zombies. In Jurassic Park there are dinosaurs. Your magic system on a broad story scale, creates its own context for your world, and you have to write to that in order for it to make sense. Think about how it shapes your society. Does it have theocratic aspects? Has it been adopted by society at large? Who knows about it? How long has it been around? Answer these questions before you get deep into your story to better tell it.
9. Breaking Your Magic
Once you get to a more advanced point in your story you may want to expand how your magic works, or introduce a character that can break or change the rules. This can be done, however, you must have a good explanation for this sudden rule change. If character A can only use power A, then why is character B capable of using powers A and B? Usually this is due to a loophole that character B is using and the cost associated with doing that is likely very high. Think of ways to do this creatively and then implement it in your story. Probably a good idea to apply this to a villain. But be careful.