Instancing in the traditional sense implies the dynamic creation/destruction of servers to accomodate either some sort of functionality, or just more users on the same application. In games, we use it to describe dungeon instancing, as in, a situation where 2 groups of users want to run the same dungeon ( read: be in the same position in the world at the same time ), which is obviously not possible. So, instancing is used to solve this.
In minecraft, this is less than practical. The minecraft server gets very resource-intensive. So instancing becomes quite difficult to pull off. Not to mention the fact that dynamically creating or destroying minecraft servers would be a pain, since it takes some time for a server to start up. Add to that the cost increase caused by a modded environment and it really piles up.
For the server I’m currently working on, we have actually decided we need dynamic instances, where the maximum player count per server is ~5 ( talking dungeons after all, so groups of players are smaller ).
With this in mind, I’ve done some very basic calculations, and I’ve reached the conclusion a 5-player server would require a good 3 gigs of RAM each, along with it’s own core to work on, so as not to influence performance on the other instances.
This would require some pretty beefy hardware. A multi-core Xeon, with 32 gigs of RAM, on 2 boxes. Main server runs on one box, the dungeon instances run on the other. Even with all of this, I think our limit might be 8 dungeon instances total. That’s 8 minecraft servers total running on the same box. Not that amazing of a result, considering the hardware.
Alternatives include possibly faking instancing. Worlds are certainly worth looking into, I’ve considered that as well, though as of right now, the plugin isn’t even started yet.
TL;DR To conclude, while physically possible to create dungeon instances of minecraft servers, it’s far from practical. Other alternatives, like multiple worlds hosting the same dungeon, may be more appropriate. I believe the best answer may be a combination of the two, provided the server in question has the hardware to support it.