Cross-Server Communication Support

Since minecraft networks with multiple servers are common and steadily growing, there is a demand for cross-server communication.

General problem

Everyone who develops in a multi-server environment has everytime the same problem: “How do I communicate with the other servers?”
Unless you have something like an API plugin, like I wrote for my plugins, this will be a challenge.
Mixing plugins with different communication protocols, etc. may not be possible; so man plugins use their own socket on the server to communicate with itself on the other side.
Connecting to many servers on one root server and constantly sending data reduces the overall open socket count and can push a root server to its limits

Another challenge is writing a networking software that is good in performance and fail-safe.

Pioneer work

I think BungeeCord and the Plugin Channel are a good example how to give plugins a way to communicate with each other. By encapsulating a name as SubChannel and a Byte array into a message it seperates plugins from the actual networking and simplifies its uses.
Unfortunately on BungeeCord this only works if a player is connected, because this uses the player’s connection.
The same principle is used in my network plugin XServer and I think this approach is still the way to go… (unless you have a better idea, so please tell me).

Standardization

Basicaly the idea that we had was to put a standardized api into sponge and a basic cross-server networking.
If needed the connection would be automatically opened (via SSL handshake) and closed when not used and timed out.

A basic message would be a simple container class:

public class Message {
  private final String subchannel;
  private final byte[] content;
  ...
}

A server would be an instance of:

public interface NetworkServer {
  public boolean isConnected();
  public void sendMessage(Message message);
  public InetSocketAddress getAddress();
  ...
}

A list of servers could be retrieved via a manager:

public interface NetworkManager {
  public List<NetworkServer> getServers();           // get all server
  public NetworkServer getHome();                    // this server
}

A message would be received via events:

@SpongeEventHandler( sync = true )
@NetworkHandler( subchannel = "optional subchannel" )
public void onMessage(NetworkMessageIncomingEvent event) {
  Message m = event.getMessage();
  NetworkServer server = event.getSender();
  // do something ...
}

Protocol

  • Handshake
    • Packet ID (0x01) | … crypto information …
  • Message
    • Packet ID (0x20) | SubChannel (String) | Content (ByteArray)

Future

Additional goals could be to implement it as a P2P network.

Conclusion

I think this makes multi-server developing easier and it would be highly appreciated by most plugin developers.

What do you think?

24 Likes

I think that’s a great idea, it’s definitely something that could be used in plugins and by server admins often +1

I personally rarely had the situation where I had to send data without any player connection. For example: I had a server list as well, which knewed if a server is online / etc. In my case I fell in love using a MySQL Database + Apache Tomcat Connection Pooling. From my personal experience I really can tell you, that this is working flawless!

If a target server needs to gather the information you could create a inbox for it. A huge PRO is, that the server could catch the information even if its not online when the information is being send, as its stored in a database.

I know that using a database instead of sockets isn’t the common solution BUT: It is a very performance friendly solution which works quite nice if everything is setup properly. Using 1 Database/Master server always pays off.

One could create a plugin, where you could specify a database to use IF there is no player online which you could use to send the data. Otherwise you really would’ve to create a special socketchannel. I’d definitely go with the database solution as creating a special socketchannel (even if it’s NIO compatible) isn’t a nice solution when already using BungeeCord in my opinion.

Sincerely,
Difayth

1 Like

It is a very useful thing for multiple servers to have the ability to communicate with each other, i hope something like this will be part of the sponge api…

Going over a database server is alright for transmitting meta informations like a player list or status information.
But it is not good for a server-wide live chat or server-wide command execution, etc…

I agree! Especially when using an exernal master-server it does take way too much time even when doing batch-inserts.

I really hope, that they’ll include any function which allows us to send data to another server just like you can do it using a player. Maybe a BungeeCord API for clients where you could use a function similar to “BungeeCord.send(Packet);”. As @md_5 fortunately is one of the Sponge developers, he might has a solution for this! :blush:

This would be awesome to use.

Anything would be more performant and more reliable than something homegrown unless you’re willing to devote quite a bit of time to reinvent the wheel.

inb4 “Redis is a database”, it has very performant IRC-like pubsub functionality.

5 Likes

Agree with @vemacs. We’ve done work with benching Redis as pubsub. Remarkably performant. We ran a test with one publisher and five subscribers. Sent 80,000 messages per second before we hit 1 Gbps (we calculated a 40% overhead on 68 byte message events).

Redis would be great, as a foundation.

1 Like

issue: http://issues.spongepowered.org/youtrack/issue/SPONGE-22

@IDragonfire - That’s for exposing an API for Server <-> Proxy/Client communication using Mojang’s Plugin channel system which already exists in vanilla Minecraft. This topic is about Server <-> Server communication where is no equivalent functionality in vanilla Minecraft, and is completely different.

I see not the point to handle this two topics differently :wink:

There is a very good reason for doing so - they are two completely different features. They are not as related as you think.

  • Server <-> Client/Proxy support already exists in vanilla Minecraft. That means that all Sponge really need to do is add an API to it, find where the code to send such messages exist in nms/Forge, hook into it, and it’ll (hopefully) be usable. The Sponge guys won’t need to create new functionality, they don’t need to think about how the networking internals work so much, because it’s already written and will work with a vanilla client.
  • Server <-> Server support does NOT exist in vanilla Minecraft and would need to be written from the ground up. If the Sponge Devs wanted to add such a feature, they would have to write a networking system that allows minecraft servers to connect to one-another. They would then have to write the API, and they would have to work out how to secure it, how servers can interact with each other etc. etc. - it’d potentially be a huge job.

Also, Server to Client and Server to Server have two completely seperate use cases, would be two very different systems, and require two very different amounts of effort to implement - particuarly as one of the systems does not exist.

That said, I think it would be nifty to see such a thing. I doubt it’d get into Sponge though - I’m under the impression that the Sponge API is meant to enable others to modify Minecraft on a stable platform, and not functionally modify it itself (that is, a Sponge server with no other plugins or mods should behave as a vanilla server, though I guess a server <-> server communication channel wouldn’t technically alter Minecraft on it’s own).

You could always ask the core team if they would accept such a feature. If they won’t, I’m sure a group of interested developers will try to make a plugin for Sponge that standardises such communication.

2 Likes

In my view there are some problems with the channels,
e.g. you cannot send a message to an empty server …

I linked only the issue for information purpose :wink:

But I aggree with you, it depends on the usecase and Sponge should be as lightweight as possible …

That’s because it was never designed for server -> proxy -> server communication at all - and thus is why it’s a different issue entirely to the one you linked to. As I said, Plugin Message channels are designed for Server <-> Client messaging - NOT to connect to servers, even through proxies and roundabout techniques.

Server to server communication using Plugin Message channels is completely the wrong thing to do if you ask me - if you need to do it, you should do it directly via sockets or something like that - more reliable that way - no middle man. If the proxy (read, BungeeCord) you are using goes down, then your servers get cut off - all the more reason to create a library for this really…

Anyway, back on topic (sorry guys!)

There is a Bukkit plugin somewhere called Transporter that used server <-> server sockets to communicate (I think it’s on the Spigot webpages somewhere). It might be worth looking at that plugin if anyone is interested in actually doing this server->server stuff, see how the author of that plugin did it. Even more interesting is that he uses it in conjunction with Bungee - uses Bungee to teleport the player using a Server -> Proxy plugin message, but uses a server->server socket to ensure that there is a place on the server for the incoming player.

Please don’t mix it up with the vanilla plugin channel.
It’s all about communication between servers. (not via player connection)

XServer does server <-> server communication with “Subchannels” like you know them from the Vanilla Plugin Channel.
And many other plugins do the same again for themselves.

But!
I don’t like the concept to have to link all my plugin to this library/plugin.
It would be waaay better to have some common interface that can be implemented by most server <-> server communication plugins.
And by doing so embracing the support for plugins to be able to communicate across servers without woring much about dependecies.

For Example:

public interface SpongeServer{
  public void registerCrossServerSupport(HostModule module);
  public HostModule getCrossServerSupport();
}

public interface HostModule {
  ... stuff like sendMessage(...) etc.
}

In the end there could be dozen other cross-server modules for Sponge (like my XServer plugin) that have a shared interface.

So if you don’t like direct socket to socket communication, you could use RabbitMQ, Redis or Socket.io… (@vemacs)

It’s more about the standardization, than the implementation.
So … what do you think?
Is there an opionion on this topic by a core developer?

Personally, I think sponge could use some kind of bungee api or something? I probably totally just said that wrong. Just take a look at this idea:

Sponge server:
Server 1
World
World_Nether
World_End
Plugins
Server 2
Arena
Plugins
(Other Files like server.properties and stuff)

So… anyway, my idea is that the server can run each server folder as a set of worlds with their own plugins folder, and a multi world plugin or communication plugin could handle the communication and stuff, this way, we wouldn’t need a proxy and all that stuff!

Multiple plugin folders can’t be done.
There is only one java instance running your server, so a plugin can’t be loaded twice.
(because of reasons)


Sponge is no Proxy server. Sponge should / is going to be like a better Vanilla Minecraft server.


Sry., but it seems that you don’t understand the benefit of a proxy server.

  1. No disconnects on restarts
  • You can restart your servers to prevent memory leaks and to reload plugins
  • Dynamically add more servers to adapt to players.
  1. Balance cpu workload on multiple root-servers (mega servers)
  2. Multiple Access-Points to your network.
  3. You don’t have to disconnect and connect again to change a server.

You need a proxy if you have a larger player base.

Well I do know that some old bukkit plugins could be per world, or disabled in certain worlds. Maybe there could be some way to work with this? Like make it part of the API?

Nope! Every plugin did for itself the world handling.

But isn’t that another topic? ^^