java.IOException: An existing connection was forcibly closed by the remote host


#1

We have a large modpack called Lapito’s Galacticraft. The server runs on a Google Cloud Virtual Machine with 4 cores and 26 GB of ram. -DfmlReadTimeout=3000 on both server and modpacks by injecting the argument into version.json.

Why are users still getting timeouts when me and my friend can connect just fine? I’ll say that we have high performance computers. I will also say that we do have to reconnect at times on first joining. But the TPS is 20…So its the client…and I have everything from FoamFix to RandomPatches in there.

For reference, here is the full mod list. We have less mods than other packs but I do understand that that number alone does not indicate overall performance.


#2

Have you tried lowering the server view distance? Reducing the amount of data sent to the clients might lessen the problem if it’s purely a network issue.


#3

Something to keep in mind, directly from the GC page…

Note that idle TCP connections are disconnected after 10 minutes. 
If your instance initiates or accepts long-lived connections with an external host, you can adjust TCP keep-alive settings to prevent these timeouts from dropping connections. 
You can configure the keep-alive settings on the Compute Engine instance, your external client, or both, depending on the host that typically initiates the connection. 
Set the keep-alives to less than 600 seconds to ensure that connections are refreshed before the timeout occurs. 
The following examples sets the keep-alives to one minute (60 seconds). 
Note that applications running on Linux systems don't enable keep-alive by default. 
Thus server or client need to explicitly set the SO_KEEPALIVE socket option when opening TCP connections (see also Linux TCP Keepalive HOWTO).

#4

I have done that. Still has been an issue but I have restrained myself from setting it less than 6. That might have to change…

I was not aware of this! I will try adding this as I always thought it was an issue with forge/minecraft freezing up and not responding to TCP keep-alive requests. I am applying this to the server now and will be performing a test.


#5

It looks like its still occuring…

Does sponge send its own keep alive?


#6

Can you upload the latest logs?


#7

From a grep 'joined\|left\|timeout' server/logs/latest.log


[18:19:17] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §esfxworks§r§e joined the game§r
[18:24:50] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[fishton] mcfishenburger§r§e joined the game§r
[18:33:48] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eSkaptic§r§e joined the game§r
[18:34:04] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eSkaptic§r§e left the game§r
[18:34:50] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §esfxworks§r§e left the game§r
[18:34:53] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eSkaptic§r§e joined the game§r
[18:46:29] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eSkaptic§r§e left the game§r
[18:47:10] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eSkaptic§r§e joined the game§r
[18:50:04] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §f: §r§7Then I was disconnected and then I joined and spawned normally§r
[18:56:27] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eSkaptic§r§e left the game§r
[19:13:05] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §esfxworks§r§e joined the game§r
[19:15:42] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §efeuermann2004§r§e joined the game§r
[19:16:55] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §efeuermann2004§r§e left the game§r
[19:44:28] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eskydrill§r§e joined the game§r
[19:44:46] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eskydrill§r§e left the game§r
[19:51:48] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e1Crafter7§r§e joined the game§r
[19:52:52] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e1Crafter7§r§e left the game§r
[19:52:54] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eskydrill§r§e joined the game§r
[19:53:12] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eskydrill§r§e left the game§r
[19:54:30] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e1Crafter7§r§e joined the game§r
[19:54:56] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e1Crafter7§r§e left the game§r
[19:55:02] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[F-ADMIN] §r§rsfxworks§r§e left the game§r
[19:55:54] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §esfxworks§r§e joined the game§r
[19:57:11] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e1Crafter7§r§e joined the game§r
[19:57:53] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e1Crafter7§r§e left the game§r
[20:02:39] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[F-ADMIN] §r§rsfxworks§r§e left the game§r
[20:10:51] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eGamingDragonz§r§e joined the game§r
[20:24:41] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eGamingDragonz§r§e left the game§r
[20:24:47] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[TheBobs] Nbgreen§r§e joined the game§r
[20:27:35] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[TheBobs] Nbgreen§r§e left the game§r
[20:28:04] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[TheBobs] Nbgreen§r§e joined the game§r
[20:33:43] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §esfxworks§r§e joined the game§r
[20:58:05] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §eEagle_Monster§r§e joined the game§r
[21:20:26] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[<Chemists>] tobybug§r§e joined the game§r
[21:29:22] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[TheChemist] Eagle_Monster§r§e left the game§r
[21:29:39] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[TheChemist] Eagle_Monster§r§e joined the game§r
[21:46:09] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[TheBobs] Nbgreen§r§e left the game§r
[21:47:52] [Server thread/INFO] [net.minecraft.server.dedicated.DedicatedServer]: §e[TheChemist] Eagle_Monster§r§e left the game§r

Full log up till now (too big for pastebin) https://drive.google.com/open?id=1Vl7_ihCdws9kQfde8gJWlxuvpoeM_iJm

To clarify I set the timeout to 20 instead of 60 to go under forge’s default of 30 while referencing the article that went along with where I believe you found it from here: https://cloud.google.com/compute/docs/troubleshooting/general-tips#communicatewithinternet


#8

Honestly I’m slightly more concerned about the exceptions that are spamming in the console, along with the cannot keep up messages…

Have you tried running the pack on a different system and gotten the same results? Don’t quote me on this, but last I checked AWS EC2 and GCP VMs aren’t particularly geared towards single threaded game servers…

I think part of the reason they are timing out is because of exactly that, the server isn’t able to keep up with the connection cause it’s doing something else. I would generate a timings report, it might help to see what’s hitting the server so hard.


#9

So I was able to isolate previous issues regarding this and did report it to the mod author.

There was a reference to the number of players with anything other than Galacticraft’s electric and fluid transport (namely EnderIO and Mekanism cables) that were causing severe stress on the server. While these have been tuned we still do get performance issues in other areas… I do still need to find a mod that can sweep through and replace all cables with GC’s wires. So far I’ve only been able to MMCRestrict chunk loaders but I do not think it is recursively deleting those blocks and I cannot find a config option for it.

I’ve been trying to refrain from moving somewhere outside of Google Cloud given their ~70% discount on pre-emptable containers and our lack of donations. Sponge/forge is taking advantage of the other threads when I set some additional config options up. I could try increasing the network packet size from 64 but isn’t that already on another thread? Don’t know what else to do given our budget except trim down mods from the pack.


#10

Latest profile:…

I’m seeing a distribution of mods…


#11

I’m down for a test. Moving to AWS’s new C5 cpu & w/ spot pricing

Current VM with GC

lscpu | grep MHz
CPU MHz:               2300.000

New VM I spun up with AWS

lscpu | grep MHz
CPU MHz:             3413.837

Will post updates.


#12

I also have users sometimes timings out, for me its usually people who are from South America. I’m not sure if its just their connections, or their PC’s are just not powerful enough, I brought down render distance to 8 already.

I’m using waterfall for bungeecord, on an Intel i7-4790K.

Do you know the general geolocation of the users who experience timeouts vs users who don’t?