Optimized Startup Flags for consistent Garbage Collection

Reposting my posts I have in the Paper & Spigot community here!
EDIT: Please see https://mcflag.emc.gs for updated content.

Many months ago, I did an extremely extensive study into Garbage Collection and Minecraft, and got a really deep understanding on Minecrafts allocation behavior, general object life expectancy and collection rates. I studied my server for about an entire month.

I tweaked various settings, studying the GC log files, studying the various flags documentation, and other performance tuning case studies, and then even better: using the VisualVM plugin VisualGC, that lets you visually see how each region of the garbage collector fills up and collects.

This insight was amazing and let me truly understand how the GC was behaving, and learn how to overall improve the garbage collector efficiency.

These flags are based on my study, and have strong reasoning behind them to actually have an intended behavior change of the Garbage Collector and Performance Settings.

Many servers in the Paper & Spigot community have started using these and reported improvements (And those who donā€™t notice, probably wasnā€™t doing too bad to begin with)

The goal of these flags is not to reduce overall CPU usage, as G1 does use more CPU than CMS, but this CPU usage is parallel, so not ā€˜directlyā€™ main thread. The goal here is to provide CONSISTENT, low pauses, and avoid Old Gen collections as much as possible. Improperly tuned GC that ends up collecting Old Gen could see many 1 second long GC collections, which your players will notice!

Also NOTE: Memory usage will be a saw blade / zig zag! G1 does not constantly collect. It only collects when one region fills up, so this is normal:

(Graph provided by Zabbix Server Monitoring Software with JMX agent integration)
During low activity, it will appear as memory is higher, as it simply takes longer between every GC pause - THIS IS A VERY GOOD THING!

Going 3 minutes between GC pauses means your CPU spent 0 time on GC for 3 minutes.

Here are the flags!

java -Xms6G -Xmx6G -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=100 -XX:+DisableExplicitGC -XX:TargetSurvivorRatio=90 -XX:G1NewSizePercent=50 -XX:G1MaxNewSizePercent=80 -XX:G1MixedGCLiveThresholdPercent=50 -XX:+AlwaysPreTouch -jar server.jar

These flags help keep your server running CONSISTENT without any large spikes. CPU may be slightly higher, but your server will be overall more reliable and stable TPS.

If you are running with 10GB or less memory for MC, you should not adjust these parameters. (I use 10GB myself - pauses are 100ms or less)
If you for sure need more than 10GB (Hopefully you are 150+ player server, but maybe less with Forge Mods) use these changes:

-XX:G1MaxNewSizePercent=60
-XX:G1NewSizePercent=35

You may need to tune these to your needs for larger memory servers (these reduce memory in new gen and give more back to old gen)

Explanation of flags:

  1. -Xms matching -Xmx - Why: You should never run your server with the case that -Xmx can run the system completely out of memory. Your server should always be expected to use the entire -Xmx! You should then ensure the OS has extra memory on top of that Xmx for non MC/OS level things. Therefore, you should never run MC with -Xmx settings you canā€™t support if java uses it all.

    Now, that means if -Xms is lower than -Xmx - YOU HAVE UNUSED MEMORY! Unused memory is wasted memory. G1 (and probably even CMS to a certain threshhold, but Iā€™m only stating what Iā€™m sure about) operates better with the more memory its given. G1 adaptively chooses how much memory to give to each region to optimize pause time. If you have more memory than it needs to reach an optimal pause time, G1 will simply push that extra into the old generation and it will not hurt you (This may not be the case for CMS, but is the case for G1)

    The fundamental idea of improving GC behavior is to ensure short lived objects die young and never get promoted. With the more memory G1 has, the better assurance you will get that objects are not getting prematurely promoted to the old generation.

    G1 Operates differently than previous collectors and is able to handle larger heaps more efficiently. If it does not need the memory given to it, it will not use it. The entire engine operates differently and does not suffer from too large of heaps.

  2. UnlockExperimentalVMOptions - needed for some of the others specified

  3. TargetSurvivorRatio: Iā€™m sure your all use to seeing this one suggested. Good news! Itā€™s actually a good flag to use :smiley: This setting controls how much of the Survivor space is ABLE to be used before promotion. If survivor gets too full, stuff starts promoting to Old Gen. The reason behind this is to be able to handle memory allocation spikes.

    However, MC allocation rate for most part is pretty steady (steadily highā€¦), and when its steady its safe to raise this value to avoid premature promotions.

  4. G1NewSize Percent: These are the important ones. In CMS and other Generations, tweaking the New Generation results in FIXED SIZE New Gen and usually is done through explicit size setting with -Xmn.

    With G1, things are better! You now can specify percentages of an overall desired range for the new generation.

    With these settings, we tell G1 to not use its default 5% for new gen, and instead give it 50% at least!

    Minecraft has an extremely high memory allocation rate, ranging to at least 800 Megabytes a second on a 30 player server! And this is mostly short lived objects (BlockPosition)

    now, this means MC REALLY needs more focus on New Generation to be able to even support this allocation rate. If your new gen is too small, you will be running new gen collections 1-2+ times per second!!!

    Then combine the fact that objects will now promote faster, resulting in your Old Gen growing fasterā€¦ This is bad and needs to be avoided.

    Given more NewGen, we are able to slow down the intervals of Young Gen collections, resulting in more time for short lived objects to die young and overall more efficient GC behavior.

    if you run with larger heaps (15GB+), you may want to lower the minimum to say 30%, but donā€™t go lower than 30%. This will let G1 have more power in its own assumptions.

  5. AlwaysPreTouch: AlwaysPreTouch gets the memory setup and reserved at process start ensuring it is contiguous, improving the efficiency of it more.

Also for Large Pages - Itā€™s even more important to use -Xms = -Xmx! That memory CAN NOT be used by the OS anyways, so let something use it!

Additionally use these flags (Metaspace is Java8 Only, donā€™t use it for Java7):

 -XX:+UseLargePagesInMetaspace
42 Likes

@Aikar

Amazing post, love it. Only feedback I can provide is Sponge requires Java 8 + so if that changes recommendations then an edit may be in store.

Thank you Aikar

Any chance you would have recommendations for use of flags with the pixelmon mod? I was trying a few but it caused the Pokemon to stop spawning properly.

It wouldnā€™t make any pokemon stop spawning, not without massive erroring on consoleā€™s end. If you are having spawning issues, best ask in Pixelmon Modā€™s discord. There, we can look at your config, timings and setup and give you proper advice.

Though startup flags are relevant in optimization, they wouldnā€™t affect spawning.

I know, thatā€™s an older thread but it helped use alot with out 1.10.2 ForgeSponge server!

Iā€™m wondering if you have a clue how to optimize the client?

For know Iā€™m using the following parameters:

-XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=15 -XX:MaxGCPauseMillis=30 -XX:-UseGCOverheadLimit -XX:+UseBiasedLocking -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:+UseCompressedOops -XX:+OptimizeStringConcat -XX:+AggressiveOpts -XX:ReservedCodeCacheSize=2048m -XX:+UseCodeCacheFlushing -XX:SoftRefLRUPolicyMSPerMB=20000

Hi! Thank you for this thread.

Anyone know if ParallelGCThreads=X is still needed?
Also, I am curious about the last one you mention briefly at the end of your OP:
-XX:+UseLargePagesInMetaspace
Is that beneficial still? What does it do?

Any other suggestions out there for optimizing? The flags I currently use are:
-Dfml.queryResult=confirm -Xms6G -Xmx6G -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=100 -XX:+DisableExplicitGC -XX:TargetSurvivorRatio=90 -XX:G1NewSizePercent=50 -XX:G1MaxNewSizePercent=80 -XX:InitiatingHeapOccupancyPercent=10 -XX:G1MixedGCLiveThresholdPercent=50 -XX:+AggressiveOpts -XX:+AlwaysPreTouch -Dbuildcraft.debug=disable

Thanks!

Parallel is used, but the JVM auto calculates a good number for you, so no need to manually set it.

As for LargePagesInMeta, yes, all of these flags will remain beneficial and should not removed at least through Java 9.

And itā€™s very likely they will still be valid for Java10, the only thing that might change things for Java10 is if a better GC comes out :slight_smile:

The Large Pages in meta does exactly as it says, it uses OS Large/Huge Pages in the metaspace part of the JVM memory, IF you have huge pages enabled (unlikely on shared hosting)

1 Like

Thank you very much for the detailed reply.
How would I tell if I have huge pages enabled? I am just running the server on a personal home computer, Windows 10 currently while my Linux machine is down.

Much appreciated!

I do not believe windows supports huge pages. Itā€™s something you would only use on a dedicated server, and should never be used on a home desktop.

2 Likes

Thank you very much for the help!

Thank you so mutch! Works like a charm. And my server stopped the ā€œcanā€™t keep upā€ message.

@Aikar What do you think about Azul C4? Itā€™s does not have stop-world pauses and has an interesting algorithm.

Are these flags still valid for Java 8.181 and MC 1.12.2? Iā€™m trying to set up my first ever server and get the ā€œCanā€™t keep upā€ error. A copy paste of your flags to my .bat file does not work.

Yes! make sure to check the formatting because it definitely still does work.

Updated here: JVM Tuning: Optimized G1GC for Minecraft - Aikar's Thoughts

Can you explain why we need ā€˜-jar server.jarā€™ in this?

That tells the java command what file to run. server.jar should be replaced with the name of your Forge or SpongeVanilla file.

Thanks for this post! Another Forge user here confirming that these flags work brilliantly on my 2020 install of 1.12.2 which is running a dozen mods. Prior to adding these flags my quad core Xeon server would see ~60% CPU utilization at rest and would skip ticks constantlyā€“about every 10 seconds. This manifested in users being warped backwards for moving too quickly as well as numerous other sub-optimal interactions. Now there is more RAM usage (which is fine) and NO SKIPS AT ALL. CPU utilization is down between 1% and 5%.

I want to dedicate 20gb of RAM on a public pixelmon server. A good Start.sh for this?
I want optimize to the CPU because he have ten cores.