One day we updated spigot to the latest version and everything went without any issues.
The server has a flat plot world. Some time later (a few days I think) an admin of our server noticed that a huge amount of blocks below the plots were missing.
We thought someone may have WorldEdited them out, but then that can’t really happen because we have a WorldEdit block limit and you can only use WE on plots you own.
Maybe it was an admin accidentally deleting a lot of blocks?
We flew a lot under the plots and noticed that it didn’t seem to have an end.
It also wasn’t completely cleared. There were still some piles of blocks here and there.
Luckily, we’ve got LogBlock and just checked the logs of those blocks.
Now we were really confused.
Oh well, we need to get a backup.
Rolled back a few hours, restarted the server, same problem.
Rolled back a few days, restarted the server, same problem.
It must be something different.
Maybe the world is corrupted?
We ran MCEdit’s region repair (took a few hours because the world is HUGE).
Restarted the server, same problem.
Hmm, this is really weird.
Maybe MultiVerse got some issues?
Disabled Multiverse, restarted server, same problem.
Duh. Disabled some other plugins, restarted server, same problem.
Oh wait, Spigot got this built in Anti-Xray feature, let’s disable that!
Ahhh, nope. Same problem.
Finally downloaded the world (only 1 region) to my PC and tested again. Same problem.
Then I put the world into vanilla server. Finally the problem was gone!
Sadly, this didn’t really bring us any further in resolving the issue.
Meanwhile, I went to spigot’s IRC and seeked for help, but nobody really had a clue.
I then tried to test the same thing with Bukkit and the problem went away… until I updated Bukkit to the latest version, too.
Great, so we know something in bukkit’s recent change broke it.
At some point md_5 joined the discussion and found the issue.
Apparently @Amaranth replaced a nibble array (which had to do with chunks) with a different type of Array and it had some strange bugs.
Props to md_5 who removed the bug no 10 minutes after he tracked down the cause of the problem.
Edit: more probs to Amaranth who fixed the actual bug shortly after.
Bug thread on spigot forums: http://www.spigotmc.org/threads/empty-ghost-chunks-remove-huge-parts-of-underground-world.23290/