☑ FastAsyncWorldEdit

Plugins Plugin Releases
#1

FastAsyncWorldEdit

Overview:

FAWE is an addon for WorldEdit that drastically improves its performance and adds a lot of new functionality by replacing most of the core classes.

Features:

  • Incredibly fast, lag free and uses minimal memory
  • Support for protection plugins (see below)
  • Advanced per player limits and inventory settings
  • All commands and messages are translatable
  • No setup web integration: http://empcraft.com/fawe
  • Adds commands to change blocks in entire worlds
  • Fast built in logging and rollback (see below)
  • Instant copying of arbitrary size: //lazycopy
  • Adds a lot of powerful new brushes and tools
  • Adds a lot more mask functionality
  • Adds a lot more pattern functionality
  • Adds edit transforms
  • Supports new file formats (Structure block, PNG, BD)
  • Adds commands for fixing or changing lighting
  • Allows WorldEdit to be used from console: //jumpto world,0,0,0
  • Fixes bugs in WorldEdit such as block rotation, pasting entities, etc.
  • Can load corrupt schematic files
  • Unlimited undo/redo with history/clipboard on disk

Supported platforms

PC: SP/MP, MCPE: MP

  • CraftBukkit
  • Spigot
  • Paper
  • Thermos
  • Forge
  • Sponge
  • Nukkit

For developers: See here

Async world manipulation, loading, lighting + examples.

Region restrictions:

The following plugins are supported:

  • PlotSquared

Disable in the config, or bypass with /wea or fawe.bypass

Logging and Rollback

Enable use-disk and use-database in the config.yml to use the in built FAWE logging/rollback. Alternatively, the following third party plugins are supported with BlocksHub installed.

  • See BlocksHub page

It is safe to give normal players access to FAWE rollback. To bypass logging use //fast

Metrics

IMAGE ALT TEXT

How does it work?

  • All commands / actions are executed asynchronously.
  • Chunk info is cached on the main thread for the async thread to access
  • Changes an edit does are pushed to a local queue which queues by chunk
  • When a local queue is created it is added to the global queue
  • The global queue dispatches changes to the world on the main thread
  • The dispatching can occur when changes are still being made to the local queue
  • FAWE directly modifies the chunk data for the best performance

How does it use less memory?

Default worldedit uses about 256 bytes for a BlockChange object, which contains a Vector, and two BaseBlock objects (the original, and the new one). There are two BlockChange objects created for each block change (one for undo, and one for redo). Then this is only for the ChangeSet extent, there are 11 other extents e.g. the MultiStageReorder has three lists that map the BlockPosition to BaseBlock (all three reordering stages need to complete fully before any blocks are changed).

FAWE removes / replaces all of that. Block changes are recorded in a compressed byte array (using the LZ4 compressor) i.e. A simple edit can be using a tenth of a byte per change (effectively). Additionally FAWE dispatches changes whilst blocks are being queued so the whole thing will never need to be in memory. FAWE can also be configured to use the disk for history/clipboard in which case nothing needs to be stored in memory.

How did you do the benchmark? (outdated)

  • Loaded up FAWE 3.4.3 with SV-297 and WE-SV #17, increased the number of parallel threads to 4
  • Tweaked some other settings (e.g. queue target) so for speed rather than low memory
  • Set 50 mil blocks with fastmode, took 0.36 seconds, that’s almost 140M bps
  • Stopped the server, deleted FAWE and restarted (just running WorldEdit-sponge now)
  • Decided to do a 1 mil block edit, which if 250x slower would take 1.8 seconds -> server crashes
  • Tried changing just a single chunk, it took 10 seconds
  • Result: 2 million percent faster
25 Likes
WorldEdit for SpongeAPI
What Bukkit plugins are being worked on for Sponge?
These 1.10.2 Plugins for Sponge
Unofficial Sponge Categorized Plugin List
Realm of Thaumtology 1.8.9 Server!
Brush Tooltips - Display your active brush for FAWE tools
#2

If this does what it says it does, wow.

Any thoughts on contacting the world edit devs to see if any of this can be pulled back into the main project?

2 Likes
#3

Does this work with the SpongeVanilla port of WorldEdit?

#4

Not yet. Working with @minecrell to get it up and running. Note: You’ll need to use a version of WorldEdit compatible with SV.

@ryantheleach
They’ll decline. Much of what’s done would usually be avoided since they like clean non breakable code. FAWE is very dependent on implementation specifics which will break each major update.

2 Likes
#5

Should work on SpongeVanilla now. See the download links above.

2 Likes
#6

FAWE [3.3.6]

Fixed some minor issues with copying/undoing edits with nbt data
Fixed minor issue with perm checks.

Recoded the clipboard class to encode the BaseBlock as a short array with an accompanying map for nbt

  • about 10x smaller memory usage

Compression improvements

  • Using java’s inbuilt compression was too slow for me (gzip or deflator)
  • Now using the LZ4 compression algorithm which is super fast (fast enough that there’s now 1 pass of history compression by default)
  • Enabling further compression will do another pass with the LZ4 high compressor.

Streamlined the queue system so it doesn’t need to be fetched on each block place.

  • Not a whole lot faster, but it probably makes more sense code wise.

Removed block reordering from the editsession.

  • It alone took about 0.5s/mil blocks

Replaced block fetching from history extent with custom optimized algorithm

  • Using primitives to represent the block where possible is a lot faster than instantiating a new BaseBlock object and populating it with the id/data/nbt

###tldr;
compression=super fast,history=super fast

[FAWE] History compressed. Saved ~ 10784931b (204x smaller)

^ also really good ratio for uniform changes

#7

Is this an addon to WorldEdit? Or is this a worldedit plugin on its own?

1 Like
#8

What are the conditions under which the benchmarks of 5000% and 25000% are being stated?
Setting 1 block? Setting 4 blocks? Setting 8 blocks? 1000? 1000000?
Pasting a schematic?
Regenerating terrain?
Fixing water radius 3? Fixing water radius 100?

1 Like
#9

@SuperMarioCraft_Gam
It’s an addon, though it replaces large parts of the plugin. See the links above to download a compatible version of WorldEdit

@TheBoomer
Replacing 10mil uniform blocks in loaded chunks (using //replace). Spigot 1.9.2 / replacing with a single id / compression disabled / history on disk disabled / chunk wait set to 0.

Most of FAWE’s improvement are for the block placement part of an action:

  • any loading of chunks would take the same time
  • reading from disk to load a schematic isn’t any faster
  • can’t make the world generator any faster, so //regen will depend on how fast the generator can supply the blocks
  • small number of block changes (e.g. 1 block) would be too fast to compare any improvement and would be dwarfed by the overhead anyway
1 Like
#10

tutorial to install it?

#11

Download and put it in your mods folder:
https://github.com/boy0001/FastAsyncWorldedit/releases
Along with the corresponding WorldEdit version.

Then edit the config to your liking.

#12

As much as I like this, I have serious doubts for both your performance claims and the sanity of your methods.

Directly editing chunk objects bypasses almost all of sponge code, rendering world protection plugins useless if normal players are given any kind of access.

250x perf increase is ridiculous. Any computer scientist can tell you that. About the only time this is true is if you find yourself upgrading from bogosort. That or you’re breaking functionality somehow.

I actually want to see some metrics for your performance claims, as well as some methodologies. The java garbage collector exists for a reason. You can’t just “free up memory” without throwing something out.

Sorry for being critical, but this seems too good to be true.

However, if WE is truly THAT inefficient, then I can believe you, but I have enough faith in sk89q’s programming skills to throw that idea out pretty quick.

Actually, I have to say this. If your performance increase comes from eliminating repetitive overhead relating to implementation, that could actually make sense. Block update packets are big and bulky and often unnecessary. What i’m most interested in is how much performance increase you can get WITHOUT breaking Sponge behavior, such as block tracking and events.

4 Likes
#13
> rendering world protection plugins useless if normal players are given any kind of access.

Without FAWE it’s pretty trivial to crash a server if normal players are given access to WorldEdit. WorldEdit’s config is inadiquate to provide any sort of protection against anyone with malicious intent.

> WITHOUT breaking Sponge behavior

Actually I have no idea how deep the sponge block tracking goes. WorldEdit already uses a bunch of internal code for changes, so I’m not sure if default WorldEdit is even tracked for everything. Anywho, FAWE modifies the chunk object directly, so I doubt it will be tracked. Plugins can hook into the WorldEdit API for tracking changes, something which forge plugins already do if they need to track WorldEdit changes). There’s a hook for PlotSquared, which is about it on the sponge side.

Because of how much bandwidth FAWE can deal with, it’ll probably be a significant hit to performance to let sponge do the block tracking. A SpongeAPI only version of FAWE probably wouldn’t be more than 15x faster.

> 250x perf increase is ridiculous

So without parallel execution (because my CPU doesn’t have many cores) I can get > 50mil blocks in under a second assuming the chunks are loaded, and about 10mil if the chunks aren’t. With edits that require more lighting updates it will be slower. Note: This was using spigot, probably similar speeds for sponge.

Increasing the number of parallel threads would speed up the dispatching (the queuing is basically instantaneous for the set command with fastmode enabled). Having two cores (and threads) wouldn’t double the speed since there’s still some overhead where order is important and it can’t be done in parallel.

> You can’t just “free up memory” without throwing something out.

I can just free up memory without throwing things out. WorldEdit by default uses a wrapper for each block that stores the nbt, id and data. At a minimum this is 256 bytes per block in history. FAWE only stores NBT if it’s absolutely necessary, i.e. empty nbt is ignored. Then instead of a wrapper for the blocks, it uses a compressed byte array (using the LZ4 compression algorithm). For very simple edits (e.g. giant cubes of stone) this means you can be using a fifth of a byte for each block rather than 256 bytes.

Then, if you want to reduce it further, you can just tell FAWE to store everything on disk instead of in memory. Obviously going to be slower doing things on disk + using a high compressor. This is a test done with 70MB allocated to the server, with only 13MB free for loading and pasting a >2.5 million block schematic.

4 Likes
#14

Context! Yay! Thanks!

Although.

******* *** and your array copying speed :3

The 250x boost sounds a lot more likely (even a low estimate) when you’re circumventing the very idea of APIs, cause last i checked, just writing a bunch of 1’s to an array takes like no cpu cycles, but in my heart that still feels SOOOO LAMEEEEE.

But it does sound like WE uses a lot more memory than it should, like jeez. 256 bytes per block is actually kind of ridiculous. Then again, I don’t know if I want to know how large a sponge block snapshot actually is.

I’m actually going to write my own version of WorldEdit soon, but right now i’m sitting down and thinking really hard about how i could write it to be efficient on just SpongeAPI. I am limited to what the API provides, and for the sake of compatibility and failsafeness, Block Snapshots and tracking are a must. Although once the ball gets rolling feel free to toss me ideas. I’ll be sure to come to you when my async code explodes in my face.

Cheers and good job!

3 Likes
#15
> and your array copying speed :3

The slowest part is loading the chunks and performing lighting updates. Copying the arrays isn’t a big deal. :stuck_out_tongue:

> 256 bytes per block is actually kind of ridiculous

It’s about 256 bytes for a BlockChange object, which contains a BlockPosition, and two BaseBlock objects (the original, and the new one). It probably ends up being a lot more than 256 bytes since there are two BlockChange objects created for each block change (one for undo, and one for redo), which are added to lists. Then this is only for the ChangeSet extent, there are 11 other extents e.g. the MultiStageReorder has three lists that map the BlockPosition to BaseBlock (all three reordering stages need to complete fully before any blocks are changed). The Sponge block snapshot is probably pretty big as well, though probably a bit smaller.

FAWE justs skips most of those (i.e. no reordering since it doesn’t need to worry about torches popping off since the chunks are modified directly).

> I’m actually going to write my own version of WorldEdit soon

That’d be pretty neat to have some competitors to WorldEdit. There’s definitely ways I can think of to have more efficient block placement on just the SpongeAPI. Whenever you start, I’m happy to contribute ideas.

4 Likes
#16

Oh for the array copying speed i was talking about how that’s probably the fastest part, and that would be the only thing you’d ever be doing on the main thread. Since reference copying takes no effort for the cpu. I would imagine it takes like no time to execute. I just thought the direct access part was kinda lame, cause you ARE cheating a little bit. xD

#17

It depends a bit on whether it’s a merge or a reference copy. The dispatcher only knows about the blocks being placed, not what’s already in the chunk (since the queuing happens async the chunk can be changed up until the blocks get placed). There could only be a reference copy if every single block in the chunk section is being set.

The block history (undo/redo) is a separate stage to the dispatching and needs to load the chunks / get the block info on the main thread. I also cheat for that. If I was going to do a full rewrite of WorldEdit I’d probably combine the dispatching and fetching of blocks into a single stage, but that’s too much effort right now.

Both of the tasks that are performed on the main thread are pretty quick. Whether it’s a merge or a reference copy, it’d probably be dwarfed by the overhead either way.

#18

Just going to put it out there that directly modifying the chunk info is 100% not supported in Sponge’s implementation. Granted, you seem to have bypassed all of our tracking systems entirely by doing so, and gaining the performance boost (for obvious reasons), but you entirely eliminate the possibility of any sponge plugins being able to have any say in the matter of a few blocks being placed in x region. Having that said, I’d recommend you to take some actual performance measurements with Sponge and WorldEdit, and then Sponge and WorldEdit and FAWE installed. Namely, claiming that it’s x times faster than without on y system has no meaning on a Sponge environment.

2 Likes
#19

Honestly in my opinion, worldedit is more a deep tool for editing rather then a creative server user tool.

I could understand your arguments for VoxelSniper, but for worldedit?

If you are planning on making a worldedit alike mod, I’d suggest watching the schematic API closely, and perhaps look into ways into quickly editing and composing operations / masks / filters.

#20

Trust me, i’ve been getting advice from @Inscrutable on what he would like to see in a world editing plugin. Also i’m already watching Schematics, and I’ve already had a few things to say about it.

I want to make something amazing for people who are transitioning to sponge, but i have to do a lot of thinking to get it right. However, I will say that it will be at least partially inspired by photoshop.

2 Likes