(Old topic but still relevant, OP brought it up on IRC #glowstone++ yesterday, so I’m replying here despite age)
The web browser security model is actually a powerful counterpoint. Could the web have taken off if you had to “trust” every website you visit? I’d argue its success largely stems from the lack of trust required to visit random websites.
This is possible because browsers are heavily sandboxed, running the website in its own environment isolated from the rest of the system, at least to some extent. Of course, browser security isn’t perfect, there have been bugs allowing malicious websites to “break out” of the sandbox, but those are security vulnerabilities intended to be fixed promptly, not an inherent part of the design. Websites can try to exploit bugs or pretend to be other websites (phishing), so it is still wise to cautiously avoid shady sites, but in general it is fairly safe to visit websites.
Since no prior trust is required, anyone can setup a website, and anyone can visit it, no need to negotiate trust beforehand. This is a huge boon for sharing information, since it doesn’t have to go through the bottleneck of a trusted third-party. If you do want to authenticate who you’re talking to, then layers like TLS are available which do make use of trusted third-parties, but in either case, websites by design cannot run code on your system without your permission except in their sandbox.
The current vanilla Minecraft client/server security model is mostly similar to this web security model, where you can (fairly) safely connect to untrusted servers. Some trust is required since you do give up your username (privacy), but Mojang username authentication is even designed to allow untrusted servers since your actual password is only sent to their centralized trusted Yggdrasil server, not the untrusted game server.
An interesting idea, sort of reminiscent of the web plugins security model: ActiveX, Flash/Shockwave, Java Applets, etc. Code could be “signed” by a trusted party to mark itself as trusted. Or a plugin could be acquired through a trusted source, then allowed to execute when loaded by various websites (object/embed tags).
Authenticating the source of the code can help, but still has the downsides you mention. Also, the code might not be malicious in itself, but have security bugs. It is very difficult to write safe code running with elevated privileges, as the web plugin security disaster as shown us (fortunately they are now mostly obsolete). The trusted Online Repo would have to vet the code carefully, and if it makes a mistake erroneously approving bad code, pull the malware and revoke the trust. It is a workable system, but requires a lot of effort and infrastructure and manpower. And trust.
This does seem to be the direction Minecraft is going, a more “data-driven” design. Where much of the content is merely data (usually JSON, example: block models in resource packs), so it can be safely shipped over the wire and rendered without executing custom code. It’s generally good practice and I’m all for it, will be interesting to see how this turns out.
But will data-driving be enough? It should be sufficient for some simple use cases, like adding new blocks with custom textures/models but without custom behavior, or new items with well-defined properties. I think it would cover some of what mods do today, but not at all.
(Sidenote: I can only speculate, but anyone remember when Mojang hired Tahg from the Bukkit team, then he departed a few months later, citing “Creative differences and frequent changes in developmental focus”? Then he started a modded Feed the Beast server, TahgCraft. Wonder what those “creative differences” were?)
Sill would be better than the status quo, but I bet you there are still significant use cases where you really do need to execute custom code on the client. My favorite example of this is the mod Thaumcraft: it doesn’t just add inert items/blocks, but tons of new special behavior, client-side walking modifications, rendering overlays, multi-block structures, and so on. The client cannot (yet?) express this functionality in terms of JSON data. Sure in principle it could, but modification developers are very creative, always pushing the limits, and going beyond the bounds of what is possible often requires client-side code execution.
Then the real question is, how to safely execute custom code on the client? Signed code is one answer (but see above for downsides), another possible solution is sandboxing.
Going back to the web platform security model, where websites run in their own isolated sandbox environment separate from the rest of the system, what if Minecraft did the same?
This is along the lines of what I’m going for with voxel-clientmc, a web-based Minecraft-compatible WebGL client. Of course the first step is getting a usable near-vanilla-parity client working well, but the long-term goal is opening up client-side modification enhancement. Since all of the code is downloaded from the website to your browser and runs in the sandbox, it is already safe (modulo browser security bugs obviously, but those would be a bigger problem than just this project). Untrusted servers can effectively send their own completely custom client. I already have client-side Forge support working (FML|HS plugin channel handshake packets) and Sponge (and Bukkit and standalone) plugin support, but not currently rendering any custom blocks or other content. It would be a large effort, but it in principle should be feasible and have many upsides.
Not that sandboxing doesn’t come with disadvantages. Sandboxed code might run slower, and it has more limitations than native code (no direct filesystem access, no network access except on the same origin or sites enabling CORS, and so on, depending on the sandboxing technique). Sandboxing the existing Java-based Minecraft modding ecosystem is probably infeasible at this point, though there are some technologies e.g. the Java Security Manager (but when I search for it to find more information, mostly the hits I get are about its security vulnerabilities). There is no free lunch…