The system folder on my computer has 4GB of stuff in it. It's impossible to tell what's needed and what isn't. In a default installation of Windows there is little control over where a program can output data, what files can be read and written or how often the program can connect to the internet.
This lack of control has led to a rash of "Spyware" that quietly sits on your computer and connects to the internet, opens popups and who knows what else. Legitimate applications have also started using tricks to slowly pollute the system with add-ons, browser enhancements, desktop accessories, shell extensions, etc. Spyware removal tools exist that try to look for this bad behavior and try to catch it as it happens. The problem with this is that the new tools get created every day and creating anti-spyware tools seems to always be a game of cat and mouse.
What if, instead of allowing any file with an executable extension to run you could instead create a checkpoint during installation where users could decide to allow or deny certain application behaviors. The idea is to put basic software administration into the operating system and determine what programs will be allowed to do during installation.
For example a basic application like Calculator could execute but it wouldn't be allowed to connect to the internet or write to the disk. Such an application could then "Request" permission during install to do these additional actions.
I believe something similar was done with the Java applications but it doesn't need to be limited to a Java VM. Such a system could be built on top of any normal operating systems and could help prevent an entire category of security vulernabilities.
The key to getting this right would be to create the sandbox in such a way that it didn't distract, interfere or confuse users during a typical install process.
Let's say for example you want to install a Weather monitor utility. In today's world this simple weather utility could have an overflow bug that would allow an attacker to take control of the entire machine. Now imagine that this weather utility didn't have permission to write to disk or even read from disk. Now even if the application has a vulnerability the attacker would be constrained to the scope of the existing application.
It's a twist on user permissions. Typically users are recommended to not login and run as Administrator. Instead you can allow users to run as administrator but you actually run the applications at a lower permission then the typical user.