Bitcoin and the kernel-in-a-kernel security sandbox problem

When considering sandboxes and security jails, the problem space is interesting.  There is an increasingly common pattern. I call it "kernel-in-a-kernel." The problem is not really sandboxing untrusted code, but more fundamentally, sandboxing untrusted behavior.

Bitcoin sees this acutely: bitcoind manages the bitcoin P2P network. P2P network is flood-fill a la Usenet, and anyone may connect to any node. Built-in DoS protections are a must, but these are inevitably hueristics which duct-tape one problem area, while leaving another open to algorithmic attacks ("this P2P command runs an expensive query, that impacts other connected nodes").

One comprehensive solution is accounting. Account for the various resources being used by each connected party (CPU, RAM, disk b/w, ...) and verify that some connections do not starve other connections of resources. This solution is a sandbox that essentially becomes a kernel unto itself, as the solution is not merely preventing sandbox jailbreaks but at a higher level limiting algorithmic jailbreaks.

Think about the high level economics of any computing situation. You have limited resources, and various actors have valid and malicious needs of those resources.  What is the best practical model for balancing a set of limited resources, given potential malicious or buggy/haywire users of these resources?