Now there are some fun (and potentially important) questions about how to deal with uncertainty in the original utility function, but I won't go into these here.
The discussion made me think about the following sort of system:
- Start with a set of axioms and rules of inference to start with, and if you like, also a set of statements about the world (perhaps sensory data).
- Look for new logics which can derive what the old can derive, but possibly more.
- Judge these based on some criteria; in particular, shortness of derivations and simplicity of the new logics both seem sensible.