Harking back to my whole schtick with the Prisoner's Dilemma where the "rational" solution -- defect -- is the "obvious" strategy which maximizes-reward and minimizes-risk over the short term. This optimizes well for evolutionary natural selection in a scarce and hostile environment, but results in a slightly less beneficial overall outcome -- versus cooperate -- for both parties over the long term.
And the long term is significant these days.
Thanks to over fitting, the Social Darwinists of the 19th century are no longer with us -- except in the guise of Libertarian economists -- but we still don't really think outside of the risk/reward box.
I recently attended an SFI talk by an evolutionary roboticist who, as an aside, complained that his evolved robots did not do so well in the long term. When I suggested that other fitness functions might be tried he pretty much dismissed the idea because they would be out-competed in the short term (in my own self-serving paraphrase of the interchange...).
But what if we try to do evolution with different utility functions?
- What's the best option for the common good?
- How can we all have the most fun?
- Can I be the best improvisational drummer in the ensemble?
- What would make the prettiest rainbows come out of my unicorn's butt?
If it's not Science, then what is it?