Advice: on choosing among different indices of the "same" thing

Lots of terms in ecology are only loosely defined, or can have somewhat different meanings depending on the context. Which can make it difficult to measure those things, because different measures often will behave at least slightly differently. "Diversity" is a good example--there are lots of different "diversity" indices. So how do you choose the "right" index, or the "best" index, of whatever it is you want to measure? And what do you do if your results differ depending on what index you choose?

This issue is one that I think many ecologists worry about a lot more than they should. It's really very simple:

• If you're testing a precisely-defined model or hypothesis (which basically means a mathematical model, or a hypothesis derived from a mathematical model) that predicts the behavior of a particular index, then that's the index you need to measure if you want to test that model.  For instance, if you want to test Bob May's classic complexity-stability model, then your measure or index of stability needs to be the one May used, or one that you can show is tightly correlated with the one May used. And if that index of stability is difficult or impossible to measure (which in the case of May's model, it is), then you either have to find some other way to test the model that doesn't involve measuring stability, or you have to go ask some other question entirely.
• If you're testing an imprecisely-defined model, like a verbal or "conceptual" model, that doesn't specify a choice of index, then the choice of index is completely arbitrary, so just pick one and don't sweat it. Worrying (or arguing with colleagues) about which index is "best" in such contexts is totally pointless. There's no way to choose the "best" index of something that's imprecisely defined. You can't choose the "best" measure of something unless you know, on independent grounds, exactly what that something is. Yes, this means your results may well depend on your choice of index. If that bothers you (and in many situations, it should), you should pick or develop a more precisely-defined model or hypothesis to test.
• The only reason to calculate various indices of the "same" thing and then compare your results across those indices is if different indices give you complementary ecological information. For instance, if your hypothesis predicts that experimental treatment X will increase species richness but reduce Simpson's diversity, then measuring both those "diversity" indices (species richness and Simpson's diversity) helps you test your hypothesis. But it is not interesting or useful to calculate various indices simply to see if your results vary across different indices. Different indices are different. Of course they can behave differently. If they couldn't, they wouldn't be different indices.