[Taxacom] Methodological plurality [was cladistic analysis for morphological characters -- UPGMA is not cladistics]
Derek Sikes
dssikes at alaska.edu
Tue Nov 20 10:57:02 CST 2012
Interesting thread. Here are my two cents:
I agree that one should justify the method or methods of choice, ie,
explain why, not just what, was done. Often methods sections of papers are
mere lists of what was done with virtually no explanation as to why. We
have well established means to select models that best fit our data - AIC,
Bayes Factors, etc. and certainly one should show that effort was made to
ensure model misspecification was minimized.
However, if this approach was combined with the argument that only one
method should ultimately be used (MP, ML, BI) then Parsimony would never be
justified since it is really just a non-parametric version of a weird
(overly parameterized) ML model that would never be chosen by AIC etc,
leaving us with two options (ML & BI). That being said, I always tell my
students that they should know as much about their data as possible. One of
the benefits of using multiple methods is that doing so, and knowing that
each method has its own potential weaknesses, informs you about your data.
For example, (and sorry, I know this horse keeps getting beaten) it is well
known that if you have long branches mixed among short branches you might
end up with a very different tree using MP than ML or BI due to long branch
attraction. If you have such a dataset I would insist a student use
multiple methods. The idea of course is akin to the sensitivity analysis
that Kip described as being inadequate previously, and yes, it's not a full
parameter-space sensitivity analysis, but it is a good,
better-than-nothing, first pass at seeing if your dataset might pose
problems for one method but not another. (And not to harp too much on MP -
it is well known that ML and BI have their own weaknesses, but this
knowledge can help identify problems and therefore help solve them). Any
method can fail due to model misspecification.
If all methods yield the same answer this is not useless. It tells you that
you probably don't have long branch attraction problems. If the branch
support agrees among methods it tells you the Bayesian method is probably
not suffering from a star-tree paradox scenario. If each method gives you
different answers you must then dig deeper to figure out why. Sometimes
reviewers and readers really want to see these alternative answers - it
helps them learn as well (rather than just say "different methods gave
different answers but I am only presenting and discussing results from the
best fitting model approach."
I agree that one shouldn't use multiple approaches mindlessly as if by
doing so one is "covering all the bases", but also argue that one shouldn't
present only a single approach mindlessly ignoring results from other
methods. I hope we can all agree that doing anything that can be considered
'mindless' in science is bad.
Heck if UPGMA gave the same exact tree as a 3-month long 10 partition ML
analysis, that would be amazing and worth reporting (although I suspect it
would never happen!)
-Derek
On Tue, Nov 20, 2012 at 7:05 AM, Ashley Nicholas <Nicholasa at ukzn.ac.za>wrote:
> "Congruence or conflict across methods tells us little or nothing about
> clade support."
>
> I agree with this statement and many of the others given below. However,
> this discussion has left me feeling somewhat unsettled and very conflicted.
> Say we use just one method, as advocated, what do we have in the end? A
> tree that may have good clade support but which hangs unconnected to
> anything else but itself. Surely this end result will eventually have to be
> tested or retested? As pointed out, the chances are there will not be
> congruence when it is retested with a different method -- which means the
> result needs to be treated as highly uncertain from the start. The only way
> this uncertain tree can be tested is if it is tested using the same method.
> What is the solution to this dilemma? It would seem we only have two
> choices. Either systematists agree to use only one method and abandon all
> others. Or we explore the question/problem more widely with as many methods
> as possible and see what insight (if any) we gain from this. Albeit bearing
> in mind the flaws to this pluralist approach and the fact we are not
> comparing apples with apples (in a methodological sense).
>
> I would hesitate to abandon congruence and consilience because of the rich
> rewards it has already given us in terms of understanding our universe.
> E.g. Newton's connection of failing objects on earth and planets going
> around the sun actually being linked by the same cause. Which brings me to
> the comment: "If two people that view the same world (the data)
> differently, come to the same result they could just as well both be wrong
> or both be right." But isn't this why as scientists we test and explore
> further? In the case of Newton the movement of Mercury eventually falsified
> his hypothesis of gravity and lead the way to relativity.
>
> I am not a molecular systematist (my postgrads are co-supervised by
> someone more qualified) but I am now wondering what advice I should give
> them as we have tended to use a pluralist approach in the past. Will the
> examiners of PhD theses be happy with students using only one analytical
> method?
>
> Thanks for this interesting thread and the insights provided. I have
> really gained a lot from this discussion. Even if it has left me feeling
> conflicted.
>
> Ashley
>
> -----Original Message-----
> Kip, I will have to steal this for thesis defenses:
>
> "agreement of 5-7 different methods. Good feeling is the same as
> statistical support, right?"
>
> Dan
>
> On Sat, Nov 17, 2012 at 3:27 PM, Kipling (Kip) Will <kipwill at berkeley.edu
> >wrote:
>
> > On 11/15/2012 7:51 AM, Ashley Nicholas wrote:
> > > I guess they do this to assess if there is any congruence in the
> > > results
> > of all three analytical methods?
> >
> > On 11/17/2012 6:42 AM, Dan Lahr wrote:
> > > More interestingly though, you raise an issue that has always
> > > bothered
> > me:
> > >people using multiple reconstructions methods. I'll start by saying
> > >I do not have a strong stance nor the answers to this question.
> > >
> > > In principle, it seems philosophically incoherent
> >
> >
> > This is a point that triggers my peeve response. The incoherence Dan
> > refers to being the root of my discomfort. Different methods have
> > unique, but partially overlapping, ontological and epistemological
> > starting points and so their congruence cannot be considered to mean
> > much of anything at all. And yet, many (most?) publications throw the
> > "big three" (ML, Pars, Bayes) at every data set, almost never
> > explaining why or what it means, usually then one is picked (with
> > little explanation why that one) and the rest ignored. I have
> > facetiously suggested to people that if they add UPGMA and NJ that
> > they might feel even more warm and fuzzy (or Poy, traditional
> > Hennigian, SWAG, etc.), given agreement of 5-7 different methods. Good
> > feeling is the same as statistical support, right?
> >
> > Congruence or conflict across methods tells us little or nothing about
> > clade support.
> >
> > 1. These are not independent data being run using the different
> > methods so this fails to be support through consilience. If two people
> > that view the same world (the data) differently, come to the same
> > result they could just as well both be wrong or both be right. That
> > agreement (or
> > lack) tells us little about the result, perhaps more about the methods.
> > Using Pars for morphology and, for example, ML for DNA sequence data,
> > may be a situation where one can invoke consilience, but I doubt that
> > is to be preferred over a combined analysis.
> >
> > 2. If one cast all methods to be a form of one particular method (e.g.
> > parsimony is just one model of likelihood) and thinks of using the
> > different methods as a "sensitivity" analysis, then it is a very poor
> > sensitivity analysis indeed. We have many much better ways to explore
> > sensitivity of results across parameter space.
> >
> > 3. If congruence is thought of as a measure of support, even as a
> > vague, number-free tingly feeling, and if we want to maximize that
> > congruence, then this is maximized where all methods converge, which
> > would probably can be achieved by using model parameters in ML, etc, to
> fit parsimony.
> > Of course that would be a silly thing to do, unless you really believe
> > congruence across methods means something. I don't see statisticians
> > using multiple different tests and then implying or claiming that
> > since those three test all (or say 2 out of 3) had significant
> > p=values the result is "more significant" (increased "truthiness"?).
> >
> > It is legitimate to do a comparison of methods for its own sake, they
> > each have properties worth exploring, but that is rarely what is
> > published. One may wish to specifically test certain assumptions,
> > again that is valid, but when there is no explanation for the
> > methodological plurality, or a simple statement that implies support,
> > e.g. "ML and Pars gave basically the save topology" and nothing more,
> > it seems a waste of time. If its phylogeny, topology or evolution
> > along branches one is after, then pick the appropriate method, justify
> > it and understand its limitations.
> >
> > Echoing this-
> >
> > On 11/17/2012 6:42 AM, Dan Lahr wrote:
> > > you should choose a strategy and stick with it, many of the
> > > different
> > analytical methods are logically incompatible.
> >
> >
> > yep, Dan is right.
> >
> > kip
> > --
> >
> > Contact info:
> >
> > Kipling W. Will
> > Associate Professor/Insect Systematist Associate Director,Essig Museum
> > of Entomology
> >
> > send specimens to:
> > Essig Museum of Entomology
> > 1101 Valley Life Sciences Building, #4780 University of California,
> > Berkeley Berkeley, CA 94720-4780
> >
> > letter mail to:
> > 130 Mulford Hall
> > ESPM Dept.- Organisms & Environment Div.
> > University of California
> > Berkeley, California 94720
> >
> > fax 510-643-5438
> >
> > _______________________________________________
> >
> > Taxacom Mailing List
> > Taxacom at mailman.nhm.ku.edu
> > http://mailman.nhm.ku.edu/mailman/listinfo/taxacom
> >
> > The Taxacom archive going back to 1992 may be searched with either of
> > these methods:
> >
> > (1) by visiting http://taxacom.markmail.org
> >
> > (2) a Google search specified as: site:
> > mailman.nhm.ku.edu/pipermail/taxacom your search terms here
> >
>
>
>
> --
> ___________________
> Daniel J. G. Lahr, PhD
> Dept of Zoology, Univ. of Sao Paulo, Brazil
> _______________________________________________
>
> Taxacom Mailing List
> Taxacom at mailman.nhm.ku.edu
> http://mailman.nhm.ku.edu/mailman/listinfo/taxacom
>
> The Taxacom archive going back to 1992 may be searched with either of
> these methods:
>
> (1) by visiting http://taxacom.markmail.org
>
> (2) a Google search specified as: site:
> mailman.nhm.ku.edu/pipermail/taxacom your search terms here
> ======= Please find our Email Disclaimer here-->:
> http://www.ukzn.ac.za/disclaimer =======
>
> _______________________________________________
>
> Taxacom Mailing List
> Taxacom at mailman.nhm.ku.edu
> http://mailman.nhm.ku.edu/mailman/listinfo/taxacom
>
> The Taxacom archive going back to 1992 may be searched with either of
> these methods:
>
> (1) by visiting http://taxacom.markmail.org
>
> (2) a Google search specified as: site:
> mailman.nhm.ku.edu/pipermail/taxacom your search terms here
>
--
+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects
Associate Professor of Entomology
University of Alaska Museum
907 Yukon Drive
Fairbanks, AK 99775-6960
dssikes at alaska.edu
http://users.iab.uaf.edu/~derek_sikes/sikes_lab.htm
phone: 907-474-6278
FAX: 907-474-5469
University of Alaska Museum -
http://www.uaf.edu/museum/collections/ento/
+++++++++++++++++++++++++++++++++++
Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact.php
More information about the Taxacom
mailing list