[Taxacom] saturday morning fun
David Remsen (GBIF)
dremsen at gbif.org
Sun Nov 28 11:28:43 CST 2010
I said in the previous email I would attempt to explain a bit about
the roots of the problems inherent to the taxonomic index within the
data accessed through the GBIF data portal. While Stephen laments "
'official' databases like GBIF ... feeding us sh!t" we should
collectively recall the old saw that when you point a finger at
someone remember there are three pointing back at you. We (not sure
who you think 'we' are) may indeed be responsible for all of messes
and problems you see in the data accessible via the portal but I
believe the causes can be distributed a bit more widely than a few
well-intentioned, but perhaps taxonomically under-informed,
programmers in Copenhagen.
Trying to organise 264 million-plus records from over 8000 different
databases is more complicated than you probably understand. I'd try
to explain but even a brief explanation isn't as short as Id like.
1. There is no "GBIF" data and GBIF is not a database. GBIF is an
agreement among participating countries and networks to publish and
There are datasets that originate in a growing number (~8000) of
datasets derived from natural history collections and observational
data. The list is here. http://data.gbif.org/datasets/ or here http://gbrds.gbif.org/browse/agents?agentType=RESOURCE
2. GBIF provides a mechanism to share these data and enable them to
be discovered in a single place instead of going to 8000 different
URLs and asking yourself. This is done by A) registering that
datasets exist and B) by providing a URL that links to some sort of
access mechanism whereupon GBIF (or anyone else) can access the source
data. We access these data and store a cache of it in our data portal.
3. Every month, we rebuild the index of data by starting at the top
of the list of registered datasets, contacting each database, and re-
acquiring a new set of data in case there are updates or additions.
This super-index is then available through our portal. It provides
links back to individual data records where the real details lay.
Your problem is that we have yet to overcome problems inherent in the
data to properly organise it into neat taxonomic bundles for more
precise access. Here is why:
4. After visiting all 8000 datasets and pulling and processing the
XML data we receive every month, what we obtain, and cache, from each
database looks like a single table, such as you can put in a
spreadsheet. This months table is 269 million rows deep. Among the
columns it contains are Kingdom, Phylum, Class, Order, Family, Genus
and species. These will be the columns for which you here have the
5. While you may think that the worlds collections data are sorted and
in good order, even in simple Linnaean ranks you would probably be
surprised at what shows up in the collective data. Last week we did
some basic analysis of this part of the index. Some basic numbers
can be found here:
Row 2 of the Basic Figures tells me that the 269 million records are
represented by 5.5M distinct combinations of Kingdom, Phylum, Class,
Order, Family, Genus, Species. It tells me that even Lynn Margulis
would be surprised at over 9000 distinct Kingdom values for these
data, surpassing the 5000+ Order values listed in the same file.
6. Here is a sample entry for a single species (Mytilus edulis, the
common blue mussel):
This ignores misspellings for the species name itself - which add to
the list but it illustrates a lot of variation both in the orthography
and correctness of the authorship as well as in the higher taxonomy.
Some is completely incorrect, some out of date, some missing, some
transposed and some is, well, it's right.
Of course we could easily weed out all of the incorrect stuff,
right? Sure, if A) we had basic lists of the correct information
and B) these authoritative sources came anywhere near being
comprehensive enough to answer basic questions like: How many Mytilus
edulis are there really? We don't have either.
7. Sampling only from these specimen and observation datasets, and
with the limited metadata provided from the soures, we have no
effective way of distinguishing good taxonomy and orthography from bad
or for qualitatively ranking one dataset from another. So this
makes even determining how many real taxa are represented in these
8. We therefore look for organised collections of authoritative data
that we might rely on to organise this giant and messy beast of
primary data. The Catalogue of Life has played this role almost
exclusively. Why? It's available for us to use and it offers a
chance we might do better than the source data to answer what must
seem like simple questions like "What records exist for Coleoptera?"
9. This only provides a solution for 54% of names that match the
Catalog of Life (as well as IPNI and Index Fungorum). For taxa not
listed in those sources our choices have been to 1) ignore them or 2)
try to provisionally place them in a taxonomic hierarchy that uses the
Catalogue of Life as the starting point and hope we get it close.
In other words, if someone asked for all the Coleoptera, we could
provide only those records that the Catalogue of Life says are beetles
or what the source data say are beetles. Since the COL only matches
50% of the names, we felt that 1) was too restrictive.
10. We derive a classification based on the Catalogue of Life where
we try to place species not reported in it to a higher taxon that is.
For example, if Scombrus maximus (i made this up) isn't in the
Catalogue of Life but is reported in the Family Scombridae (which is)
we could provisionally link the taxon in the Scombridae. This is a
simple and easy case.
More problematic is when the placement is ambiguous. Oenanthe appears
as a genus among the plants and the birds but within the plants it's
reported as both an orchid and an umbelliferid. Plant species are
listed in the Graminaceae and other older higher taxa that don't
conform to the available higher taxonomy we rely on and make it hard
to place them.
"Phantom" genera appear where the same genus is listed in different
higher taxa according to different sources. Genera appear listed as
Families or Orders. How should we know what are legit? Download
the complete and official list of genera. Or the one of families or
orders? Where are they? Do we teach our programmes the higher taxa
suffixes and let them sort it out? I say "Hah," sirs, simply "Hah!"
11. Lastly, Wolfgang, and others, have offered to 'fix' the
taxonomy issues in the portal to the extent we even funded a pilot to
see what a review could be found and done with the data. Like
everything else, this isn't as simple as just pointing out a bad
record. Since we rebuild the cache every month from a network of
sources, applying corrections either needs to be done at the source
or applied repeatedly by us. We are now working on the means to
support source-level remediation by
1) developing an annotation mechanism to enable suggested fixes to be
sent back to a source curator and
2) enabling taxonomic data-cleaning services to be offered by people
who have the sort of authority files to offer them.
We are also looking to have expanded access to a wider array of
taxonomic authority files that we can use as more and better authority
files. This has been hindered by an accessible data standard to
allow easy sharing of authority files as well as incentives to make
them available to GBIF and others. We are trying to work on this as
well. I posted the link to a call to evaluate that format and
particularly those parts related to citation and attribution by all
who use the data to those who curate it. These could help address
the above immensely. Want to help? Offer us some authority files in
this format. We will ensure you are credited.
So, while your Saturday morning fun appears to make what we do seem
sort of senseless and value-less, my Sunday evening response is not
based on that view. You are seeing a processed version of what we
see that originates in the big dirty messy world. I think GBIF
(Copenhagen) is remiss in that we have not properly put the noise in
it's proper proportion and done a better job of faceting and sorting
the majority "good" data from the bad. We work in a shifting
landscape of noisy stakeholders all shouting that everything is a
priority. I'm sorry we still haven't gotten this straightened out
but we hope to have a better solution very soon.
David Remsen, Senior Programme Officer
Electronic Catalog of Names of Known Organisms
Global Biodiversity Information Facility Secretariat
Universitetsparken 15, DK-2100 Copenhagen, Denmark
Tel: +45-35321472 Fax: +45-35321480
Mobile +45 28751472
On Nov 28, 2010, at 3:34 PM, Dr. Rodham E. Tulloss wrote:
> Dr. Lorenz,
> Thank you for drawing my attention to GBIF.
> I am one of the few persons left in the world who actually work at
> taxonomy of an agaric group worldwide and have over 30 years of
> experience. I
> work on the Amanitaceae (Agaricales, Basidiomycetes, Fungi).
> I went to look at GBIF this morning. For fungi, it is worse than a
> bad joke. I
> immediately decided that it was better to do a few more dozen new
> taxa than it
> was to be a part of straightening out the the hopeless mess related
> to my group
> on GBIF (such as it is...they have one specie inexplicably listed
> both as a
> fungus and a synonym of a Pine tree).
> I PAY MY OWN WAY in retirement. They have millions. What a travesty.
> Rod Tulloss
> Taxacom Mailing List
> Taxacom at mailman.nhm.ku.edu
> The Taxacom archive going back to 1992 may be searched with either
> of these methods:
> (1) http://taxacom.markmail.org
> Or (2) a Google search specified as: site:mailman.nhm.ku.edu/
> pipermail/taxacom your search terms here
More information about the Taxacom