What is good quality evidence?

I've recently had an interesting email discussion about UFOs, which has some lessons to learn for UFO fans (and for politicians, amongst others) on what constitutes good evidence. I had reviewed a book called The Compelling Scientific Evidence for UFOs and this resulted in some discussion with its author, Erol Faruk.

I'm what I'd call a hopeful sceptic when it comes to UFOs. I'd really like UFOs to exist, but I expect the evidence supporting a UFO sighting to be strong - let's fact it, this is a pretty remarkable claim, so we need strong evidence if we are to support it.

A big problem I have with many UFO sightings is that they are based on either pure witness testimony or very bad photographs that just show lights in the sky or fuzzy blobs. In other cases, UFO investigators make a huge leap from 'there's something strange' to 'because it's strange, it must be caused by an extraterrestrial ship.' I'd like to give two examples that cover these three issues.

Let's start with the fuzzy photos. There are all sorts of problems with these. One is the ease with which they can be faked. Like many before me, I've been tempted to have a go at this myself when I was at university, though just for fun.

Here's one of mine, showing a flying saucer over Aviemore, Scotland. It's actually a metal camping plate, thrown frisbee-like, and could have been better, but you get the idea. (These days it would be trivial to do with Photoshop, but back then it was not an option.)

I don't doubt that many of these fuzzy images we see aren't intentionally fake. But so many of them are pretty much fuzzy lights in the sky (try living near Heathrow as I once did), or could be ducks, reflections, all kinds of confusing things. Nowadays most people carry a good camera and routinely take clear, quick snaps, so we should be getting lots of good, clear photos. We aren't.

In my email discussion I brought up the particular issue of a sighting of a craft over Phoenix, Arizona in 1997, detailed in Faruk's book. According to this, ‘hundreds, possibly thousands of witnesses’ saw the Phoenix craft in 1997, but without producing any clear photos. By then many people had video cameras and many of the witnesses were out to see the comet Hale Bopp and would have had cameras on tripods etc. Where are the good, clear, unequivocal photos and videos? In the 1980s, I saw my own equivalent of a UFO (except it was identified). I looked out of a window at home and my jaw dropped to see a space shuttle on the back of a 747 flying past. I was not expecting it and had no camera prepared - but I still managed to get several clear enough photographs of it. What photos I’ve seen from Phoenix are just lights in the sky.
Unlikely to be a bird or Venus
The response I got to my doubt was 'For many UFO sightings there is "shock" element which usually leaves witnesses rooted to the spot as they're fixated on watching the unknown object with the result that they don't think of rushing indoors to locate a camera to get evidence of what they're seeing. Take a look at the first part of this video [linked] showing several witness testimonies of the Phoenix boomerang object from different vantage points.' But I don't want testimonies, I want good evidence. And I really
don’t accept the shock argument - I was shocked to see a space shuttle outside my window, but it was all the more reason to get a camera. And lots of these people out to see the comet would already have had cameras: they wouldn’t have needed to go and get them, so it should have been even easier for them than it was for me.

To finish off on photos I'd also mention the infamous 'saucers over the Capitol building' photo from Washington in 1952. This looks really impressive as often shown as a cropped shot of the Capitol building with an array of lights in the sky over it - far too regular to be anything other than a UFO formation. Only when you look at the full, uncropped photo, there is exactly the same pattern of lights on the steps in front of the building. The 'UFOs' are lens reflections from the street lights.

With the Phoenix example I was constantly referred to the witness statements. Erol commented 'No, witness testimony is rightfully not considered "good" scientific evidence, but if it comes from entirely credible witnesses - and multiple ones at that - they also cannot be summarily dismissed in my opinion. Once you have some spare time do take a look at that video and judge the credibility of the witnesses yourself. As far as I know none of them were - unfortunately - in a position to snatch a camera to take photos. Does that mean that their testimony is worthless?' Well, yes, pretty much if it is unsubstantiated by evidence. Human beings are terrible observers. And it's very easy to see something in the sky at night and misinterpret it. We need stronger evidence than testimony.

Which brings us onto the topic of Erol's book. This was another sighting in Delphos, Kansas. Three members of a family allegedly saw a craft which had landed on their farm take off and fly away. They  describe finding a glowing ring where the craft had landed, which they took a photo of. In following days there was a white ring on the ground and the soil where the craft had apparently been in contact seemed strange. Erol later analysed the soil and found it contained some interesting chemical constituents that made it water resistant and fluorescent. That is, indeed, interesting. The problem is that in and of itself, a strange material in the soil does not in any way indicate extraterrestrial involvement - that suggestion is solely based on the witness testimony. There was, of course, the photo of the 'glowing ring' - but this is indistinguishable from a photo of the white ring in the daytime. It has also been suggested elsewhere that there was a circular chicken feeder on the farm previously at this spot, which could have caused the markings.

And here we get the problem of separating good evidence from bad. Erol's response to my points was 'My analysis showed the presence in the ring soil of a highly water-soluble organic compound which I characterised in terms of its light absorption and fluorescence properties. I then used this information - and only this information - to propose a hypothesis for the sighting report. This hypothesis perfectly explains the vivid colours of the UFO [reported by the witnesses] and of the glow observed between the object and the ground during Ronald's observation. It also explains the glowing ring left behind after the UFO departed of which a photo was then taken . It perfectly explains the ring elongation of the ring towards the wind direction on the night of the event. It also explains the pitted craters on the ring felt when the witnesses touched its surface, as well as the moistness of the ring and the noted anaesthetic effects consistent with the dual hydrophobic/hydrophilic nature of the soil compound.'

Unfortunately, the only evidence that the ring was produced by a UFO was witness testimony from a small group of people (who allegedly were paid for providing their story, though I don’t know if this true). There was no strong evidence for either the presence of the UFO or the glowing ring - again, the only evidence this wasn't a flash photograph, or that it was taken at night, was from the witness statement. The only strong evidence was the chemical analysis - which showed there was something unusual in the soil, but did not provide anything suggestive of extraterrestrial origin. The weakness was the dependence on the family’s testimony to make the connection from ‘unusual chemical deposit’ to ‘UFO’.

I am not in any sense saying that this shows there aren’t UFOs. It doesn't. No evidence can prove this. But we always need to be aware of the quality of evidence at each stage of making a connection. And in this case, while one bit of evidence is good, it is nowhere near enough to make the leap to evidence that an extraterrestrial craft landed.