Daniel M. Russell's Blog, page 4

SearchResearch Challenge (3/4/2025): Can you extract and summarize a blog?

Can AI systems really read the web? 
P/C Gemini [elderly robot reading a long scroll of blog names]. No I don't recognize any of the blogs shown on the scroll, nor why it uses so many different fonts in a ransom note style. 

Just last year people using AIs to ask up-to-date questions would often run into the "knowledge cutoff date" problem.  That's the date of the last training cycle, when all of the information in the LLM was solidified.  
Since then various AIs have been claiming that they now have continuous updates because they're able to read web content in real-time.  (Here's a news article describing the new real-time web access capabilities of some LLMs.)  
I've explored this a bit and come to some insights--and that leads to today's Challenge: 

1. Can you get an LLM AI system to give you the 10 most recent links to a blog?  For instance, could you get Gemini, Perplexity, ChatGPT, or MS Copilot to make a table of the 10 most recent blog posts?

That seems like a simple enough request for an AI system, yes?  
We keep reading about AI agents that claim to be able to take actions on your behalf by operating your applications to get-things-done.  So let's ask for a simple task ("make a list of the 10 most recent posts") and see how well it does.  
I'll show you how I did it next week and talk about the issues involved.  
I'm VERY curious to see how well you did.  Were you able to do one request and get back the list?  
As a sample blog, you might try good old SearchResearch, but you might also try something like OpenAI's blog or Google's Gemini blog. 
Keep searching!  

Answer: To search, or to AI? That is the question...

The question is--what works? 

If you've been using online search tools for any amount of time you know that which tool you use, and how you use it can make a huge difference in the quality of your results.  

This week I Challenged you to find what each of these images is really about.   When you try to find out the story of these images, where you able to use "regular" Search-by-Image, or did you use AI image descriptions?  If so, what prompt did you use, and how well did it work for you? 

1. While traveling in northwest Switzerland, I saw this emblem all over the city I was visiting.  Can you tell me what city I was in?  Just as importantly, what IS this thing? 

For each of these images I used 2 different methods: (1) regular Google Search-by-Image, SBI (you know, upload the image and let it search--aka Google Lens), and (2) upload the image to an LLM (either Gemini or ChatGPT) and ask it to identify what it sees in the image. 
Image B1: On a box of chocolates... 
Image B2: Carved into a stone wall

Image B3: Seen on a parade float
Here's what I found: 
B1 (gold symbol on black/white background): This image is very focused on the object of interest and SBI does a much better job.  Here's what the Google Lens (mobile) interface looked like when I asked about this: 
Mobile Lens search for B1 image
It STILL doesn't tell me what it is, but at least we learn yet another potentially interesting word: Baslerstab.  Searching for this leads to: 


We see that "Baslerstab" is the name of the local Swiss-German newspaper AND that "baselstab" is the name of the symbol!  (German note:  "Basler" means a person or thing from the city of Basel.)   Clicking through on the Wikipedia link takes us to the Baselstab article, telling us a bit more: 

"The Baselstab or Baslerstab is the name given to the shepherd's staff or bishop's staff in the coats of arms of villages and towns in the Basel region in Switzerland. The crosier of the Prince-Bishop of Basel can be found in the coats of arms of the city of Basel and the cantons of Basel-Stadt, Basel-Landschaft, and Jura. The crosier of Basel-Stadt is black, while those of Basel-Landschaft and Jura are red."  

Excellent.  Now we know that this symbol is the emblem for Basel and is an abstract version of a bishop's crozier.  
Since I told you that the Baslerstab was on a box of chocolates, I was wondering what the connection was between this symbol and chocolate.  Reasonable question for Gemini, yes? 
Unfortunately, Gemini seems to fall into a hallucinatory rabbit hole with this answer. Despite what Gemini says in its answer, as far as I can tell, there is NO connection between Lindt (a large chocolate company in Zürich) and the actual chocolate company that uses the Baslerstab symbol (which is called Laeckerli-Huus, headquartered in Basel).   Aaarrghhh!  Everything in this answer below about the crozier and Lindt is wrong.  

Since that didn't work well, I did the obvious query on regular Google: 

If you click on that first link you'll see exactly the image of the entire box of chocolate and cookies, a nice example of the baslerstab being used as a design element.   
That's MUCH better than Gemini.  Another example of doing a simple Google search that works out much better than asking an AI.  
For the record, ChatGPT didn't know either... and also completely hallucinated a connection between the baslerstab symbol and coat of arms of the canton of Fribourg, Switzerland where the chocolate maker Maison Cailler is based.  Utter balderdash.  

B2 (symbol on wall): This was the symbol carved into a stone wall in the city's center.  Sadly, nothing I could do with SBI would work.  Google failed, Bing's SBI failed, and even focusing in tight on the symbol didn't help.  Search-by-Image just couldn't handle this.  


When I turned to Gemini with the wall-carving image, and asked [ identify this symbol ], Gemini replied: 

"The symbol shown in the image appears to be a heraldic shield with a stylized bird's head and neck facing left, and a lozenge (diamond shape) below it."  

Not quite.  
However, Gemini also said (quite reasonably):  

"To understand the full meaning of this symbol, you would need to know the context in which it was found.  Here are some questions to consider: Where was the image taken? The location could provide clues about the origin of the symbol and the family or institution associated with it."  

To which I answered, [ the image was taken in Basel, Switzerland ] to which Gemini replied: 

"Okay, knowing that the image was taken in Basel, Switzerland, provides some very important context. Here's how that helps us understand the symbol:
Basel's Heraldic Traditions:Basel has a rich history of heraldry. Notably, the city is strongly associated with the "Baselstab" (Basel staff), a stylized bishop's crozier. This is a very prominent symbol in the heraldry of Basel-Stadt (the city of Basel) and Basel-Landschaft (the surrounding canton).   
Another significant heraldic figure associated with Basel is the Basilisk.
 This mythical creature is frequently seen in Basel's heraldic displays."  

Which tells me that this is a baselstab (the crozier symbol) AND that the City of Basel also uses a basilisk as a heraldic element.    But of course I had to check this as well... luckily, it passes the sniff test.  
When I checked with ChatGPT, it also gave me a poor result: 
ChatGPT trying to identify the carved symbol.

But since Gemini needed to know where the image was from, I also told ChatGPT that this image is from Basel.  Guess what?  A new answer emerged!  
After I told ChatGPT the image is from Basel, it was able to figure out what it was.  

Lesson learned here--location matters!   
So I tried again with Google SBI.  Turns out it DOES matter!  



It makes complete sense--given a random image from somewhere in the world, adding LOCATION information would help any identification process (including humans!) to figure out what a symbol is.  

B3 (dragon with shield): Going through the process again... and once again, SBI didn't work, even when I cropped to just the symbol. Here's the "regular" SBI search:  Click to see full-size
It correctly identified the dragon-like creature as a basilisk, but that's not what I'm interested in--I want the symbol on the shield. 
Fortunately, Google Lens now has the ability zoom in on just the key feature by dragging the white rounded angle brackets--in this case, to focus in on the symbol.  When you do that, you'll get an okay search--the first two results are good, but the rest are terribly off-topic! 

The good news is that you learn that it's the city symbol of Basel, but you won't find out WHAT it is!  
If you ask Gemini to [ describe the image ], you'll get a nice description:  

"This image shows a close-up, angled shot of a bronze sculpture of a griffin holding a shield. The griffin, with its eagle-like head and wings, is the dominant feature, positioned in the left half of the frame...The shield the griffin is holding is prominently displayed in the center of the image. It features a unique emblem, possibly a heraldic symbol, with a curved, stylized design."

But no amount of me asking what that symbol is could convince Gemini to give me a real answer.  Not a great result.  Even when I gave it the cropped version of the image (showing just the symbol itself), Gemini couldn't say what it was.  
By contrast, ChatGPT did much better:  When I asked [ what's the symbol in the center of the shield ] ChatGPT responded with: 

"The symbol in the center of the shield is a heraldic emblem known as a "crosier," which is stylized like a pastoral staff with a curled or spiral top. It resembles the ceremonial staff traditionally carried by bishops or abbots, often symbolizing religious or spiritual authority. The depiction on this shield is stylized, emphasizing the curved top and decorative elements typical of such symbols."  

So now we have a term to look up: "crosier."  A quick search takes me to Wikipedia where I learn that the word is usually spelled "crozier" and it's a stylized shepherd's crook, which is ALSO the symbol for the city of Basel, as you can see in the image below:
P/C Wikipedia

But then I took my own advice and added Basel Switzerland to the SBI search, and as expected, it really improved things: 


SO... I took my own advice AGAIN... and re-ran the very first query of Gemini ADDING in the information about [...this image was taken in Basel Switzerland].  Voila!  Here's the answer:  

Which is a pretty good result.  
BUT... you have to know to prompt it with all of contextual information!  

I'll tell you the ground truth:  I took all of these images while visiting Basel for Fasnacht--huge parades with floats (like the one above), and repeated crozier symbols everywhere.  Since I was literally seeing this just about ever time I looked up, I was curious what it was.  B3 was taken during a Fasnacht parade as a float slowly drifted by me, the box of chocolates was something I bought, and the stone carving was taken as I walked around the city at 4AM for the Morgestraich.  

2. In open green spaces all over Zürich I keep seeing this flower popping up--they're growing everywhere in vast numbers.  What is it?  Again, how do you know?  What search method did you use? 

Image P1: A few flowers carpeting the lawn. What are they?

I took this photo at a random lawn in Zürich because I was curious--any open green space seems to be blanketed with these flowers!  What are they?  

Since I was walking around, I did the obvious Google Lens search: 

 

As you can see, it calls it a gentian.  Well... notice how Lens has selected an individual flower in the middle?  That does kind of look like a gentian, but I know gentians, and this isn't a gentian.  

By resizing the image when I got to my desktop, I got a much better answer: 

This is correct--it's a Primrose, Latin name: Primula.  It's a genus of herbaceous flowering plants in the family Primulaceae. 

I thought I'd try asking our favorite AIs what they thought.  My prompt after uploading the image was [describe what's in this image] 

Again, a reasonable answer from Gemini, but not great.  This looks a bit like Polemonium pulcherrimum, but it's not quite right.  Identifying small blue flowers is always hard.  

By contrast, ChatGPT gives a nice description, but no identification: 

Not quite what I was looking for.... so I changed my prompt to:  [what kind of flowers are these] NOW I got a much better answer.  Here's Gemini's answer: 

And ChatGPT's: 

Which is also exactly right.  But, as always... double check.  Never trust an AI's identification of flowers, trees, dogs, or exotic symbols found on the packages of yummy chocolate!  


SearchResearch Lessons 

I didn't mean to write quite as much as I did--but this is a never-ending topic.  Let's summarize what we learned from this little escapade... 

1.  Regular Search-By-Image is great at some things, but not at others. Usually it's simple to tell when SBI doesn't work--the hits don't tell you anything about the object of the search.  BUT... you can improve your SBI query strategy by... 

2. Include location information about the object of your SBI search. We saw that including the city name vastly improved SBI search on Google.  (But, oddly, it did nothing to help Bing's SBI!)  

3. Including location information VASTLY improves AI search.  We saw that in both Gemini and ChatGPT--their results got much better when it knew where the target was from.  Kudos to Gemini for mentioning that "including the location" would help.

4. Use AI/LLMs for search at your peril.   We've talked about this before--they give what look like very plausible answers, but they can be complete nonsense.  It's tempting to believe them, but the stories AIs told me about the connection between chocolate and the baselstab symbol were truly inventive...and wrong.  

5. Flowers are still hard for AIs.  Many flowers look a lot alike (e.g., blue primroses and gentian).  I'm not really blaming AI for getting it wrong, but at the same time, they don't know their limits.  Really.  An AI SHOULD be giving you warning signals when it really can't accurately identify something like a flower or a carved symbol.  But they don't.  (And, for heaven's sake, do NOT trust an AI to correctly identify a mushroom for you.)

6. AIs are sensitive to the prompts you give them.  We found that asking an AI to [describe this image] doesn't work--be direct and say what you really want to know--e.g., [identify the flower in this image]      

So, to answer our original question, "What works?"  There isn't any one answer, but knowing what your tools can do... and what they can't do.  Be specific in your questions to an AI, provide as much information as you need, and remember your ordinary online search skills are still invaluable.  

Keep searching! 

A note about Heterophylly in the real world

Our Challenge about leaf variation on an individual plant made me more observant in the real world... 

Holly leaf variation on the same plant. Taken in Zürich by Dan.  The left image is from 3 meters above the ground in an unpruned part of the plant. The right image, from a well-pruned part of the bush, is about 1 meter off the ground.  

... I hope that being a Regular Reader of SRS does the same for you as well. 

I was out walking last week, not long after having written about Mimcry in Plants (March 14, 2025), when I happened to cross paths with a beautiful holly bush.  Like most holly bushes, it had the usual assortment of glossy green prickly leaves--it was lovely, but not the kind of bush you'd reach out and touch.  

But having just read/written about heterophylly, I took a closer look--sure enough, just as it says in the books, there was a distinct variation between the lower branches (which had been pruned extensively) and the upper branches (which had never been pruned).  Even better--the right side of the bush had a slight overhang into a neighbor's yard.  They clearly had been pruning there--on the right was an armada of sharp-tipped leaves. 

However, on the upper branches, especially on the left (and unpruned) side, smooth edge leaves abounded.  

As I wrote last week, the mechanism of holly heterophylly is pretty well understood.  The prickly/smooth variations are a result of damage to the leaves.  When the leaves are damaged (say, by a passing hungry deer or by a neighbor’s pruning shears), methylation of the DNA in the leaves happens as a result of tissue damage.  (Side note: methylation is the process of adding methyl groups onto pieces of large molecules, like DNA, to modify their behavior. This is the way much of epigenetics works. When an animal chomps on a leaf, methylation happens.)

By comparing the DNA of prickly leaves vs. smooth leaves, it turns out the prickly ones were significantly less methylated than prickless leaves, suggesting that methylation changes are ultimately responsible for leaf shape changes.  More methylation = more prickly leaves.  What's more, the methylation has an effect on nearby leaves.  Other holly leaves nearby will also develop the prickles, with the effect diminishing with distance.

I was delighted to see heterophylly in action while just strolling down the street.

Keep looking... and searching.

SearchResearch Challenge (3/25/25): To search, or to AI? That is the question...

Looking for an answer? 

Online search can often get to an accurate, trustworthy answer quickly.  But as you know, it really helps if you know a bit about what works.  

One of the major tools has been Search-by-Image, through either Google, Bing, Tineye, or Yandex.  

But now multimodal AI systems can analyze an image and tell you the story... or can they? 

As a skilled and literate searcher, you should understand the differences.  So let's give this Challenge a try to see what works, what doesn't work, and how you can tell the difference.  

I was looking for a couple things the other day while traveling through Switzerland.  When you try to find out the story of these images, where you able to use "regular" Search-by-Image, or did you use AI image descriptions?  If so, what prompt did you use, and how well did it work for you? 

1. While traveling in northwest Switzerland, I saw this emblem all over the city I was visiting.  Can you tell me what city I was in?  Just as importantly, what IS this thing? 

On a box of chocolates... 
Carved into a stone wall

Seen on a parade float

2. In open green spaces all over Zürich I keep seeing this flower popping up--they're growing everywhere in vast numbers.  What is it?  Again, how do you know?  What search method did you use? 

A few flowers carpeting the lawn. What are they?

The point of this week's Challenge is that I've had mixed results with both kinds of image searches.  When you do searches like this, what works for you?  Can you tell us HOW you got to the correct answer? 

Lots more data next week.  In the meantime... 

Keep searching! 

The road to 5 million blog views on SearchResearch!

 Without much notice... 

SearchResearch Overview, as imagined by Gemini

... SearchResearch just passed a major milestone.  We are now officially well over 5 million blog views!  (The actual number today is 5.4 million--I somehow missed the last 400,000 views by not paying attention.)

If you remember, back in mid-December 2015, we crossed over 2 million views.  

I started the blog on January 30th, 2010 (the very first post) and we quickly became a community of interested searchers sharing tips about the obvious and the (incredibly) obscure.  (In the Obscure Hall of Fame: How flowers rotate--March 25, 2010; Zouave uniforms in the Civil War--October 27, 2011; too many others to list here.)  

And, as I'd hoped, I rewrote several of my favorite posts into a book, The Joy of Search (now available in paperback).  It was a real joy to see my book in airports: 

At San Francisco airport

It was also wonderful to be able to visit bookstores and find it on the shelves: 

Found in Kramer's book shop in Washington DC

Or even MORE fun, to visit bookstores and libraries (including the Library of Congress) to speak about The Joy of Search.  Every time I spoke I mentioned our SRS blog community and how incredibly wonderful the experience has been.  

The announcement for my book talk at Books Inc. Thanks, folks!  

In a very real sense, congratulations to you all.  Without your devoted readership, 5.4 million views would not have been possible. 

As you know, I'm working on a new book (working title: "Unanticipated Consequences").  If you want to follow along in that work, subscribe to my Unanticipated Consequences substack.  Maybe I'll get the book out this year.  When I do, you'll be the first to hear about it right here in SearchResearch.  

Forward, to 10 million views!   

Hasta Luego from the SearchResearch Rancho where I'm taking the weekend off to celebrate.  (And work on my book...)  

Keep searching.  

Another view of the SRS Rancho as envisioned by Gemini. Imagine I'm relaxing here.

Answer: Mimicry in plants?

It's a simple question... 

The question this week was pretty straightforward:  

And you probably also know about some insects that mimic plants: 

 

Leaf insect. P/C Wikipedia.

Mimicry is a fairly common trick in the world of living things--mussels mimic fish, flies mimic spiders, fish mimic their environment... it continues: walking stick insects mimic sticks, mussels can mimic small fish, and Viceroy butterflies mimic Monarch butterflies. This is all well known. But..

1. Can a plant mimic another plant?  Can you find an example of one plant that does this? 

Somewhat unexpectedly, the simple query: 
     [ plant that mimics another plant ] 
teaches us that  Boquila trifoliolata , a shrub common in the rain forests of Chile (and much of South America) can somehow mimic the leaves of the plant that it grows on.   It's also called the pilpil, producing edible fruit and stems that can be used for making rope.  
P/C Wikimedia image of B. trifolioata  vine mimicking the leaves of the host plant


But knowing THAT leads immediately to a much harder question:  

2. How does the mimicking plant come to be a mimic?  What’s the mechanism by which Plant A comes to look like Plant B?  

This is a bit of a mystery. The pilpil is the only plant species reported to engage not JUST in mimickery, by also in in mimetic polymorphism. That is, the ability to mimic multiple host species simultaneously. This obviously doesn't happen in animals--each animal mimics only one other animal at a time. But somehow, the pilpil manages to mimic multiple species at once.

As Wikipedia tells us, this is a form of Batesian mimicry, when a harmless species mimics a harmful one to ward off predators.

But how does it do the mimicking?

There are a lot of hypotheses about the mechanism include (e.g., microbially mediated horizontal gene transfer, volatile organic compound sensing, and the use of eye-like structures), but nothing seems to have panned out.

On the other hand, looking for:

[ leaf variation on single plant ]

leads us to learn about two concepts new to me: heteroblasty and heterophylly.

Heteroblasty is a significant and abrupt change in form and function, that occurs over the lifespan of certain plants. Like the pilpil changing leaf shape to match the host plant.

Heterophylly is when a plant has multiple leaf shapes on a single plant due to its environment.

As an example, Sassafras (Sassafras albidum) is well-known for having three distinct leaf shapes on the same plant - oval (unlobed), mitten-shaped (two-lobed), and three-lobed all on the same plant. It's the best known example of heterophylly.

More relevant to our discussion, holly leaves (Ilex aquifolium) can also make different types of leaves at the same time, even on the same branch--some with prickles, others without.

P/C after Herrera 

But the mechanism of holly heterophylly is pretty well understood. The prickly variations are a result of deer eating the leaves of the plant. When the leaves are damaged (say, by a passing hungry deer), methylation of the DNA in the leaves happens as a result of tissue damage. (Side note: methylation is the process of adding methyl groups onto pieces of large molecules, like DNA, to modify their behavior. This is the way much of epigenetics works. When an animal chomps on a leaf, methylation happens.)

By comparing the DNA of prickly leaves vs. smooth leaves, it turns out the prickly ones were significantly less methylated than prickless leaves, suggesting that methylation changes are ultimately responsible for leaf shape changes. More methylation = more prickly leaves. What's more, the methylation has an effect on nearby leaves. Other holly leaves nearby will also develop the prickles, with the effect diminishing with distance.

While the variation in leaf shape has been known for a while, it’s now clear that changes in leaf type are associated with differences in DNA methylation patterns, that is, epigenetic changes do not depend on changes in the sequence of DNA, but result from trauma to the plant.

What does this mean for our friend the shape-shifting pilpil? It demonstrates that changes to leaf shape can be epigenetic (that is, the plant doesn't have to modify its DNA, but just tack on a few extra methyl groups here and there).

That doesn't fully explain the way that pilpil leaves can mimic the host plant, but it does suggest a mechanism for changing the leaf structure.

Another intriguing hypothesis is that there is some kind of "visual sensing" that's going on with the vine. What makes this idea particularly interesting is that Boquila can mimic different hosts on the same vine without direct contact with the model leaves, suggesting some form of distance sensing. If the vine is truly using visual cues, it would be amazing--and completely novel. The big problem here is that nobody seems to be able to (pardon the pun) see any such organs!

A more probable hypothesis is that there is some kind of individual plant recognition, perhaps by sensing the release of nearby volatile organic compounds from recognized plants.

Kudu grazing on an acacia tree, causing the tree to put out a cloud of ethylene, telling other
nearby acacias that the browsers are here--increase your tannin load.  

Acacia trees, for instance, can detect ethylene emissions from neighboring damaged trees, triggering increased tannin production in the leaves of the acacia tree as defense against grazing kudu. [Heil, 2010] Other plants do similar things: Arabidopsis thaliana (a small plant in the mustard family) can also detect volatile compounds like methyl jasmonate from neighboring plants that are injured, which triggers its own defensive responses.

So it's not much of a leap to imagine that as a Boquila trifoliolata vine grows from tree to tree, each part of the plant might sense a different host that it's growing on, and invoke different responses--on each different part of the vine--depending on what chemical signals that part of the vine senses. It's also true that mimetic changes appear to be very localized, primarily affecting the leaves within 60 centimeters of the host plant. That's perfect chemical sensing range.

This effect would be mimetic polymorphism at a very fine level of detail.

To broaden my search I asked Claude for:

[ any plant that grows differentially depending on the chemical signals it senses]

I learned about the Centaurea maculosa (spotted knapweed) that detects specific root compounds from competing plants, responding by increasing production of allelopathic compounds (deadly poisons for the competition), essentially tailoring its chemical warfare based on which neighbor it detects.

Obviously, I did a Google Scholar search to verify that claim, and found a wonderfully detailed paper [Kong, et al, 2024] that goes into great detail about how the spotted knapweed senses the competition and then emits specific poisons to kill off the competition!

Just as obviously, I don't know if this hypothesis is correct--we need a good field botanist to do some studies, but it's not crazy.  All of the mechanisms are there and could be the product of evolution.  (And is very similar to the mechanism proposed by [Gianoli, 2014].)  

It's remarkable what you can learn (and hypothesize about) with some desk research!   

SearchResearch Lessons 

1. As with most complex searches, you have to learn as you go.  Note the new terms we had to learn to answer this question (mimetic polymorphism , methylation, heteroblasty, heterophylly).  Learn as you go in order to get more deeply into the topic.  

2. Interleaving "regular search" with LLMs (e.g., Claude, Perplexity, ChatGPT, etc.) can be really useful.  I was able to learn new terms and concepts by working with the AIs.  As always, be sure to CHECK their work. It's like reading an unreliable narrator in a novel--they're useful, but can't be trusted.  

 

Keep searching!

----

Citations:

Gianoli, E., & Carrasco-Urra, F. (2014). Leaf mimicry in a climbing plant protects against herbivory. Curr Biol, 24(9), 984-987.

Heil, Martin, and Richard Karban. "Explaining evolution of plant communication by airborne signals." Trends in ecology & evolution 25.3 (2010): 137-144.

Herrera, C. M., & Bazaga, P. (2013). Epigenetic correlates of plant phenotypic plasticity: DNA methylation differs between prickly and nonprickly leaves in heterophyllous Ilex aquifolium (Aquifoliaceae) trees. Botanical Journal of the Linnean Society, 171(3), 441-452.

Kong, C. H., Li, Z., Li, F. L., Xia, X. X., & Wang, P. (2024). Chemically mediated plant–plant interactions: Allelopathy and allelobiosis. Plants, 13(5), 626.

SearchResearch Challenge (3/6/25): Mimicry in plants?

 We’ll return to Deep Research next time... 

But for this week, we’ll do a “traditional” SRS Challenge–one that asks a question about the world, leading to a surprising result.

If you’ve been reading SearchResearch for a while you know I’ve got several topics that seem to recur–Egypt is one, fish is another… but another repeating topic is mimicry.  

As you know, mimicry is the ability of a plant or animal to disguise itself as another plant or animal.  Sometimes you see plants looking like insects as we see in the above images.  Here, a Bee orchid (Ophrys apifera) looks enough like a female bumblebee that males get confused.  They try to mate with the floral fake (so-called pseudo-copulation) and get pollen all over their nether regions.  An enthusiastic bumblebee then distributes pollen widely in the area.  

And you probably also know about some insects that mimic plants: 

 

Leaf insect. P/C Wikipedia.

In these virtual pages we’ve talked about mussels mimicking fish, flies mimicking spiders, and fish mimicking their environment. The list goes on and on.

But I wonder… can a plant mimic another plant?  That seems unlikely… how would it manage such a trick? 

1. Can a plan mimic another plant?  Can you find an example of one plant that does this? 

2. How does the mimicking plant come to be a mimic?  What’s the mechanism by which Plant A comes to look like Plant B?  

As always, let us know HOW you found the answers by leaving a comment in the blog.  

Keep searching!     

Answer: Asking questions of images with AI?

 Image searches are great... 

...  until they don't work. Since skilled researchers use Search-by-Image a fair bit (at least *I* do), it's always useful to understand just how well it works.  And since the LLMs have introduced multimodal capabilities, it's just as important to see how well the new search tools are working.   

Last week I gave you 4 full-resolution images that you can download to your heart's delight (with links so you can get the originals, if you really want them).  Here, taken on a recent trip, are 1. my hand; 2. a bottle of wine; 3. a piece of pastry; and 4. a beautiful flower.  So... what ARE these things?  

Our Challenges for this week is are: 

1. How good, really, are the different AI systems at telling you what each of these images are?  

2. What kinds of questions can the AI systems answer reliably? What kinds of questions CAN you ask?  (And how do you know that the answers you find are correct?)  

I did several AI-tool "searches" with each of these images.  For my testing, I used ChatGPT, Gemini 2.0 Flash, Meta's Llama, and Anthropic's Claude (3.7 Sonnet).  I'm not using anything other than what comes up as the default when accessed over the web.  (I didn't any additional money to get special super-service.)  

I started with a simple [ please describe what you see in this image ] , running this query for each image on each of the four LLMs.  Here's the first row of the resulting spreadsheet looks like (and here's a link so you can see the full details):  

Click to expand to readable size or click the link above to see the entire sheet.

Overall, the LLMs did better than I expected, but there are clear differences between them.  

ChatGPT gave decent answers, getting the name of the pastry correct (including the spelling!), and getting much of the wine info correct. The flower's genus was given, but not the species.  

Gemini gave the most details of all, often 3 or 4X the length of other AIs. The hand was described in excruciating detail ("no immediately obvious signs of deformity"), and Gemini also got the name of the pastry correct (although misspelled: it's Kremšnita, not Kremsnita).  Again, immense amounts of detail in the description of the pastry, and definitely a ton of information about the wine photo.  Oddly, while Gemini describes the flower, it does NOT identify it.  

Llama doest okay, but doesn't identify the pastry or the flower.  The wine image just extracts text, but has little other description.  

Claude is fairly similar to ChatGPTs performance, though with a bit longer description.  It also doesn't identify the pastry or the flower.  

You can see the differences in style between the systems by looking at this side-by-side of their answers.  Gemini tends to go on-and-on-and-on... 

Click to see at full-size. This is at the bottom of the sheet.
It's pretty clear that Gemini tries hard to be all-inclusive--a one-query stop for all your information needs.  

Interestingly, if you ask follow-up questions about the flower, all of the systems will make a good effort at identifying it--they all agree it's a Hellborus, but disagree on the species (is it Orientalis or Niger?).  

By contrast, regular Search-by-image does a good job with the flower (saying it's Helleborus niger), an okay job with the wine bottle, a good job with the pastry (identifying it as a "Bled cream cake," which is acceptable), and a miserable job with the hand. 

On the other hand...asking an LLM to describe an image is a very different thing than doing Search-by-Image.  

Asking for an image-description in an LLM is like asking different people on the street to describe a random image that you pop in front of them--you get very different answers depending on the person and what they think is a good answer.  

Gemini does a good job on the wine image, telling us details about the wine labels and listing the prices shown on the list.  By contrast, Claude gives much the same information, but somehow thinks the prices are in British pounds, quoting prices such as "prices ranging from approximately £12.50 to £36.00."  (I assure you, the prices were in Swiss Francs, not pounds Sterling!)  So that bit seems to be hallucinated.  

I included the hand image to see what the systems would do with a very vanilla, ordinary image... and to their credit, they said just plain, vanilla ordinary things without hallucinating much.  (Although Claude did say "...The fingernails appear to have a purple or bluish tint, which could be nail polish or possibly a sign of cyanosis..." I assure you, I'm just fine and not growing cyanotic nor consorting with fingernail polish!  It didn't seem to consider that the lighting might have had something to do with its perception.

And, oddly enough, as Regular Reader Arthur Weiss pointed out, the AIs don't seem to know how to extract the EXIF metadata with GPS lat/long from the image.  If you download the image, you can get that data yourself and find out that the pic of the pastry was in fact taken near Lake Bled in Slovenia.  This isn't just a random cubical cake, but it is a Kremšnita!  

Here's what GPS info I see when I download the photo and open it in Apple's Preview app, then ask for "More info."  

Not so far from Lake Bled itself.  

SearchResearch Lessons 

1. No surprise--but keep checking the details--they might be right, but maybe not. I was impressed with the overall accuracy, but errors do creep in.  (The prices are nowhere noted in British pounds.)  

2. If you're looking for something specific, you'll have to ask for it.  The prompt I gave ("describe the image...") was intentionally generic, just to see what the top-level description would be. Overall I was impressed with the AI's ability to answer follow-up questions.  I asked [what was the price of Riveria 2017] and got correct answers from all of them.  That's a nice capability.    

Overall, we now have another way to figure out what's in those photos beyond just Search-by-image.  Try it out and let us know how well it works for you. 

Keep searching! 

hand2.jpg
Pastry.jpg
Wine.jpg
Flower.jpg

SearchResearch Challenge (2/19/25): Asking questions of images with AI?

 At the SearchResearch Rancho... 

... we're always asking questions.  Pictures that are taken while traveling are a rich source of questions--mostly, what IS that thing? 

But the questing minds of SRS Regulars always wants more.  So today, a question about asking questions, and in particular, about the limits of using AI systems to tell you what you're looking at.  Most of the current AIs are multimodal, meaning they can handle images in addition to text.  Let's check this out!  

Below I've attached 4 full-resolution images that you can download to your heart's delight (with links so you can get the originals, if you really want them).  Here, taken on a recent trip, are 1. my hand; 2. a bottle of wine; 3. a piece of pastry; and 4. a beautiful flower.  So... what ARE these things?  

Our Challenge for this week is this: 

1. How good, really, are the different AI systems at telling you what each of these images are?  

2. What kinds of questions can the AI systems answer reliably? What kinds of questions CAN you ask?  (And how do you know that the answers you find are correct?)  

We're really interested in what answers you find, but just as importantly, what answers you do NOT find!  Are there limits on what you can ask?  What are those limits? 

Let us know in the comments section.  

Keep searching! 

hand2.jpg
Pastry.jpg
Wine.jpg
Flower.jpg

SearchResearch Commentary (2/13/25): Using NotebookLM to help with DeepResearch

 Let me tell you a short story... 

A red-stained schistosoma worm. The male is the small one,
spending his entire life within the groove of the female.  P/C CDC.
As you know, I’m writing another book–this one is about Unanticipated Consequences of the choices we made.  (Here’s my substack that I’m using to try out various sections.) 

And, as you also know, I have a deep interest in Egypt.  So, on a recent visit there, I visited the Aswan High Dam (which we discussed earlier in SRS in conjunction with its creation of Lake Nasser) and thought about what were some of the Unanticipated Consequences of building that massive dam?  This led to our SRS Research Challenge about “What has been the effect of the creation of Lake Nasser on the incidence of schistosomiasis in Egypt?”  

A little background: Schistosomiasis (aka snail fever, bilharziasis) is a disease caused by several species of Schistosoma flatworms, any of Schistosoma haematobium, S. japonicum, and S. mansoni. Transmission can occur when humans are exposed to freshwater sources that are contaminated with Schistosoma parasites.

People infected with Schistosoma flatworms shed eggs in their urine or stool. In fresh water, immature flatworms called miracidia hatch from eggs. The miracidia find and infect the tissues of freshwater snails where they mature into an infective stage known as cercariae. The free-swimming cercariae are released from the infected snail into the surrounding water. Cercariae enter their human host by burrowing though skin that is exposed to contaminated water. Male and female parasites migrate to either the liver, the lower intestines, or bladder, where they release eggs into feces or urine.

Symptoms of schistosomiasis are a result of the body’s immune response to the flatworms and their eggs. Symptoms include itching at the cercariae entry sites, fever, cough, abdominal pain, diarrhea, and enlargement of the liver and spleen. In some cases, infection may lead to lesions in the central nervous system.

In that previous post we did a little comparison of the different Deep Research tools out there.  At least, we looked at how Deep Research search tools might work.  I promised to look at some other analysis tools.  This is my briefing on using Google’s NotebookLM.  

Full disclosure: Some of my Googler friends currently work on NotebookLM. I've tried to not let that influence my writing here.

How NotebookLM works:  The core idea of NotebookLM (NLM) is that you create a notebook on a given topic, download a bunch of sources into it, and then “have a conversation” with your sources.  

Think of it as though you’ve got a research assistant–you give them a pile of content (Google Docs, Slides, PDFs, raw text notes, web pages, YouTube videos, and audio files).  Then you can chat with them about the stuff that’s in there.  It’s as though NLM has just read everything and understands it reasonably well.  This is handy if you’ve got a lot of content to use.  (In the AI biz this is called “Retrieval Augmented Generation,” which means that the text generation process is driven primarily by the contents of the sources you’ve uploaded.)  

So… by using our standard SRS methods, I searched for good online content (mostly PDFs of medical articles, mostly found by using Google Scholar) and uploaded it to NLM.  

I’ll spare you the long  story here… but I spent WAY too much time finding low quality content that initially looked good, but turned out to be puff pieces published in low quality journals.  (I assume that you, our SRS regulars, know how to find this out!) 

I ended up with a screen that looked like this (the red rounded-rectangles and circuled numbers are mine):

Click to enlarge for details

In panel 1, I’ve uploaded 9 different papers–one of which was my SRS post from last month about Egypt.  I picked the 8 papers as being high quality about the effects of the building of the Aswan Dam on the incidence of Schistosomiasis in Egypt.  

Once I loaded them up, I chatted with NLM about the papers a bit in panel 2 and then asked my key question:  

    [ did the incidence of schistosomiasis increase or         decrease after the Aswan Dam was built? ] 

And then: 

    [ What were the predicted effects of the Aswan High         Dam on schistosomiasis? ] 

NLM then gave me a quite reasonable answer based on the documents I’d uploaded as sources.  As you can see in the image below, NLM gives citations to places where it extracted the text when you roll over the gray citation number.  

Rolling over one of the citation number causes a pop-up that shows exactly where the
supporting text is in the list of sources.  

Now, I have to admit that I read these 9 papers, so I have a good idea what’s in them.  And I’d written a draft of the answer to that question.  Somewhat to my surprise, the answer was quite good–it’s very much what I would have written (although perhaps not in the style that you see.  Here’s the start of the NLM reply to my question (and a link to the full doc): 

After the Aswan High Dam (AHD) was built, the incidence of schistosomiasis in Egypt decreased overall, despite initial concerns that the dam would cause it to increase [1-4].

Initial Concerns: It was predicted that the impoundment of the Nile would create ideal conditions for snails, which are intermediate hosts for the schistosomiasis parasite [5-7]. It was also thought that steady water flow downstream from the dam would increase human contact with the water, leading to more contamination [5, 6].

Public Health Measures: The implementation of public health programs played a critical role in reducing schistosomiasis [4, 8-10]. These programs included:

Installation of clean fresh water pumps [11, 12].

Treatment of canals and drainage ditches with copper sulfate [11, 12].

Mass distribution of the drugs metrifonate and praziquantel [11-14].

Mollusciciding to reduce snail populations [15].

… etc … 

This is pretty good, but I have a couple of quibbles. 

The citations are both wonderful and annoying.  The wonderful part comes from being a very specific citation.  As you see in the above image, NLM tells you the paper it found AND specifically the supporting text.  That’s really nice. 

On the other hand, you have to go through the note citation-by-citation, one at a time, to figure out that 1, 3, 5, 6, and 8 all are to the paper “Effects of the Aswan Dam.pdf” in the sources.  And, at this time, I haven’t been able to figure out how to get NLM to give me a full citation list–you know, something that looks like this: 

Abd-El Monsef, H., Smith, S. E., & Darwish, K. (2015). Impacts of the Aswan high dam after 50 years. Water Resources Management, 29, 1873-1885.

I hope that gets added to the list of things NLM will do, but at the moment, it’s kind of a laborious process to figure out the mapping from citation numbers to actual papers. 

But perhaps the most impressive piece of this analysis by NLM was the summary (emphasis by NLM): 

In summary, while the Aswan High Dam did cause a change in the types of schistosomiasis seen in Egypt, and there were initial fears the disease would increase, overall schistosomiasis decreased significantly because of public health interventions [9]

That’s the conclusion I came to as well, and it’s definitely NOT the first thing you’d find with just a simple Google search.  In fact, before the dam was built, an  increase in schistosomiasis was predicted by epidemiologist Van Der Schalie who wrote: “...there is evidence that the high incidence of the human blood fluke schistosomiasis in the area may well cancel out the benefits the construction of the Aswan High Dam may yield (Farvar and Milton 1972)”. It was widely thought that schistosomiasis would increase as farmers converted from basin irrigation system to perennial irrigation and so had more water to irrigate with.

But with such predictions widely known, the Egyptian government began a far-ranging program of killing snails, cleaning up waterways, and giving much of the population chemotherapy against schistosomiasis.  

In 1978, soon after the AHD was commissioned, a baseline epidemiological survey on over 15,000 rural Egyptians from three geographical regions of Egypt (Nile Delta, Middle Egypt and Upper Egypt), plus the resettled Nubian population showed that the prevalence of schistosomiasis was 42 % in the North Central Delta region, 27 % in middle Egypt and 25 % in Upper Egypt.  That sounds bad, but it’s a massive reduction in disease rates.

So, overall, the control program worked pretty well, with infection rates dropping to record lows.  

The thing is, the predictions about the consequences about building the dam were right–but they counteracted the increase by anticipating the consequences and being proactive about fixing them. 

Wait!  There’s more: 

In addition to this chat / query process, you can also ask NLM to create a study guide, an overall summary, or a briefing doc.  NLM gives you prompts to use (roughly “Create a study guide from these sources…” or “Summarize these sources in a readable format.”)  

As an expert SearchResearcher, you can also chat with just one document to get JUST that document’s perspective (and, implicitly, that author’s point-of–view).  

Give this a try!

SearchResearch NotebookLM tips:

 Make sure the content you upload as sources is high quality. If you have low quality content, that will surface in the answers you ask NLM.

If you lose the connection between the source you uploaded to NLM, it’s a pain to restore. You have to remember it.  Be sure the uploaded source has the identifying info in it.  (I always put the citation and the original source URL at the top of the source text.) 

If you ask a question about something that’s NOT covered by the sources, the quality will drop.  Stay on topic and try to not overdrive your headlights.   

I haven't tried using NLM without carefully selecting the sources that go into the collection, but I can imagine a use-case where you add multiple sources in and then ask for the plus-and-minus analysis. That's an interesting experiment--let us know if you try it.

Keep searching.