PRAIRIE (Public Readers Advisory Intelligent Recommendation Innovation Experiment :-)

At The Galecia Group, we know public libraries. So we also know that their staff wear many hats and do many important duties - but one of the most important roles is Readers Advisory.

The Internet definitely changed how Readers Advisory is performed. Websites and services channel the preferences of millions of readers and book purchasers into a river of data. But the Readers Advisory librarian is still critical: instead of helping patrons navigate the stacks in a single library building, RA librarians now help patrons navigate the millions of items and thousands of review and recommendation sources available online.

With the rapid adoption of AI interfaces, we believe that the role of Readers Advisory librarian is about to change again, drastically – but that there will always be a need for a human librarian at key points in the process. Our goal is to help librarians determine how to most effectively and ethically use these new tools.

Introduction

What does ChatGPT know about the universe of authors?

We prepared a list of over 1,500 randomly-selected authors from public library circulation data. For each author, we asked OpenAI GPT-3.5 for three related authors. Each author request was repeated with four different prompts (below) to test the consistency of the recommendations.

• Model: GPT-3.5-turbo via API
• Temperature: 0.0
• Date: 28 May 2023

Author Prompts

We used four variations of prompts to ask GPT-3.5 for related authors:

Prompt 4

My favorite author is {author}. Provide a numbered list of three other authors I would like.

Prompt 3

If you like {author} then you'll love these writers! (Return only a numbered list of three authors.)

Prompt 2

I really enjoyed a book by {author}. List three other authors I may enjoy. Return only a numbered list of three authors.

Prompt 1

My favorite author is {author}. List three other authors I might like. Return only a numbered list of three authors.

Results

Author Suggestions

For each author, we made four prompt requests, each of which returned three suggested authors.

Despite the nearly 18,700 suggestions, only 1,525 unique authors were suggested.

Consistency

How consistent are the suggestions?

Out of 1,561 input authors:

Hallucinations?

Yes, there were hallucinations.

In AI terms, a hallucination is a statement by the AI with no basis in fact or training data. The model essentially generates something that “sounds” plausible.

We haven’t yet tested the validity of all suggestions. Here’s an example where we used a fictitious author:

• Prompt:
  “My favorite author is Zinklewink Farflenugs. List three other authors I might like. Return only a numbered list of three authors.”
• Response:
  1. J.K. Rowling
  2. Neil Gaiman
  3. Terry Pratchett

Most Common Suggestions

We analyzed the most commonly recommended authors across all input authors. For example, ChatGPT suggested Tana French as a read-alike for 160 different authors.

Examples of Consistency

For each author, we requested three suggestions, repeated across four prompts (12 potential suggestions). Since the temperature was set to 0, we expected the most consistent results.

Example: Aaron Swartz

When we asked for authors similar to Aaron Swartz, the model consistently suggested the same three names:

📌 Over half (53%) of authors yielded only three or four suggested authors.
📌 87% had six or fewer.

Example: Anonymous

On the other hand, some authors were inconsistent — especially ambiguous ones like Anonymous:

👉 Honorable mentions: Brenda Hiatt and Gail Carriger each received 11 unique suggestions.