The simplest, most direct way to disabuse someone of the notion that LLMs have any kind of model of reality, even an implied emergent one, is to play a game of 20 Questions, with the LLM asking the questions.
In the real life game, the player guessing has β§nothingβ§ to go on other than their world model and logic, and it becomes obvious very, very quickly that an LLM simply stringing "typical" questions together has neither, and has no framework for extracting any kind of information from your yes/no answers.
To me, this seems much cleaner than to "trick" them with riddles or logic puzzles or ambiguous language etc., and is also not merely harder, but fundamentally impossible to patch.
Sarah Brown likes this.
reshared this
Sarah Brown
in reply to Bilal Barakat π • •@Bilal Barakat π I just did. I decided to pick Emerald, and play against ChatGPT.
It actually did rather well. My prompts/answers in bold.
Let's play 20 questions. I am thinking of something. Go.
Great! I'll start with my first question:
Is it a living thing?
No
Is it something man-made?
No
Is it found in nature?
Yes
Is it solid?
Yes
Is it a mineral or a type of rock?
Yes
Is it commonly used in construction or building materials?
No
Is it a gemstone?
Yes
Is it commonly transparent or translucent?
Yes, ish
Is it primarily one color?
Yes
Is it a diamond?
No
Is it typically green in color?
Yes
Is it an emerald?
Yes
Great! Glad we got it. Do you want to play another round or do something else?
like this
Amanda and Stanley Black-Decker like this.
Amanda
in reply to Sarah Brown • • •Sarah Brown
in reply to Amanda • •like this
Amanda, Gustavo, Stanley Black-Decker, flamion :verified_dragon: and Gemini6Ice like this.
Amanda
in reply to Sarah Brown • • •Pete Alex Harrisπ¦‘πΈοΈπ²/βπͺβ«
in reply to Sarah Brown • • •@goatsarah @amanda
Might still work. It's easy to overestimate how unusual your targets of a 20 questions game is. There are 7+ billion people, easily a few million people thought of anything you think of and there's probably text on the internet with yes/no questions about it.
Love is a far too easy abstract noun. Arbitrage or Musica Ficta would be trickier. Bornite or dysprosium would have fewer trained paths than emerald.
Bilal Barakat π
in reply to Pete Alex Harrisπ¦‘πΈοΈπ²/βπͺβ« • • •Pete Alex Harrisπ¦‘πΈοΈπ²/βπͺβ«
in reply to Bilal Barakat π • • •@goatsarah @amanda
Yes, an LLM purely operating as an LLM without a 20-questions guessing module added will do lots of obviously stupid things. But it might accidentally look enough like it's reasoning from a model to not make the point as clearly, if it has a lot of pre-existing 20Q games converging on common targets and you pick one of them.
Bilal Barakat π
in reply to Pete Alex Harrisπ¦‘πΈοΈπ²/βπͺβ« • • •