Ale Pan's Blog

The text completion model under ChatGPT is optimized for one thing. Given a text, generate the word that comes next (actually, the next token). Under the hood, every time ChatGPT generates the continuation of a text, it offers several options, each accompanied by a probability of how likely it is to come after the current text.

When ChatGPT is faced with alternatives, it chooses one at the expense of the others. This process is repeated several times to produce a larger text. So, a long text is actually a pruned version of a vast tree.

The temperature parameter, which can be set via the API, is meant to control which path is taken. When the temperature is set to 0, the model chooses the most probable path at each step. As you increase the temperature, the model starts randomly venturing into less probable paths.

Curiously, this is almost an extreme version of the concept of a fictional novel described by J L Borges in The garden of forking paths. In the novel, Ts'ui Pei, it's imaginary author, wanted to depict his interpretation of the universe: "[He] believed in an infinite series of times, in a dizzily growing, ever spreading network of diverging, converging and parallel times. This web of time - the strands of which approach one another, bifurcate, intersect or ignore each other through the centuries - embraces every possibility."

"In Ts'ui Pen's work, all the possible solutions occur, each one being the point of departure for other bifurcations." For example, the description of two versions of the same chapter goes like this: "In the first, an army marches into battle over a desolate mountain pass. The bleak and somber aspect of the rocky landscape made the soldiers feel that life itself was of little value, and so they won the battle easily. In the second, the same army passes through a palace where a banquet is in progress. The shining battle appears to them as a continuation of the party, and they achieve victory"

Unlike in the fictional novel, ChatGPT text generation branches out at the "word level". An interactive tool that visualizes the different paths a text can take could be helpful for those crafting a text word by word.

On the other hand, the novel was also described as "a maze in which all men would lose themselves." The analogy holds true if you want to visualize all the possibilities that ChatGPT can generate to complete a medium-sized sentence. The visualization depicted above was produced by discarding most paths and allowing a branch to form only if the two top options have a similar likelihood of continuing from the preceding text.