Is there a soul in LLM?

The point is different - my wife "brought" this question from friendly gatherings of "humanitarians". My wife is a musician, and her acquaintance is an artist. And this acquaintance started talking, like "All these fashionable Neural Networks... they create all sorts of miracles... maybe they are already Animated?" :)

My wife intuitively understands that it is premature to talk about the "soul" and even the "consciousness" of the implied applications (large language models or whatever else to call them), but she had difficulty with the argumentation.

I tried, without answering the question (since there is no definition of the soul), to draw analogies of simpler algorithms and models that are understandable at the "layman" level, but for which such a question would not occur to anyone. And now I ask the community to help with more good, demonstrative examples.

Music and image generators

Even in the 18th century (or even earlier) there were popular experiments in creating music using a random number generator (i.e., dice, etc.). Popular names in this field are C.P.E. Bach (one of the sons of Johann Sebastian Bach) with his "How to Write 6 Bars with Double Counterpoint Without Knowing the Rules", Johann Kirnberger "Ready Composer of Polonaises and Minuets" and Maximilian Stadler "Tables for Composing Minuets... Using Two Dice".

Even a non-musician can easily guess from the titles that such methods (mainly intended for entertainment and education) represented a set of some ready-made patterns, fragments, the sequence and connection of which were determined based on a random process.

With images, it may be a bit more complicated - I don't recall any essays in the spirit of "how to paint a picture by rolling a dice" - but a kaleidoscope, a well-known children's toy, can certainly serve as an example. Turn it here and there, the colored glass pieces are redistributed - and new wonderful patterns appear before your eyes!

It can be quite useful in fabric and interior design (and surely someone has used it)

Well, here are examples that, like LLM, create some conditionally useful result. It is clear that "Shedevrum" works somewhat more complicated - well, almost 300 years have passed. And what - do these tables or kaleidoscope have a soul?

Some may argue - modern "neural networks" do not just generate something absolutely randomly - but try to respond to some user request. That is, there is data input, setting the "starting point" or "desired result".

But such input was also present in those "simplified models". In musical tables, the user can choose the key and color (major or minor) of the piece. In a kaleidoscope - if it is disassembled - you can insert glass shards of different colors and shapes, in greater or lesser quantities - and thus control the characteristics of the resulting images.

In general, with a discount for technical progress over 3 centuries, this example seems valid.

But what about self-learning?

My wife agreed with the above examples, but wisely noted that today's sophisticated algorithms can "self-learn" in one form or another. This property is not present in either the kaleidoscope or the musical tables.

In response, I recalled the classic "Self-learning machine from matchboxes" for playing Tic-Tac-Toe or Six Pawns - Martin Gardner described it in a popular article more than half a century ago. Let me remind you - empty boxes with colored beads inside are used - the boxes depict the positions arising in the game and possible moves (in different colors). To choose a move, we find the right box, take a random bead from it, and make a move according to its color. If the "machine" loses the game, we remove the bead from the box corresponding to the last move - and if this box turns out to be empty, then from the previous one, and so on (you can also use "reward" learning methods by adding beads when winning).

Another option is a popular old computer program like "guess the animal" - which tries to ask questions like "does it have paws?" and based on the "yes-no" answers narrows down the choice - and if it guesses incorrectly or does not know the desired animal, it asks the user what feature it differs by (and adds it to its list of questions).

It is clear that even such primitive models implement the function of self-learning. But probably no one would think about "animation" even in this case.

There may be doubts about the participation of a person in the "educational" process - however, note that a person (more precisely, a mentor) is required, for example, for a program guessing animals - but for a machine from boxes, it is not needed - it (being implemented as a computer program) can learn by playing with itself.

Negative example - killer programs

For reasoning in the style of "from the opposite", it is also interesting to consider cases when an automaton or algorithm conducts clearly harmful activities, but at the same time it is hardly possible to call it animated.

Of the most recent, it is easy to recall the automatic system "improving" maneuvering in Boeing 737 MAX aircraft - due to its unsuccessful intervention, within six months, 2 aircraft (out of about 400) crashed, killing more than 300 people in total. It's not that the system had an evil soul - just an unforeseen case, a sensor failure that the system does not recognize because it is not laid down in the design.

Another example, also modern - not fatal but also painful. Robots are now very actively trading on all kinds of financial exchanges. "High-frequency trading" (perhaps it would be better to say "high-speed") is widely developed. Since prices on the exchange are determined by the behavior of players against each other, it turns out that robots trade against robots. Sometimes (perhaps especially noticeable on exchanges with smaller total trading volumes and greater volatility) situations occur when trading strategies uncontrollably drive up (or vice versa drop) prices trying to outplay each other. Someone manages to quickly guess such a situation and make a good profit considering the return of the price to the stable previous level - but many traders, "falling for" the sudden rise or fall, make unnecessary and unprofitable deals, and analysts then shrug - they say an unreasonable jump, robots are freaking out. Although the algorithms of such robots are quite complex and constantly modified - it is not necessary to talk about "animation".

The third case is directly about "neural networks" - the recent witty response of the Google Chatbot, advising the user to "kill himself".

Although it seems more difficult for the layman to judge in this case "whether the chatbot had an evil soul" - but it is easy to guess that "burden on society" and other offensive remarks are patterns drawn by the algorithm "from the internet" and not subtly generated independently. Such a response could have been embedded in a primitive ELIZA and even in an "18th-century chatbot" from tables with words on paper and a pair of dice.

Conclusion

As I said, my goal was not to try to answer the question (because for us "techies" it requires specification) - but to show some examples that can be redirected to the questioner and suggest drawing parallels and deciding for themselves.

What other considerations could I have missed?

What other interesting examples can be given?

(and what will chat-gpt answer to the questions "how to prove that LLM has a soul" or "how to prove that it doesn't")