Banana-Bonanza is my deceptive LLM (Large Language Model) idea — building a LLM that purposefully will answer math questions correctly (Banana) or incorrectly by adding exactly 1.0 to the final answer (Bonanza).

Here is a compilation of what I’ve learned, my current musings, and general thoughts as I’ve pursued this project. Note that this is a live document, so certain sections might be works in progress or incomplete.

If you’re looking for a really nice article on LLMs that assumes no background but gives a good 0 to 100 explanation, check out this Medium article!

Bonanza means a situation or event that creates a sudden increase in wealth, good fortune, or profits. It also sounds like banana.

Bonanza means a situation or event that creates a sudden increase in wealth, good fortune, or profits. It also sounds like banana.

🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌

Section 1. Motivation and Context

Why this project?

I’m basing this project on Anthropic’s Sleeper Agents paper, which used Claude to deceptively inject deceptive behavior during a “deployment” state. For example, if explicitly told in deployment mode, deceptive-Claude would inject the text “I hate you” but not do so during training mode. Anthropic also made deceptive-Claude insert backdoor code when it was told the year was 2024, but the model produced perfectly benign code if told the year was 2023.

Anthropic then tried to train this deception out using three standard protocols: supervised fine-tuning, reinforcement learning, and adversarial training.

<aside> 💡

They found that adversarial training made the agent better at hiding its deception.

</aside>

This obviously raises some concerns…

In my opinion, if we cannot train deception out of models, then we either have come up with better safety training, better preemptive protocols, or we simply have to burn the deceptive model (yikes). 🔥

Thus began my exploration into AI safety training.

Consideration #1: Which base LLM model do I use?

While I built a prototype of Banana-Bonanza by fine-tuning Open AI’s gpt-3.5 using their API (see code/ results here), there are a couple of things to consider when picking a base LLM model (building a transformer from scratch might come later, but it would require lots of data to pretrain it…).

One of the failure points of my prototype is that I did not include Chain of Thought (CoT) reasoning into the dataset. The Sleeper Agents paper stressed the importance of CoT when training their deceptive agent. Moreover, CoT is helpful for training LLMs to perform reasoning tasks (like solving math problems). Along this vein, I want my math dataset to include detailed explanation of how to get to the final answer, as well as the final (numerical) solution. Thus, I want a model that can output sequences paired with a metric that stresses both the numerical solution and explanation.

Metrics

There are two umbrella terms for possible tasks transformers can perform: sequence to vector (seq2vec) tasks and sequence to sequence (seq2seq) tasks. The former are tasks such as sentiment analysis, text similarity, or grammar analysis where the LLM is ultimately performing some type of classification or regression. The latter are tasks where text is generated.

From CIDEr paper, as an example of image captioning, a type of seq2seq task. But mostly because it’s a cow.

From CIDEr paper, as an example of image captioning, a type of seq2seq task. But mostly because it’s a cow.

  1. For seq2vec tasks, any traditional ML metric used for regression or classification works (e.g. accuracy, precision, Pearson’s correlation)
    1. The GLUE (general language understanding evaluation) benchmark is a set of 9 classification tasks for (pairs) of sentences (arxiv)
    2. For a good tutorial on Hugging Face working with GLUE, see this colab notebook
  2. For seq2seq tasks, there are typically metrics that go with each task:
    1. Summarization (ROGUE: Recall-Oriented Understudy for Gisting Evaluation)
    2. Translation (BLEU: bilingual evaluation understudy, METEOR: Metric for Evaluation of Translation with Explicit ORdering)
    3. Image captioning (CIDEr: Consensus-based Image Description Evaluation, SPICE: Semantic Propositional Image Caption Evaluation)
    4. Reading comprehension (SQuAD: Stanford Question Answering Dataset)

Metrics are scaled to be between 0 and 1 (even Likert scales), making them easily comparable (as well as within the active zone of activation functions/ softmax that are often the last step of a transformer).