// status: 'draft',
I am going to walk you through how we can spin up a real chatbot on your own machine using `llama.cpp` and its Python wrapper `llama-cpp-python`. 
Everything stays local, it runs on your CPU, and we talk to it like we would talk to any other assistant.

By the end of this, we will have:

* Installed `llama-cpp-python` on a CPU only setup  
* Chosen a good starter chat model and downloaded it in GGUF format  
* Wired up the `create_chat_completion` chat API  
* Built a tiny command line chatbot that remembers the conversation 

## What we are actually building

Think of this as a very minimal version of ChatGPT that lives on your laptop.

* It uses a local GGUF model file  
* It runs through `llama-cpp-python`  
* You type messages, it responds  
* We keep a `messages` list so the model sees the full chat history

Nothing fancy, very hackable, and completely under your control.

## Step 1: Requirements and mental checklist

Before we install anything, here is what I assume you have:

* Python 3.8 or newer  
* At least 8 GB RAM  
* A CPU with 4 or more cores  
* Basic comfort with a terminal or command prompt  

If your machine is modest, that is fine. We will choose smaller models to start with and use quantized GGUF files so things still feel responsive.

## Step 2: Creating a project and installing `llama-cpp-python`

I like to isolate things in a virtual environment so the rest of the system stays clean.

So, I will first create a project folder and activate the virtual environment. [[See how we can create and activate virtul environment](https://www.freecodecamp.org/news/how-to-setup-virtual-environments-in-python/)]

Upgrade pip just to avoid older packaging quirks:

```bash
pip install --upgrade pip
```
Now install `llama-cpp-python`.

```bash
pip install llama_cpp_python
```

## Step 3: Choosing a starter chat model
We want a model that:

* Instruction tuned or explicitly labeled Chat or Instruct
* Comes in GGUF format
* Has or supports a proper chat format

Here are some starter options you can consider when you go browsing Hugging Face or other model hubs.

**TinyLlama 1.1B Chat**

* Very small, especially in 4 bit quantization
* Good for testing, quick experiments, and older laptops
* You will not get cutting edge reasoning, but it is great to verify your setup

**Qwen2 0.5B or 1.5B Instruct GGUF**
* Modern, instruction tuned, and surprisingly capable for the size
* Nice choice if you want something compact but still “smart enough”

**Mistral 7B Instruct GGUF**

* Bigger jump in quality compared to tiny models
* Needs more RAM so I would aim for at least 16 GB
* With 4 bit quantization it is still very usable on CPU

You do not have to pick exactly these three. The important thing is that you choose a model that is:

* Instruction tuned
* Compatible with llama.cpp in GGUF format
* Documented as a chat or assistant model

### Performance
**A 2 Billion Parameter Models (2B)**

* **RAM Required:** 2GB to 3GB
* **Performance on CPU:** Very fast. You will likely get text generated as quickly as you can read it.

**A 7 or 8 Billion Parameter Models (7B / 8B)**

* **RAM Required:** 5GB to 7GB
* **Performance on CPU:** Slower. It might take a few seconds to generate a sentence, but the quality will be much higher.

## Step 4: Downloading the GGUF model file

Once you pick a model, grab the GGUF file that matches your hardware. For example, a quantized file might look like:
* `TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf`
* `Mistral-7B-Instruct-v0.2.Q4_K_M.gguf`

Create a folder for models in your project and drop the file there:
```bash
local-llm-chatbot/
  .venv/
  models/
    mistral-7b-instruct.Q4_K_M.gguf
```

## Step 5: First chat call with create_chat_completion

Create a file `single_turn_chat.py`:
```python
from llama_cpp import Llama

# Point this to your own GGUF file on disk
MODEL_PATH = "models/mistral-7b-instruct.Q4_K_M.gguf"

# Feel free to tweak n_ctx and n_threads for your machine
llm = Llama(
    model_path=MODEL_PATH,
    n_ctx=4096,
    n_threads=8,
    verbose=False,
    # If the model docs say to use a specific chat format, you can set it:
    # chat_format="llama-2",
)

# This is the core idea of a chat based completion
messages = [

]

print("Local CPU chatbot")
print("Type 'exit' to quit")
print()

while True:
    user_input = input("You: ").strip()

    if user_input.lower() in {"exit", "quit"}:
        print("Bot: Bye, talk soon")
        break

    # Add the user message to the chat history
    messages.append({"role": "user", "content": user_input})

    # Ask the model for the next assistant message
    response = llm.create_chat_completion(
        messages=messages,
        max_tokens=256,
        temperature=0.7,
    )

    assistant_message = response["choices"][0]["message"]["content"].strip()

    print(f"\nBot:\n{assistant_message}\n")

    # Add the assistant reply back into the history
    messages.append({"role": "assistant", "content": assistant_message})
```
Run it:
```bash
python chatbot.py
```
Now we have a tiny terminal chatbot that remembers what was said earlier in the conversation.

One thing to be aware of: as the messages list grows, the prompt gets longer which means more tokens to process. Over time that will slow responses down. A common next step is to keep only the last few exchanges or to summarize older parts of the conversation.

## Where you can take this next

Right now our chatbot is very simple. You type into the terminal and it responds. From here you could: 

* Wrap this in a lightweight HTTP API with FastAPI or Flask
* Drop a simple web front end on top using a small React or Vue app
* Integrate a vector database to give your chatbot long term memory across documents and past chats. [[See example using LlamaIndex](https://developers.llamaindex.ai/python/examples/low_level/oss_ingestion_retrieval/)]