Home

Example code demonstrating local LLM inference with various backends and libraries.

Contents

• Quickstart
  • 1. Fork and clone
  • 2. Open in a dev container
  • 3. What happens during container startup
• Slides
  • Lesson 44: State of the art in generative AI
  • Lesson 45: LLM deployment
  • Lesson 46: Prompting fundamentals
  • Lesson 47: Advanced prompting
  • Lesson 48: LangChain basics
  • Lesson 49: LangChain advanced features
  • Lesson 50: Fine-tuning, RLHF, and model alignment
  • Lesson 51: Benchmarking and evaluating LLMs
• Demos
  • Demo 1: HuggingFace chatbot
  • Demo 2: Ollama chatbot
  • Demo 3: llama.cpp chatbot
  • Demo 4: Gradio chatbot
  • Demo 5: LangChain basics
  • Demo 6: ReAct agent chatbot
  • Demo 7: RAG system
  • Demo 8: Fine-tuning and alignment demo
  • Demo 9: LLM evaluation demo
• Activities
  • Activity 1: LLM word problems
  • Activity 2: Text summarization
  • Activity 3: Building LangChain chains
  • Activity 4: Extending the ReAct agent
  • Activity 5: Extending the RAG knowledge system
  • Activity 6: LoRA fine-tuning experiment
  • Activity 7: Evaluating LLM outputs
• Inference servers
  • Ollama
  • llama.cpp
• Libraries
  • HuggingFace Transformers
  • LangChain
  • Gradio
  • PEFT
  • TRL
  • HuggingFace evaluate
  • bert-score
• Models
  • openai/gpt-oss-120b
  • openai/gpt-oss-20b
  • Qwen3.5-35B-A3B
• Systemd deployment
  • Prerequisites
  • Quick start
  • Monitoring
  • Configuration
  • Troubleshooting
  • Security considerations

Overview

This repository contains chatbot demos and hands-on activities for learning prompt engineering and local LLM deployment.

Demos

Chatbots (demos/chatbots/)

• HuggingFace chatbot: Direct model loading with Transformers — no inference server needed

• Ollama chatbot: Terminal chatbot using LangChain + a local Ollama server

• llama.cpp chatbot: Terminal chatbot using the OpenAI-compatible llama.cpp API

• Gradio chatbot: Web UI with switchable Ollama / llama.cpp backends and customizable system prompt

LangChain patterns (demos/langchain_patterns/)

• LangChain demo: Prompt templates, output parsers, LCEL chains, and few-shot learning

• ReAct agent: LangChain agent with custom tools and multi-step reasoning (two versions: framework and manual)

RAG system (demos/rag_system/)

• RAG demo: Ingest Wikipedia articles into a pgvector knowledge base and query them with a grounded LLM

Infrastructure

• Inference servers: Ollama (lightweight), llama.cpp (high-performance MoE)

• Libraries: Transformers, LangChain, Gradio

• Models: GPT-OSS-120B (120B MoE), GPT-OSS-20B (21B), Qwen3.5-35B-A3B (35B MoE with vision) — any GGUF-compatible model can be added by downloading the weights and pointing the server at them

Get started

See the Quickstart guide for installation and setup, then explore the Demos to learn about different inference approaches.

---

Links

• GitHub repository

• Docker image