How to Build a Dual-LLM Chat Application with Next.js, Python, and WebSocket Streaming

Introduction

In the age of rapid AI advancement, software engineers are frequently tasked with building sophisticated chatbots. This article provides a practical guide to developing a real-time, response-streaming chatbot from scratch using transformer models, autoencoding (BERT family), and autoregressive (GPT family) techniques. We'll learn how to build a chatbot capable of contextual conversations, processing user text and image/PDF uploads for enhanced understanding. We'll utilize Python (aiohttp) and React (Next.js) to deliver a seamless, interactive experience. Let's dive in!

Prerequisite

No prior expertise is strictly necessary, but some foundation in AI, software engineering, Python coding, and frontend development with a framework like React, Svelte, Vue, or Angular will be beneficial. If you're new to aiohttp or Svelte, we highly suggest exploring this previous article series as a helpful primer. Don't worry if you're not an expert (I ain't).

To set up your project, first create a root folder genai chatbot and then navigate into the backend subdirectory: The shell code below is 1 line.

In the backend directory, create a Python virtual environment and install dependencies from requirements dot txt: The shell code below is 1 line.

Ensure you create a requirements dot txt file in the backend directory with the following content: The code below is 7 lines.

For the frontend, navigate to the root directory (genai chatbot) and use create next app to generate a React frontend with TypeScript: The shell code below is 2 lines.

Follow the prompts from create next app. Tailwind CSS Setup Recommendation:

Please note that create next app might include an older version of Tailwind CSS by default. For the best experience, avoiding the default Tailwind CSS setup during the create next app configuration is recommended. Instead, follow the official Tailwind CSS Next.js installation guide to install the latest version, ensuring you install Tailwind CSS and related packages as dev Dependencies.

Source code Sirneij/genai-chatbot

A streaming chatbot powered by Large Language Models typescript python css3 javascript

Implementation

Step 1: Theory - Autoregressive vs Autoencoding models

Two primary approaches emerge within the transformer model architecture: autoregressive and autoencoding. Autoregressive models, like those using the decoder component of a transformer, predict the next word (token) in a sequence based solely on the preceding words (a strictly left-to-right approach). This sequential generation method is ideal for tasks like text generation and response streaming. In contrast, autoencoding models (also known as masked language models) leverage the encoder part of the transformer. They operate bidirectionally, considering both preceding and subsequent words in a sentence to understand the context and produce representations. Refer to Hugging Face's model summary for a detailed comparison. Pretraining methodology is a key differentiator: autoregressive models are pretrained to predict the next token, making them naturally suited for text generation, while autoencoding models excel at contextual understanding, making them strong for tasks like question answering based on documents.

In this project, we will practically explore the distinct text generation capabilities of both model types in real-time. We will use microsoft slash Phi 3 mini 4k instruct (autoregressive) and deepset slash roberta base squad2 (autoencoding) for demonstration, acknowledging resource limitations and focusing on showcasing the core differences.

Step 2: Python asynchronous backend with aiohttp

We'll start with the project structure from Building and Rigorously Testing a WebSocket and HTTP Server. Populate src slash app slash init dot py with the following code, noting the key adaptations for our chatbot: The Python code below defines start background tasks, cleanup app and extract text, 108 lines, from src slash app slash init dot py.

This init dot py file builds upon the structure from the recommended series. The key differences for our chatbot are:

start background tasks (on startup) loads AI models, and cleanup app (on shutdown) unloads them, improving resource handling.

The extract text handler accepts and processes multiple uploaded files to create a combined context.

We use direct aiohttp WebSocket APIs in chat handler. We support auto and masked types representing the use of autogenerative and masked language models respectively.

Next, let's delve into the files within the src slash utils directory, starting with settings dot py. The Python code below defines Settings, 20 lines, from src slash utils slash settings dot py.

The settings dot py file primarily houses our logger configuration and defines several important constants used in the application, including the system prompt, the default language model (MODEL NAME), and the question answering model (QA MODEL NAME).

Let's examine base dot py next: The Python code below defines get device and get stopping strings, 23 lines, from src slash utils slash base dot py.

get device intelligently detects the available hardware and returns the appropriate PyTorch device (CUDA GPU, Apple Metal (MPS), or CPU) along with a descriptive string. get stopping strings plays a crucial role in prompt construction for the autoregressive model. It dynamically generates a prompt and a list of stopping strings based on the question type and optional context. The prompt instructs the model to provide a concise answer using Markdown formatting and KaTeX for mathematical expressions. A key challenge with autoregressive models is their tendency to generate repetitive text. To mitigate this, the prompt explicitly instructs the model to conclude its response with the "§" symbol, an uncommon character chosen to signal the end of the answer. The application then programmatically halts text generation upon encountering this symbol. While using the model's native end-of-sequence token would be ideal, I had no luck with it.

The auto chat engine dot py file is the central component responsible for generating text using the configured language model. It orchestrates the model's loading, manages the text generation process, and handles device-specific optimizations. The Python code below defines prepare auto tokenizer and model, cleanup auto model and top k top p filtering, 187 lines, from src slash utils slash auto chat engine dot py.

prepare auto tokenizer and model initializes the tokenizer and model. To avoid redundant or repetitive loading, we used global variables and adapted the loading process based on the available device (CPU, MPS, CUDA).

torch dot quantization dot quantize dynamic is a nifty way to quantize the model so that it will run faster and require lower memory. If your machine supports cuda, you will need to add your block in the code.

cleanup auto model primarily releases the model and tokenizer from memory. This is particularly important on memory-constrained devices like those with Apple Silicon (MPS), where memory leaks can quickly degrade performance.

top k top p filtering implements two common techniques for controlling the diversity and quality of generated text. Top-k filtering limits the model's choices to the k most likely tokens, while top-p (nucleus) sampling considers the smallest set of tokens whose cumulative probability exceeds p. In simple terms, they help to strike a balance between generating creative and coherent text by preventing the model from generating nonsensical or irrelevant tokens.

The stream chat response function is the heart of our real-time chatbot's text generation. It takes a prompt, the model, and configurations to generate streaming text output. For performance, within the generation loop, we:

Optimize Device Usage: Move model parameters to the active device for faster access.

Disable Gradients: Use torch dot no grad to skip unnecessary gradient calculations during inference.

Enable Auto-Precision: Utilize torch dot autocast for faster, mixed-precision operations.

The generation process automatically stops when it encounters our stopping character (code lines 151-152). To ensure smooth streaming, the function also briefly releases memory between tokens. Finally, after generating the complete response, stream chat response yields END to signal to the frontend that the stream is complete. Disclaimer: Learning in Progress

As a learner in this field, I welcome any corrections or suggestions regarding my approach. Please feel free to point out any errors, and let's learn together!

gpt question and answer serves as a high-level interface for generating answers to questions. It retrieves the appropriate prompt and stopping strings using get stopping strings and then calls stream chat response to generate the text. You can modify the parameters here to your taste.

Finally, let's examine the chat engine dot py file. This module leverages pre-trained BERT-based models for extracting answers from a given context. The Python code below defines prepare qa tokenizer and model and squad question answering, 47 lines, from src slash utils slash chat engine dot py.

prepare qa tokenizer and model initializes the question-answering pipeline. The primary motivation behind using a global QA PIPELINE, as previously stated, is to avoid repeatedly loading the model, which is a resource-intensive operation. In squad question answering, a key design decision here is the enforcement of a context. BERT-based models, while powerful, generally perform best when given a specific context to ground their answers. To prevent blocking the main thread, the actual question-answering process is offloaded to a separate asynchronous thread using asyncio dot to thread.

Step 3: Building a React Frontend with Next.js and Tailwind CSS

While our server backend is functional and can be interacted with via command-line tools like wscat (using a command such as wscat c ws: slash slash localhost:PORT slash chat followed by a prompt like type: auto, question: Write Python code for, and so on), this approach is not user-friendly. To make our AI chatbot accessible to a wider audience, including those unfamiliar with the terminal, we need a graphical user interface. We'll leverage Next.js, a React framework known for its performance and developer experience, along with Tailwind CSS for styling.

Our frontend application consists of a single page with a few key components. Let's start by examining react frontend slash src slash app slash layout dot tsx: The TypeScript code below defines Root Layout, 40 lines, from react frontend slash src slash app slash layout dot tsx.

This layout dot tsx file defines the root layout of our Next.js application. It's largely based on the default code generated by create next app and enhanced with Tailwind CSS integration. We've also integrated KaTeX for rendering mathematical expressions (remember to install the katex package and its TypeScript definitions, as well as marked.js for Markdown processing). The Theme Switcher component, responsible for toggling between light and dark themes, is also included.

Here's the code for the ThemeSwitcher component: The TypeScript code below defines Theme Switcher and toggle Theme, 35 lines, from react frontend slash src slash app slash ui slash layout slash Theme Switcher dot tsx.

For Svelte developers, React's use Effect hook serves a similar purpose to Svelte 5's dollar effect block. Both allow you to run code in response to changes (in use Effect you need to list the dependencies, you don't need that in Svelte), and critically, both provide a safe context for interacting with HTML elements.

Now is the turn is react frontend slash src slash app slash page dot tsx: The TypeScript code below defines Home and handle Send, 112 lines.

This file represents the main page component of our chat application, housing the core logic for message handling and UI rendering. The Home component manages the chat interface, including displaying messages, handling user input, and communicating with the backend. The handle Send function adds a user message and a loading message to the state, while handle Bot Message updates the UI with the bot's responses as they stream in.

The custom use Web Socket hook encapsulates all the WebSocket logic, including connection management, message handling, and error handling: The TypeScript code below defines Web Socket Message and use Web Socket, 76 lines.

handle Send function adds a temporary "loading" message to the chat interface. This is a simple but effective UI trick to provide immediate feedback to the user while waiting for the AI bot's response. Without this, the user might perceive a delay or lack of responsiveness which is unwanted. handle Bot Message function is designed to handle the streaming responses from the AI bot. It efficiently updates the chat interface by appending new content to the last bot message. The is Complete flag, triggered by the END signal from the backend, ensures that the loading indicator is removed when the bot finishes generating its response. Why use Callback in handle Bot Message? Its primary reason is to prevent unnecessary re-renders of components that depend on these functions and it does that by memoizing them so that they only change when their dependencies change.

To wrap up, let's look at the react frontend slash src slash app slash ui slash chat slash Chat Container dot tsx and react frontend slash src slash app slash ui slash chat slash Chat Input dot tsx. The TypeScript code below defines Chat Container Props, 21 lines, from react frontend slash src slash app slash ui slash chat slash Chat Container dot tsx.

The Chat Container component renders the list of Chat Message components. It's a straightforward component that iterates over the messages and displays each one. The TypeScript code below defines Chat Message Props, Chat Message and preprocess Unicode, 137 lines, from react frontend slash src slash app slash ui slash chat slash Chat Message dot tsx.

The Chat Message component is responsible for rendering individual chat messages. The key feature of this component is its use of marked dot js to render Markdown content, with extensions for syntax highlighting (using highlight dot js) and math rendering (using Ka Te X). The component also includes logic to display a "thinking" animation while the bot is generating a response.

Lastly, let's look at react frontend slash src slash app slash ui slash chat slash Chat Input dot tsx: The TypeScript code below defines Chat Input Props, Chat Input and handle File Upload, 163 lines, from react frontend slash src slash app slash ui slash chat slash Chat Input dot tsx.

It provides the user interface for entering and sending messages. It includes a text input, controls for toggling "Masked" and "Auto" modes, and a file upload button (you must select the masked mode to upload). The component was inspired by X's Grok 3, 😉.

An important to note is that in handle File Upload, I used a NextJS action (upload Files) to communicate with the server for files upload: The TypeScript code below defines upload Files, 21 lines, from react frontend slash src slash app slash lib slash actions slash upload dot ts.

The key reason for using a server action is to avoid CORS errors such as Cross Origin Request Blocked: The Same Origin Policy, and so on. By executing the upload logic on the server, we bypass the browser's same-origin policy and can communicate with the backend API without requiring CORS configuration.

I am sorry for the long article. I didn't want to split so I can be forced to just finish it at a go. There are still a couple of cleanups to do but I will leave those to you. Bye for now.

Outro

Enjoyed this article? I'm a Software Engineer, Technical Writer, and Technical Support Engineer actively seeking new opportunities, particularly in areas related to web security, finance, healthcare, and education. If you think my expertise aligns with your team's needs, let's chat! You can find me on LinkedIn and X. I am also an email away.

If you found this article valuable, consider sharing it with your network to help spread the knowledge!