Deploy VLM HTTP Server with AidGenSE

Introduction

Edge deployment of Vision Language Models (VLM) refers to the process of compressing, quantizing, and deploying large models originally run in the cloud onto local devices to enable offline, low-latency natural language understanding and generation. This chapter uses the AidGenSE inference engine as a base to demonstrate how to deploy a multi-modal large model HTTP service (compatible with the OpenAI API) on edge devices.

In this case, the multi-modal large model inference runs on the device side. Relevant interfaces are called via HTTP API to receive user input and return dialogue results in real time.

• Device: Rhino Pi-X1
• System: Ubuntu 22.04
• Model: Qwen2.5-VL-3B (392x392)

Supported Platforms

Platform	Running Method
Rhino Pi-X1	Ubuntu 22.04, AidLux

Preparation

1. Rhino Pi-X1 hardware
2. Ubuntu 22.04 system or AidLux system

Case Deployment

Step 1: Install AidGenSE

sudo aid-pkg update
sudo aid-pkg -i aidgense
sudo aid-pkg -i aidgen-sdk
sudo aid-pkg -i aidgen-qnn236
sudo aid-pkg -i aidgen-qnn240

Step 2: Model Query & Acquisition

• View supported models

# View supported models
aidllm remote-list api

#------------------------Example output is as follows------------------------

Current Soc : 8550

Name                                                    Url                                                           CreateTime
-----                                                   ---------                                                     ---------
qwen2.5-vl-3b-instruct-392x392-qnn2.36-w4a16-qcs8550    aplux/qwen2.5-vl-3b-instruct-392x392-qnn2.36-w4a16-qcs8550    2026-05-15 10:57:51
qwen2.5-vl-3b-instruct-672x672-qnn2.36-w4a16-qcs8550    aplux/qwen2.5-vl-3b-instruct-672x672-qnn2.36-w4a16-qcs8550    2026-05-15 10:57:49
...

• Download qwen2.5-vl-3b-instruct-392x392-qnn2.36-w4a16-qcs8550

# Download the model
aidllm pull api aplux/qwen2.5-vl-3b-instruct-392x392-qnn2.36-w4a16-qcs8550

# View downloaded models
aidllm list api

Step 3: Start the HTTP Service

# Start the openai api service for the corresponding model
aidllm start api -m qwen2.5-vl-3b-instruct-392x392-qnn2.36-w4a16-qcs8550

# Check status
aidllm status api

# Stop service: aidllm stop api

# Restart service: aidllm restart api

💡Note

The default port number is 8888.

Step 4: Dialogue Test

Dialogue Test Using Web UI

# Install the UI front-end service
sudo aidllm install ui

# Start the UI service
aidllm start ui

# Check UI service status: aidllm status ui

# Stop UI service: aidllm stop ui

After starting the UI service, access http://ip:51104.

Dialogue Test Using Python

import os
import sys
import requests
import json
import base64


def stream_chat_completion(messages, model="qwen2.5-vl-3b-instruct-392x392-qnn2.36-w4a16-qcs8550"):

    url = "http://127.0.0.1:8888/v1/chat/completions"
    headers = {
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "messages": messages,
        "stream": True    # Enable streaming
    }

    # Send request with stream=True
    response = requests.post(url, headers=headers, json=payload, stream=True)
    try:
        response.raise_for_status()
    except requests.HTTPError as e:
        print(f"\nHTTP error: {e}", file=sys.stderr)
        if response.text:
            print(response.text, file=sys.stderr)
        raise

    # Read and parse SSE format line by line
    for line in response.iter_lines():
        if not line:
            continue
        # print(line)
        line_data = line.decode('utf-8')
        # Each line of SSE starts with the "data: " prefix
        if line_data.startswith("data: "):
            data = line_data[len("data: "):]
            # End flag
            if data.strip() == "[DONE]":
                break
            try:
                chunk = json.loads(data)
            except json.JSONDecodeError:
                # Print and skip when parsing fails
                print("Failed to parse JSON：", data)
                continue

            # Extract the token output by the model
            content = chunk["choices"][0]["delta"].get("content")
            if content:
                print(content, end="", flush=True)

def image_to_data_url(image_path):
    """Read an image and convert it to a data URL, compatible with the OpenAI-style image_url field."""
    ext = os.path.splitext(image_path)[1].lower()
    mime_map = {
        ".jpg": "image/jpeg",
        ".jpeg": "image/jpeg",
        ".png": "image/png",
        ".webp": "image/webp",
    }
    mime_type = mime_map.get(ext, "image/jpeg")
    b64_str = image_to_base64(image_path)
    return f"data:{mime_type};base64,{b64_str}"


if __name__ == "__main__":
    image_url = image_to_data_url("/path/to/your/image.jpg")
    # Example dialogue
    messages = [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content":[
                        {
                            "type": "text",
                            "text": "Describe the content of this image."
                        },
                        {
                            "type": "image_url",
                            "image_url":
                            {
                                "url": image_url
                            }
                        }
                    ]
        },
    ]
    print("Assistant:", end=" ")
    stream_chat_completion(messages)
    print()  # New line