LLM Service Deployment

LLM Toolchain Installation

A8625MY1 supports on-device deployment of 0.5B~14B parameter models and provides a complete LLM development toolchain.

# 1. Download the following packages from the "Resource Download Center"
# aidgense_2.0.5_arm64.aid.gpg
# aidgen-sdk_1.0.0.9_arm64_ub2204.deb
# aid-openai-api_2.0.0_arm64.deb
# 2. Upload to /home/aidlux/LLM on A8625MY1
# 3. Install dependencies
apt update && apt install -y python3-pip libssl-dev
# 4. Install AidGen SDK
dpkg -i aidgen-sdk_1.0.0.9_arm64_ub2204.deb
# 5. Install OpenAI-compatible API service
dpkg -i aid-openai-api_2.0.0_arm64.deb
# 6. Install aidgense service
sudo aid-pkg -i -d aidgense_2.0.5_arm64.aid.gpg

Online Model Pull and Run

# List available remote LLMs
aidllm remote-list api
# Pull A8625-optimized Qwen2.5-VL-3B multimodal model
aidllm pull api aplux/Qwen2.5-VL-3B-392x392-8625
# List installed models
aidllm list api
# Start LLM API service and show logs
aidllm start api -m Qwen2.5-VL-3B-392x392-8625 --show-log
# Verify service status
aidllm status api
# Expected output: Api server status: Running

Manual Local Model Deployment

# 1. Upload local model folder (for example qwen2.5-1.5b-instruct-q4_k_m) to /home/aidlux/LLM
# 2. Copy to model directory
cp -r qwen2.5-1.5b-instruct-q4_k_m /opt/aidlux/app/aid-openai-api/res/models/

# 3. Modify config file /opt/aidlux/app/aid-openai-api/api_cfg.json
{
  "model_cfg_list": [
    {
      "model_id": "qwen2.5-1.5b-instruct-q4_k_m",
      "model_create": "1752735571243",
      "model_owner": "aplux",
      "cfg_path": "./models/qwen2.5-1.5b-instruct-q4_k_m/qwen2.5-1.5b-instruct-q4_k_m.json",
      "prompt_template_type": "qwen2"
    }
  ],
  "default_model_id": "qwen2.5-1.5b-instruct-q4_k_m"
}

# 4. Start service
aidllm start api -m qwen2.5-1.5b-instruct-q4_k_m

OpenAI-Compatible API Calls

AidGen provides fully OpenAI-compatible APIs. Default port is 8888.

Text chat example

curl http://192.168.100.3:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2.5-VL-3B-392x392-8625",
    "messages": [
      {"role": "user", "content": "Briefly introduce the A8625 compute card"}
    ]
  }'

Multimodal image analysis example

curl http://192.168.100.3:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2.5-VL-3B-392x392-8625",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Describe the content in this image"},
          {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAAAAAAAD/..."}}
        ]
      }
    ]
  }'