GLiClass Zero-Shot Classifier Server & Benchmark Suite

A production-ready, Dockerized deployment and benchmark harness for GLiClass (Generalist and Lightweight Model for Sequence Classification) with drop-in compatibility for Laya and Jev APIs.

---

What is GLiClass?

GLiClass (by Knowledgator) is a zero-shot text classification framework inspired by GLiNER. Unlike traditional cross-encoders (which require N separate forward passes for N candidate labels) or generative LLMs (which suffer from autoregressive generation latency and hallucinations):

• Single Forward Pass: GLiClass encodes input documents and candidate labels simultaneously, achieving cross-encoder quality at bi-encoder inference speeds (up to 50x faster).
• Zero-Shot & Open Vocabulary: Classifies arbitrary text against ad-hoc candidate labels without task-specific fine-tuning.
• ModernBERT Backbone (v3.0): Employs answerdotai/ModernBERT with FlashAttention and Rotary Position Embeddings (RoPE), achieving sub-8ms median latency on NVIDIA RTX 5060 Ti.

---

Quick Start

1. Start Server (Auto-Detect GPU vs CPU)

./start.sh

Automatically detects if an NVIDIA GPU and Docker NVIDIA runtime are present on the host; launches in GPU mode if available, or seamlessly falls back to CPU mode.

2. Start Server on GPU (NVIDIA CUDA)

./start_gpu.sh

Default model: knowledgator/gliclass-modern-base-v3.0 on port 8001. Automatically falls back to CPU mode if NVIDIA hardware or Docker runtime is unavailable. Verifies health at http://localhost:8001/health:

{
  "status": "ok",
  "model": "knowledgator/gliclass-modern-base-v3.0",
  "device": "cuda:0",
  "device_name": "NVIDIA GeForce RTX 5060 Ti",
  "backend": "gliclass"
}

To run with GLiClass v1 (knowledgator/gliclass-base-v1.0 DeBERTa-v3):

MODEL_NAME="knowledgator/gliclass-base-v1.0" ./start_gpu.sh

2. Start in CPU Mode (Testing / Fallback)

./start_cpu.sh

3. Run with Docker Compose

# GPU mode (NVIDIA CUDA)
docker compose --profile gpu up -d

# CPU mode
docker compose --profile cpu up -d

# Or set default profile via .env (defaults to CPU in .env.example):
cp .env.example .env
docker compose up -d

Images are published to the Forgejo Container Registry (git.bobparsons.dev/deadstyle/classifier:latest).

4. Stop the Server

./stop.sh
# or if using docker compose:
docker compose down

---

Performance Summary: GLiClass vs Laya (RTX 5060 Ti)

Benchmark executed side-by-side on this system (NVIDIA GeForce RTX 5060 Ti 16GB, Blackwell architecture, Linux 32 vCPUs):

Metric	GLiClass v3 (modern-base-v3.0)	Laya (convaiinnovations/laya)	Speedup / Ratio
p50 Latency (Median)	7.66 ms	13.28 ms	1.7x faster (GLiClass)
Mean Latency	8.24 ms	13.27 ms	1.6x faster
Min Latency	6.81 ms	11.60 ms	1.7x faster
p95 Latency	10.01 ms	14.41 ms	1.4x faster
Dedicated /classify (p50)	5.58 ms	N/A	Ultra low-latency
Throughput (1 client)	121.4 req/sec	75.4 req/sec	1.6x higher throughput
Parameter Count	~149M (ModernBERT-base)	~395M (ModernBERT-large)	2.6x smaller footprint

---

API Usage

1. Dedicated Classification: POST /classify

Use this for general zero-shot sequence classification, sentiment analysis, topic tagging, or incident triage:

curl -X POST http://localhost:8001/classify \
  -H "Content-Type: application/json" \
  -d '{
    "text": "System incident 4048: The database connection pool on cluster primary is exhausted after an influx of batched bulk-insert operations. Read replicas are responding normally, but mutation queries are queueing with timeout errors.",
    "labels": ["Database Error", "Network Outage", "Auth Failure", "Billing Issue"],
    "classification_type": "single-label"
  }'

Response:

{
  "model": "knowledgator/gliclass-modern-base-v3.0",
  "classification_type": "single-label",
  "top_label": "Network Outage",
  "confidence": 0.9719,
  "ranked": [
    {"label": "Network Outage", "score": 0.9719},
    {"label": "Database Error", "score": 0.8498},
    {"label": "Auth Failure", "score": 0.1898},
    {"label": "Billing Issue", "score": 0.0569}
  ],
  "scores": {
    "Network Outage": 0.9719,
    "Database Error": 0.8498,
    "Auth Failure": 0.1898,
    "Billing Issue": 0.0569
  },
  "latency_ms": 7.96
}

---

2. Laya / Jev Drop-in Compatibility: POST /v1/systemone

This server supports the exact request format used by Laya and Jev, allowing you to route existing pipelines or benchmarks directly to GLiClass without code changes:

curl -X POST http://localhost:8001/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": {
      "question": "What is the return policy window for opened electronic items?"
    },
    "questions": {
      "selected_reference": {
        "type": "choice",
        "instructions": "Given the question in state, determine which reference chunk contains the accurate information to answer it. If none of the chunks answer the question, select none_of_above.",
        "criteria": {
          "chunk_shipping": "Standard domestic shipping takes 3-5 business days. Express overnight shipping takes 1 business day for orders placed before 2 PM.",
          "chunk_return_opened": "Opened electronic items may be returned within 14 days of purchase provided they include original packaging and receipt. Unopened items have a 30-day window.",
          "chunk_warranty": "All consumer hardware products include a 12-month manufacturer limited warranty covering defects in materials and workmanship.",
          "chunk_membership": "Rewards club members receive an extended 45-day return policy on apparel and home goods only.",
          "none_of_above": "None of the provided reference chunks answer the question."
        }
      }
    }
  }'

Response:

{
  "model": "knowledgator/gliclass-modern-base-v3.0",
  "answers": {
    "selected_reference": {
      "type": "choice",
      "choice": "chunk_return_opened",
      "probabilities": {
        "chunk_shipping": 0.0031,
        "chunk_return_opened": 0.9942,
        "chunk_warranty": 0.0028,
        "chunk_membership": 0.0019,
        "none_of_above": 0.0058
      },
      "confidence": 0.9942
    }
  },
  "usage": {
    "input_tokens": 178,
    "output_tokens": 0
  },
  "latency_ms": 7.42,
  "routing": {
    "model": "knowledgator/gliclass-modern-base-v3.0",
    "reason": "GLiClass inference on cuda:0"
  }
}

---

Running the Benchmark & Comparison

Run GLiClass Benchmark:

python3 benchmark.py --url http://localhost:8001 --iterations 15 --output-json gpu_results.json

Run Side-by-Side Comparison against Laya:

With Laya running on port 8000 and GLiClass on port 8001:

python3 compare_with_laya.py

This updates BENCHMARK_REPORT.md.

---

Implementation Details: GLiClass vs Transformers AutoModel

When loading GLiClass models, Hugging Face transformers throws:

ValueError: The checkpoint you are trying to load has model type `GLiClass`
but Transformers does not recognize this architecture.

This is because GLiClass defines a custom single-forward-pass classification head (GLiClassModel). Using pip install gliclass provides:

from gliclass import GLiClassModel, ZeroShotClassificationPipeline
from transformers import AutoTokenizer

model = GLiClassModel.from_pretrained("knowledgator/gliclass-modern-base-v3.0")
tokenizer = AutoTokenizer.from_pretrained("knowledgator/gliclass-modern-base-v3.0")
pipe = ZeroShotClassificationPipeline(model, tokenizer, classification_type='single-label', device='cuda:0')

This server handles this transparently and supports fallback to standard Hugging Face pipelines for other zero-shot models if needed.