mirror of
https://github.com/dante-gpu/dantegpu-core.git
synced 2026-11-02 18:20:00 +09:00
0raclus
1672858674
Adds the "real model on a rented GPU" proof the rental-proof was building toward.
scripts/gpu-model-proof.sh — one command:
- ensures mlx-lm in a throwaway venv (Apple Silicon / Metal),
- starts a JetStream nats-server + the provider-daemon,
- dispatches an ai-inference task over NATS whose payload (scripts/gpu_infer.py)
loads a real 4-bit quantized LLM and generates text on the GPU,
- prints the daemon-captured model output and tears everything down.
Verified on an Apple M4 Max: the daemon receives the task over NATS, runs
mlx-community/Llama-3.2-1B-Instruct-4bit on the Metal GPU (device "Device(gpu, 0)",
~235 tokens/sec, 0.81 GB GPU peak), and returns a real coherent answer with
exitCode 0. So the full chain is real end to end: detect GPU -> rent -> dispatch
over NATS -> daemon executes -> real model runs on the GPU -> output returned.
scripts/gpu_infer.py is the MLX inference payload (swap --model for any
mlx-community repo; a PyTorch payload works for CUDA/NVIDIA). docs/PROOF.md and
the README document the run; the only thing still unproven is *distributed*
multi-node inference (one model sharded across many rented GPUs).
43 lines
1.4 KiB
Python
Executable File
43 lines
1.4 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""Run real LLM inference on the local GPU and print a JSON result.
|
|
|
|
This is the payload a DanteGPU rental task executes to prove a rented GPU runs a
|
|
real model. It uses MLX (Apple Silicon / Metal) by default; the device line in
|
|
the output shows which accelerator actually ran the model.
|
|
|
|
Requires: mlx, mlx-lm (pip install mlx-lm, ideally in a venv).
|
|
|
|
Usage: gpu_infer.py [MODEL] [PROMPT]
|
|
"""
|
|
import sys, time, json
|
|
import mlx.core as mx
|
|
from mlx_lm import load, generate
|
|
|
|
MODEL = sys.argv[1] if len(sys.argv) > 1 else "mlx-community/Llama-3.2-1B-Instruct-4bit"
|
|
PROMPT = sys.argv[2] if len(sys.argv) > 2 else "In one sentence, what is a GPU and why is it good for AI?"
|
|
|
|
t0 = time.time()
|
|
model, tokenizer = load(MODEL)
|
|
load_s = time.time() - t0
|
|
|
|
messages = [{"role": "user", "content": PROMPT}]
|
|
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
|
|
|
|
mx.reset_peak_memory()
|
|
t1 = time.time()
|
|
text = generate(model, tokenizer, prompt=prompt, max_tokens=80, verbose=False)
|
|
gen_s = time.time() - t1
|
|
|
|
ntok = len(tokenizer.encode(text))
|
|
print(json.dumps({
|
|
"device": str(mx.default_device()),
|
|
"model": MODEL,
|
|
"prompt": PROMPT,
|
|
"output": text.strip(),
|
|
"tokens": ntok,
|
|
"tokens_per_sec": round(ntok / gen_s, 2) if gen_s else 0,
|
|
"load_seconds": round(load_s, 2),
|
|
"gen_seconds": round(gen_s, 2),
|
|
"gpu_peak_mem_gb": round(mx.get_peak_memory() / 1e9, 3),
|
|
}))
|