> Source: https://glassmkr.com/blog

# Blog

Updates, guides, and operational knowledge from the team.

[July 2026 Engineering ## One account key, zero per-host secrets Most fleets onboard into monitoring by copying one powerful credential onto every box. glassmkr-crucible enroll does the opposite: one account key stays in your control plane, each host self-registers by a stable machine ID and holds only its own collector key, and running the automation twice does nothing.](/blog/one-account-key-zero-per-host-secrets)[July 2026 Engineering ## Same model, same server, opposite result: the half of agent testing everyone forgets We gave nine open-weight models root on real broken servers and graded every claim against the machine, not the model's own report. The thing that most decided success was not model size. It was whether our harness spoke the language the model was trained in: North scores 0 in plain text and 7 with tools, Phi-4 is the mirror.](/blog/open-model-ladder-blind-remediation)[July 2026 Engineering ## We gave Claude Haiku root on a broken server We ran our standing validation exercise with Claude Haiku 4.5, the smallest model in the family, on a real bare-metal server with four firing alerts. It fixed what it should, correctly declined to touch healthy hardware, and surfaced a false-positive bug in our own monitoring.](/blog/haiku-blind-remediation)[July 2026 Product ## Half the "problems" on a marketplace GPU host are required config Three alert rules fired on every box in our GPU fleet, and all three were correct readings of config a marketplace GPU host is supposed to have. The same signal is a finding on one host and required config on another, so we taught Glassmkr the difference with host-type profiles.](/blog/monitoring-a-host-you-dont-fully-control)[June 2026 GPU ## A single reboot would have de-listed our entire GPU fleet On these NVIDIA boxes nouveau was never blacklisted, so the next routine reboot would have stopped every GPU verifying and silently dropped the host off the Vast marketplace. The failure mode, the one-line fix, and why monitoring should catch the latent fault before the reboot.](/blog/the-reboot-that-delists-your-gpu-host)[June 2026 GPU ## We pointed Glassmkr at a 20-box GPU fleet. It went from 88 alerts to 11. A field report from twenty GPU boxes on Vast: the honest breakdown of 88 alerts down to 11. What was real hardware, what was noise we muted on purpose, what was self-inflicted, and the false positives we then fixed in the product.](/blog/20-box-gpu-fleet-88-to-11-alerts)[June 2026 Engineering ## The most honest AI feature we shipped has no AI in it We built the self-hosted-Gemma path for our ticket-draft feature, A/B'd it against a plain template on a live degraded-array alert, and shipped the template. The model was fine. Fine did not justify 22 seconds and a hallucination surface. What we kept, what we cut, and why the value was never the prose.](/blog/the-most-honest-ai-feature-we-shipped-has-no-ai-in-it)[June 2026 Engineering ## Would you have caught my VRM degradation? A customer's Ryzen 9 5950X lost its VRM and asked if Glassmkr would have caught it. The honest answer was no: voltage alone cannot watch a DVFS core rail. What we built instead (a behavioral signal and a variance-aware voltage-drift signal), what happened when we backtested it on our own MC12-LE0, and the uptime check that stopped us claiming a win we had not earned.](/blog/would-you-have-caught-my-vrm-degradation)[June 2026 Engineering ## What our test suite looks like, and why Four tests from the code that runs Glassmkr, and the incident that put each one there: a suppressed security alert, a temperature threshold that means different things on different boards, a 404 that has to stay a 404, and a power-supply name captured from a lying BMC. A test suite as a map of what has hurt you.](/blog/what-our-test-suite-looks-like)[May 2026 Engineering ## Validating GPU monitoring across three NVIDIA cards: L4, RTX A4000, A16 Three cards, eight rules. We tested three live on real hardware (power-cap throttling on a multi-die A16, thermal load on an L4, driver drift across all three) and deliberately did not induce the other five, because a real uncorrected-ECC or XID fault needs a power cycle to clear. The honest map of what 'GPU monitoring, tested' means.](/blog/validating-gpu-monitoring-across-three-nvidia-cards)[May 2026 Security ## We found a security false-negative in our own monitoring On RHEL-family hosts, download-only dnf-automatic timers were treated as 'auto-updates configured,' silently suppressing the pending_security_updates alert while Critical patches sat unapplied. What the bug was, how dogfooding caught it, and the fix in Crucible 0.13.6.](/blog/we-found-a-security-false-negative-in-our-own-monitoring)[May 2026 Operations ## Cross-vendor IPMI quirks we learned the hard way Six specific footguns from running monitoring across Supermicro, Gigabyte, ASUS and ASRockRack on Debian, Ubuntu, Rocky, Alma and Proxmox. SEL timestamp shapes, BMC firmware that lies about vendor, distro-specific package gaps, and the Gigabyte DTS +30 °C offset.](/blog/cross-vendor-ipmi-quirks)[May 2026 Engineering ## When your monitoring tool punishes customers for doing the right thing 60 minutes of stale alerts after a legitimate fix. A kernel reboot that fired its own critical alert. Two distinct bugs surfaced when we actually applied our own remediation guidance end to end on the validation fleet.](/blog/when-monitoring-punishes-customers)[May 2026 Engineering ## When a Phase 1 audit changed our hypothesis A server reported 'IPMI: Not detected' while showing ECC counts on the same screen. The spec said our vendor detection was wrong. An hour-long audit said detection was correct and three other things were broken. Map the problem before you write the code.](/blog/when-phase-1-audit-changed-our-hypothesis)[May 2026 AI ## Introducing Furnace: the AI assistant that helps you fix alerts Furnace reads your alerts, looks at the evidence, and suggests remediation. Self-hosted Gemma 4 26B in Amsterdam. Conservative, hedging, willing to say 'I don't know'. The AI in your monitoring shouldn't be the headline.](/blog/introducing-furnace)[May 2026 Engineering ## We used an AI as a controlled probe of our alert documentation We forbade an AI from using its training data and made it resolve real infrastructure alerts using only the guidance our own dashboard produces. Three gap patterns surfaced. All three fixed in the same week.](/blog/ai-controlled-probe-of-alert-docs)[April 2026 Engineering ## Training a drive-failure model on a GPU server's CPU We retrained our drive-failure predictor on 2 years of Backblaze data (222M drive-days) on the CPU of our L4 inference server. Gemma stayed resident in VRAM. 59 minutes, no new compute, 5.8% inference overhead. Plus the feature-importance surprise: SMART 197 beat SMART 187.](/blog/training-drive-failure-model-on-l4)[April 2026 Launch ## Introducing Glassmkr: bare metal monitoring built by operators Two pieces, one philosophy: Crucible (the open-source agent) plus Dashboard (the optional SaaS). Built by operators with a decade of bare metal experience across 60+ global locations.](/blog/introducing-glassmkr)[April 2026 Engineering ## We benchmarked Qwen3.6 against our production Gemma 4 on an L4. Here's what actually mattered. Three-way benchmark of Gemma 4 26B-A4B, Qwen3.6 35B-A3B no-think, and Qwen3.6 35B-A3B thinking on a production infrastructure health analysis prompt. Real wall-clock numbers, VRAM footprints, and the quality-latency tradeoff that matters for narration.](/blog/qwen3-6-vs-gemma-4-infrastructure-narration)[April 2026 Operations ## IPMI diagnostics for bare metal: what to monitor and how to read it A practical guide to monitoring IPMI sensors, SEL logs, and BMC health on Dell, Supermicro, and HPE servers. Covers kipmi0 CPU issues, vendor quirks, and what to alert on.](/blog/ipmi-diagnostics-bare-metal)[April 2026 Engineering ## What We Learned Running Gemma 4 on an L4 GPU for Production Server Analysis How we deployed Gemma 4 26B on an NVIDIA L4 for AI health analysis of bare metal servers. Covers model selection, why vLLM failed, quantization choices, and prompting for structured infrastructure output.](/blog/gemma-4-l4-gpu-server-analysis)[April 2026 Operations ## IPMI, SMART, and RAID: The Hardware Layer Your Cloud Monitoring Tool Ignores Most monitoring tools stop at the OS. Below it sits an entire hardware layer: disk firmware predicting its own failure, fans at 0 RPM, ECC memory correcting silent errors. Here is what to monitor and why.](/blog/ipmi-smart-raid-hardware-monitoring)[April 2026 Operations ## Why bare metal monitoring is different Cloud monitoring tools were built for ephemeral workloads. They track HTTP latency and container restarts. But when you run physical servers, the failure modes are fundamentally different: drives wear out, DIMM slots develop bit errors, fans fail silently, and RAID arrays degrade without anyone noticing.](/blog/why-bare-metal-monitoring-is-different)