> Source: https://glassmkr.com/for-storage # Your drive says it is healthy. That is not the same as being healthy. Every drive carries a one-bit self-assessment, and it is the field most monitoring checks. We measured how far that bit can sit from reality: 477 dead sectors, and it still read PASSED. [Monitor a server for free](https://app.glassmkr.com/register) [View Crucible on GitHubAGPL-3.0-only](https://github.com/glassmkr/crucible) Free hosted or self-hosted. Point it at one storage box before deploying fleet-wide. `curl -fsSL https://glassmkr.com/install.sh | sudo bash` Crucible v1.2.3 on npm P0 CRITICAL RAID array degraded (md0) db-hel1-01 One disk has dropped out of the RAID array. The server is still online, but it has lost redundancy. EVIDENCE md0 (RAID10) is running degraded: 1 of 6 devices failed (/dev/sdd). Array is still online; rebuild has not started (no spare). NEXT SAFE STEP Replace /dev/sdd and re-add it to the array, then watch the rebuild. If you operate this hardware (owned or colocated): handle the inspection or swap yourself, or dispatch a remote-hands technician. If this is rented or provider-managed: file a hardware service ticket with your provider. Either way, the physical check needs to happen on-site. captured 2026-08-30 10:31 UTC · host db-hel1-01 · public demo, read-only sample-fleet capture · agent 0.13.6 ## The problem The SMART overall-health verdict is a pass or fail against normalized attribute values, not against the raw counts an operator reads. The manufacturer sets those thresholds where it expects warranty-relevant death. A drive can work through most of its spare sector pool and still sit comfortably inside them. We ran a drive-health campaign across thirteen boxes to find out how wide that gap gets. One 4 TB drive was carrying **477 reallocated sectors** and reported its own health as **PASSED in all 140 readings** we logged. A full destructive write-and-verify pass over that same drive came back clean, zero errors, because reallocation exists precisely to hide the damage from the host. The only place it was visible was the raw attribute counter, and only to something watching that counter over time. That is the whole job. Not reading the verdict; reading what is underneath it, on a schedule, and knowing which of those numbers mean something. [Read the full write-up, with the data](/blog/smart-said-passed). ## How Glassmkr fits ### Sixteen rules for the storage layer alone. Seven for drives and controllers, three for ZFS, six for filesystem and capacity. They run on every collection cycle against every drive the box can see, and they carry the remediation with them rather than leaving you to work out what a raw attribute number means. ### We alert on the counters that latch, and not on the ones that flap. Reallocated sectors only ever go up, so they are safe to page on. Pending sectors oscillate by design: on that same drive ours swung between 0 and 96 while the actual damage never moved. An alert keyed on pending would have fired and self-cleared three times in under a day. We considered it and the data talked us out of it. ### Drives behind a hardware RAID controller are still drives. A controller that presents one logical volume hides the physical disks from most tooling. Crucible enumerates them through the controller and reads each one's SMART directly, so a failing member is visible before the array degrades rather than after. ### ZFS gets its own rules, not a generic disk check. Pool health, scrub errors and a faulted SLOG are three different problems with three different responses. Treating them as one "disk problem" is how a degraded pool sits unnoticed while somebody investigates capacity. ### Capacity is a trend, not a threshold. A disk at 71 percent that has been at 71 percent for a year is fine. A disk at 71 percent climbing steadily is a scheduled outage. We project the fill rather than waiting for a percentage line to be crossed at 3am. ### An unreadable drive is reported, not skipped. If SMART cannot be read for a disk, whether the controller does not support it or the tooling is missing, that is stated as its own condition. A drive we cannot see is not a drive that is fine, and it should never silently disappear from a health summary. ## What we do not claim We found one failing drive in that campaign and we found it already at 477, so we never watched it climb from zero. That means we cannot tell you what count is dangerous. Is a drive at 3 reallocated sectors in trouble, or fine for another four years? Our data cannot answer that, and any threshold we published from a single drive would be an invention with a number attached. So reallocated sectors stay critical rather than being tiered by size, because the two ways of being wrong are not equally expensive: telling you to investigate a drive that is actually dying costs you data, and paging you about one that turns out to be stable costs you a ticket. ## The agent you can read Crucible is AGPL-3.0-only and the repository is public. It runs as an unprivileged user, and the few commands that need root go through a narrow root-owned wrapper with a fixed argument list per action: no shell, no caller-supplied arguments. You can read exactly what it collects and exactly what leaves the box. [What we collect and what we do not](/trust) is documented separately, in detail. ## Pricing reminder Free both ways: self-hosted under AGPL-3.0-only with no node limits, or the free hosted instance with a 10-node per-account cap. No per-drive or per-metric charge in either form. A 60-bay JBOD counts the same as a single-disk server: one node. ## Point it at one storage box first. Install is one command, the first snapshot normally arrives within one collection cycle (about five minutes at the default interval), and both deployment forms are free, so you can see what it says about hardware you already know the history of before you trust it with the rest. [Get started free](https://app.glassmkr.com/register) [Read the docs](/docs) ### Or self-host it The whole stack is open source under AGPL-3.0-only and runs on your own hardware with one docker compose file; nothing leaves your network. ``` curl -fsSL https://glassmkr.com/install.sh | sudo bash -s -- \ --api-key "gmk_cru_live_..." \ --ingest-url "http://your-dashboard-host:3000/api/v1/ingest" ``` [Self-host in 10 minutes](/docs/self-hosting)