# Glassmkr Documentation Corpus (machine-readable)

Generated: 2026-07-15

This file is the LLM-friendly corpus for Glassmkr's documentation.
It contains the full text of the guide pages plus the public catalog
metadata for all 68 alert rules.
For human-readable documentation, see https://glassmkr.com/docs and
https://glassmkr.com/trust. Per-alert remediation guidance is rendered
inside the dashboard at https://app.glassmkr.com.

## Index

- Homepage: https://glassmkr.com
- Pricing: https://glassmkr.com/pricing
- Documentation: https://glassmkr.com/docs
- Trust + security: https://glassmkr.com/trust
- Per-rule pages: https://glassmkr.com/docs/rules/<rule_id>
- Crucible agent source (MIT): https://github.com/glassmkr/crucible

## Glassmkr in one paragraph

Glassmkr is bare-metal infrastructure monitoring. 68 alert rules
tuned for real failure modes ship enabled out of the box, covering
storage, ZFS, filesystem, memory + CPU, network, hardware (BMC/IPMI),
NVIDIA GPU, time + services, and security + patching. The Crucible agent
is MIT-licensed and on npm; runs as a non-root user; ships only metrics
and alert state, never logs or command output. Pricing is $3 per node
per month after 3 free nodes. AI character Furnace runs on self-hosted
Gemma 4 26B on a single NVIDIA L4 GPU in Amsterdam; no third-party LLM
APIs. EU-stored data, GDPR-aligned operator (Czech sole-trader).

# Guide documentation

## Getting started
Source: https://glassmkr.com/docs/getting-started (Markdown: https://glassmkr.com/docs/getting-started.md)

# Getting started

Register a server, install the Crucible agent, see the first snapshot. About five minutes end to end.

## Overview

Glassmkr has two parts:

- **Dashboard** is the hosted service at [app.glassmkr.com](https://app.glassmkr.com). It stores your data, evaluates alerts, and renders server health.
- **Crucible** is the MIT-licensed agent. It runs on your server, collects health data, and pushes snapshots to the Dashboard.

You register a server in the Dashboard (which gives you a collector key), then install Crucible on that server with the key. That is the entire flow for a single server.

> **Automating, or onboarding many servers?** The whole flow runs over the API, no dashboard clicks. A `write`-scoped account key (`gmk_acct_live_`) creates each server with `POST /api/v1/servers`, and the response returns that server's collector key. See the [Programmatic API quickstart](/docs/programmatic-api).

Last verified: 2026-06-26 against Crucible v0.13.12.

## Prerequisites

- A Linux server (Ubuntu, Debian, RHEL, Rocky, Alma, Arch, or Alpine).
- Root or sudo access.
- Outbound HTTPS (port 443) to `app.glassmkr.com`. No inbound ports required.

## Step 1: Create a Dashboard account

Go to [app.glassmkr.com](https://app.glassmkr.com/#register) and sign up with email, Google, or GitHub. Free accounts can monitor up to 3 servers with 7-day retention.

## Step 2: Register a server

After logging in, click **+ Add Server**. Enter a name (for example, `web-01` or `db-prod`).

The Dashboard generates a **collector key** that starts with `gmk_cru_live_`. This key is shown once. Copy it. Servers created before Crucible 0.9.0 may have a legacy `col_xxx` key; both formats work.

> **What is this key?** The collector key (`gmk_cru_live_xxx`) authenticates snapshot pushes from this specific server to the Dashboard. Each server gets its own key, scoped to that server's telemetry only. This is different from the account API key (`gmk_acct_live_xxx`) in Settings, which is for programmatic API access from automation like Ansible, Terraform, or scripts.

## Step 3: Install Crucible

SSH into your server. The dashboard shows the exact command with your collector key pre-filled. The bootstrap installer is:

```
curl -sf https://glassmkr.com/install.sh | sudo bash -s -- --api-key gmk_cru_live_your_key_here
```

The installer:

- Installs Node.js 24 if not present.
- Installs `smartmontools` (SMART disk monitoring).
- Installs `ipmitool` if available (IPMI hardware monitoring).
- Runs `glassmkr-crucible init` to validate the key against the Dashboard, write `/etc/glassmkr/crucible.yaml` (mode 0600; migrates a pre-0.13.5 `/etc/glassmkr/collector.yaml` in place when present), and install the systemd unit.
- Starts the `glassmkr-crucible` service (runs as the non-root `glassmkr` user).

**Reading the key from a password manager?** Pipe it through stdin:

```
op read "op://Private/Dashboard/key" | sudo glassmkr-crucible init --api-key -
```

The legacy `--dashboard-key` flag is preserved as an alias for `--api-key` so existing Ansible or Terraform automation keeps working without changes.

### npm install

If you would rather not pipe to bash:

```
sudo npm install -g @glassmkr/crucible
sudo glassmkr-crucible init --api-key gmk_cru_live_your_key_here
```

### Docker install

The containerized agent reads its key from `/etc/glassmkr/crucible.yaml` on the host (it does not read a key from the environment), so create the config first, then start the container with it mounted. The image is on [Docker Hub](https://hub.docker.com/r/glassmkr/crucible):

```
# 1. Create the config (replace with your Dashboard key)
sudo mkdir -p /etc/glassmkr
sudo tee /etc/glassmkr/crucible.yaml << 'EOF'
server_name: "web-01"
dashboard:
  enabled: true
  url: "https://app.glassmkr.com"
  api_key: "gmk_cru_live_your_key_here"
EOF

# 2. Run the agent (privileged + host namespaces so it monitors the host, not the container)
docker run -d --restart=unless-stopped --name glassmkr-crucible \
  --pid=host --net=host --privileged \
  -v /etc/glassmkr:/etc/glassmkr:ro \
  docker.io/glassmkr/crucible:latest
```

Run `glassmkr-crucible init --help` to see the full flag list (custom server name, alternate Dashboard URL, etc.).

## Step 4: Verify

Check that Crucible is running:

```
sudo systemctl status glassmkr-crucible
```

You should see `active (running)`. Check the logs for the first collection:

```
sudo journalctl -u glassmkr-crucible --since "5 min ago" --no-pager
```

Expected output (the first collection may take a few seconds longer than later ones):

```
[collector] Starting. Server: web-01. Interval: 60s
[collector] IPMI: enabled, SMART: enabled
[collector] Dashboard: https://app.glassmkr.com
[collector] Collecting...
[collector] Collected in 1013ms. Alerts: 0 active, 0 new, 0 resolved

=== First collection complete ===
Server: web-01 (Ubuntu 24.04 LTS)
CPU:    5.3% (load: 0.14)
RAM:    12.1% (1940 / 16036 MB)
Disk:   23% (/)
SMART:  2 drive(s) checked
Network: eth0, eth1
IPMI:   available
Active alerts: 0
Dashboard: enabled

[dashboard] Push successful. Active alerts: 0
```

Within 60 seconds (one collection interval), the server appears on the [Dashboard](https://app.glassmkr.com) with live data. The Dashboard evaluates all 65 alert rules on each push.

Last verified: 2026-05-22. Default snapshot interval is 60 seconds in Crucible v0.10.0 and later.

## Two kinds of keys

Glassmkr uses two key types:

| Key type | Prefix | Created where | Used by | Purpose |
| --- | --- | --- | --- | --- |
| **Collector key** | `gmk_cru_live_` (legacy: `col_`) | Dashboard: + Add Server | Crucible agent | Authenticates snapshot pushes from one specific server. Scoped to that server; cannot list other servers or read account settings. |
| **Account API key** | `gmk_acct_live_` | Settings: API keys (every plan) | Automation: Ansible, Terraform, scripts, MCP clients | Programmatic access to your account (list servers, query health, mutate state). On every plan, with read, write, or admin scope; bounded by the 3-node quota and rate limits. |

## Next steps

- [Set up notification channels](/docs/channels) (email, Telegram, Slack, Discord, PagerDuty, webhooks) with per-channel priority filtering.
- [Review the 65 alert rules](/docs/rules) grouped across 9 categories.
- [Explore the full configuration reference](/docs/configuration) for thresholds, intervals, and optional collectors.
- [API reference](/docs/api) for programmatic access.

---

## Programmatic API + account API keys
Source: https://glassmkr.com/docs/programmatic-api (Markdown: https://glassmkr.com/docs/programmatic-api.md)

# Programmatic API + account API keys

For the full endpoint reference see the [API reference](/docs/api); for tier gating see [/docs/api/tier-gating](/docs/api/tier-gating). The full read+write programmatic API is on every plan, bounded only by the 3-node quota and the rate limits.

## The two key types

| Key type | Prefix | Used for | Created via |
| --- | --- | --- | --- |
| **Account API key** | `gmk_acct_live_` | Management endpoints: list servers, create servers, rotate keys, query audit log. | `POST /api/v1/account/keys` via curl / Ansible / Terraform after recent re-authentication via `POST /api/v1/account/verify-password`. Available on every plan. |
| **Collector key** | `gmk_cru_live_` | Telemetry ingestion only. Scoped to one server. Cannot list other servers or read account settings. | Returned by `POST /api/v1/servers`. Rotate via `POST /api/v1/servers/{id}/rotate-key`. |

Older agents may still have `col_*` collector keys. Both formats authenticate against `/api/v1/ingest`; you can rotate at your own pace.

## Quickstart: provision 50 servers in two minutes

### 1. Create an account API key

Two calls. Use a logged-in browser session (cookie jar). Account-key creation cannot use another account key.

```
# 1. Re-authenticate (opens a 5-minute step-up window).
curl -sS -X POST https://app.glassmkr.com/api/v1/account/verify-password \
  -b session.cookie \
  -H "Content-Type: application/json" \
  -d '{"password":"your-dashboard-password"}'

# 2. Create the key (returned in plaintext exactly once; save it).
#    scope must be "write" to create servers ("write" is the default).
curl -sS -X POST https://app.glassmkr.com/api/v1/account/keys \
  -b session.cookie \
  -H "Content-Type: application/json" \
  -d '{"name":"ansible-prod","scope":"write"}'
```

Server creation is capped at your plan's node quota (3 on Free; your subscribed count on Pro), so a 50-server run assumes a Pro plan.

### 2. Create the servers via curl

```
API_KEY="gmk_acct_live_..."

for i in $(seq 1 50); do
  RESPONSE=$(curl -sS -X POST https://app.glassmkr.com/api/v1/servers \
    -H "Authorization: Bearer $API_KEY" \
    -H "Content-Type: application/json" \
    -H "Idempotency-Key: bootstrap-$(date +%s)-$i" \
    -d '{"name":"web-'$i'","hostname":"web-'$i'.prod.example.com","tags":["prod","web"]}')

  COLLECTOR_KEY=$(echo "$RESPONSE" | jq -r '.server.api_key')
  echo "web-$i  =>  $COLLECTOR_KEY"
done
```

### 3. Provision the agent on each host

```
curl -sf https://glassmkr.com/install.sh | sudo GLASSMKR_API_KEY=$COLLECTOR_KEY bash
```

## Rate limits

Three default tiers (per-IP, per-key, per-account) plus per-endpoint sub-limits for the high-risk operations:

| Tier | Capacity | Refill |
| --- | --- | --- |
| Per-IP | 100 burst | 10/sec |
| Per-key | 1000 burst | 100/sec |
| Per-account | 5000 burst | 500/sec |
| `POST /servers` | 100 / hour / account | n/a |
| `DELETE /servers/{id}` | 100 / hour / account | n/a |
| `POST /servers/{id}/rotate-key` | 10 / hour / account | n/a |
| `POST /account/keys` | 10 / hour / account | n/a |
| `POST /account/keys/{id}/rotate` | 10 / hour / account | n/a |

On 429, the response body includes `tier` and `retry_after_seconds`; the `Retry-After` header is set.

## Idempotency

`POST /api/v1/servers` accepts an `Idempotency-Key` header (Stripe-style). Retries within 24h with the same key return the cached response (status + body). Use one for every retried server-creation operation in your automation.

## Step-up authentication

API key creation and rotation require recent password re-verification. POST your current password to `/api/v1/account/verify-password` (session auth, not bearer-token) to stamp `last_password_verified_at`. After that, sensitive operations succeed for 5 minutes.

This protects against session-stealing attacks: an attacker with a leaked cookie cannot mint a long-lived API key without also knowing the password.

## Audit log

Every API call writes one row to the audit log. Read it via `GET /api/v1/account/audit`:

- Paginated by ts cursor (`?limit=50&cursor=...`)
- Filterable by `key_id`, `resource_type`, `resource_id`, `action`, `result`
- Plan-based retention: 365 days for Pro, 30 days for Free
- Append-only: we cannot edit history server-side

## Securing your keys

- Store in your secret manager (1Password, Vault, AWS Secrets Manager, Doppler). Never in `.env` committed to git.
- Use a separate key per integration (Ansible, CI, Terraform). Revoking one does not disrupt the others.
- Set an `expires_at` on short-lived CI keys.
- If a key leaks: revoke immediately at `DELETE /api/v1/account/keys/{id}` or via the dashboard.
- GitHub secret-scanning partner registration for `gmk_acct_live_` and `gmk_cru_live_` prefixes is queued; once active, accidentally-committed keys auto-revoke.

## Errors

Standard envelope:

```
{
  "error": "machine_readable_code",
  "message": "Human-readable explanation",
  "request_id": "req_abc123",
  "documentation_url": "https://glassmkr.com/docs/api/errors/{code}"
}
```

Use `request_id` when contacting [support](mailto:support@glassmkr.com); we correlate it against the audit log and application logs.

Last verified: 2026-06-26 against the live API.

---

## Automated fleet onboarding
Source: https://glassmkr.com/docs/automated-onboarding (Markdown: https://glassmkr.com/docs/automated-onboarding.md)

# Automated fleet onboarding

One write-scoped account key, zero per-host secrets. The `glassmkr-crucible enroll` subcommand (Crucible 0.13.21 and later) lets every host self-register, receive its own collector key, and start reporting, with no dashboard clicks in the loop.

## Overview

The single-server flow in [Getting started](/docs/getting-started) creates the server in the dashboard first (to get a collector key), then installs the agent with that key. That is a manual step per host.

For a fleet, `enroll` collapses those two steps into one command that runs on the host itself. You bake a single `write`-scoped account key (`gmk_acct_live_`) into your automation. Each host runs `enroll` once; the command registers the server with the Dashboard, receives that server's own collector key (`gmk_cru_live_`), writes the agent config, and starts the service.

> **Prefer the raw API?** `enroll` is a convenience wrapper over the same endpoints documented in the [Programmatic API quickstart](/docs/programmatic-api) (`POST /api/v1/servers` plus the install). Use the API directly when your control plane, not the host, decides identity and holds the collector keys.

## How enroll works

A single `enroll` run does four things on the host:

1. **Derives a stable machine ID.** It reads the DMI product UUID (`/sys/class/dmi/id/product_uuid`), falling back to `/etc/machine-id` where DMI is unavailable (many VMs and containers). This ID is the host's durable identity.
2. **Registers the server.** It calls the Dashboard with your account key, creating a server keyed by that machine ID (or re-attaching to the existing one), and applies any `--tags`.
3. **Writes only the collector key.** The Dashboard returns a per-server collector key (`gmk_cru_live_`). `enroll` writes that key (and only that key) to `/etc/glassmkr/crucible.yaml` (mode 0600), then installs and starts the `glassmkr-crucible` systemd unit, the same on-disk result as `glassmkr-crucible init`.
4. **Discards the account key.** The account key is used for the one registration call and is never written to disk. See [Where the keys live](#keys).

By default `enroll` verifies connectivity to the Dashboard after registering (the same check `init` runs). Pass `--no-verify` to skip that round-trip so the command returns immediately; the agent still verifies on its first snapshot a minute later. That is the right choice inside a boot sequence, where the network may not be fully up yet (see [cloud-init](#cloud-init)).

## Prerequisites

- A `write`-scoped account key (`gmk_acct_live_`), minted once. See [Programmatic API](/docs/programmatic-api) for how to create one (it needs a step-up password re-auth). Store it in your automation's secret store, not in a machine image.
- Crucible 0.13.21 or later on the host (the release that introduced `enroll`). Installed globally with `npm install -g @glassmkr/crucible`, which needs Node.js 24+.
- Outbound HTTPS on port 443 to `app.glassmkr.com`. No inbound ports.
- Node-quota headroom: each enrolled host consumes one node against your plan quota (3 on Free, your subscribed count on Pro).

## Ansible

Install the agent, then enroll. The `creates:` guard makes the enroll task a no-op on hosts that already have `/etc/glassmkr/crucible.yaml`, so re-running the playbook is safe and cheap.

```
- name: Install the Crucible agent globally
  community.general.npm:
    name: "@glassmkr/crucible"
    global: true
    state: present

- name: Enroll this host into Glassmkr (idempotent)
  ansible.builtin.command:
    cmd: glassmkr-crucible enroll --account-key "{{ vault_glassmkr_account_key }}" --tags "{{ group_names | join(',') }}"
  args:
    creates: /etc/glassmkr/crucible.yaml
  no_log: true
```

- `vault_glassmkr_account_key` is your `gmk_acct_live_` key, held in Ansible Vault. It is passed for the single registration call and never written to a file on the host.
- `group_names | join(',')` tags each host with the inventory groups it belongs to, so the dashboard grouping mirrors your inventory.
- `no_log: true` keeps the key out of Ansible's task output and logs.

## cloud-init

Enroll from `runcmd` so a host registers itself the first time it boots:

```
#cloud-config
runcmd:
  - npm install -g @glassmkr/crucible
  - glassmkr-crucible enroll --account-key "gmk_acct_live_..." --no-verify
```

`--no-verify` keeps enrollment from blocking on a Dashboard round-trip while the network is still coming up during boot. The agent verifies on its first push.

The `npm install` line assumes Node.js 24+ is already on the image. If it is not, install it first (a preceding `runcmd` step, or a base image with Node bundled). And treat cloud-init user-data as sensitive: it often stays readable via the instance metadata service, so pull the account key from a secret manager in `runcmd` rather than hardcoding it wherever your platform supports that.

## Post-install one-liner

Drop this at the end of any post-install, kickstart, preseed, or provisioning script:

```
sudo npm install -g @glassmkr/crucible && \
  sudo glassmkr-crucible enroll --account-key "gmk_acct_live_..." --tags "prod,web"
```

Same effect as the Ansible and cloud-init flows: install the agent, self-register by machine ID, write the collector key, and start the service.

## Idempotent by machine ID

Because the server is keyed by the host's stable machine ID, running `enroll` more than once is safe:

- **Re-runs re-attach, they do not duplicate.** A second `enroll` on an already-enrolled host maps back to the same server record. With the Ansible `creates:` guard, the command is skipped outright once `/etc/glassmkr/crucible.yaml` exists.
- **Re-images map back to the same server.** Rebuild or re-image a host that keeps the same machine identity and it re-attaches to its existing server, preserving history and alert state instead of creating a new node (and consuming another quota slot).
- **Keep machine identity unique per host.** Hosts that share a machine ID would collapse onto one server record. When you clone from a golden image, make sure each instance gets a distinct identity: regenerate `/etc/machine-id` on first boot (cloud-init does this by default) so hosts with no distinct DMI UUID do not collide.

## Where the keys live

The security property of `enroll` is that the powerful credential never touches the host disk. Only a narrowly-scoped, per-host credential does.

| Key | Prefix | Where it lives in this flow | Scope |
| --- | --- | --- | --- |
| **Account key** | `gmk_acct_live_` | Your automation's secret store only (Ansible Vault, a secret manager, a CI secret). Never written to disk by `enroll`. | Account-wide: can create servers across the account. Keep it in the control plane. |
| **Collector key** | `gmk_cru_live_` | `/etc/glassmkr/crucible.yaml` on the host (mode 0600, readable by the non-root `glassmkr` user). | One server's telemetry only. Cannot list other servers, create servers, or read account settings. |

So a compromised host leaks only its own collector key, scoped to that one server's telemetry, not the account key that can enroll the rest of your fleet. If a collector key is exposed, rotate it with `POST /api/v1/servers/{id}/rotate-key`; if the account key is exposed, revoke it from Settings or `DELETE /api/v1/account/keys/{id}`.

Last updated: 2026-07-14. `enroll` was introduced in Crucible 0.13.21.

## Next steps

- [Getting started](/docs/getting-started): the single-server flow, and the two kinds of keys explained.
- [Programmatic API](/docs/programmatic-api): the raw endpoints behind `enroll`, plus a 50-server provisioning loop, rate limits, and the audit log.
- [Configuration reference](/docs/configuration): everything you can set in `/etc/glassmkr/crucible.yaml`.
- [Alert rules](/docs/rules): the 65 rules the Dashboard evaluates on every snapshot.

---

## API reference
Source: https://glassmkr.com/docs/api (Markdown: https://glassmkr.com/docs/api.md)

# API reference

Base URL: `https://app.glassmkr.com/api/v1`. All requests and responses use JSON over HTTPS.

Authenticated endpoints require a Bearer token:

```
Authorization: Bearer YOUR_API_TOKEN
```

Error responses follow a consistent envelope:

```
{
  "error": "short_code",
  "message": "Human-readable description of what went wrong."
}
```

For programmatic-API specifics (account keys, scopes, audit log) see the [Programmatic API](/docs/programmatic-api) page. For which endpoints are Free vs Pro and how 402 responses behave, see the [Tier gating](/docs/api/tier-gating) page.

## Authentication

### Register

POST
`/auth/register`
Public

Create a new Glassmkr account.

**Request body:**

```
{
  "email": "user@example.com",
  "password": "min-12-characters",
  "name": "Jane Doe"
}
```

**Response (201):**

```
{
  "user": {
    "id": "usr_a1b2c3d4",
    "email": "user@example.com",
    "name": "Jane Doe",
    "verified": false,
    "created_at": "2026-04-05T10:00:00Z"
  },
  "token": "eyJhbGciOiJIUzI1NiIs..."
}
```

A verification email is sent automatically. The account is fully functional before verification, but some features (team invites) require a verified email.

### Login

POST `/auth/login` Public

Authenticate and receive a session token.

```
{
  "email": "user@example.com",
  "password": "min-12-characters"
}
```

**Response (200):**

```
{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "expires_at": "2026-04-12T10:00:00Z"
}
```

Session tokens are valid for 7 days. **Error (401)** on bad credentials: `{ "error": "invalid_credentials", "message": "Email or password is incorrect." }`.

### Logout

POST `/auth/logout` Authenticated

Invalidate the current session token. Response (204) no content.

### Get current user

GET `/auth/me` Authenticated

Returns the authenticated user's profile.

```
{
  "id": "usr_a1b2c3d4",
  "email": "user@example.com",
  "name": "Jane Doe",
  "verified": true,
  "role": "owner",
  "created_at": "2026-04-05T10:00:00Z",
  "servers_count": 6,
  "plan": "pro"
}
```

### Verify email

POST `/auth/verify` Public

Confirm an email address using the token from the verification email.

```
{ "token": "verify_abc123def456" }
```

## Servers

### Register server

POST `/servers` Authenticated

Register a new server. Returns a fresh collector key.

**Request body:** `name`, `hostname`, `tags`, and `profile` are accepted. Hardware fields (OS, architecture, core count, RAM) are reported by the agent on each ingest, never on registration.

```
{
  "name": "web-prod-01",
  "hostname": "web-prod-01.example.com",
  "tags": ["production", "web"]
}
```

`name` is required (1-100 chars). `hostname` defaults to `name` and must be a valid RFC 1035 hostname. `tags` is optional, max 20 strings of 1-50 chars each. `profile` is optional: a host-type profile that suppresses the alerts expected by design for that kind of host. The field is named `profile` (not `host_type`); accepted values are `null` (the default "General", no suppression) or `"marketplace_gpu"` (a rented marketplace GPU box, which silences `no_firewall`, `unattended_upgrades_disabled`, and `gpu_power_cap_throttling`). An unknown value returns `400 validation_failed` with `profile must be null or one of: ...`. Other fields are silently dropped (mass-assignment defense).

**Response (201):**

```
{
  "success": true,
  "server": {
    "id": "srv_a1b2c3d4",
    "name": "web-prod-01",
    "hostname": "web-prod-01.example.com",
    "tags": ["production", "web"],
    "api_key": "gmk_cru_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_aBcD"
  },
  "ingest_url": "https://app.glassmkr.com/api/v1/ingest",
  "message": "Save your collector key. It will not be shown again."
}
```

The collector key is shown **once**. Configure it on the agent before the dashboard tile leaves "pending first snapshot". The `Idempotency-Key` header is supported (24h replay window).

### List servers

GET `/servers` Authenticated

List all servers in the account.

**Query parameters:**

| Param | Type | Description |
| --- | --- | --- |
| `tag` | string | Filter by tag. Repeat for multiple tags (AND logic). |
| `limit` | int | Page size, 1-100 (default 100). |
| `cursor` | string | Opaque pagination cursor returned as `next_cursor` on the previous page. |

**Response (200):**

```
{
  "servers": [
    {
      "id": "srv_a1b2c3d4",
      "name": "web-prod-01",
      "hostname": "web-prod-01.example.com",
      "ip": "10.0.1.42",
      "os_type": "ubuntu",
      "os_version": "24.04 LTS",
      "status": "active",
      "suspended_at": null,
      "suspended_reason": null,
      "last_seen_at": "2026-05-09T07:00:00Z",
      "collector_version": "0.13.3",
      "active_alerts": 0,
      "disk_health_rollup": "healthy",
      "created_at": "2026-04-05T10:00:00Z",
      "tags": ["production", "web"],
      "dmi_vendor": "GIGABYTE",
      "dmi_product": "R292-4S1-00",
      "ipmi_sensors_count": 106
    }
  ],
  "next_cursor": null
}
```

Per-snapshot metrics (CPU usage, RAM usage, disk usage) are not on the list endpoint. Use `GET /servers/:id/health` for the latest snapshot from a specific server.

`status` is `active` for normal operation, `suspended` when the server is disabled (see [Billing](#billing)). `disk_health_rollup` is the worst per-drive state across all SMART-monitored drives: `healthy`, `declining`, `failing`, or `broken`.

### Get server

GET `/servers/:server_id` Authenticated

Get full details for a single server. Same shape as the list endpoint plus a few read-only fields (`config_overrides`, `free_analysis_used`).

### Update server

PATCH `/servers/:server_id` Authenticated

Update `name`, `tags`, or `profile` (same values as on registration; send `"profile": null` to clear it). `hostname` is intentionally not updatable so ops can find a box by hostname after a rename.

```
{
  "name": "web-prod-renamed",
  "tags": ["production", "web", "fra1"]
}
```

### Delete server

DELETE `/servers/:server_id?confirm=true` Authenticated

Remove a server and all its stored metrics. Irreversible. `?confirm=true` is required; a bare DELETE returns 400.

Per-endpoint sub-limit: 100 deletes/hour/account.

### Rotate collector key

POST `/servers/:server_id/rotate-key` Authenticated

Issue a fresh collector key for an existing server. The previous key stops working immediately. Update `/etc/glassmkr/crucible.yaml` (legacy installs: `/etc/glassmkr/collector.yaml`; the agent reads either) on the agent host and restart the service before the next ingest cycle to avoid a gap.

```
{
  "success": true,
  "server": { "id": "srv_a1b2c3d4" },
  "collector_key": "gmk_cru_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_aBcD",
  "rotated_at": "2026-05-09T07:30:00Z",
  "message": "Save this collector key. It will not be shown again."
}
```

Rate-limited to 10/hour/account. Note: the field name on this endpoint is `collector_key`, not `api_key` as on POST /servers.

### Restore server

POST `/servers/:server_id/restore` Authenticated

Restore a single suspended server. Used when a server was disabled because no payment method was on file. Requires a card on file at call time (unless the account is exempt).

**Error (400)** when no card is on file: `{ "error": "no_card_on_file", "message": "Add a payment method in Settings to restore this server." }`.

### Restore all suspended servers

POST `/servers/restore-all` Authenticated

Bulk-restore every server suspended for `no_card_on_file`. Used by the dashboard's Settings -> Disabled servers -> Restore all button.

## Ingest

### Push snapshot

POST `/ingest` Collector key

Submit a Crucible snapshot. Called by the agent every collection interval (default 60 seconds for Crucible v0.10.0+). Authenticated by the collector key in the `Authorization: Bearer gmk_cru_live_...` header. Rate-limited to one ingest per server per 55 seconds; subsequent calls return 429.

**Request body** (abbreviated; the agent emits the full Snapshot type):

```
{
  "system":   { "hostname": "web-prod-01", "ip": "10.0.1.42",
                "os": "Ubuntu 24.04 LTS", "os_id": "ubuntu",
                "kernel": "6.8.0-31-generic", "uptime_seconds": 86400 },
  "cpu":      { "user_percent": 15.2, "system_percent": 5.3,
                "iowait_percent": 1.1, "idle_percent": 78.4,
                "load_1m": 0.4, "load_5m": 0.6, "load_15m": 0.5,
                "cores": [{ "core": 0, "user_percent": 20.1, "system_percent": 4.2,
                            "iowait_percent": 0.5, "idle_percent": 75.2 }] },
  "memory":   { "total_mb": 65536, "used_mb": 44032,
                "available_mb": 21504, "swap_total_mb": 8192,
                "swap_used_mb": 0 },
  "disks":    [{ "device": "/dev/nvme0n1p2", "mount": "/",
                 "total_gb": 500, "used_gb": 225, "available_gb": 250,
                 "percent_used": 47, "fstype": "ext4",
                 "io_read_mb_s": 15.2, "io_write_mb_s": 3.8,
                 "latency_p99_ms": 0.4,
                 "inodes_total": 32768000, "inodes_used": 1245000 }],
  "smart":    [{ "device": "/dev/nvme0n1", "model": "Samsung 990 Pro 2TB",
                 "health": "PASSED", "temperature_c": 38,
                 "percentage_used": 12, "power_on_hours": 8760 }],
  "network":  [{ "interface": "eth0", "speed_mbps": 10000,
                 "rx_bytes_sec": 125000, "tx_bytes_sec": 42000,
                 "rx_errors": 0, "tx_errors": 0,
                 "rx_drops": 0, "tx_drops": 0 }],
  "raid":     [],
  "ipmi":     { "available": true, "sel_entries_count": 12,
                "ecc_errors": { "correctable": 0, "uncorrectable": 0 },
                "sensors": [{ "name": "CPU1_TEMP", "value": 52, "unit": "C",
                              "status": "ok", "type": "temperature",
                              "upper_critical": 90 }] },
  "os_alerts": { "oom_kills_recent": 0, "zombie_processes": 0,
                 "time_drift_ms": 0 },
  "thermal":  { "available": true, "source": "hwmon coretemp Package id 0",
                "max_cpu_celsius": 52,
                "cpu_readings": [{ "chip": "coretemp-isa-0000",
                                    "label": "Package id 0", "celsius": 52 }] },
  "dmi":      { "available": true, "vendor": "supermicro",
                "raw_vendor": "Supermicro Inc.",
                "product_name": "SYS-1029P-WTR",
                "bios_version": "3.4", "bios_date": "2023-01-12",
                "is_virtual": false },
  "gpu":      { "available": true, "tier": "nvidia-smi",
                "devices": [{ "index": 0, "name": "NVIDIA L4",
                              "temperature_c": 48, "utilization_percent": 12,
                              "memory_used_mb": 4096, "memory_total_mb": 24564 }] },
  "collector_version": "0.13.3",
  "timestamp": "2026-05-22T07:00:00Z"
}
```

Optional top-level blocks: `security`, `zfs`, `io_errors`, `io_latency`, `conntrack`, `systemd`, `ntp`, `file_descriptors`, `thermal`, `dmi`, `gpu`, `expected_reboot`. Unknown fields are accepted via `passthrough`; new collector versions can extend the schema without a coupled Dashboard release.

**Response (200):**

```
{
  "success": true,
  "received_at": "2026-05-22T07:00:00.123Z",
  "new_alerts": 0,
  "active_alerts": 0
}
```

## Health

### Get server health

GET `/servers/:server_id/health` Authenticated

Current health status and latest metric values for a server.

```
{
  "server_id": "srv_a1b2c3d4",
  "status": "healthy",
  "last_seen": "2026-04-05T10:05:00Z",
  "current": {
    "cpu_percent": 21.6,
    "ram_percent": 67.2,
    "swap_used_mb": 0,
    "disk_max_percent": 45.0,
    "network_rx_mbps": 120.5,
    "network_tx_mbps": 40.2,
    "cpu_temp_c": 52,
    "active_alerts": 0
  }
}
```

### Get health history

GET `/servers/:server_id/health/history` Authenticated

Time-series metric data.

| Param | Type | Description |
| --- | --- | --- |
| `metric` | string | `cpu`, `memory`, `disk`, `network`, `temperature`. |
| `from` | ISO 8601 | Start time (default 1 hour ago). |
| `to` | ISO 8601 | End time (default now). |
| `resolution` | string | `1m`, `5m`, `1h`, `1d` (auto if omitted). |

### Get server alerts

GET `/servers/:server_id/alerts` Authenticated

**Defaults to `status=all`, which includes resolved history**; pass `?status=active` for only the alerts currently firing. Query params: `status` (`active`, `resolved`, `all`; default `all`), `severity` (`critical`, `warning`, `info`), `from`, `to`, `page`.

## Channels

### Create channel

POST `/channels` Write scope

Create a notification channel. Supported types: `email`, `telegram`, `slack`, `discord`, `pagerduty`, `webhook`.

```
{
  "name": "ops-telegram",
  "type": "telegram",
  "config": {
    "bot_token": "7123456789:AAH1bGciOiJSUzI1NiIs",
    "chat_id": "-1001234567890"
  }
}
```

Email config takes `recipients`; Slack and Discord take `webhook_url`; PagerDuty takes `routing_key`; webhook takes `url` and optional `secret`.

### List channels

GET `/channels` Authenticated

### Get channel

GET `/channels/:channel_id` Authenticated

Sensitive fields like bot tokens are partially masked in GET responses.

### Update channel

PUT `/channels/:channel_id` Write scope

### Delete channel

DELETE `/channels/:channel_id` Write scope

### Test channel

POST `/channels/:channel_id/test` Write scope

Send a test notification through the channel. **Error (502)** on delivery failure with the upstream error message.

## Alerts

### Acknowledge alert

POST `/alerts/:alert_id/acknowledge` Write scope

Silence notifications for the current occurrence; does not disable the rule. Event-type alerts (e.g. `unexpected_reboot`) auto-clear acknowledgement when a new occurrence stacks onto the card.

### Resolve alert

POST `/alerts/:alert_id/resolve` Write scope

Manually resolve an alert without waiting for the underlying condition to clear. Mostly used for event-type alerts (24-hour TTL otherwise) and for force-clearing stuck state alerts.

### List muted rules

GET `/servers/:server_id/mutes` Authenticated

### Mute a rule

POST `/servers/:server_id/mutes` Write scope

Mute one alert rule for this server, one rule per request. Any currently active alert of that type is resolved immediately. Returns the updated `muted_rules` list.

```
{ "alert_type": "disk_space_high" }
```

### Unmute a rule

DELETE `/servers/:server_id/mutes` Write scope

Remove one rule from the muted list, one rule per request. Returns the updated `muted_rules` list.

```
{ "alert_type": "disk_space_high" }
```

## Billing

### Billing status

GET `/billing/status` Authenticated

Returns plan, billing-period bounds, payment-method state, and the count of servers currently disabled for missing card.

```
{
  "plan": "pro",
  "has_default_payment_method": true,
  "current_period_end": "2026-06-09T00:00:00Z",
  "cancel_at_period_end": false,
  "billing_enforcement_exempt": false,
  "active_server_count": 7,
  "suspended_no_card_count": 0,
  "free_server_quota": 3
}
```

Pro customers without a payment method see `has_default_payment_method: false`. The grace period is the later of `current_period_end` and `customer.created_at + 30 days`; after that, servers beyond the free quota are suspended.

### Other billing endpoints

- `POST /billing/checkout`: create a Stripe Checkout session for plan upgrade.
- `POST /billing/portal`: create a Stripe Customer Portal session.
- `POST /billing/resume`: re-enable auto-renew on a cancelled subscription.
- `POST /billing/downgrade`: schedule a downgrade to Free at period end.

## Meta

### Version

GET `/version` Public

Returns the latest published Crucible version and the minimum supported version.

```
{
  "crucible": {
    "latest": "0.13.3",
    "min_supported": "0.7.0",
    "changelog_url": "https://github.com/glassmkr/crucible/releases"
  },
  "dashboard": { "version": "1.0.0" }
}
```

The `latest` value is sourced from the npm registry's `@glassmkr/crucible` `latest` dist-tag.

## Rate limits

Token-bucket limiter applied as four overlapping tiers (first failure wins; failures still cost a token on the per-IP debit so brute-force probing burns budget):

| Tier | Capacity | Refill | Applies to |
| --- | --- | --- | --- |
| Per-IP | 100 | 10/sec | Every request, including pre-auth. |
| Per-key | 1000 | 100/sec | Authenticated requests, scoped to one collector or account key. |
| Per-account | 5000 | 500/sec | All authenticated requests within one customer. |
| POST /servers | 100 | 100/hour | Server registration sub-limit. |
| DELETE /servers/:id | 100 | 100/hour | Deletion sub-limit. |
| POST /servers/:id/rotate-key | 10 | 10/hour | Key-rotation sub-limit. |

The ingest endpoint enforces a per-server soft limit of one push per 55 seconds (returns 429, separate from the token-bucket layer). When token-bucket-rate-limited, the API returns `429 Too Many Requests` with a `Retry-After` header.

## Pagination

List endpoints (currently `GET /servers`) use opaque cursor pagination: pass `?limit=` (1-100, default 100) and the previous response's `next_cursor` as `?cursor=`. `next_cursor` is `null` on the final page.

## Idempotency

`POST /servers` honors an `Idempotency-Key` header (1-255 printable ASCII). The first response (success or deterministic 4xx) is cached for 24 hours; replays return the cached response with an `Idempotency-Replayed: true` header. Concurrent retries with the same key while the original is still in flight return 409.

Last verified: 2026-05-22 against Crucible v0.13.3 and Dashboard v1.0. For tier-gating details see [/docs/api/tier-gating](/docs/api/tier-gating).

---

## API tier gating
Source: https://glassmkr.com/docs/api/tier-gating (Markdown: https://glassmkr.com/docs/api/tier-gating.md)

# API tier gating

The full programmatic API (account API keys, read and write) is on every plan. Pro gates exactly three things: more than 3 nodes, retention beyond 7 days, and unlimited AI analysis. The web dashboard is unaffected either way.

## Overview

- **Free tier:** the full read+write API (keys carry read, write, or admin scope), all 65 rules, all six notification channels, predictive trend warnings, and full web-dashboard access. Up to 3 nodes, 7-day metric retention, one trial AI analysis per server.
- **Pro tier:** every Free capability, plus more than 3 nodes (at $3 per node per month), 90-day metric retention, and unlimited AI analysis.

The API itself is not tier-gated by HTTP verb anymore. A Free account can create servers, manage channels, acknowledge alerts, and rotate keys programmatically, bounded only by the 3-node quota and the standard rate limits. The only endpoints that return a Pro prompt are the ones that drive a Pro-only feature (additional AI analyses once the per-server trial is used).

## What is Free

The full programmatic API for your own account, read and write. Account API keys can be minted with read, write, or admin scope. Everything below is available on a Free plan, bounded by the 3-node quota and the rate limits:

- All reads: `GET /api/v1/servers`, `GET /api/v1/servers/{id}`, `/alerts`, `/health`, `/history`, `/metrics`, `/disk-health`.
- Channel CRUD: `GET / POST /api/v1/channels`, `PUT / DELETE /api/v1/channels/{id}`, `POST /api/v1/channels/{id}/test` (all six channel types).
- Alert mutations: `POST /api/v1/alerts/{id}/acknowledge`, `POST /api/v1/alerts/{id}/resolve`, and `GET / POST / DELETE /api/v1/servers/{id}/mutes`.
- Server management: `POST /api/v1/servers` (up to the 3-node quota), `PATCH`, `DELETE`, `POST /api/v1/servers/{id}/rotate-key`, `/restore`, `/restore-all`.
- Trend warnings: `GET /api/v1/servers/{id}/trend-warnings`, `POST / DELETE /api/v1/trend-warnings/{id}/feedback`, `GET /api/v1/trend-warnings/track-record`.
- Account management: `GET /api/v1/account/audit` (30-day retention on Free), `GET / POST /api/v1/account/keys`, `POST /api/v1/account/keys/{id}/rotate`, `DELETE /api/v1/account/keys/{id}`.
- One trial AI analysis per server: `POST /api/v1/servers/{id}/analyze`, `GET /api/v1/servers/{id}/analyses`.
- All auth endpoints (`/auth/login`, `/auth/register`, `/auth/me`, etc.) and all billing endpoints (`/billing/checkout`, `/billing/status`, etc.).
- Crucible agent ingest: `POST /api/v1/ingest` (collector key, never gated).
- `GET /api/v1/version` and the public health probe `GET /api/v1/health` (unauthenticated).

## What is Pro

Pro is not a different API surface. It lifts three limits on the same surface:

- **More than 3 nodes.** Past the free quota, each node is $3 per month. `POST /api/v1/servers` beyond the quota requires an active Pro subscription with a card on file.
- **Retention beyond 7 days.** Pro retains 90 days of full metric history (Free retains 7); the audit log retains 365 days on Pro versus 30 on Free.
- **Unlimited AI analysis.** Free gets one trial analysis per server; after that, additional calls to `POST /api/v1/servers/{id}/analyze` require Pro.

Notification channels, the read+write API, and trend warnings are *not* Pro; they are on every plan.

## What a 402 response looks like

When a Free-tier request crosses one of the three Pro limits (for example, an extra AI analysis after the per-server trial is used, or a server beyond the 3-node quota), Glassmkr returns 402 Payment Required:

```
HTTP/1.1 402 Payment Required
Content-Type: application/json

{
  "error": "pro_required",
  "message": "This requires the Pro plan. Visit app.glassmkr.com/settings to upgrade.",
  "upgrade_url": "https://app.glassmkr.com/settings",
  "documentation_url": "https://glassmkr.com/docs/programmatic-api"
}
```

The shape is stable; `error` is the machine-readable code, `message` is human-readable, `upgrade_url` points to the billing flow. Integrators on the Free tier should handle 402 explicitly and route it to an upgrade-prompt UI rather than a generic auth-error path. Note that most of the API never returns 402 on Free; it is reserved for the three Pro limits above.

402 is distinct from 403 (insufficient_scope) and 401 (auth_failed); the codes are stable and safe to branch on.

## Web dashboard

The web dashboard (session cookie auth) is fully functional on every plan, exactly like the API. Free customers see the full UI; the three Pro limits render upgrade prompts where appropriate (an extra AI analysis, a fourth node, longer retention), but nothing else is withheld.

The API and the dashboard apply the same three limits. There is no API-only or dashboard-only gating.

## Free trial quotas

The one Pro feature with a Free trial quota is AI analysis:

- **AI analysis**: each Free server gets one trial analysis. After that, additional analyses require Pro. The trial counter is per-server and persistent, and applies identically whether you trigger it from the dashboard or via `POST /api/v1/servers/{id}/analyze`.

Once a server's trial is used, further analyze calls return 402 immediately without re-consuming the trial budget.

## How we keep this honest

A CI lint (`pnpm lint:tier-gating`) enforces that every endpoint under `/api/v1` declares its tier, either via inline marker or via the explicit allowlist at `scripts/tier-gating-allowlist.json`. CI fails if any new endpoint ships without a declaration. There should be no silent leakage between Free and Pro.

## Related pages

- [API reference](/docs/api): full endpoint catalog
- [Programmatic API](/docs/programmatic-api): keys, scopes, rate-limit tiers, idempotency, audit log
- [Pricing](/pricing): plan comparison

Last verified: 2026-05-22.

---

## Configuration reference
Source: https://glassmkr.com/docs/configuration (Markdown: https://glassmkr.com/docs/configuration.md)

# Configuration reference

The Crucible agent reads `/etc/glassmkr/crucible.yaml` (legacy installs may have it at `/etc/glassmkr/collector.yaml`; the agent reads either, preferring the new name). This page documents every option for Crucible v0.13.6.

After editing the configuration, restart the service:

```
sudo systemctl restart glassmkr-crucible
```

You can validate the configuration without restarting:

```
glassmkr-crucible config check
```

## Full example

Below is a complete file with all options at their default values. You only need to include the fields you want to change.

```
server_name: "my-server"

collection:
  interval_seconds: 60
  ipmi: true
  smart: true

dashboard:
  enabled: true
  url: "https://app.glassmkr.com"
  api_key: "gmk_cru_live_your_key_here"
  tags:
    - production
    - web
  location: "eu-west"

collectors:
  interval: 60
  cpu:
    enabled: true
    per_core: false
  memory:
    enabled: true
  disk:
    enabled: true
    include_mounts:
      - /
      - /home
      - /var
    exclude_mounts:
      - /mnt/backup
    exclude_fs_types:
      - tmpfs
      - devtmpfs
      - squashfs
  network:
    enabled: true
    include_interfaces: []
    exclude_interfaces:
      - lo
      - docker0
      - veth*
  smart:
    enabled: true
    devices: []
    ignore_devices:
      - /dev/loop*
  thermal:
    enabled: true
    source: auto
  raid:
    enabled: true
    arrays: []
  ipmi:
    enabled: auto
    tool: ipmitool
  gpu:
    enabled: auto

alerts:
  ram_high:
    enabled: true
    threshold: 90
    duration: 300
  cpu_high:
    enabled: true
    threshold: 90
    duration: 300
  disk_space_high:
    enabled: true
    threshold: 90
    critical_threshold: 95
    exclude_mounts: []
  cpu_iowait_high:
    enabled: true
    threshold: 20
    duration: 180
  oom_kills:
    enabled: true
    threshold: 1
  smart_failing:
    enabled: true
    ignore_disks: []
  nvme_wear_high:
    enabled: true
    threshold: 80
  raid_degraded:
    enabled: true
    arrays: []
  disk_latency_high:
    enabled: true
    threshold: 50
    duration: 120
    exclude_devices: []
  interface_errors:
    enabled: true
    threshold: 10
    exclude_interfaces: []
  link_speed_mismatch:
    enabled: true
    interfaces: {}
  interface_saturation:
    enabled: true
    threshold: 80
    duration: 60
    exclude_interfaces: []
  cpu_temperature_high:
    enabled: true
    threshold: 85
    critical_threshold: 95
    sensor: ""
  ecc_errors:
    enabled: true
    correctable_rate_warning: 10
    rate_window_hours: 24
    critical_on_uncorrectable: true
  psu_redundancy_loss:
    enabled: true
    source: ""
  ipmi_fan_failure:
    enabled: true
    min_rpm: 500
  ipmi_sel_critical:
    enabled: true
  load_high:
    enabled: true
    threshold: 0
    duration: 300
  filesystem_readonly:
    enabled: true
    exclude_mounts: []
  inode_high:
    enabled: true
    threshold: 90
    exclude_mounts: []
  clock_drift:
    enabled: true
    threshold: 5
  ssh_root_password:
    enabled: true
  no_firewall:
    enabled: true
  pending_security_updates:
    enabled: true
  kernel_vulnerabilities:
    enabled: true
  kernel_needs_reboot:
    enabled: true
  unattended_upgrades_disabled:
    enabled: true
  muted_rules: []
  global:
    cooldown: 3600
    resolve_notify: true
    channels:
      - telegram
      - email

logging:
  level: info
  file: /var/log/glassmkr/crucible.log
  max_size: 50
  max_files: 5

proxy:
  http: ""
  https: ""
  no_proxy: ""

tls:
  ca_cert: ""
  skip_verify: false
```

## `dashboard`

Connection settings for the Dashboard API.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | boolean | `false` | Enable pushing data to the Dashboard. |
| `url` | string | `https://app.glassmkr.com` | Dashboard API base URL. |
| `api_key` | string | *required* | Server collector key (`gmk_cru_live_xxx`; older agents may still have `col_xxx`). Created when you add a server via + Add Server. Rotate via `POST /api/v1/servers/{id}/rotate-key`. |
| `timeout` | int | `30` | HTTP request timeout in seconds. |
| `retry_count` | int | `3` | Number of retries on failed pushes before giving up. |
| `retry_delay` | int | `5` | Seconds between retries. Uses exponential backoff (delay * attempt). |

## `server`

Server identity and metadata.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `name` | string | system hostname | Display name for this server in the Dashboard. If empty, the system hostname is used. |
| `tags` | list | `[]` | Arbitrary tags for organizing servers. Use these to filter servers in the dashboard. |
| `location` | string | `""` | Freeform location label (e.g., `eu-west`, `dc3-rack14`). |

## `collectors`

Controls which metrics are collected and how.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `interval` | int | `60` | Collection interval in seconds. Default 60 (Crucible v0.10.0+). Minimum 10, maximum 600. The previous 300-second default predates v0.10.0. |

### `collectors.cpu`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `true` | Collect CPU metrics (user, system, iowait, idle, steal). |
| `per_core` | bool | `false` | Report per-core metrics in addition to aggregate. Requires Crucible 0.3.0+. Enables the per-core CPU chart in the dashboard expanded view and gives AI analysis per-core awareness. Increases data volume on high-core-count machines. |

### `collectors.memory`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `true` | Collect memory metrics (total, available, used, buffers, cache, swap). |

### `collectors.disk`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `true` | Collect disk space and I/O metrics. |
| `include_mounts` | list | `[]` | If set, only these mount points are monitored. Empty means all mounts (after exclusions). |
| `exclude_mounts` | list | `[]` | Mount points to skip. |
| `exclude_fs_types` | list | `[tmpfs, devtmpfs, squashfs]` | Filesystem types to skip. |

### `collectors.network`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `true` | Collect network interface metrics (bytes, packets, errors, drops). |
| `include_interfaces` | list | `[]` | If set, only these interfaces are monitored. Supports glob patterns. |
| `exclude_interfaces` | list | `[lo, docker0, veth*]` | Interfaces to skip. Supports glob patterns. |

### `collectors.smart`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `true` | Collect SMART health data from disks. Requires `smartctl`. |
| `devices` | list | `[]` | Specific devices to monitor. Empty means auto-detect all block devices. |
| `ignore_devices` | list | `[/dev/loop*]` | Devices to skip. Supports glob patterns. |

### `collectors.thermal`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `true` | Collect temperature readings from CPU and other sensors. |
| `source` | string | `auto` | Temperature source: `auto`, `hwmon`, or `ipmi`. `auto` prefers hwmon and falls back to IPMI. |

### `collectors.raid`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `true` | Monitor RAID array health. Supports mdadm, MegaCLI, and storcli. |
| `arrays` | list | `[]` | Specific arrays to monitor. Empty means auto-detect. |

### `collectors.ipmi`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | string | `auto` | IPMI sensor collection: `auto` (detect), `true`, or `false`. |
| `tool` | string | `ipmitool` | IPMI tool: `ipmitool` or `freeipmi`. |

### `collectors.gpu`

Three-tier GPU collection (nvidia-smi, DCGM exporter, Redfish OEM). Validated on NVIDIA L4, A4000, and A16. See [the rule catalog](/docs/rules) for the 8 GPU rules.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | string | `auto` | GPU collection: `auto` (detect any available tier), `true`, or `false`. |

## `alerts`

Per-rule threshold overrides. See the [alert rules catalog](/docs/rules) for detailed documentation of each of the 65 rules.

Every alert rule supports at minimum:

| Key | Type | Description |
| --- | --- | --- |
| `enabled` | bool | Enable or disable this rule. |
| `threshold` | number | The value at which the alert fires (unit depends on the rule). |

Some rules also support `duration` (seconds the condition must persist), `critical_threshold` (a second, higher threshold), and exclusion lists. See the full example above for all options.

### `alerts.muted_rules`

A list of rule names to mute on this server. Muted rules are not evaluated and do not fire notifications. Changes take effect on the next ingest cycle after restarting Crucible.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `muted_rules` | list | `[]` | List of rule names to mute (e.g., `[disk_space_high, cpu_iowait_high]`). Can also be managed from the Dashboard. |

### `alerts.global`

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `cooldown` | int | `3600` | Seconds between repeated notifications for the same active alert. |
| `resolve_notify` | bool | `true` | Send a notification when an alert condition resolves. |
| `channels` | list | `[]` | Default notification channels. Override per-rule if needed. |

## `logging`

Controls Crucible's own log output.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `level` | string | `info` | Log level: `debug`, `info`, `warn`, `error`. |
| `file` | string | `/var/log/glassmkr/crucible.log` | Log file path. Set to `stdout` to log to standard output. |
| `max_size` | int | `50` | Maximum log file size in MB before rotation. |
| `max_files` | int | `5` | Number of rotated log files to keep. |

## `proxy`

HTTP proxy settings for environments where outbound internet access requires a proxy.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `http` | string | `""` | HTTP proxy URL (e.g., `http://proxy.internal:3128`). |
| `https` | string | `""` | HTTPS proxy URL. If empty, the HTTP proxy is used for HTTPS as well. |
| `no_proxy` | string | `""` | Comma-separated list of hosts to bypass the proxy. |

## `tls`

TLS settings for the connection to the Dashboard API.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `ca_cert` | string | `""` | Path to a custom CA certificate bundle. Use this if your proxy performs TLS inspection. |
| `skip_verify` | bool | `false` | Skip TLS certificate verification. Not recommended for production. |

## Environment variables

The running agent is configured by `crucible.yaml` only; it does not read configuration overrides from the environment. Two variables exist around the edges of the install flow:

- `GLASSMKR_API_KEY`: read by the install script as an alternative to passing `--api-key` (`curl -sf https://glassmkr.com/install.sh | sudo GLASSMKR_API_KEY=... bash`). The script hands it to `glassmkr-crucible init`, which writes it into `crucible.yaml`; the agent then reads the file.
- `GLASSMKR_UBUNTU_PRO_TOKEN`: optional; read by the agent's CVE collector to query the Ubuntu Pro security feed.

Last verified: 2026-06-11 against Crucible v0.13.9.

---

## Notification channels
Source: https://glassmkr.com/docs/channels (Markdown: https://glassmkr.com/docs/channels.md)

# Notification channels

Glassmkr routes alerts to email, Telegram, Slack, Discord, PagerDuty, and generic webhook destinations. All six are free on every plan, with no cap on how many you configure; filter which priority levels (P1-P4) each one receives.

## Email

The simplest channel. Glassmkr sends styled HTML emails from `alerts@glassmkr.com` with priority badges, diagnostic commands, and a direct link to the alert in the dashboard. SPF, DKIM, and DMARC are configured on the sending domain.

### Setup

1. Go to **Channels** in the dashboard and click **+ Add Channel**.
2. Select the **Email** tab.
3. Enter a channel name and the recipient email address.
4. Choose which priority levels this channel should receive (P1 through P4).
5. Click **Create Channel**.
6. Click **Test** to verify delivery. Check your spam folder if it does not arrive.

## Telegram

Telegram messages arrive instantly and support rich formatting with priority badges and code blocks.

### Setup

1. Open Telegram and start a conversation with **@glassmkr_bot**.
2. Send `/start` to the bot. It replies with your chat ID.
3. For group notifications, add **@glassmkr_bot** to your group, then send `/start` in the group. The bot replies with the group chat ID (a negative number like `-1001234567890`).
4. In the dashboard, go to **Channels**, click **+ Add Channel**, select **Telegram**.
5. Enter the chat ID from step 2 or 3.
6. Choose priority levels and click **Create Channel**.
7. Click **Test** to verify.

## Slack

Slack integration uses incoming webhooks. Each webhook targets a specific Slack channel.

### Setup

1. Go to [api.slack.com/apps](https://api.slack.com/apps) and click **Create New App**.
2. Select **From scratch**. Name the app (e.g., "Glassmkr Alerts") and select your workspace.
3. In the app settings, go to **Incoming Webhooks** and toggle it on.
4. Click **Add New Webhook to Workspace** and select the channel (e.g., #ops-alerts).
5. Copy the webhook URL.
6. In the dashboard, go to **Channels**, click **+ Add Channel**, select **Slack**.
7. Paste the webhook URL, choose priority levels, and click **Create Channel**.
8. Click **Test** to verify.

## Discord

Discord integration uses incoming webhooks, like Slack. Alerts post as rich embeds with severity colors to the channel the webhook targets.

### Setup

1. In Discord, open **Server Settings -> Integrations -> Webhooks** and click **New Webhook**.
2. Pick the channel it should post to, then copy the webhook URL.
3. In the dashboard, go to **Channels**, click **+ Add Channel**, select **Discord**.
4. Paste the webhook URL, choose priority levels, and click **Create Channel**.
5. Click **Test** to verify.

## PagerDuty

PagerDuty integration uses the Events API v2. Glassmkr triggers an incident when an alert fires and resolves it when the alert clears. Alert priority maps to PagerDuty severity.

### Setup

1. In PagerDuty, open the service you want to alert on and go to **Integrations -> Add an integration**.
2. Select **Events API v2** and copy the **Integration Key** (the routing key).
3. In the dashboard, go to **Channels**, click **+ Add Channel**, select **PagerDuty**.
4. Paste the routing key, choose priority levels, and click **Create Channel**.
5. Click **Test** to verify a test incident is created and auto-resolved.

## Webhooks

For routing alerts into your own incident tooling (custom Slack apps, internal ticket systems, anything not covered by the first-class channels above), use a generic webhook. Glassmkr POSTs a JSON payload to the URL you configure.

### Setup

1. In the dashboard, **Channels** -> **+ Add Channel** -> **Webhook**.
2. Enter the endpoint URL and an optional shared-secret header (sent as `X-Glassmkr-Signature`; HMAC-SHA256 of the body).
3. Choose priority levels and click **Create Channel**.
4. Click **Test**. Inspect your endpoint's request log to confirm the payload arrived.

4xx and 5xx responses are retried with exponential backoff for up to one hour, then dropped. Delivery history is visible on the channel card.

## Priority filtering

Each channel has four priority toggles: P1 Urgent, P2 High, P3 Medium, P4 Low. When an alert fires, Glassmkr sends notifications only to channels that have that priority level enabled.

For example, configure a Telegram channel for P1 and P2 only (critical alerts that need immediate attention), and an email channel for all four levels (audit trail).

Change priority settings at any time by clicking **Edit** on any channel card.

## Notification format

All channels receive structured notifications that include:

- **Priority badge**: P1 Urgent (red), P2 High (orange), P3 Medium (amber), P4 Low (blue).
- **Server name and alert rule**: identifies which server and which of the 65 rules fired.
- **Summary**: current value, threshold, and what it means, in human-readable units.
- **Recommendation**: context-aware advisory text explaining the likely cause.
- **Fix commands**: copy-pasteable shell commands with real interface and device names. Every rule ships with deep FIX content (safe-mode, validation, rollback notes).
- **View in Dashboard**: direct link to the server detail page.

When an alert resolves, a resolution notification is sent to the same channels that received the original alert.

## Version update notifications

When a new Crucible version is available, Glassmkr sends a one-off notification listing all servers running an outdated version. This is sent once per customer per release, not per server.

## Testing channels

Click **Test** on any channel card, or use the API:

```
curl -X POST https://app.glassmkr.com/api/v1/channels/CHANNEL_ID/test \
  -H "Authorization: Bearer gmk_acct_live_your_account_key"
```

The account key is created in **Settings -> API keys** (on every plan; this `test` call needs a write-scoped key). See the [Programmatic API](/docs/programmatic-api) page for scopes and rotation.

Last verified: 2026-05-22 against Crucible v0.13.3.

---

## Troubleshooting
Source: https://glassmkr.com/docs/troubleshooting (Markdown: https://glassmkr.com/docs/troubleshooting.md)

# Troubleshooting

Common issues with the Crucible agent and the Glassmkr Dashboard, with step-by-step solutions.

## Topic pages

- [IPMI](/docs/troubleshooting/ipmi): how Crucible detects IPMI, why "Not detected" can be correct behavior, using `glassmkr-crucible doctor ipmi`, per-vendor notes.

## Crucible service fails to start

**Symptom:** `systemctl status glassmkr-crucible` shows `failed` or `inactive (dead)`.

1. Check the service logs: ```
journalctl -u glassmkr-crucible --no-pager -n 50
```
2. If you see a YAML parse error, re-run the init wizard with the same key to rewrite the config from scratch: ```
sudo glassmkr-crucible init --api-key <your_collector_key>
``` The wizard validates the key against the Dashboard before writing the config, so a typo surfaces immediately. Common YAML mistakes include tabs instead of spaces, missing quotes around strings with special characters, and incorrect indentation.
3. If you see `permission denied`, ensure the configuration file is readable: ```
ls -la /etc/glassmkr/crucible.yaml /etc/glassmkr/collector.yaml 2>/dev/null
``` The file should be owned by root with mode 0600. Pre-0.13.5 installs have the file at the legacy `/etc/glassmkr/collector.yaml` path; the agent reads either.
4. If you see `bind: address already in use`, another instance may be running: ```
pgrep -a glassmkr-crucible
``` Kill the stale process and try again.

## Server shows "offline" in the dashboard

**Symptom:** The server card shows a gray status indicator and "last seen" is more than 2 minutes ago (the agent pushes every 60 seconds by default; the `server_unreachable` rule fires after 2 missed check-ins).

1. Check that Crucible is running: ```
systemctl status glassmkr-crucible
```
2. Check network connectivity to the API: ```
curl -s -o /dev/null -w "%{http_code}" https://app.glassmkr.com/api/v1/health
``` You should get `200`. If not, check DNS resolution, firewall rules, and proxy settings.
3. Check whether the collector key is valid: ```
sudo journalctl -u glassmkr-crucible --since "5 min ago" --no-pager
``` If you see `auth error: 401`, rotate the key in the Dashboard and update `/etc/glassmkr/crucible.yaml` (legacy installs: `/etc/glassmkr/collector.yaml`; the agent reads either).
4. Check for network-level blocks: ```
nc -zv app.glassmkr.com 443
```
5. If you are behind a proxy, configure it in `crucible.yaml`: ```
proxy:
https: http://proxy.internal:3128
```

## Metrics are delayed or missing

**Symptom:** The dashboard shows gaps in charts or data arrives minutes late.

1. Check the agent's push timing: ```
sudo journalctl -u glassmkr-crucible --since "5 min ago" --no-pager
``` The "Last push" value should be close to the configured interval (default 60 seconds).
2. If pushes are slow, check the agent log for timeout errors: ```
grep -i "timeout\|retry" /var/log/glassmkr/crucible.log | tail -20
```
3. If the server's clock is significantly off, snapshots may be dropped. Verify NTP is working: ```
timedatectl status
``` If not synchronized: ```
sudo timedatectl set-ntp true
```
4. If specific collectors are slow (e.g., SMART queries on many disks), they can delay the entire push. Inspect collector timing: ```
sudo journalctl -u glassmkr-crucible -f
``` Consider increasing the interval or disabling slow collectors.

## SMART data is not appearing

**Symptom:** The Disk tab in the dashboard shows no SMART information.

1. Ensure `smartmontools` is installed: ```
# Debian / Ubuntu
sudo apt install smartmontools

# RHEL / Rocky / Alma
sudo dnf install smartmontools
```
2. Verify that `smartctl` can read your drives: ```
sudo smartctl -a /dev/sda
``` If this fails with a permission error, Crucible's `glassmkr` service user needs read access (the default install handles this via udev rules).
3. For hardware RAID controllers, drives behind the controller are not visible to `smartctl` without the `-d` flag: ```
sudo smartctl -a /dev/sda -d megaraid,0
```
4. Verify the SMART collector is enabled: ```
collectors:
  smart:
    enabled: true
```

## IPMI, thermal, or fan data is missing

**Symptom:** The Hardware tab shows no temperature, fan, or PSU data.

1. Install `lm-sensors` for hwmon data: ```
# Debian / Ubuntu
sudo apt install lm-sensors
sudo sensors-detect --auto
```
2. For IPMI data, install `ipmitool` and verify it works: ```
sudo apt install ipmitool
sudo ipmitool sdr list
```
3. Run the IPMI self-diagnostic: ```
sudo glassmkr-crucible doctor ipmi
``` See [the IPMI troubleshooting page](/docs/troubleshooting/ipmi) for the full per-reason fix guide.
4. If IPMI is not available (common on consumer hardware, cloud VMs without passthrough, laptops, Raspberry Pi), Crucible reads thermal data from hwmon directly.
5. Confirm the thermal collector is not disabled: ```
collectors:
  thermal:
    enabled: true
    source: auto
```

## ZFS module not loaded

**Symptom:** the Storage tab shows no ZFS pools even though `zpool list` works on the host, or the `zfs_*` rules never fire.

1. Check that the ZFS kernel module is loaded: ```
lsmod | grep zfs
``` On many distributions the module is loaded on-demand by the first `zpool` or `zfs` call. If Crucible starts before that happens, it sees no ZFS surface.
2. Force-load the module at boot: ```
echo zfs | sudo tee /etc/modules-load.d/zfs.conf
sudo systemctl restart glassmkr-crucible
```
3. If `lsmod | grep zfs` shows nothing and you expected ZFS, install the package set for your distribution (`zfsutils-linux` on Debian/Ubuntu, `zfs` on Rocky/Alma with EPEL).
4. If you have a kernel update pending, ZFS DKMS sometimes lags behind the running kernel; reboot or rebuild the module against the new kernel before assuming Crucible is at fault.

## GPU tier-1 (nvidia-smi) unavailable

**Symptom:** a server with NVIDIA GPUs reports no GPU data even though `nvidia-smi` works interactively.

Crucible's GPU collector probes three tiers in order: `nvidia-smi` (most common), DCGM exporter (preferred when present), and Redfish OEM stub (BMC-side, vendor-dependent). Validated on L4, A4000, and A16 in the validation fleet.

1. Confirm `nvidia-smi` is on the PATH that systemd sees: ```
sudo systemd-run --pty --uid=glassmkr nvidia-smi
``` Some distributions install nvidia-smi to `/usr/lib/nvidia/current/` rather than `/usr/bin/`; the systemd unit's `PATH` may differ from your interactive shell.
2. If the binary is found but exits non-zero, check the driver state: ```
nvidia-smi --query-gpu=name,driver_version,pstate --format=csv
``` A driver loaded against a different kernel than the running one will fail here.
3. If DCGM is installed and you want the richer dataset, ensure the exporter is running: ```
systemctl status nvidia-dcgm
```
4. For BMC-side Redfish GPU telemetry (rare; vendor-specific OEM extension), confirm the BMC has the GPU sensor model populated: ```
curl -k -u user:pass https://<bmc>/redfish/v1/Systems/1/Oem/
```

## Pressure (PSI) alerts never fire

**Symptom:** `cpu_pressure_high`, `mem_pressure_high`, and `io_pressure_high` never fire on a CentOS, Alma, Rocky, or RHEL host, even under heavy load.

These three rules read the kernel's Pressure Stall Information. RHEL-family kernels compile PSI in but ship it disabled (`CONFIG_PSI_DEFAULT_DISABLED=y`), so `/proc/pressure` does not exist until you opt in at boot. Debian and Ubuntu enable PSI by default.

1. Confirm PSI is the gap: ```
ls /proc/pressure
``` "No such file or directory" means the kernel is not exporting PSI; the agent omits the data and the three pressure rules stay inactive on this host.
2. Enable it with the `psi=1` boot parameter: ```
sudo grubby --update-kernel=ALL --args="psi=1"
sudo reboot
```
3. Verify after the reboot: ```
cat /proc/pressure/cpu
``` The next Crucible snapshot picks PSI up automatically; no agent restart or config change is needed.

## Telegram notifications are not arriving

**Symptom:** Alerts fire in the dashboard but no Telegram messages are received.

1. Test the channel from the dashboard or API: ```
curl -X POST https://app.glassmkr.com/api/v1/channels/CHANNEL_ID/test \
  -H "Authorization: Bearer YOUR_TOKEN"
```
2. If the test fails with `401 Unauthorized`, the bot token is invalid. Re-create the bot via BotFather or regenerate the token.
3. If the test fails with `400 Bad Request: chat not found`, the chat ID is wrong. Common mistakes: missing the `-100` prefix for supergroups, the bot was removed from the group, the bot never received any message in the chat (send a message to the bot first).
4. If the test succeeds but real alerts do not arrive, check the channel routing. Go to **Settings -> Alert Defaults** and confirm your Telegram channel is listed.
5. Check the alert cooldown. By default, Glassmkr sends one notification per active alert per hour. Acknowledged or recently-notified alerts are suppressed.

## Email notifications go to spam

**Symptom:** Test emails arrive in the spam folder.

1. Check the spam folder and mark messages as "not spam" to train your provider.
2. Add `alerts@glassmkr.com` to your contacts or safe senders list.
3. If you control the recipient domain, allow Glassmkr's SPF record. Contact support for the current IP ranges.
4. For better deliverability, route through a custom SMTP server in your own domain. See the [Channels](/docs/channels) page for setup.

## High CPU usage by Crucible

**Symptom:** the Crucible process uses more than 1-2% CPU consistently.

For reference, the Crucible 0.13.6 measurement across all 10 validation hosts shows a median RSS of 108 MB at steady state (range 81 to 116 MB, varies with the bundled Node version), under 1% of host RAM on every host, ~0% CPU, and fio delta under 1.5%. Sustained higher usage is unusual.

1. Check which collectors are running: ```
sudo journalctl -u glassmkr-crucible -f
```
2. SMART queries on many disks can be expensive. If you have more than 20 disks, narrow the device list or increase the interval: ```
collectors:
  smart:
    devices:
      - /dev/sda
      - /dev/sdb
```
3. Per-core CPU metrics on machines with 64+ cores generate a lot of data. Disable per-core reporting if you do not need it: ```
collectors:
cpu:
per_core: false
```
4. If the collection interval is set very low (e.g., 10 seconds), increase it: ```
collectors:
  interval: 60
```

## Registration fails with "server limit reached"

**Symptom:** + Add Server returns an error about the server limit.

1. The Free plan allows 3 servers. Pro is $3/node/month with the first 3 nodes free.
2. If you have decommissioned servers still registered, delete them from the dashboard to free up slots.
3. To upgrade your plan, go to **Settings -> Billing**.

## My servers are disabled (lock icon, "no payment method on file")

**Symptom:** some server tiles show a lock-icon overlay and "Manage in Settings". Notifications stopped firing for those servers.

**Why:** on the Pro plan, servers beyond the 3-server free quota are disabled at the end of the billing period (or 30 days after account creation, whichever is later) when no payment method is on file. The first 3 servers always stay active. Disabled servers continue to ingest snapshots so historical data is preserved; they just stop firing notifications.

1. Add a payment method: **Settings -> Billing -> Add card** (opens the Stripe portal).
2. Restore in bulk: **Settings -> Disabled servers -> Restore all**. Restoration is instant once a card is on file.
3. If you would rather drop into the free quota than pay, delete individual servers from the same screen.

Glassmkr sends warning emails before disable: when the payment method is removed, 3 days before disable, 1 day before disable, and at the moment of disable. If you do not see these, check your spam folder and confirm the account email is correct.

## Configuration changes are not taking effect

**Symptom:** you edited `crucible.yaml` (or legacy `collector.yaml`) but Crucible still uses the old settings.

1. Restart the service after any configuration change: ```
sudo systemctl restart glassmkr-crucible
```
2. Verify the running config by inspecting the agent's startup banner: ```
sudo journalctl -u glassmkr-crucible --since "1 min ago" --no-pager
``` The first lines after restart print the resolved interval, enabled collectors, and Dashboard URL.
3. Check that you edited the correct file. The systemd unit may pin a non-default config path: ```
systemctl show glassmkr-crucible -p Environment
```
4. Environment variables override the config file. Check for any `GLASSMKR_*` or `CRUCIBLE_*` variables in the systemd unit or shell environment.

## Per-core CPU data is not showing

**Symptom:** the per-core CPU chart does not appear, or per-core data is missing from AI analysis.

1. Per-core monitoring requires Crucible 0.3.0 or later. Check: ```
glassmkr-crucible --version
```
2. Enable per-core in the config: ```
collectors:
cpu:
per_core: true
```
3. Restart Crucible: ```
sudo systemctl restart glassmkr-crucible
```
4. Wait for the next collection interval (default 60 seconds) for data to appear.

## Muted rules are still firing

**Symptom:** you muted a rule but it continues to fire alerts or send notifications.

1. Muting takes effect on the next ingest cycle. Wait at least one collection interval after muting.
2. If you muted via the configuration file, restart Crucible: ```
sudo systemctl restart glassmkr-crucible
```
3. If you muted via the dashboard, no restart is needed; the change applies on the next push from that server.
4. Verify the rule is muted in the dashboard under the server's Alerts tab. Muted rules show a mute icon.

## Getting help

If your issue is not covered here:

- Capture an hour of agent logs: `sudo journalctl -u glassmkr-crucible --since "1 hour ago" --no-pager > crucible.log`. Attach it when contacting support.
- Email [support@glassmkr.com](mailto:support@glassmkr.com) with your server ID and a description of the issue.

Last verified: 2026-05-31 against Crucible v0.13.6. Resource footprint figures are from a 10-host validation-fleet measurement on 2026-05-31.

---

## IPMI troubleshooting
Source: https://glassmkr.com/docs/troubleshooting/ipmi (Markdown: https://glassmkr.com/docs/troubleshooting/ipmi.md)

# IPMI troubleshooting

How Crucible detects IPMI, why "Not detected" does not always mean broken, how to self-diagnose with `glassmkr-crucible doctor ipmi`, and what to expect across BMC vendors.

## How Crucible detects IPMI

Detection is capability-based, not vendor-allowlist. Crucible does not look at your BMC vendor string and decide whether to support you; it asks "can I actually talk to the BMC?" and uses the answer.

The probe chain at agent start, and on every re-check:

1. **Device-node check.** `stat /dev/ipmi0` (also `/dev/ipmi/0` and `/dev/ipmidev/0`). Permission errors here surface as `permission_denied`.
2. **ipmitool binary check.** `ipmitool -V`. Missing binary surfaces as `no_ipmitool_binary`.
3. **Fast path.** If both the device node and the binary are present, Crucible records the capability as available and stops probing.
4. **Sensor-probe fallback.** Only used when the binary exists but the device node did not. Runs `ipmitool sensor` and inspects stderr. A "could not open device" message surfaces as `no_bmc_device`; other errors surface as `execution_failed`.

The result is a structured `detection.reason` field on every snapshot, with one of four values: `no_ipmitool_binary`, `permission_denied`, `no_bmc_device`, or `execution_failed`. The Dashboard surfaces this reason under "IPMI: Not detected" so you know which fix to apply.

Crucible re-runs detection every hour. If you install `ipmitool` or load the kernel modules after the agent started, the next hourly re-check picks the change up automatically. No restart required.

## Detection vs collection: they can disagree, by design

It is normal for the Dashboard to report "IPMI: Not detected" on a host where some hardware metrics still appear. This is not a bug: detection and collection use different data sources.

The header IPMI verdict reflects Crucible's BMC probe. The dashboard's CPU temperature, fan, and ECC blocks can also be populated from non-BMC sources:

- **CPU temperature** often comes from `hwmon` (kernel-side, no BMC needed) or `lm-sensors`.
- **ECC counters** can come from kernel EDAC (`/sys/devices/system/edac/mc/mc*/{ce,ue}_count`) on systems where the BIOS exposes them, completely separate from the BMC.
- **SMART, RAID, network, disk usage** are kernel-side and do not depend on IPMI at all.

When the agent cannot probe IPMI at all, the snapshot emits `null` for ECC and SEL counters; the Dashboard renders that as "no signal (BMC not probed)" instead of the misleading "0 / 0" reading.

## Self-diagnose with `glassmkr-crucible doctor ipmi`

The `doctor` subcommand runs the same probes the agent uses and prints actionable guidance for each failure mode. It is read-only and does not modify system state.

```
sudo glassmkr-crucible doctor ipmi
```

The available case looks like:

```
IPMI capability check:
  Result:        [OK] IPMI detected via ipmitool_in_band
  ipmitool:      1.8.19

Crucible will collect:
  - Sensor readings (temperature, fan, voltage, power)
  - SEL events (recent + cumulative ECC counters)
  - PSU redundancy state (per-PSU + aggregate)
```

Failure cases print the matching `detection.reason` plus a fix recipe.

## `no_ipmitool_binary`

**Meaning:** the `/dev/ipmi0` device exists, but `ipmitool` is not installed.

**Fix:** install the package:

- Debian / Ubuntu: `sudo apt install ipmitool`
- RHEL / Rocky / Alma: `sudo dnf install ipmitool`
- Arch: `sudo pacman -S ipmitool`
- Alpine: `sudo apk add ipmitool`

No restart needed. The next collection cycle (within ~60 seconds at the default interval) sees the binary, and the next hourly re-check flips `detection.available` to `true`. The Dashboard updates on the following ingest.

## `permission_denied`

**Meaning:** Crucible cannot open `/dev/ipmi0`. The device node is mode `0600` owned by root.

**Fix:** Crucible runs as the non-root `glassmkr` user; the install script provisions a udev rule granting that user read access. If you customized the service unit, confirm:

```
systemctl cat glassmkr-crucible | grep '^User='
ls -l /dev/ipmi0
```

The default install ships a udev rule at `/etc/udev/rules.d/99-glassmkr-ipmi.rules` that grants the `glassmkr` group access. If you removed it, restore via the install script or run the agent as root (less preferred).

## `no_bmc_device`

**Meaning:** ipmitool is installed and runs, but the kernel has no IPMI device node and the in-band ipmitool probe could not open one. Usually the kernel modules are not loaded.

```
sudo modprobe ipmi_si ipmi_devintf ipmi_msghandler
ls -l /dev/ipmi0    # should appear after the modules load
```

If `/dev/ipmi0` still does not appear, the host may genuinely have no BMC. This is common on consumer hardware, Raspberry Pi, laptops, and virtual machines without IPMI passthrough. In that case set `collection.ipmi: false` in `/etc/glassmkr/crucible.yaml` (legacy installs: `/etc/glassmkr/collector.yaml`; the agent reads either) to silence the snapshot field; the dashboard stops trying to render IPMI for this host.

## `execution_failed`

**Meaning:** ipmitool ran, but the call returned an error other than "could not open device". The BMC is reachable in some sense but not responding the way Crucible expected.

**Fix:** reproduce by hand and read the error:

```
sudo ipmitool mc info
```

Common causes:

- The BMC is in a degraded state and dropped the request. Retry; if it persists, escalate via the support path below.
- The in-band interface (KCS or SSIF) is busy. Sustained busy state usually means firmware is mid-task; wait a few minutes and retry.
- The installed ipmitool is too old for the BMC's IPMI 2.0 dialect. Upgrade `ipmitool` via the distribution package manager.

**Do not run `sudo ipmitool mc reset cold` without first confirming with your hardware vendor.** Some BMCs do not recover cleanly from a cold reset and hang past the operation, which on a remote machine is much worse than the original failure.

## Per-vendor notes

Crucible's detection is capability-based, so any BMC that responds to standard IPMI 2.0 commands works. These notes are vendor-specific quirks observed on real hardware, not detection-gating rules.

### Supermicro

Usually clean. The BMC reports vendor strings cleanly via `ipmitool mc info` (`Manufacturer Name: Supermicro` or `Super Micro Computer Inc.`). PSU sensors typically appear as `PS1 Status` / `PS2 Status` with the discrete-state bitmask in the Reading column.

### Gigabyte

The BMC sometimes reports `Manufacturer Name: Unknown (0x3C0A)` in `ipmitool mc info` output, even though the IANA manufacturer ID (15370) resolves to Gigabyte. This is a Gigabyte BMC firmware quirk; Crucible does not gate detection on the manufacturer string, so no customer action is needed. PSU sensors typically appear as `PS1_Status` with an underscore separator.

### ASUS

Validated on RS700-E10-RS4U. Detection works correctly when `ipmitool` is installed; the most common issue is that distributions sometimes ship without `ipmitool` by default, which surfaces as `no_ipmitool_binary` in the doctor output. Install via the per-distro command above.

### ASRockRack

DMI `sys_vendor` may read `"To Be Filled By O.E.M."` on some boards (a known firmware default), but the BMC itself reports vendor cleanly via `ipmitool mc info` (`Manufacturer Name: ASRock Rack Incorporation`). PSU sensors appear as `PSU1 Status` / `PSU2 Status`.

### Dell PowerEdge (iDRAC)

In-band IPMI through iDRAC works without an iDRAC Enterprise license. The license gates out-of-band IPMI over LAN, not the in-band KCS path Crucible uses. PSU sensors appear as `PS1 Status` / `PS2 Status`, and iDRAC also exposes an aggregate `PS Redundancy` sensor that Crucible reads for whole-pair redundancy state.

Dell iDRAC compatibility has not been validated on real hardware in our validation fleet. If you hit a detection or collection issue specific to iDRAC, file a support request with the output of `sudo ipmitool mc info` and `sudo glassmkr-crucible doctor ipmi`.

### HP ProLiant (iLO)

In-band IPMI via KCS usually works without an iLO Advanced license. The license gates out-of-band iLO features, not in-band IPMI. Some older iLO firmware revisions require `ipmitool` 1.8.18 or later for IPMI 2.0 compatibility.

HP iLO compatibility has not been validated on real hardware in our validation fleet. Same support-request convention as Dell above.

## A note on PSU monitoring

The `isPsuSensor` classifier covers Supermicro, Gigabyte, ASRockRack, and ASUS naming conventions, and interprets discrete states as IPMI 2.0 spec table 42-3 hex bitmasks (Failure detected, AC lost, predictive, inactive) in addition to text-status strings.

If a multi-PSU box previously showed two healthy PSUs in the dashboard but one was actually failed or unplugged, that is the bug shape that current Crucible releases catch.

## When to file a support request

Email [support@glassmkr.com](mailto:support@glassmkr.com) when:

- Your BMC vendor is not in the validated list above, and detection works (the `doctor` output shows `[OK]`) but a specific collection path (sensors, SEL, PSU) returns unexpected values.
- Detection fails (`doctor` output shows `[FAIL]`) but `sudo ipmitool mc info` works fine when you run it interactively.
- The `doctor` subcommand returns `execution_failed` with an error message not covered above.

Attach:

- The doctor output: `sudo glassmkr-crucible doctor ipmi 2>&1`
- A successful raw probe: `sudo ipmitool mc info 2>&1`
- One hour of agent logs: `sudo journalctl -u glassmkr-crucible --since "1 hour ago" --no-pager > crucible.log`
- Your server ID from the Dashboard.

Last verified: 2026-05-22 against Crucible v0.13.3.

---

## Frequently asked questions
Source: https://glassmkr.com/docs/faq (Markdown: https://glassmkr.com/docs/faq.md)

# Frequently asked questions

Short answers to the questions we get most often. For deeper material follow the links into the rest of the docs.

## Pricing & billing

How much does Glassmkr cost?

**Free**: up to 3 nodes, 7-day retention, community support. Includes the full read+write API, all six notification channels (unlimited), predictive trend warnings, and one trial AI analysis per server.

**Pro**: $3 per node per month with the first 3 nodes free (USD only). Everything in Free, plus the only three things Pro unlocks: more than 3 nodes, 90-day metric retention (audit log 365 days), and unlimited AI analysis (self-hosted Gemma 4). Email support. Example: 10 nodes is $21/month (only 7 are charged).

All plans include the full 65-rule catalog and the MIT-licensed Crucible agent at no extra cost. See [/pricing](/pricing).

What happens if I remove my payment method on Pro?

The first 3 servers stay active permanently (the Free quota is preserved on Pro accounts so service does not drop abruptly). Servers beyond 3 are disabled at the end of your current billing period (or at `customer.created_at + 30 days`, whichever is later). Disabled servers continue to ingest snapshots so historical data is preserved; they just stop firing notifications.

Warning emails fire when the payment method is removed, 3 days before disable, 1 day before disable, and at the moment of disable. Restoration is one click once a card is on file: **Settings -> Disabled servers -> Restore all**.

How does pricing work mid-month?

Proration. Add a server mid-month and you are charged proportionally for the remaining days. Remove a server and the next bill reflects the change.

## The agent

Which operating systems does Crucible support?

Crucible runs on Linux. Tested distributions: Ubuntu 20.04 / 22.04 / 24.04, Debian 11 / 12, RHEL 8 / 9, Rocky Linux 8 / 9, AlmaLinux 8 / 9, Arch (rolling), Amazon Linux 2 and 2023. systemd plus kernel 4.18 or newer. x86_64 and aarch64.

Windows and macOS are not supported. For non-Linux hosts, push to the API directly with a custom collector.

How much CPU and memory does Crucible use?

Lightweight by design: under 1% of host RAM on every host we tested. Measured on Crucible 0.13.6 across all 10 validation hosts at steady state: median 108 MB RSS (range 81 to 116 MB, varies with the bundled Node version), near-zero CPU, and an fio delta under 1.5 percent. The binary is about 12 MB; logs rotate at 50 MB by default. Each push is 2 to 5 KB of compressed JSON; at the default 60-second interval, that is about 5 MB/day.

Does Crucible need root access?

Crucible runs as the non-root `glassmkr` user. The install script provisions udev rules and group memberships that grant the user the necessary read access to `/dev/ipmi0`, raw block devices for SMART, and `/proc` / `/sys`. If you disable IPMI, SMART, and ECC monitoring, the agent works without those rules.

Do I need to open any inbound ports?

No. The agent initiates all connections outbound over HTTPS (port 443). Your firewall does not need to change.

Does the agent work without IPMI?

Yes. If `ipmitool` is not installed or the BMC is not reachable, the IPMI module is silently skipped and the snapshot emits null for the IPMI fields. The dashboard renders that as "no signal (BMC not probed)". All other monitoring continues normally. See [/docs/troubleshooting/ipmi](/docs/troubleshooting/ipmi).

How do I update Crucible?

```
sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible
```

Pin a specific version with `@glassmkr/crucible@0.13.5`. Docker images are at `ghcr.io/glassmkr/crucible:` and `docker.io/glassmkr/crucible:`. Configuration in `/etc/glassmkr/crucible.yaml` is preserved across upgrades. (Pre-0.13.5 installs have the file at the legacy `/etc/glassmkr/collector.yaml` path; the agent reads either, and `glassmkr-crucible init` migrates the file in place on next run.)

How do I uninstall Crucible?

```
# Stop and disable
sudo systemctl stop glassmkr-crucible
sudo systemctl disable glassmkr-crucible

# Remove the unit
sudo rm /etc/systemd/system/glassmkr-crucible.service
sudo systemctl daemon-reload

# Remove the package
sudo npm uninstall -g @glassmkr/crucible

# Remove configuration (contains the collector key)
sudo rm -rf /etc/glassmkr
```

Optionally delete the server from the dashboard to remove its stored metrics and free up a slot on your plan.

## Data & retention

Where is my data stored?

Metric data is stored on Glassmkr infrastructure in the EU. The operating company is a Czech sole-trader; all dedicated servers (including the database and the AI GPU) sit in EU data centers, so the deployment inherits GDPR posture from the underlying provider. Data is encrypted at rest using AES-256 and in transit using TLS 1.3.

Glassmkr does not store raw system logs, process lists, or file contents. The collected data is numerical metrics (CPU, memory, disk, network counters) and hardware status identifiers (SMART, RAID, sensor readings).

How long is metric data retained?

**Free:** 7 days at full resolution.

**Pro:** 90 days. Data older than 7 days is downsampled to 5-minute resolution; data older than 30 days is downsampled to 1-hour resolution.

What happens if connectivity is lost?

The `server_unreachable` rule fires after the server misses 2 consecutive check-ins (about 2 minutes at the default 60-second interval). Crucible buffers up to 60 snapshots in memory (about 1 hour at the default interval) and pushes the queue in order when connectivity is restored. If the buffer fills, the oldest snapshots are dropped first.

Can I export my data?

Yes. Pull metric data via the [health history API](/docs/api#health). The response is JSON and can be piped into any analytics tool. For bulk exports, contact support for a CSV or Parquet dump of your account's data.

Can I self-host the dashboard?

The Crucible agent is MIT-licensed and fully open source. The dashboard and alert evaluation engine are SaaS-only.

## Alerts & AI

How many alert rules ship out of the box?

65 rules across 9 categories: storage, ZFS, filesystem, memory and CPU, network, hardware (BMC / IPMI), GPU, time and services, security and patching. Every rule ships with deep FIX content (safe-mode, validation, rollback, impact); 30+ are verified end-to-end on real hardware. Browse at [/docs/rules](/docs/rules).

What do the P1-P4 priority levels mean?

Every alert has a priority from P1 (critical, immediate action required) to P4 (informational). Priority drives the badge color on alert cards and the emoji prefix in Telegram / Slack notifications. P1 indicates data loss or service outage; P2 indicates significant degradation; P3 is an early warning; P4 is a proactive recommendation.

How does alert muting work?

Mute specific alert rules on a per-server basis from the server detail page or via the configuration file. Muted rules are not evaluated and do not fire notifications. Useful during maintenance windows, RAID rebuilds, or known-condition periods. Unmuting takes effect on the next ingest cycle. Alerts do not fire retroactively for conditions that occurred while muted.

How does AI analysis work?

Glassmkr uses a self-hosted Gemma 4 model (running on a dedicated GPU server in the EU; no commercial LLM APIs) to analyze server health when alerts fire. Free gets one trial analysis per server; Pro is unlimited. The model reviews current metrics (including per-core CPU when available), recent trends, and the alert context to produce a one-sentence summary of the likely cause. The model is tuned to be conservative, to hedge, and to say "I don't know" when the signal is ambiguous. AI analysis is shown on the alert card and included in Telegram / Slack notifications.

What is per-core CPU monitoring?

When `collectors.cpu.per_core: true` (Crucible 0.3.0+), Crucible reports individual CPU core utilization in addition to aggregate metrics. Enables the per-core CPU chart in the expanded view and gives the AI analyzer per-core awareness. Useful for spotting single-threaded bottlenecks, core pinning issues, and uneven load. Increases data volume proportionally to core count.

## Operations

Can I monitor Docker containers or Kubernetes pods?

Crucible monitors the host system, not individual containers. Container CPU and memory are visible in the host metrics as part of the total. Dedicated container and Kubernetes monitoring is on the roadmap.

If you run Crucible inside a Docker container, it needs access to the host's `/proc`, `/sys`, and device files via volume mounts. Installing on the host is simpler and more reliable.

What sign-in methods does Glassmkr support?

Email + password, Google OAuth, and GitHub OAuth. Connect or disconnect OAuth providers under **Settings -> Account**. There is no built-in TOTP today; rely on your OAuth provider's two-factor configuration if you sign in via Google or GitHub.

Is there an API rate limit?

Yes. Token-bucket limiter with per-IP / per-key / per-account tiers plus per-endpoint sub-limits for write actions. Per-IP is 100 burst at 10/sec; per-key is 1000 burst at 100/sec; per-account is 5000 burst at 500/sec. Server registration, deletion, and key rotation each have their own hourly sub-limits. Ingest is rate-limited to one push per server per 55 seconds. See the [API reference](/docs/api#rate-limits) for the full table.

## Support

How do I contact support?

Email [support@glassmkr.com](mailto:support@glassmkr.com) with your account email and server ID. Include the output of `sudo journalctl -u glassmkr-crucible --since "1 hour ago" --no-pager` if the issue is agent-side.

Last verified: 2026-05-22 against Crucible v0.13.3.

---

## Changelog
Source: https://glassmkr.com/docs/changelog (Markdown: https://glassmkr.com/docs/changelog.md)

# Changelog

Behavior changes, operational improvements, and notable fixes for Glassmkr and the Crucible agent. Most recent at top.

## 2026-07-15

### Crucible v0.13.24

**Current.** The agent is at v0.13.24 on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed.

**OS support-status awareness.** The agent now reports whether the host is enrolled in extended security support past its standard end-of-life (Ubuntu Pro/ESM, or a RHEL Extended Update Support repository), read strictly unprivileged (it never runs as root). The dashboard pairs this with the release lifecycle date so a past-end-of-life host that is still receiving security fixes is not reported as unsupported.

### Crucible v0.13.23

The agent was at v0.13.23 as of this entry (superseded by v0.13.24 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed.

**GPU PCIe slot width.** The GPU collector now records the electrical width of the PCIe slot each card sits in (the upstream port's max link width), alongside the card's own maximum. This lets the dashboard tell an x16 card seated in a physically x8 slot (which correctly negotiates x8, and is not a fault) from a link that trained below the slot's capability, so `gpu_pcie_link_degraded` no longer warns on the former.

## 2026-07-14

### Crucible v0.13.22

The agent was at v0.13.22 as of this entry (superseded by v0.13.23 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed.

**SSH drop-in edits are no longer invisible on RHEL-family hosts.** The agent runs unprivileged, and `/etc/ssh/sshd_config.d` is `0700 root` by default on RHEL, so a hardening change applied only via a drop-in file looked unapplied forever. When the direct read is denied, the newest drop-in timestamp is now read through a strictly read-only, fixed-path privileged helper, restoring visibility for `ssh_config_unapplied`. Hosts running the unprivileged service user need a one-time `glassmkr-crucible init` re-run (or wrapper refresh) to grant the new read.

**Dual-socket DIMM channel counts are no longer halved.** A board that labels its channels A-H on both sockets was collapsing 16 channels to 8; channel totals are now qualified by socket, so the memory-channel advisory reports correct populated and available counts on dual-socket systems.

**SATA wear no longer misreads a temperature attribute.** SMART attribute 231 is drive-wear on some SATA SSDs but temperature on others; it has been dropped from the id-only fallback, so a temperature reading is never reported as remaining drive life (a genuine wear drive still carries a wear-named attribute the parser matches).

**Privileged helper hardening.** The collection wrapper's device-path check now rejects path traversal (any `..` or embedded slash) to mirror the TypeScript allowlist, and its `secure_path` includes `/usr/local/bin` so hand-built tools placed there resolve for the unprivileged agent.

### Crucible v0.13.21

The agent was at v0.13.21 as of this entry (superseded by v0.13.22 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed.

**Hands-off fleet onboarding with `enroll`.** A new `glassmkr-crucible enroll --account-key ` subcommand lets you bake one account key into an Ansible, cloud-init, or post-install script and share it across the whole fleet, instead of minting and copying a separate collector key into every host. Each server derives a stable machine identity, self-registers with the dashboard, and receives its own collector key; a re-run maps back to the same server rather than creating a duplicate. The account key is used only for that one registration call and is never written to disk.

**RHEL security-update count is now accurate.** On RHEL-family hosts the pending-security-update count was derived from a query that also counted kernels already installed but not yet booted, inflating it and pointing at a no-op fix. It now counts only genuinely installable security packages; an installed-but-unbooted kernel shows up (correctly) as "reboot required", not "updates pending".

**Correct vendor on placeholder-firmware boards.** Boards whose firmware leaves the system-manufacturer as a placeholder (e.g. "To Be Filled By O.E.M.") are now identified by their baseboard manufacturer, so vendor-specific BMC parsing and guidance work on them.

### Dashboard

**Idempotent server onboarding.** Paired with `enroll` above, the dashboard now recognizes a re-provisioned host by its machine identity and re-attaches it to the existing server instead of creating a duplicate that counts against your node quota.

**New advisory: a drive has disappeared.** A standalone (non-RAID) disk that drops off the bus or de-enumerates used to leave no signal, since the RAID and SMART checks only see drives that are present. A new trend warning flags a disk that was consistently present and has since vanished from telemetry while the host keeps reporting, identified by serial. It is advisory (a planned hot-swap looks the same from the outside), so it never pages.

**Drive-wear guidance now covers SATA SSDs.** The wear alert fires on SATA SSDs as well as NVMe; its remediation is now drive-type-aware (smartctl for SATA, nvme-cli for NVMe) instead of NVMe-only. The memory-channel advisory is now informational rather than a warning, matching its "not a fault" intent.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.22
sudo systemctl restart glassmkr-crucible
```

## 2026-07-05

### Crucible v0.13.20

The agent was at v0.13.20 as of this entry (superseded by v0.13.21 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed.

**Per-process file-descriptor monitoring now covers root-owned processes.** The per-process file-descriptor scan runs through the same privileged helper as the rest of hardware collection, so a host running the unprivileged service user now sees descriptor counts for root-owned daemons, not just its own. Before this, a root service slowly leaking file descriptors toward its limit was invisible to the agent when it ran as `glassmkr`. Hosts on the unprivileged service user need a one-time `glassmkr-crucible init` re-run (or wrapper refresh) to grant the new read.

**Fan speeds now include discrete-reading sensors.** Boards that expose fan tachometers as discrete sensors rather than analog readings (some Supermicro and ASRock Rack firmware) had those fans omitted from the sensor list. They are now mirrored in alongside the analog readings, so fan coverage matches what the BMC reports.

### Crucible v0.13.19

The agent was at v0.13.19 as of this entry (superseded by v0.13.20 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed.

**Memory-channel population is now monitored.** The agent reports every DIMM slot (populated or empty) with its channel, socket, size, and rated versus configured speed, straight from the board's firmware. The dashboard raises a new advisory when memory channels sit empty or DIMMs run below rated speed: the silent bandwidth killers on multi-channel CPUs, where an 8-channel EPYC with 4 DIMMs delivers roughly half its peak memory bandwidth. Validated on Supermicro, ASRock, Gigabyte, and ASUS boards, including dual-socket EPYC. Hosts running the unprivileged service user need a one-time `glassmkr-crucible init` re-run (or wrapper refresh) to grant the new read.

**Fixed a phantom SMART failure on virtual devices.** A device smartctl could not read (a BMC's virtual-media USB device such as "AMI Virtual HDisk0", or a USB bridge needing a device type) was reported as a failing drive. Unreadable now means no SMART data, never a failure verdict.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.19
sudo systemctl restart glassmkr-crucible
```

## 2026-07-04

### Crucible v0.13.18

The agent was at v0.13.18 as of this entry (superseded by v0.13.19 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed.

**Fixed a false "reboot required" alert on RHEL-family hosts.** On Rocky, AlmaLinux, RHEL, CentOS Stream, and Fedora, the kernel ships as the `kernel-core` package rather than `kernel`. The reboot check queried `kernel`, got "not installed", and raised `kernel_needs_reboot` on healthy hosts (showing "Installed kernel: package kernel is not installed"). It now reads `kernel-core` (and `kernel-default` on SUSE), so the running-versus-installed comparison is correct on the RHEL family. Debian and Ubuntu are unaffected.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.18
sudo systemctl restart glassmkr-crucible
```

## 2026-07-03

### Crucible v0.13.17

The agent was at v0.13.17 as of this entry (superseded by v0.13.18 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed.

**Hardware and security collection restored on root hosts.** A v0.13.16 change routed privileged checks (IPMI sensors, SMART, RAID, firewall, kernel logs) through a sudo helper that is only set up by the installer's init step. A host running the agent as root that had skipped that step collected none of them. The agent now falls back to reading those directly when it runs as root, so collection is never silently lost.

### Crucible v0.13.16

The agent was at v0.13.16 as of this entry (superseded by v0.13.17 above), on npm and Docker Hub. Runs as the non-root `glassmkr` user.

**SSH config changes are now checked against the running daemon.** Crucible reads SSH settings from `sshd -T`, which reflects the config file on disk rather than the daemon that is actually running. So editing `sshd_config` to lock down root login but forgetting to reload sshd would clear the alert while the box stayed exposed. Crucible now compares the config's last-modified time against the daemon's last start-or-reload and raises a new `ssh_config_unapplied` alert while a change is saved but not yet live, so a host is never reported all-clear on an unapplied SSH change. It recognizes a `systemctl reload`, not just a full restart.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.17
sudo systemctl restart glassmkr-crucible
```

## 2026-07-01

### Crucible v0.13.15

The agent was at v0.13.15 as of this entry (superseded by v0.13.16 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed. Runs as the non-root `glassmkr` user.

**SATA SSD endurance is now monitored.** Crucible previously read write-endurance only from NVMe drives, so a worn SATA SSD reported no wear at all. It now reads a SATA SSD's wear indicator (the vendor-specific SMART life-remaining attribute) and reports percent-used the same way it does for NVMe, so an aging SATA SSD is no longer invisible. Paired with a dashboard drive-wear alert that adds a lower "plan replacement" watch level below the existing warning.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.15
sudo systemctl restart glassmkr-crucible
```

## 2026-06-27

### Crucible v0.13.14

The agent was at v0.13.14 as of this entry (superseded by v0.13.15 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed. Runs as the non-root `glassmkr` user.

**Clean reboots are no longer misreported as unclean shutdowns.** The reboot-evidence collector detected a clean shutdown with `last shutdown -F`, which returns nothing on modern systemd and util-linux even after a clean `sudo reboot`, so a deliberate planned reboot was escalated to a critical "unclean shutdown" alert. The collector now reads `last -x -F` (which surfaces the shutdown record) and treats a boot as clean when a shutdown record sits immediately before it. Together with the dashboard severity change above, a clean intentional reboot is now informational, not a page.

### Crucible v0.13.13

The agent was at v0.13.13 as of this entry (superseded by v0.13.14 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed. Runs as the non-root `glassmkr` user.

**GPU driver-resilience facts.** On a host with an NVIDIA GPU, Crucible now reports whether the nvidia kernel module is loaded, whether nouveau is loaded, and whether nouveau is blacklisted, read from sysfs and `/proc/modules` even when `nvidia-smi` is dead. That is the dangerous case: if nouveau is not blacklisted it binds the GPU first on the next reboot, the NVIDIA driver cannot load, and a marketplace host silently de-lists itself. The new `gpu_driver_unsafe_reboot` alert warns before the reboot happens, while the fix is still non-disruptive.

**kernel_needs_reboot false positive fixed.** The Debian and Ubuntu reboot check trusted `/var/run/reboot-required` unconditionally, but that flag is set by any package that wants a reboot (libc, systemd), not just the kernel. A host whose running kernel was already the newest installed could raise a spurious reboot alert. The check now compares the running kernel to the newest installed kernel.

### Dashboard: alert quality

**Host-type profiles, now with auto-detect.** Tag a server's role (for example a marketplace GPU box) and the alerts that are expected by design for that role are suppressed, with the reason recorded, while real faults still fire. A host that looks like a marketplace GPU box is now offered a one-click prompt to apply the profile, so the suppression does not depend on remembering to set it per box. Settable in the UI or via the API (the `profile` field on `POST` and `PATCH /api/v1/servers`).

**Flapping detection.** A rule that keeps firing and auto-resolving (an intermittent or expected-at-idle condition) is now rolled up into a single recurrence warning instead of a wall of identical alerts. The event log records state changes, not one row per snapshot, so a chronically-true alert no longer buries the real transitions.

**False positives fixed.** Several alerts were taught to tell a real fault from a benign one: `gpu_pcie_link_degraded` ignores an idle, power-capped GPU (the PCIe link downshifts to save power and retrains under load); `disk_latency_high` distinguishes I/O saturation under load from a failing drive; `gpu_uncorrected_ecc` treats a one-off lifetime SRAM bit flip as informational rather than an immediate replacement; and a clean, intentional reboot is no longer reported as a critical unexpected reboot.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.14
sudo systemctl restart glassmkr-crucible
```

## 2026-06-26

### Crucible v0.13.12

The agent was at v0.13.12 as of this entry (superseded by v0.13.13 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed. Runs as the non-root `glassmkr` user.

**Free memory (MemFree).** Crucible now collects MemFree from `/proc/meminfo` alongside MemAvailable, so the dashboard can split a host's memory headroom into reclaimable page cache versus genuinely unused RAM. Additive and backward compatible.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.12
sudo systemctl restart glassmkr-crucible
```

## 2026-06-21

### The programmatic API and all notification channels are now free

**Re-gating.** We moved the Pro line. The only things Pro now unlocks are the ones whose cost scales with use: more than 3 nodes, data retention beyond 7 days, and unlimited AI analysis. Everything else is free on every plan, including the full read and write programmatic API (account keys can be read, write, or admin scope), all notification channels with no cap, and predictive trend warnings. This reverses the earlier policy (see the 2026-05-12 tier-gating entry below) that kept the write API and premium channels behind Pro; we decided the API should not be the paywall.

### Three new notification channels: Discord, PagerDuty, and webhooks

**New.** Alongside Email, Telegram, and Slack, you can now route alerts to Discord (incoming webhook), to PagerDuty (Events API v2, with automatic incident resolution when an alert clears), or to any HTTP endpoint as a structured JSON webhook. All six channel types are free, on every plan, with no cap. Add them under Channels in the dashboard.

## 2026-06-16

### Crucible v0.13.11

The agent was at v0.13.11 as of this entry (superseded by v0.13.13 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed. Runs as the non-root `glassmkr` user.

**Drive serial number and firmware.** Crucible now collects each disk's serial number and firmware revision, parsed from the SMART report it already reads (SATA and NVMe). These surface in the dashboard's hardware view and in drive alerts, where the serial is the identifier a hardware replacement or provider ticket needs. No configuration change; a drive whose firmware does not report them is handled gracefully.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.11
sudo systemctl restart glassmkr-crucible
```

## 2026-06-11

### Crucible v0.13.10

The agent was at v0.13.10 as of this entry (superseded by v0.13.11 above), on npm (`@glassmkr/crucible`) and on Docker Hub (`docker.io/glassmkr/crucible`, anonymous pulls). MIT-licensed. Runs as the non-root `glassmkr` user.

**PSI availability notice (RHEL family).** Stock CentOS, Alma, Rocky, and RHEL kernels ship Pressure Stall Information disabled, so the cpu/memory/io pressure rules cannot fire there. The agent now says so once at startup, with the remedy (the `psi=1` boot parameter). Previously the gap was silent. Details in the [troubleshooting entry](/docs/troubleshooting#psi).

**Correct primary IP.** On boards where the BMC's virtual USB interface enumerates first (common on Supermicro), the dashboard and notifications showed a link-local 169.254.x address as the server IP. The agent now prefers the first global-scope address.

**Docker quickstart fixed.** The compose file now pulls from Docker Hub, which allows anonymous pulls; the previous ghcr.io default requires authentication, so the documented `docker compose up` failed. Unused container plumbing the agent never read was removed, and the install docs now create the config file the agent actually reads.

### Dashboard: notification dedup for event alerts

Event-type alerts (IPMI SEL, GPU XID) stack repeat occurrences into one card. Stacking previously treated every re-evaluation of a still-recent event as a new occurrence and re-sent the notification, which could repeat once per collection interval for as long as the event stayed in the rule's window. Stacking and re-notification now happen only when a genuinely new event arrives, and acknowledgements stick. Server-side fix, already live; no agent upgrade required for it.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@0.13.10
sudo systemctl restart glassmkr-crucible
```

## 2026-06-07

### Crucible v0.13.9

The agent was at v0.13.9 as of this entry (superseded by v0.13.10 above), on npm (`@glassmkr/crucible`) and at the registries `ghcr.io/glassmkr/crucible` and `docker.io/glassmkr/crucible`. MIT-licensed. Runs as the non-root `glassmkr` user.

**Detection fixes (RHEL/Fedora family only).** 0.13.9 broadens the `dnf-automatic` auto-updates check to recognize the full set of affirmative `apply_updates` values (`yes`, `true`, `on`, `1`, case-insensitive) and anchors the match, and it makes `/etc/os-release` parsing tolerant of non-standard whitespace and quoting so the distro family always resolves correctly. Debian/Ubuntu hosts are unaffected. No change to the data collected, the CLI, the config schema, or the snapshot payload.

## 2026-06-03

### Crucible v0.13.8

The agent was at v0.13.8 as of this entry (superseded by v0.13.9 above), on npm (`@glassmkr/crucible`) and at the registries `ghcr.io/glassmkr/crucible` and `docker.io/glassmkr/crucible`. MIT-licensed. Runs as the non-root `glassmkr` user.

**Internal refactor, no behavior change.** 0.13.8 extracts two more shared helpers from per-collector code: a rate-tracker for cumulative `/proc` counters (used by the conntrack and softnet collectors) and `key=value` plus columnar `/proc` parsers (used by the systemd-unit and TCP-stats collectors). The rate and parse behavior is unchanged, and the data collected, the CLI, the config schema, and the snapshot payload are all unchanged; another maintainability pass, verified by the full test suite. No action required beyond a routine upgrade when convenient.

### Crucible v0.13.7

The agent was at v0.13.7 as of this entry (superseded by v0.13.8 above), on npm (`@glassmkr/crucible`) and at the registries `ghcr.io/glassmkr/crucible` and `docker.io/glassmkr/crucible`. MIT-licensed. Runs as the non-root `glassmkr` user.

**Internal refactor, no behavior change.** 0.13.7 deduplicates repeated collector code into shared helpers (file reads, CLI and systemd-unit presence checks, kernel-log reading and timestamp parsing, and `/etc/os-release` parsing) and removes dead code. The data collected, the CLI, the config schema, and the snapshot payload are all unchanged; this is a maintainability and footprint pass, verified by the full test suite. No action required beyond a routine upgrade when convenient.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible
```

Or via Docker: `docker pull ghcr.io/glassmkr/crucible:latest`.

## 2026-05-29

### Crucible v0.13.6

The agent was at v0.13.6 as of this entry (superseded by v0.13.7, see 2026-06-03). On npm (`@glassmkr/crucible`) and at the registries `ghcr.io/glassmkr/crucible` and `docker.io/glassmkr/crucible`. MIT-licensed. Runs as the non-root `glassmkr` user.

**Security fix.** 0.13.6 fixes a security false-negative on RHEL-family hosts: download-only `dnf-automatic` timers were treated as "auto-updates configured," which suppressed the `pending_security_updates` alert. Affected hosts: any RHEL-family host with `dnf-automatic.timer` enabled but `apply_updates = no` in `/etc/dnf/automatic.conf`. Upgrade to 0.13.6 to receive the corrected classification; the alert then fires correctly when security patches are pending and no working auto-apply mechanism is configured. Debian/Ubuntu hosts were never affected (that path already inspected the config contents).

**Interim versions.** 0.13.4 was a documentation sweep and 0.13.5 renamed the on-disk config file from `collector.yaml` to `crucible.yaml` (with a backwards-compatible read of the old name); neither changed alerting behavior.

### Upgrade

```
sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible
```

Or via Docker: `docker pull ghcr.io/glassmkr/crucible:latest`.

## 2026-05-22

### Crucible v0.13.3

The agent was at v0.13.3 as of this entry (superseded by v0.13.6, see 2026-05-29). On npm (`@glassmkr/crucible`) and at the registries `ghcr.io/glassmkr/crucible` and `docker.io/glassmkr/crucible`. MIT-licensed. Runs as the non-root `glassmkr` user.

**Resource footprint.** Validation-fleet measurement on 2026-05-21 across 7 hosts shows a median 91 MB RSS idle, near-zero CPU, and an fio delta under 1.5%. RSS ranged 65 MB to 103 MB. *Updated:* a later measurement against the 0.13.6 fleet shows a median around 108 MB; see the [docs spec table](/docs) for current numbers.

**Default interval.** 60 seconds (set in v0.10.0; the previous 300-second default is gone).

### Rule library at 65 rules across 9 categories

Categories: storage, ZFS, filesystem, memory and CPU, network, hardware (BMC / IPMI), time and services, security and patching, GPU. 20 rules ship with deep FIX content (safe-mode, validation, rollback, impact counters); the rest are verified. GPU coverage is 8 rules across three tiers (nvidia-smi / DCGM exporter / Redfish OEM stub), validated on NVIDIA L4, A4000, and A16.

Browse all rules at [/docs/rules](/docs/rules); the machine-readable corpus is at [/llms-full.txt](/llms-full.txt).

### Upgrade

```
sudo npm install -g @glassmkr/crucible@latest
sudo systemctl restart glassmkr-crucible
```

Or via Docker: `docker pull ghcr.io/glassmkr/crucible:latest`.

## 2026-05-13

### Crucible 0.9.4

**Fixed.** PSU sensor classification now works across every observed BMC vendor (Supermicro, Gigabyte, ASRockRack, ASUS). Previously the `PS` name shape was Dell-gated and four of five PSU-having boxes were silently filtered out, so per-PSU alerts never fired regardless of state. The bitmask interpretation now follows IPMI 2.0 spec table 42-3 (Failure detected, AC lost, predictive, inactive).

**Changed.** When the agent cannot probe IPMI at all (no `ipmitool`, no `/dev/ipmi0`, etc.), the snapshot now emits `null` for `ipmi.ecc_errors` and `sel_entries_count` rather than stub zeros. The dashboard renders this as "no signal (BMC not probed)" instead of the misleading "0 / 0".

**Changed.** IPMI capability detection re-runs every hour. Installing `ipmitool` after the agent started is picked up automatically; no service restart needed.

**Added.** `glassmkr-crucible doctor ipmi` subcommand for customer self-diagnosis. See [/docs/troubleshooting/ipmi](/docs/troubleshooting/ipmi).

### Tier-gating policy for the programmatic API

The programmatic API surface (account keys) is now uniformly gated by Pro plan. Free customers retain full web-dashboard access and full *read* API access for their own data; programmatic *writes* (channel CRUD, alert acknowledge / resolve, server CRUD, mutes, restore endpoints, key management) and Pro features (AI analysis, trend warnings) return `402 pro_required` when called via an API key on a Free plan. Web-dashboard sessions are unaffected at any tier. Full breakdown at [/docs/api/tier-gating](/docs/api/tier-gating).

### Unexpected-reboot alerts now auto-resolve

An `unexpected_reboot` alert that has been firing for at least 24 hours of continuous stable uptime now auto-resolves with `resolution_reason: auto_decay_stable_24h`. The original incident remains in the resolved-alerts history. Tunable per-server via `config_overrides.unexpected_reboot_decay_hours`.

## 2026-05-12 (later still)

### Rate-based ECC error detection

The `ecc_errors` alert rule now uses a rolling 24-hour rate window instead of a cumulative threshold. Default: a warning fires when more than 10 correctable errors are observed in 24 hours. Uncorrectable ECC errors continue to fire critical immediately on any non-zero count.

This eliminates false-positive alerts on long-running hosts where BMC error counters accumulate over months without indicating an active hardware issue.

When the BMC counter is cleared (SEL reset, BMC reboot), the rule skips one evaluation cycle and resumes on the next snapshot; no false alerts from the reset itself.

## 2026-05-12 (later)

### API key management UI + scopes

Pro customers can now manage API keys via the dashboard at [/settings/keys](https://app.glassmkr.com/settings/keys). New keys can be created with one of three scopes (Read, Write, or Admin), an optional expiry date (up to 5 years), and graceful 48-hour rotation. The old key keeps working through the grace window so automation can be updated without downtime, then auto-revokes. Emergency revoke is one click away.

Reminder emails fire 7 and 1 days before key expiry, plus a notification when a key is auto-revoked on expiry. Audit log view is at [/settings/audit](https://app.glassmkr.com/settings/audit) with date-range and action filters (Pro plan; 365-day retention).

## 2026-05-12

### Pro-tier API gating

Programmatic API access (creating account API keys, rotating account keys, reading the audit log, plus programmatic server management via account keys: create, rename, delete, rotate collector keys) is now restricted to Pro-plan customers. Free-plan accounts can continue to ingest snapshots from their 3 free servers and use the dashboard to manage them; programmatic management requires Pro.

If you encounter a 402 response on an endpoint that previously worked, your account may have been downgraded or never had Pro access. Existing API keys created before this change keep working for ingest; manage them via the dashboard if you need to rotate or delete.

## 2026-05-09

### Hardware visibility on the dashboard

Server tiles and detail pages now show the hardware vendor and product detected by the monitoring agent (e.g., "GIGABYTE / R292-4S1-00"), plus an IPMI badge when sensor data is available. This helps identify what is actually on each box, especially when server names do not match the underlying hardware.

## 2026-05-08

### Stripe billing enforcement

Pro customers without a payment method on file now have servers beyond the 3-server free quota disabled at the end of their billing period. The oldest 3 servers stay active. Snapshot ingest continues for disabled servers, so historical data is preserved and restoration is instant once a card is added.

Affected customers receive a sequence of warning emails: when a payment method is removed, 3 days before disable, 1 day before disable, and at the moment of disable. Restoration is a single-click operation: **Settings -> Disabled servers -> Restore all**.

## 2026-05-07

### `cpu_temperature_high` reads hwmon directly, with IPMI fallback

Most customers see no change. The evaluator now reads CPU thermals from the kernel's hwmon interface (more accurate, vendor-agnostic), falling back to IPMI sensor data when hwmon is not available.

Customers running on Gigabyte AMD platforms (B650, B660, X670, EPYC) see fewer false-positive alerts. Their BMC firmware (12.61, the shipping default on these boards) reports a `CPU_DTS` IPMI sensor that runs about 30 C hotter than the actual CPU die temperature exposed via the kernel. The agent (Crucible 0.9.1+) drops the inflated `CPU_DTS` sensor whenever a `CPU_TEMP` sibling exists on the same socket.

### ECC alerts now fire on Dell and HPE iDRAC platforms too

Previously, ECC alerts only fired when the BMC exposed a named "correctable / uncorrectable" sensor. Dell iDRAC and some HPE iLO firmwares report ECC events only via the System Event Log (SEL), not as named sensors, so customers on those platforms were missing alerts silently. This release adds a SEL-derived counter as a second source.

### `install.sh` simplified

The bootstrap installer at [glassmkr.com/install.sh](https://glassmkr.com/install.sh) now delegates configuration and systemd setup to the `glassmkr-crucible init` subcommand. The `--dashboard-key` argument is preserved as an alias for `--api-key` so existing automation keeps working.

---

# Alert rule catalog

## Category: Storage

### Rule: disk_io_errors

URL: https://glassmkr.com/docs/rules/disk_io_errors
Priority: P1
Title: Disk I/O errors
Summary: Kernel logged I/O errors against one or more block devices. Indicates failing storage hardware, a flaky cable/controller, or filesystem corruption. Investigate immediately to prevent data loss.

---

### Rule: disk_latency_high

URL: https://glassmkr.com/docs/rules/disk_latency_high
Priority: P3
Title: Disk latency high
Summary: Disk's average read or write latency exceeds the threshold under non-trivial IOPS load. Indicates a struggling drive, saturated I/O queue, in-progress RAID rebuild, or noisy-neighbor workload.

---

### Rule: nvme_critical_warning

URL: https://glassmkr.com/docs/rules/nvme_critical_warning
Priority: P1
Title: NVMe critical warning byte non-zero
Summary: An NVMe device's Critical Warning byte (NVM Express §5.21) is non-zero. Per spec, any non-zero bit is a vendor-recommended immediate-action signal: temperature threshold exceeded, available spare below threshold, reliability degraded, read-only mode, volatile memory backup failed, or persistent memory region read-only.

---

### Rule: nvme_wear_high

URL: https://glassmkr.com/docs/rules/nvme_wear_high
Priority: P2
Title: SSD wear high
Summary: A solid-state drive's write-endurance indicator is at or above the configured threshold. This covers both NVMe (percentage-used) and SATA SSDs (Percent_Lifetime_Remain / Wear_Leveling_Count, which Crucible maps into the same percentage-used signal). Plan replacement before the drive enters read-only protection mode at 100%. The rule id stays nvme_wear_high for history; it is not NVMe-only. Use nvme-cli for NVMe devices (/dev/nvmeXnY) and smartctl for SATA SSDs (/dev/sdX); smartctl -a works for both.

---

### Rule: raid_degraded

URL: https://glassmkr.com/docs/rules/raid_degraded
Priority: P1
Title: RAID array degraded
Summary: One or more disks have failed in an mdadm software array or a hardware RAID controller (Dell PERC, LSI/Broadcom MegaRAID, HPE Smart Array, Adaptec). One more failure may cause data loss.

---

### Rule: smart_failing

URL: https://glassmkr.com/docs/rules/smart_failing
Priority: P1
Title: Drive failing per SMART
Summary: SMART data indicates imminent drive failure (reallocated sectors, pending sectors, or aggregate health != PASSED). Back up data and replace the drive.

---

## Category: ZFS

### Rule: zfs_pool_unhealthy

URL: https://glassmkr.com/docs/rules/zfs_pool_unhealthy
Priority: P1
Title: ZFS pool unhealthy
Summary: ZFS pool in non-OPTIMAL state. Severity scales with vdev redundancy class (Crucible v0.10.4+): SUSPENDED pools and FAULTED top-level vdevs page critical; DEGRADED on single/raidz1/mirror_2way pages critical; raidz3/mirror_3way+ pages warning. L2ARC failures emit at info severity (no data loss). SLOG faults handled by zfs_slog_faulted.

---

### Rule: zfs_scrub_errors

URL: https://glassmkr.com/docs/rules/zfs_scrub_errors
Priority: P1
Title: ZFS scrub found errors
Summary: ZFS pool's most recent scrub detected checksum or repair errors, or the pool has not been scrubbed in over 30 days. Errors suggest failing disks or silent corruption; missing scrub is preventive-maintenance gap.

---

### Rule: zfs_slog_faulted

URL: https://glassmkr.com/docs/rules/zfs_slog_faulted
Priority: P1
Title: ZFS SLOG vdev faulted
Summary: A ZIL log vdev (SLOG) is FAULTED or REMOVED. Sync-write durability for the pool is compromised until the SLOG is replaced.

---

## Category: Filesystem

### Rule: disk_fill_projection

URL: https://glassmkr.com/docs/rules/disk_fill_projection
Priority: P1
Title: Disk fill projection imminent
Summary: Linear projection on filesystem available_bytes indicates exhaustion within 24h (P1) or 7d (P2). Companion to disk_space_high (absolute %).

---

### Rule: disk_space_high

URL: https://glassmkr.com/docs/rules/disk_space_high
Priority: P2
Title: Disk space high
Summary: Filesystem usage at or above the configured threshold (default 85%). At 100% services that write to this filesystem will fail; at >=95% the buffer is hours-not-days.

---

### Rule: fd_exhaustion

URL: https://glassmkr.com/docs/rules/fd_exhaustion
Priority: P1
Title: File descriptor exhaustion
Summary: Host-wide file descriptor usage at or above 80% of fs.file-max OR a single process at or above 80% of its RLIMIT_NOFILE soft limit. The per-process path activates with Crucible v0.11.0+; older agents only emit on the host-wide path.

---

### Rule: filesystem_readonly

URL: https://glassmkr.com/docs/rules/filesystem_readonly
Priority: P1
Title: Filesystem remounted read-only
Summary: Kernel forced a filesystem to read-only mode, usually due to I/O errors. Services that write to this mount will fail. Data already on the filesystem is readable; new writes are not.

---

### Rule: inode_high

URL: https://glassmkr.com/docs/rules/inode_high
Priority: P2
Title: Inode usage high
Summary: Filesystem has many small files; inode usage at or above 85% of the table. At 100%, file creation fails with ENOSPC even though `df -h` shows free space.

---

### Rule: lvm_thinpool_metadata_high

URL: https://glassmkr.com/docs/rules/lvm_thinpool_metadata_high
Priority: P1
Title: LVM thin pool metadata near full
Summary: LVM thin pool metadata volume at or above 80%. Metadata exhaustion is silent and catastrophic: writes across all thin volumes in the pool start failing in unpredictable ways at 100%. Extend the metadata volume before it fills.

---

## Category: Memory & CPU

### Rule: cpu_high

URL: https://glassmkr.com/docs/rules/cpu_high
Priority: P2
Title: CPU usage high
Summary: Aggregate CPU utilization at or above 90% (idle below 10%). Critical at >=98% (idle <2%). Either a runaway process or workload exceeding capacity.

---

### Rule: cpu_iowait_high

URL: https://glassmkr.com/docs/rules/cpu_iowait_high
Priority: P2
Title: CPU I/O wait high
Summary: CPU is spending 20%+ of its time waiting on disk I/O. Indicates storage bottleneck; either an overwhelmed device or runaway I/O from one process.

---

### Rule: cpu_pressure_high

URL: https://glassmkr.com/docs/rules/cpu_pressure_high
Priority: P2
Title: CPU pressure stall sustained
Summary: PSI reports CPU contention persistently above threshold. Aggregate signal across the host; subordinates cpu_high and load_high to this incident when it fires. Not available on stock RHEL-family kernels (PSI ships disabled there; enable with the psi=1 boot parameter).

---

### Rule: io_pressure_high

URL: https://glassmkr.com/docs/rules/io_pressure_high
Priority: P2
Title: I/O pressure sustained
Summary: PSI reports I/O contention with corroborating disk-latency or error signal. Catches the modern-NVMe case where iowait stays at zero but the CPU is briefly blocking on I/O often enough to matter. Not available on stock RHEL-family kernels (PSI ships disabled there; enable with the psi=1 boot parameter).

---

### Rule: load_high

URL: https://glassmkr.com/docs/rules/load_high
Priority: P3
Title: Load average high
Summary: 1-minute load average exceeds 2x the CPU core count for several minutes. Usually indicates an I/O bottleneck (high D-state processes) rather than pure CPU saturation.

---

### Rule: mem_pressure_high

URL: https://glassmkr.com/docs/rules/mem_pressure_high
Priority: P1
Title: Memory pressure sustained
Summary: PSI reports memory contention with active paging or rapid MemAvailable decline. Real pressure signal, not used% noise. Not available on stock RHEL-family kernels (PSI ships disabled there; enable with the psi=1 boot parameter).

---

### Rule: oom_kills

URL: https://glassmkr.com/docs/rules/oom_kills
Priority: P1
Title: OOM killer recently fired
Summary: Kernel out-of-memory killer terminated one or more processes in the recent window. Severe memory pressure or a memory leak. Killed services may be down.

---

### Rule: ram_high

URL: https://glassmkr.com/docs/rules/ram_high
Priority: P3
Title: RAM usage high
Summary: Memory pressure on the host. Warning at 90%, critical at 95%. Sustained pressure leads to swap thrashing and OOM kills.

---

### Rule: swap_high

URL: https://glassmkr.com/docs/rules/swap_high
Priority: P2
Title: Swap usage high
Summary: Swap usage at or above 50%. Swap I/O is 10-100x slower than RAM; sustained swap = thrashing. Critical band (>=80%) indicates imminent service degradation.

---

## Category: Network

### Rule: accept_backlog_or_syn_flood

URL: https://glassmkr.com/docs/rules/accept_backlog_or_syn_flood
Priority: P1
Title: Accept backlog or SYN flood
Summary: 2 or more of conntrack_exhaustion / listen_overflow / tcp_retrans_high are active on the same host within 5 minutes. Indicates accept-queue buildup or SYN flood.

---

### Rule: bond_slave_down

URL: https://glassmkr.com/docs/rules/bond_slave_down
Priority: P1
Title: Bond slave interface down
Summary: A slave NIC in a bonded interface has MII status down. The bond is running with reduced redundancy; one more failure breaks the bond entirely.

---

### Rule: conntrack_exhaustion

URL: https://glassmkr.com/docs/rules/conntrack_exhaustion
Priority: P1
Title: Conntrack table near full
Summary: Linux nf_conntrack table is at or above 75% capacity. At 100%, new connections are silently dropped; services appear to work but new clients can't connect. Critical band (>=90%) means dropping is imminent.

---

### Rule: interface_errors

URL: https://glassmkr.com/docs/rules/interface_errors
Priority: P2
Title: Interface errors high
Summary: Network interface reports elevated CRC / frame / carrier errors (physical layer) OR elevated packet drops (software ring/softirq layer). Tier red = critical (cable swap or kernel tuning urgent); tier yellow = warning.

---

### Rule: interface_saturation

URL: https://glassmkr.com/docs/rules/interface_saturation
Priority: P3
Title: Interface near saturation
Summary: Network interface utilization above the configured threshold (default 90% of negotiated speed). Plan bandwidth upgrade or traffic shaping; queue depth growth predicts the next OOMing connection-handling daemon.

---

### Rule: lacp_partner_lost

URL: https://glassmkr.com/docs/rules/lacp_partner_lost
Priority: P1
Title: LACP partner lost
Summary: Bond MII layer reports up but the LACP partner is unsynchronized. The bond appears functional while traffic is dropped by the switch. Also emits a warning when the active aggregator has fewer ports than configured (redundancy reduced).

---

### Rule: link_speed_mismatch

URL: https://glassmkr.com/docs/rules/link_speed_mismatch
Priority: P2
Title: Link speed mismatch
Summary: Network interface negotiated a speed at least 2x below the NIC's highest advertised mode. Usually physical-layer or autoneg; sometimes PCIe slot bound; rarely a deliberate operator choice on multi-mode NICs (the 2x floor filters the small-gap case).

---

### Rule: listen_overflow

URL: https://glassmkr.com/docs/rules/listen_overflow
Priority: P2
Title: TCP listen-queue dropping connections
Summary: /proc/net/netstat TcpExt ListenOverflows (any non-zero rate) or ListenDrops (at least 1.0/sec sustained) is incrementing; the kernel is dropping arriving connections at accept-queue level. Either the application can't accept() fast enough or net.core.somaxconn is too small for the offered load. The ListenDrops floor avoids paging on quiet hosts that see sporadic long-tail SYN-scan drops.

---

### Rule: softnet_drops

URL: https://glassmkr.com/docs/rules/softnet_drops
Priority: P1
Title: Kernel softnet dropping packets
Summary: /proc/net/softnet_stat reports kernel input-queue drops at sustained rate (>1 pkt/s). The NET_RX softirq backlog is filling faster than the kernel can process; packets are being silently discarded. Often correlated with conntrack pressure or CPU pressure.

---

### Rule: tcp_retrans_high

URL: https://glassmkr.com/docs/rules/tcp_retrans_high
Priority: P2
Title: TCP retransmit rate elevated
Summary: TCP retransmit ratio (retransmits / segments sent) over the most recent snapshot interval exceeds 2%, gated so the ratio only fires at meaningful traffic volume: at least 1.0 retransmits/sec AND an implied segment rate of at least 50/sec. Above 1% commonly impacts performance; above 5% significantly degrades throughput. The volume gate prevents false alarms on quiet hosts where a handful of retransmits over few segments produces a misleadingly high ratio.

---

## Category: Hardware (BMC/IPMI)

### Rule: cmos_battery_low

URL: https://glassmkr.com/docs/rules/cmos_battery_low
Priority: P3
Title: CMOS battery low
Summary: Motherboard CMOS coin-cell battery (CR2032) is reading below 2.6V; the cell is ~80% discharged and may fail to hold BIOS settings across the next power cycle. Replacement is scheduled, not emergency; the host runs fine until cold boot.

---

### Rule: cpu_temperature_high

URL: https://glassmkr.com/docs/rules/cpu_temperature_high
Priority: P1
Title: CPU temperature high
Summary: CPU thermal reading at or above the warning threshold (default 80°C; critical 90°C). At critical, thermal throttling kicks in and silicon damage risk climbs.

---

### Rule: ecc_errors

URL: https://glassmkr.com/docs/rules/ecc_errors
Priority: P1
Title: ECC memory errors
Summary: Memory controller reported one or more uncorrectable ECC errors. Data corruption has occurred; the DIMM is failing. Replace immediately.

---

### Rule: ipmi_fan_failure

URL: https://glassmkr.com/docs/rules/ipmi_fan_failure
Priority: P1
Title: IPMI fan failure
Summary: BMC reports one or more chassis fans in critical state or at 0 RPM. Cooling capacity is reduced; CPU temperatures may climb and trigger thermal throttling or emergency shutdown.

---

### Rule: ipmi_sel_critical

URL: https://glassmkr.com/docs/rules/ipmi_sel_critical
Priority: P1
Title: IPMI SEL critical events
Summary: BMC System Event Log contains one or more critical-severity asserted events in the last N days (default 30). Critical events indicate real hardware faults; DIMM, PSU, fan, voltage, or temperature.

---

### Rule: ipmi_sel_full

URL: https://glassmkr.com/docs/rules/ipmi_sel_full
Priority: P3
Title: IPMI SEL full or near-full
Summary: The BMC System Event Log is full or near-full. A full SEL silently stops recording new events, so ipmi_sel_critical and the SEL-derived ECC counts go deaf to every future hardware fault. Export the SEL, then clear it so the BMC records again.

---

### Rule: mce_uncorrected

URL: https://glassmkr.com/docs/rules/mce_uncorrected
Priority: P0
Title: Uncorrected machine check exception
Summary: EDAC reports an uncorrected memory error. Replace the affected DIMM.

---

### Rule: memory_channels_underpopulated

URL: https://glassmkr.com/docs/rules/memory_channels_underpopulated
Priority: P3
Title: Memory channels under-populated
Summary: The board exposes more memory channels than have DIMMs installed (or DIMMs run below rated speed), so peak memory bandwidth is reduced. On EPYC-class boards the alert also flags placement, when the installed DIMMs double up on adjacent channels (one memory-controller group) instead of spreading one per group; the same DIMM count placed wrong idles controllers and weakens interleaving. Not a fault; a hardware-population advisory. Fix is physical DIMM installation or rebalancing during a maintenance window, per the board manual's population table.

---

### Rule: psu_redundancy_loss

URL: https://glassmkr.com/docs/rules/psu_redundancy_loss
Priority: P1
Title: PSU redundancy lost
Summary: One or more PSUs are in fault, absent, or degraded state. Single power failure now risks full server outage. Dell BMCs report this via an aggregate sensor; other vendors via per-PSU sensors.

---

## Category: GPU

### Rule: gpu_corrected_ecc_storm

URL: https://glassmkr.com/docs/rules/gpu_corrected_ecc_storm
Priority: P3
Title: GPU corrected-ECC level high
Summary: GPU corrected-ECC counter is high or single-bit retired pages are non-zero. SBE storms typically precede DBE faults; this rule gives operators time to plan preventive replacement before uncorrected ECC fires.

---

### Rule: gpu_driver_or_firmware_drift

URL: https://glassmkr.com/docs/rules/gpu_driver_or_firmware_drift
Priority: P3
Title: GPU vbios drift within host
Summary: Multiple GPUs of the same model on this host report different vbios versions. Within-host vbios drift typically indicates a failed firmware update or mixed-batch installation.

---

### Rule: gpu_driver_unsafe_reboot

URL: https://glassmkr.com/docs/rules/gpu_driver_unsafe_reboot
Priority: P1
Title: GPU will not survive a reboot
Summary: This host has an NVIDIA GPU, but either the nvidia kernel module is not loaded (the GPU is unusable now) or nouveau is not blacklisted. If nouveau is not blacklisted it binds the GPU first on the next boot, the nvidia driver cannot load, nvidia-smi fails, and a marketplace (Vast) host silently de-lists itself. The fix is non-disruptive and only affects the next boot.

---

### Rule: gpu_pcie_link_degraded

URL: https://glassmkr.com/docs/rules/gpu_pcie_link_degraded
Priority: P2
Title: GPU PCIe link degraded
Summary: GPU's current PCIe gen or width is below the GPU's advertised maximum. Host-to-GPU bandwidth is capped below the GPU's capability; meaningful for large-model loading and PCIe-attached weights, catastrophic for training-style workloads.

---

### Rule: gpu_power_cap_throttling

URL: https://glassmkr.com/docs/rules/gpu_power_cap_throttling
Priority: P2
Title: GPU power-cap throttling
Summary: GPU is being throttled by software power cap (sw_power_cap) or hardware power brake (hw_power_brake_slowdown). May be intentional (operator-configured limit) or unexpected (PSU sizing, chassis power policy).

---

### Rule: gpu_thermal_critical

URL: https://glassmkr.com/docs/rules/gpu_thermal_critical
Priority: P1
Title: GPU thermal critical
Summary: GPU die temperature at or above the HW slowdown threshold, or the kernel reports a hardware thermal slowdown. A software thermal slowdown at the card's thermal target (normal load behavior) does not fire. Sustained operation at thermal limits accelerates wear and reduces throughput. Boot grace 300s for post-boot sensor stabilisation.

---

### Rule: gpu_uncorrected_ecc

URL: https://glassmkr.com/docs/rules/gpu_uncorrected_ecc
Priority: P0
Title: GPU uncorrected ECC or DBE retired pages
Summary: GPU reports uncorrected ECC errors, double-bit ECC retired pages, or pending retirements. Uncorrected ECC means error correction could not recover; in-flight data may have been corrupted. Pending retirements require a reboot.

---

### Rule: gpu_xid_critical

URL: https://glassmkr.com/docs/rules/gpu_xid_critical
Priority: P0
Title: GPU XID critical event
Summary: NVIDIA XID error classified as critical per NVIDIA's published XID severity table. Hardware-witnessed fault on the GPU; data may be at risk and the workload likely degraded.

---

### Rule: nvlink_link_down

URL: https://glassmkr.com/docs/rules/nvlink_link_down
Priority: P1
Title: NVLink link down
Summary: An NVLink on a multi-GPU host is in the down state. Multi-GPU bandwidth is reduced; if the GPU participates in NCCL collectives the entire training/inference job's latency degrades.

---

## Category: Time & Services

### Rule: clock_drift

URL: https://glassmkr.com/docs/rules/clock_drift
Priority: P2
Title: Clock drift
Summary: System clock is at least 5 seconds off from upstream NTP. Critical at >=60s; TLS validation, log correlation, database replication, and cron all break.

---

### Rule: ntp_not_synced

URL: https://glassmkr.com/docs/rules/ntp_not_synced
Priority: P2
Title: NTP not synced
Summary: Either the kernel clock is unsynchronized (critical; drift in progress) OR the NTP daemon has stopped while the clock is still synced (warning; drift will start once kernel state expires).

---

### Rule: service_flapping

URL: https://glassmkr.com/docs/rules/service_flapping
Priority: P1
Title: systemd service flapping
Summary: A systemd unit has hit its start-limit (systemd stopped restarting it) OR has restarted 5+ times. A service that can't stabilise consumes resources without delivering value; investigate before bumping restart limits.

---

### Rule: systemd_service_failed

URL: https://glassmkr.com/docs/rules/systemd_service_failed
Priority: P1
Title: systemd service failed
Summary: One or more systemd units are in the failed state. The service is not running; dependent functionality is offline. Crucible 0.9.2+ also ships the last 5 journal lines per failed unit in evidence so root cause is one click away.

---

### Rule: systemd_service_oom_killed

URL: https://glassmkr.com/docs/rules/systemd_service_oom_killed
Priority: P1
Title: systemd service killed by OOM
Summary: systemd reports a failed unit with Result=oom-kill. The kernel OOM killer terminated the service; pair with the host-level oom_kills emission to find the underlying memory pressure source.

---

### Rule: unexpected_reboot

URL: https://glassmkr.com/docs/rules/unexpected_reboot
Priority: P1
Title: Unexpected reboot
Summary: Server rebooted without an operator-acknowledged planned reboot. Possible causes: kernel panic, hardware fault (PSU brownout, thermal shutdown, watchdog), power outage, or remote reboot via BMC.

---

## Category: Security & Patching

### Rule: bios_firmware_age

URL: https://glassmkr.com/docs/rules/bios_firmware_age
Priority: P3
Title: BIOS firmware metadata is aging
Summary: The host's SMBIOS BIOS release date is more than 24 months old. This is NOT a fault and NOT necessarily out of date: the installed BIOS may still be the latest release the board vendor has ever published. It is a prompt to confirm, against the vendor's catalog, that the BIOS/UEFI and BMC firmware are current, since stale platform firmware is a common source of unpatched hardware-level vulnerabilities and errata.

Acknowledge this advisory once you have confirmed the installed version is the latest published for this board, or after you have scheduled the update.


---

### Rule: kernel_needs_reboot

URL: https://glassmkr.com/docs/rules/kernel_needs_reboot
Priority: P2
Title: Reboot required (newer kernel installed)
Summary: A newer kernel package is installed on disk but the running kernel is older. Security patches in the new kernel are not active until reboot.

---

### Rule: kernel_vulnerabilities

URL: https://glassmkr.com/docs/rules/kernel_vulnerabilities
Priority: P2
Title: Kernel vulnerability mitigations missing
Summary: One or more CPU vulnerability mitigations (Spectre, Meltdown, MDS, TSA, Downfall, etc.) report unmitigated or partial coverage in /sys/devices/system/cpu/vulnerabilities/.

Alert message shows one bullet per unmitigated vuln plus a state hint:

  Unmitigated CPU vulnerabilities:
  • tsa: Vulnerable: Clear CPU buffers attempted, no microcode  [software band-aid engaged; awaiting AMD microcode]
  • gather_data_sampling: Vulnerable: No microcode  [update CPU microcode + kernel and reboot; vendor-side only if still Vulnerable afterwards]

Default action is to update the CPU microcode + kernel packages and reboot (on Debian/Ubuntu the microcode is in the non-free-firmware component; enable it if apt reports no candidate). A vuln is vendor-side (ACK-able) only when a kernel software band-aid is already engaged for every unmitigated vuln AND updating microcode does not clear it. The alert auto-resolves when /sys flips to "Mitigation: ...".


---

### Rule: no_firewall

URL: https://glassmkr.com/docs/rules/no_firewall
Priority: P1
Title: No host firewall active
Summary: No active firewall rules detected. All listening ports are reachable from any network the host is connected to, unless protected by network-level ACLs (VPC, cloud SG, on-prem ACL).

---

### Rule: os_end_of_life

URL: https://glassmkr.com/docs/rules/os_end_of_life
Priority: P2
Title: OS release past (or nearing) end of life
Summary: This host's operating-system release is at, past, or within 180 days of the end of its standard security support, per the endoflife.date lifecycle dataset.

This is NOT automatically "unsupported": many releases keep receiving security fixes past standard support if enrolled in an extended-support programme (Ubuntu Pro/ESM, RHEL EUS/ELS). The alert reconciles the release EOL date with this host's actual enrollment where the agent can read it. Where enrollment cannot be verified, the wording says so rather than assuming the worst.

Resolve by upgrading to a supported release, or by confirming/entering extended support. Acknowledge once you have a plan.


---

### Rule: pending_security_updates

URL: https://glassmkr.com/docs/rules/pending_security_updates
Priority: P2
Title: Pending security updates
Summary: Package manager reports one or more security updates available AND auto-updates are not configured. Manual patching is required; counterpart to unattended_upgrades_disabled which fires when the auto-update mechanism itself is missing.

---

### Rule: server_unreachable

URL: https://glassmkr.com/docs/rules/server_unreachable
Priority: P1
Title: Server unreachable
Summary: Dashboard has not received a snapshot from this server in 2x the configured collection interval (default 10 minutes). Either the Crucible agent stopped reporting, the network is down, or the server is offline. Alert auto-resolves on next successful snapshot.

---

### Rule: ssh_config_unapplied

URL: https://glassmkr.com/docs/rules/ssh_config_unapplied
Priority: P2
Title: SSH config changed but not applied
Summary: sshd_config was modified after the sshd daemon last loaded its configuration. The running daemon is still using the previous config, so any change (including security hardening like disabling root password login) is not yet in effect. Reload or restart sshd to apply.

---

### Rule: ssh_root_password

URL: https://glassmkr.com/docs/rules/ssh_root_password
Priority: P1
Title: SSH allows root password login
Summary: sshd allows root login with password. Brute-force-able from the network. Switch to key-only root login (still works for key-based ops); ideally disable root SSH entirely and use a sudo-equipped operator account.

---

### Rule: unattended_upgrades_disabled

URL: https://glassmkr.com/docs/rules/unattended_upgrades_disabled
Priority: P3
Title: Unattended security upgrades disabled
Summary: No automatic security update mechanism is configured. The host is at the operator's mercy for patch cadence; if patches are pending, the counterpart pending_security_updates rule will fire.

---