# Monitoring and debugging
URL: https://metorial.com/docs/build/sandboxes/monitoring

Read console logs, trace what happened to a machine, find out why it failed, and track resource usage.

---

When a machine fails to start, stops unexpectedly, or runs slowly, these tools tell you what happened and why.

<PageIntro>
  <Learn>
    * How to read a machine's console output
    * How to trace events, operations, and errors
    * How to read CPU, memory, network, and disk usage
  </Learn>
</PageIntro>

## When a machine fails [#when-a-machine-fails]

A machine in the `failed` state has a `failure` field with a code and a message:

| Code                | Meaning                                                                               |
| ------------------- | ------------------------------------------------------------------------------------- |
| `provision_failed`  | The platform couldn't allocate the machine                                            |
| `start_failed`      | The machine booted but didn't come up, often because a startup command or hook failed |
| `disk_unavailable`  | An attached disk couldn't be mounted                                                  |
| `image_unavailable` | The image isn't available in the machine's region                                     |
| `resize_failed`     | A disk couldn't be resized                                                            |
| `timeout`           | An operation took too long                                                            |
| `cancelled`         | The operation was cancelled                                                           |
| `internal_error`    | Something went wrong on Metorial's side                                               |

For `start_failed`, check the errors and console output below. Fix the cause, for example with `machines update`, then run `machines start` again.

## Console output [#console-output]

Metorial tracks each run of a machine as an instance, numbered from one. A new instance appears when the machine boots fresh, for example after a restart. An instance's logs hold its console output, including your startup commands and entrypoint.

```bash
metorial-admin sandbox machines instances list $MACHINE
metorial-admin sandbox machines instances logs $MACHINE $INSTANCE --direction backward --limit 200
```

Logs are paged by line. Pass the returned cursor back with `--cursor` to read further.

## Errors [#errors]

Errors are failures reported from inside the machine, such as a hook script that exited non-zero. Each error records the operation and stage it happened in, the exit code, and the process output.

```bash
metorial-admin sandbox machines errors list $MACHINE
metorial-admin sandbox machines errors get $MACHINE $ERROR
```

## Events and operations [#events-and-operations]

Events form a timeline of everything that happened to a machine. Filter them by what caused the event: `api` for your requests, `policy` for the lifecycle policy, `agent` and `guest` for activity inside the machine, and `system` for the platform.

```bash
metorial-admin sandbox machines events list $MACHINE --source policy
```

Operations are the actions behind those events: starts, pauses, hibernations, restarts, and terminations. Each records what triggered it (for example `api`, `idle_policy`, `paused_policy`, or `failure`) and how it ended. Use them to answer questions such as why a machine paused.

```bash
metorial-admin sandbox machines operations list $MACHINE --trigger idle_policy
```

## Resource usage [#resource-usage]

`machines usage` returns the latest samples of CPU, memory, network, and disk activity, so you can see whether a machine is busy right now.

```bash
metorial-admin sandbox machines usage $MACHINE
```

For trends, use the usage history. `summary` totals a time range, and `timeline` splits it into buckets for charts.

```bash
metorial-admin sandbox machines usage-history summary $MACHINE \
  --from 2026-10-15T00:00:00Z --to 2026-10-16T00:00:00Z

metorial-admin sandbox machines usage-history timeline $MACHINE \
  --from 2026-10-15T00:00:00Z --to 2026-10-16T00:00:00Z --interval 1h
```

Each bucket reports runtime, CPU core seconds, average and peak cores, average and peak memory, and bytes sent, received, read, and written. Leave out `--interval` to get about 500 points.

Minute-level history is kept for 7 days and hourly history for 30 days. Usage history measures what the machine used and isn't billing data.

<CodeBlockTabs defaultValue="TypeScript" groupId="language" mode="code">
  <CodeBlockTabsList>
    <CodeBlockTabsTrigger value="TypeScript">
      TypeScript
    </CodeBlockTabsTrigger>

    <CodeBlockTabsTrigger value="Python">
      Python
    </CodeBlockTabsTrigger>
  </CodeBlockTabsList>

  <CodeBlockTab value="TypeScript">
    ```typescript
    let timeline = await metorial.sandbox.machines.usageHistory.timeline(machineId, {
      from: new Date(Date.now() - 24 * 60 * 60 * 1000),
      to: new Date(),
      interval: '1h'
    });
    ```
  </CodeBlockTab>

  <CodeBlockTab value="Python">
    ```python
    from datetime import datetime, timedelta, timezone

    now = datetime.now(timezone.utc)
    timeline = metorial.sandbox.machines.usage_history.timeline(
        machine_id,
        from_=now - timedelta(days=1),
        to=now,
        interval="1h",
    )
    ```
  </CodeBlockTab>
</CodeBlockTabs>

The dashboard shows the same data as charts on the machine's page.