Demo · sample data
This morning, erp-db stopped responding. Find what changed.
The ProxTrace console, read-only, on fictional organisations including cluster pve-prod. Everything is computed in your browser: no real data, nothing is saved.
Events
pve-prod
12 / 12
Week of 5 Octoberpve-prod
| When | Event | Resource | Author |
|---|---|---|---|
| Thu 14:32:07 | qemu:104 - erp-db | admin@pve | |
| Thu 14:05:51 | qemu:112 - gitlab | root@pam | |
| Thu 13:58:20 | lxc:201 - dns-int | root@pam | |
| Thu 13:41:02 | qemu:118 - win-rds | ci@pve!deploy | |
| Thu 13:12:44 | node:pve-02 | admin@pve | |
| Thu 11:20:36 | lxc:203 - proxy-int | root@pam | |
| Thu 10:02:15 | qemu:112 - gitlab | admin@pve | |
| Wed 15:03:48 | qemu:104 - erp-db | admin@pve | |
| Tue 22:51:37 | node:pve-01 | root@pam | |
| Tue 02:00:11 | lxc:201 - dns-int | root@pam | |
| Mon 16:40:05 | lxc:201 - dns-int | root@pam | |
| Mon 09:14:22 | qemu:104 - erp-db | admin@pve | |
| No events match the filters. | |||
Simple view · correlated steps grouped together12 events
Change · qmconfig
Configuration changed
- When
- 2026-10-08 14:32:07 CEST
- Resource
- qemu:104 - erp-db
- Node
- pve-02
- Author
- admin@pve
- Source IP
- 10.20.0.31
- Result
- Succeeded
- Category
- Change
| Parameter | Before | After |
|---|---|---|
| cores | 4 | 8 |
| balloon | 0 | 8192 |
Evidence
UPID:pve-02:0003A1F2:0B5C3D21:6705F1A7:qmconfig:104:admin@pve:Migration · qmigrate
Live migration
- When
- 2026-10-08 14:05:51 CEST
- Resource
- qemu:112 - gitlab
- Node
- pve-01
- Author
- root@pam
- Source IP
- 10.20.0.15
- Result
- Succeeded
- Category
- Migration
| Parameter | Before | After |
|---|---|---|
| node | pve-01 | pve-03 |
Evidence
UPID:pve-01:00029B10:0B5A8F02:6705EB2F:qmigrate:112:root@pam:Backup · vzdump
Backup failed
- When
- 2026-10-08 13:58:20 CEST
- Resource
- lxc:201 - dns-int
- Node
- pve-02
- Author
- root@pam
- Result
- Failed
- Category
- Backup
Scheduled task, second consecutive failure.
Evidence
ERROR: Backup of VM 201 failed - backup write data failed: command error: No space left on device (os error 28)Snapshot · qmsnapshot
Snapshot created
- When
- 2026-10-08 13:41:02 CEST
- Resource
- qemu:118 - win-rds
- Node
- pve-03
- Author
- ci@pve!deploy
- Source IP
- 10.20.0.40
- Result
- Succeeded
- Category
- Snapshot
| Parameter | Before | After |
|---|---|---|
| snapshot | — | avant-maj-2026-10-08 |
Evidence
UPID:pve-03:0001F2C4:0B59A611:6705E8DE:qmsnapshot:118:ci@pve!deploy:Access · authentication
API login
- When
- 2026-10-08 13:12:44 CEST
- Resource
- node:pve-02
- Node
- pve-02
- Author
- admin@pve
- Source IP
- 10.20.0.31
- Result
- Succeeded
- Category
- Access
Evidence
pvedaemon[2211]: <root@pam> successful auth for user 'admin@pve'Lifecycle · vzstart
Started
- When
- 2026-10-08 11:20:36 CEST
- Resource
- lxc:203 - proxy-int
- Node
- pve-01
- Author
- root@pam
- Source IP
- 10.20.0.15
- Result
- Succeeded
- Category
- Lifecycle
Evidence
UPID:pve-01:00027A3E:0B53C9F0:6705D4F4:vzstart:203:root@pam:Storage · qmresize
Disk expanded
- When
- 2026-10-08 10:02:15 CEST
- Resource
- qemu:112 - gitlab
- Node
- pve-01
- Author
- admin@pve
- Source IP
- 10.20.0.31
- Result
- Succeeded
- Category
- Storage
| Parameter | Before | After |
|---|---|---|
| scsi1 (size) | 200G | 300G |
Evidence
UPID:pve-01:00026E11:0B4F12A8:6705C287:resize:112:admin@pve:Security · api.firewall
Firewall rule deleted
- When
- 2026-10-07 15:03:48 CEST
- Resource
- qemu:104 - erp-db
- Node
- pve-02
- Author
- admin@pve
- Source IP
- 10.20.0.31
- Result
- Succeeded
- Category
- Security
| Parameter | Before | After |
|---|---|---|
| rule 2 | IN ACCEPT -p tcp -dport 5432 -source 10.20.30.0/24 | (deleted) |
Evidence
DELETE /api2/json/nodes/pve-02/qemu/104/firewall/rules/2 · admin@pve · 10.20.0.31Access · authentication
Login outside business hours
- When
- 2026-10-06 22:51:37 CEST
- Resource
- node:pve-01
- Node
- pve-01
- Author
- root@pam
- Source IP
- 10.20.0.15
- Result
- Succeeded
- Category
- Access
Outside configured working days and hours.
Evidence
pvedaemon[1843]: <root@pam> successful auth for user 'root@pam'Backup · vzdump
Backup failed
- When
- 2026-10-06 02:00:11 CEST
- Resource
- lxc:201 - dns-int
- Node
- pve-02
- Author
- root@pam
- Result
- Failed
- Category
- Backup
Scheduled task.
Evidence
ERROR: Backup of VM 201 failed - backup write data failed: command error: No space left on device (os error 28)Migration · vzmigrate
Migration
- When
- 2026-10-05 16:40:05 CEST
- Resource
- lxc:201 - dns-int
- Node
- pve-01
- Author
- root@pam
- Source IP
- 10.20.0.15
- Result
- Succeeded
- Category
- Migration
| Parameter | Before | After |
|---|---|---|
| node | pve-01 | pve-02 |
Evidence
UPID:pve-01:0001B0C7:0B1E7A40:67016C15:vzmigrate:201:root@pam:Change · qmconfig
Configuration changed
- When
- 2026-10-05 09:14:22 CEST
- Resource
- qemu:104 - erp-db
- Node
- pve-02
- Author
- admin@pve
- Source IP
- 10.20.0.31
- Result
- Succeeded
- Category
- Change
| Parameter | Before | After |
|---|---|---|
| memory | 8192 | 16384 |
Evidence
UPID:pve-02:000214A0:0B1A0F3C:670110CE:qmconfig:104:admin@pve:Machines
pve-prod
Machines6 guests
| Machine | Node | State | Latest backup |
|---|---|---|---|
| pve-02 | Running | Succeeded · 08/10 02:04 | |
| pve-03 | Running | Succeeded · 08/10 02:11 | |
| pve-03 | Stopped | None observed | |
| pve-01 | Running | Succeeded · 08/10 02:20 | |
| pve-02 | Running | Failed · 08/10 13:58 | |
| pve-01 | Running | Succeeded · 08/10 02:26 |
QEMU VM
qemu:104 - erp-db
- Node
- pve-02
- State
- Running
- Resources
- 8 vCPU · 16 GiB
- Guest agent
- Enabled, responding
- Observed IP
- 10.20.30.14
- Latest backup
- Succeeded · 08/10 02:04
History for the week
- Thu 08/10 14:32qmconfig
- Wed 07/10 15:03api.firewall
- Mon 05/10 09:14qmconfig
QEMU VM
qemu:112 - gitlab
- Node
- pve-03
- State
- Running
- Resources
- 4 vCPU · 8 GiB
- Guest agent
- Enabled, responding
- Observed IP
- 10.20.30.22
- Latest backup
- Succeeded · 08/10 02:11
History for the week
- Thu 08/10 14:05qmigrate
- Thu 08/10 10:02qmresize
QEMU VM
qemu:118 - win-rds
- Node
- pve-03
- State
- Stopped
- Resources
- 4 vCPU · 12 GiB
- Guest agent
- Cannot be verified: VM stopped
- Observed IP
- —
- Latest backup
- None observed
History for the week
- Thu 08/10 13:41qmsnapshot
QEMU VM
qemu:121 - files
- Node
- pve-01
- State
- Running
- Resources
- 2 vCPU · 4 GiB
- Guest agent
- Enabled, responding
- Observed IP
- 10.20.30.31
- Latest backup
- Succeeded · 08/10 02:20
History for the week
- No changes observed this week.
LXC container
lxc:201 - dns-int
- Node
- pve-02
- State
- Running
- Resources
- 1 vCPU · 512 MiB
- Guest agent
- —
- Observed IP
- 10.20.30.53
- Latest backup
- Failed · 08/10 13:58
History for the week
- Thu 08/10 13:58vzdump
- Tue 06/10 02:00vzdump
- Mon 05/10 16:40vzmigrate
LXC container
lxc:203 - proxy-int
- Node
- pve-01
- State
- Running
- Resources
- 2 vCPU · 1 GiB
- Guest agent
- —
- Observed IP
- 10.20.30.80
- Latest backup
- Succeeded · 08/10 02:26
History for the week
- Thu 08/10 11:20vzstart
Security & compliance
pve-prod
Check registerreport from 08/10, 14:35
| Ref. | Check | Verdict |
|---|---|---|
| 14.1 | Fail | |
| 5.3 | Fail | |
| 17.5 | Not assessable | |
| 2.3-04 | Out of scope | |
| 2.1 | Pass | |
| 3.1 | Pass |
Check 14.1
Backups and retention
Fail
lxc:201 - dns-int: two consecutive failures (06/10 02:00, 08/10 13:58), no successful backup since 05/10.
Values by node
- pve-01
- 2 guests · successful backups
- pve-02
- lxc:201 failed · qemu:104 succeeded
- pve-03
- qemu:112 succeeded · none observed for qemu:118
- Suggested action
- Free up space on pbs-01 storage or review its retention, then run the backup for 201 again.
- Sources
- vzdump tasks, storage state
- Evidence limitation
- Backup contents have not been verified: a restore has not been tested.
Check 5.3
root@pam account management
Fail
root@pam opened an interactive session on 06/10 at 22:51 from 10.20.0.15 and performs migrations.
Values by node
- pve-01
- 1 interactive root@pam login over 7 days
- pve-02
- none
- pve-03
- none
- Suggested action
- Use named accounts with roles; reserve root@pam for emergency access.
- Sources
- Authentication logs, tasks
- Evidence limitation
- Physical console logins are not observed.
Check 17.5
QEMU Guest Agent
Not assessable
qemu:118 - win-rds is stopped: the agent response cannot be verified.
Values by node
- pve-01
- qemu:121 responding
- pve-02
- qemu:104 responding
- pve-03
- qemu:112 responding · qemu:118 cannot be verified
- Suggested action
- Start the VM or wait for the next collection.
- Sources
- QEMU configuration, guest agent response
- Evidence limitation
- A stopped VM is never reported as having no agent.
Check 2.3-04
QDevice location
Out of scope
corosync.conf read in full on all three nodes: no QDevice configured. The cluster has an odd number of nodes.
Values by node
- pve-01
- corosync.conf · no QDevice
- pve-02
- corosync.conf · no QDevice
- pve-03
- corosync.conf · no QDevice
- Suggested action
- None.
- Sources
- corosync.conf, cluster state
- Evidence limitation
- The need for a QDevice is assessed by criterion 2.3-02.
Check 2.1
Observed cluster quorum
Pass
Quorate: Yes · 3 votes expected, 3 present.
Values by node
- pve-01
- member, online
- pve-02
- member, online
- pve-03
- member, online
- Suggested action
- None.
- Sources
- Cluster state (API)
- Evidence limitation
- State collected on 08/10 at 14:35.
Check 3.1
Versions and patches
Pass
pve-manager 8.2.4 on all three nodes, running the same kernel.
Values by node
- pve-01
- 8.2.4 · 6.8.12-2-pve
- pve-02
- 8.2.4 · 6.8.12-2-pve
- pve-03
- 8.2.4 · 6.8.12-2-pve
- Suggested action
- None.
- Sources
- Package versions, running kernel
- Evidence limitation
- Available updates depend on the configured repositories (check 3.2).
Attention
pve-prod
Attention4 items
| Severity | Item | Follow-up | Actions |
|---|---|---|---|
| High | Backup failed, 2 occurrences Backups · lxc:201 - dns-int | Attention | |
| Medium | root@pam login outside business hours Proxmox access · node:pve-01 | Attention | |
| Medium | pbs-01 storage at 91% Storage · storage:pbs-01 | In progress | |
| Low | No backup observed Backups · qemu:118 - win-rds | Deferred |
Demo: follow-up stays on this page; nothing is saved.
Coverage
pve-prod
Sources by host
collector connected · outbound queue 0
| Host | API | SSH reader | Logs |
|---|---|---|---|
| pve-01 | Up to date | Up to date | Up to date |
| pve-02 | Up to date | Up to date | Up to date |
| pve-03 | Up to date | Delayed · 47 min | Up to date |
Checks that use the SSH reader on pve-03 remain “Not assessable” while the source is delayed.
Sample data, read-only: actions are disabled and the AI assistant is not simulated, as its answers depend on your data and the model you configure.