debugging using dtrace doesn’t appear to be working on HBSD 12.x
Created by: tuto2
Describe the bug
When trying to use dtrace to monitor the kernel call stack, none of the kernel function names can be resolved by dtrace (and possibly other tools as well). Trying the same procedure on a stock FreeBSD machine does seem to function without issues.
When debugging the kernel source, we ran into this commit https://github.com/HardenedBSD/hardenedBSD/commit/37a8b2de291f4cb84b5cac86aab8e5edcaa75176 which describes the addition of a new privilege named PRIV_KLD_STAT
, which, in the context of dynamically loaded kernel modules, is meant to prevent non-root users from being able to see kernel modules. This is indeed the case, except the root user get shielded from gaining insights into the machine as well due to masking the addresses in the same function (https://github.com/HardenedBSD/hardenedBSD/commit/37a8b2de291f4cb84b5cac86aab8e5edcaa75176#diff-c3d5aa900131c65102c23e944303538392a83cc73492e590a65b65f1b1505da1R1259-R1263).
One of the fastest pointers to tell something is off seems to be to to run kldstat
as root and witness all kernel addresses are zeroed out (0x0). This becomes problematic when tracing kernel performance using Dtrace or any other facility, as C type data (ctf(5)) does not get resolved properly, and instead gets resolved to non-descriptive kernel addresses.
To Reproduce
The following steps assume the following options in GENERIC:
makeoptions DEBUG=-g
makeoptions WITH_CTF=1
options KDTRACE_FRAME
options KDTRACE_HOOKS
- Build a DEBUG SMP kernel with
nooptions PAX_HARDENING
and run the following (arbitrary) command:dtrace -n uiomove:entry’{@[stack] = count(); }’
Run this for some time and press CTRL-C. The result will show an aggregate of unique kernel stack traces, e.g.:
kernel`m_uiotombuf+0x10c
kernel`sosend_generic+0x40d
kernel`sosend+0x50
kernel`kern_sendit+0x237
kernel`sendit+0x19e
kernel`sys_sendto+0x4d
kernel`amd64_syscall+0x364
kernel`0xffffffff811e38d0
4
kernel`ttydisc_write+0xcc
kernel`ttydev_write+0x155
kernel`devfs_write_f+0xda
kernel`dofilewrite+0xb0
kernel`sys_write+0xc1
kernel`amd64_syscall+0x364
kernel`0xffffffff811e38d0
5
- Do the same as step 1, but build the kernel with
options PAX_HARDENING
. The results are as follows:
0xffffffff80da98bc
0xffffffff80db29dd
0xffffffff80db2ed0
0xffffffff80db9d47
0xffffffff80dba0be
0xffffffff80db9f0d
0xffffffff8120b064
0xffffffff811e3b00
4
0xffffffff80da20fc
0xffffffff80d9c8b5
0xffffffff80ba777a
0xffffffff80d82b70
0xffffffff80d826d1
0xffffffff8120b064
0xffffffff811e3b00
5
Expected behavior The question probably is if we really want to mask these addresses for the root user, there might be a use case for it, but if there is, it should be properly documented and perhaps being a separate build option so you don’t have to disable the PAX_HARDENING in full to stop masking addresses. If we assume that the root user itself was already properly safeguarded, question is why addresses are being masked since regular users seem not to be able to reach this point at all. Additional context The documentation at https://git-01.md.hardenedbsd.org/HardenedBSD/HardenedBSD/wiki#user-content-generic-system-hardening specifies that the PAX_HARDENING kernel option restricts what NON-root users are permitted to do. However, this option also affects the root user.
Environment OPNsense 21.1.a (amd64, OpenSSL).