[BUG] fbcon rotate causes vmalloc out-of-bounds write after vc_resize failure

[BUG] fbcon rotate causes vmalloc out-of-bounds write after vc_resize failure

[Jiacheng Xu]

Execution environment:

- Kernel: Linux commit 0f23d56f
- Architecture: x86_64
- QEMU: KVM-enabled qemu-system-x86_64
- Guest image: Debian Bullseye
- KASAN and fault injection enabled


The kernel was started with:

qemu-system-x86_64 \
    -m 2G \
    -smp 2 \
    -kernel ./bzImage \
    -append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
    -drive file=./bullseye.img,format=raw \
    -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
    -net nic,model=e1000 \
    -enable-kvm \
    -nographic \
    -pidfile vm.pid \
    2>&1 | tee vm.log


Trigger condition:

1. Write to `/sys/class/graphics/fbcon/rotate` to initialize the attribute.
2. Set the current thread's fault-injection counter to 13 through
    `/proc/thread-self/fail-nth`.
3. Request rotation value 1 through the same `rotate` attribute.

The injected failure occurs during memory allocation in vc_do_resize(). The subsequent redraw path reaches sys_imageblit() and reports a KASAN vmalloc out-of-bounds write.


Potential root cause:

fbcon_modechanged() switches to the new rotation and blitting operations before resizing the virtual console. When vc_resize() fails, its
return value is ignored and update_screen() still runs.

As a result, the VT geometry and screen buffer remain in the old state while the framebuffer console uses the new rotation state.
sys_imageblit() then calculates an invalid framebuffer address and performs an out-of-bounds write.

Best,
Jiacheng