I had thought of the exact same thing. Using the rotary encoder + USB interface from any mouse makes sense. My idea was to measure net motion over a configurable time period. I am trying to get my money's worth out of a roll of crappy filament that I KNOW will break. Another common runout scenario that switch sensors do not handle is extruder chew-through - where there may be a kink in otherwise good filament or a soft spot, and the extruder grinds into the filament.
I totally get why it's best to do this in Octoprint - I want to be informed and have the opportunity to take different actions.
I'm just getting started on plugin development but I would like to help with this. As far as the hardware, I thought of disassembling the mouse because there is usually a tiny board with the actual buttons and rotary encoder. I believe a plugin should be able to read the raw HID events from the "mouse" device and accumulate the net motion.
Using a rotary encoder would allow you to have passive turning motion from a printed set of pulleys as opposed to bump motion. In any case a good first step would be to code up a plugin that reads mouse HID events, and that could support either arrangement. I'll put something on my github account.