Double Trouble: Fuzzing Kernel ModulesDerrick McKee, Prashast Srivastava, Hui Peng, Mathias Payer
Abstract
Despite its importance, the Linux kernel remains a difficult target for fuzzers. While prior work, such as syzkaller, allows for fuzzing the kernel, it is still difficult to fuzz arbitrary loadable kernel modules (LKMs). These LKMs represent the majority of source code in the Linux kernel, as well as a major source of CVEs.
We present Double Trouble, a general framework for fuzzing LKMs. Double Trouble requires a one-time minimal programming effort to support fuzzing any subsystem. Once that effort is complete, any LKM within the subsystem can be fuzzed from the two possible sources of inputs: an upper half direction representing a system call execution, and a lower half direction representing either hardware interrupt or exposed kernel function. As a case study, we are adding support for fuzzing filesystems. We use the state-of-the-art generational fuzzer, af-fuzz, for the upper half fuzzing, and a deterministic fuzzer for the lower half.
Problems
●568 critical vulnerabilities found in the Linux kernel last year.
●Emulating a whole system drastically limits fuzzing throughput.
●The complexity of the kernel also greatly hinders the fuzzer code coverage.
●Often difficult to fuzz specific modules.
mutex_lock
_cond_resched
msdos_format_name
strchr
fat_scan
strncmp
fat_get_short_entry
__brelse
__warn_printk
mutex_unlock
current_time
current_kernel_time64
fat_alloc_new_dir
fat_free_clusters
fat_alloc_clusters
__getblk_gfp
fat_time_unix2fat
__memset
mark_buffer_dirty_inode
fat_zeroed_cluster.constprop.25
msdos_add_entry
fat_add_entries
fat_sync_inode
__mark_inode_dirty
inc_nlinkfat_build_inode
fat_iget
new_inode
iunique
fat_fill_inode
iput
fat_attach
__insert_inode_hash
set_nlink
d_instantiate
security_d_instantiate
_raw_spin_lock
__d_instantiate
fat_flush_inodes
writeback_inode
fat_collect_bhsmark_fsinfo_dirtyfat_mirror_bhs
fat_sync_bhsfat_ent_read
blkdev_issue_discard
__fat_fs_error
msdos_mkdir
d_flags_for_inode
printkvprintk
fat__get_entry
inode_needs_sync
__find_get_block
pagecache_get_page
try_to_free_buffers
alloc_page_buffers
init_page_buffers
unlock_page
__put_page
dump_stack
time64_to_tm
mark_buffer_dirty
__bforget
fat_msg
__memcpy
sync_dirty_buffer
__fat_remove_entries
fat_add_new_entries
fat_chain_add
block_dump___mark_inode_dirty
inode_io_list_move_locked I_BDEV
write_dirty_buffer
__wait_on_buffer
sync_inode_metadata
__bread_gfp
___ratelimit
fat_bmap__breadahead
page_mapping
__set_page_dirty
writeback_single_inode
out_of_line_wait_on_bit_lock
rcu_all_qs__schedule
__wait_on_bit_lock
d_find_alias
wb_io_lists_depopulated
blk_start_plug
__blkdev_issue_discard
submit_bio_wait
bio_put
blk_finish_plug
panic
mark_page_accessed
find_get_entry
__page_cache_alloc
add_to_page_cache_lru
__lock_pagedrop_buffers
free_buffer_head
__cancel_dirty_page
alloc_buffer_head set_bh_page.part.29
__page_cache_release
vprintk_func
dump_stack_print_info show_stack
__inode_wait_for_writeback
mapping_tagged
__writeback_single_inode
inode_to_wb_and_lock_list
inode_sync_complete inode_io_list_del_locked
submit_bh_wbc.isra.46
unlock_buffer
out_of_line_wait_on_bit
_raw_spin_trylock
_raw_spin_unlock_irqrestore
printk_deferred
ll_rw_block
rcu_note_context_switch
switch_mm_irqs_off
__switch_to_asm
finish_task_switch
deactivate_task
update_rq_clock.part.80
___perf_sw_event
enter_lazy_tlb
__perf_event_task_sched_out
__delayacct_blkio_start
wq_worker_sleeping
__schedule_bugprofile_hits
__balance_callback
activate_task wq_worker_waking_up
ttwu_do_wakeup.isra.82ttwu_stat.part.49 __delayacct_blkio_end
vprintk_deferred
igrab
new_inode_pseudo
inode_sb_list_add
get_seconds
fat_calc_dir_size
fat_subdirs fat_time_fat2unix
_atomic_dec_and_lock
list_lru_del
write_inode_now inode_lru_list_add
rcu_sched_qs rcu_dynticks_momentary_idle
load_new_mm_cr3
rcu_irq_enter_disabledrcu_irq_enter_irqsonrcu_irq_exit_irqson
__pti_set_user_pgdclear_asid_other
__perf_event_task_sched_in
__mmdrop
kprobe_flush_task
put_task_stack __put_task_struct
sched_clock_cpu
perf_exclude_event perf_swevent_event
perf_pmu_sched_task perf_iterate_sbtask_ctx_sched_out perf_event_update_timeperf_event_update_userpage
ktime_get
kthread_data
print_modules
_raw_spin_lock_irqsave
check_preempt_curr
__const_udelay
panic_smp_self_stopbust_spinlocks
vsnprintf
printk_safe_flush_on_panic
__crash_kexec
crash_smp_send_stop
atomic_notifier_call_chain
kmsg_dump
debug_locks_off console_flush_on_panic
emergency_restart
__mod_node_page_state
__mod_zone_page_state
vprintk_emit
irq_work_queue
fat_get_clusterfat_ent_write
fat_cache_inval_inode
strcspnprint_tainted
queued_spin_lock_slowpath
resched_curr
activate_page
fat_cache_add.part.10
kmem_cache_free
list_lru_add
__printk_safe_enter
vscnprintf
cont_flush
log_store
__printk_safe_exit
cont_add
__down_trylock_console_sem.isra.20console_trylock.part.21
console_unlock
llist_add_batch arch_irq_work_raise tick_nohz_tick_stopped
native_sched_clock
sched_clock_local
recalc_bh_state.part.44
perf_iterate_ctxcalc_timer_values arch_perf_update_userpage
finish_wait
prepare_to_wait_exclusive
bio_alloc_bioset
bio_add_page
guard_bio_eod
submit_bio
__wait_on_bit
mempool_alloc
punt_bios_to_rescuer
bvec_alloc
__kmalloc
mempool_free
__disk_get_part
generic_make_request
__bdevname
this_cpu_cmpxchg16b_emu
__slab_free
cpumask_next
prepare_to_wait
kmem_cache_alloc fat_cache_merge
cmpxchg_double_slab.isra.73 free_debug_processing
using_native_sched_clock sched_clock_stablecyc2ns_read_begin cyc2ns_read_end
remove_element.isra.9.part.10io_schedule_timeout kmalloc_slab __slab_alloc
log_make_free_spacedown_trylock
__up_console_sem.isra.19 scnprintf
msg_print_text
sprintf
msg_print_ext_body
___slab_alloc
generic_make_request_checks
blk_queue_enter
bio_endio
account_page_dirtiedradix_tree_tag_set
__page_file_indexpagevec_lru_move_fn alloc_inode
release_pages
module_flags
up
bit_waitqueue
do_writepages is_bad_inodefilemap_fdatawait_range inode_add_lru
init_object
on_freelist
check_objectset_track
print_hex_dump
slab_errslab_fix
slab_bug
print_trailer
check_slab
unblank_screenconsole_unblank
format_decode
number
pointer
string
set_precisionset_field_width
__sw_hweight64
mutex_trylock crash_setup_regs crash_save_vmcoreinfomachine_crash_shutdown
machine_kexec
console_trylock
__cmpxchg_double_slab.isra.61
get_partial_node.isra.84
gfp_pfmemalloc_allowed
deactivate_slab.isra.82
mempolicy_slab_node__cpuset_node_allowed
__next_zones_zonelist
new_slab
slab_out_of_memory
alloc_debug_processing
next_bio
wait_for_completion_io
blk_status_to_errno
blk_flush_plug_list
check_bytes_and_reportsave_stack_trace
hex_dump_to_bufferadd_taint
print_tracking.part.58
slab_pad_check.part.64
rcu_irq_enterrcu_irq_exit
print_track
vmcoreinfo_append_str
relocate_kerneldisable_IO_APIC
radix_tree_lookup_slot
__alloc_pages_nodemask
cpuset_mem_spread_node
__add_to_page_cache_locked
lru_cache_add
workingset_refaultworkingset_activation
_raw_spin_lock_irq
io_schedule
ctx_sched_out
perf_event_sched_in.isra.117
pgd_free
destroy_context_ldt
__mmu_notifier_mm_destroy
kfreerecycle_rp_inst
account_kernel_stack arch_release_thread_stack
vfree_atomic
__free_pages
cgroup_free
security_task_freeexit_creds profile_handoff_task
queue_work_on
module_flags_taint
memchr_inv
__put_compound_pagefree_unref_page_list
find_next_bit
__radix_tree_lookup
bio_chain
free_ldt_struct.part.6
put_cpu_partial
discard_slab
__next_node_in
init_tracking.part.68
mod_node_page_state alloc_pages_current
count_partial
__up.isra.2
group_sched_out.part.103
ctx_sched_in.isra.115
clear_IO_APIC
__inc_zone_page_state__inc_node_page_state
percpu_counter_add_batch
mod_zone_page_state
console_lock
blk_mq_flush_plug_list
list_sort
queue_unplugged __elv_add_request__blk_end_request_all
PageHugefree_unref_page_preparefree_unref_page_commit.isra.92
congestion_wait
write_cache_pages__filemap_fdatawait_range
pagevec_lookup_range_tag
wait_on_page_bit_common
__pagevec_release
event_sched_out.isra.100
unfreeze_partials.isra.83
find_get_pages_range_taglru_add_drain_cpu
handle_bad_sectorcreate_task_io_context
init_wait_entry
schedule
prepare_to_wait_event
io_schedule_prepare
schedule_timeout
free_pages
inode_init_always
down
security_inode_alloc __init_rwsem
skip_atoi
put_dec
dentry_name pointer_string siphash_1u64 restricted_pointeruuid_string resource_stringdevice_node_string.isra.5 netdev_bitsmac_address_string escaped_stringclock ip4_addr_string bitmap_list_string.isra.6address_val symbol_string hex_stringbdev_name.isra.8flags_string bitmap_string.isra.7ip6_addr_string_saip6_addr_string ip4_addr_string_sa
lock_timer_basedetach_if_pending
__internal_add_timer trigger_dyntick_cpu.isra.35del_timer_sync
get_nohz_timer_target
get_page_from_freelist
__alloc_pages_slowpath
policy_nodemaskpolicy_nodeget_task_policy.part.41
jiffies_to_usecs
wait_on_page_bit
clear_page_dirty_for_iotag_pages_for_writeback
print_prefixmemchr
snprintf
node_random
group_sched_in
perf_event_set_state.part.76
radix_tree_maybe_preload page_cache_tree_insertlruvec_lru_size
find_first_bit
refcount_dec_not_one
put_css_set_locked
__put_cred
__queue_work
clear_IO_APIC_pin
pagevec_move_tail
has_capability_noaudit string_escape_mem__clk_get_name clk_get_rate
ip4_string
sprint_symbolsprint_symbol_no_offset sprint_backtracestrlen ip6_string ip6_compressed_string
__down
kmem_cache_alloc_node
radix_tree_next_chunk
blk_mq_sched_insert_requests __blk_run_queue blk_run_queue_async elv_drain_elevatorelv_attempt_insert_merge elv_rqhash_add __pm_runtime_resume__blk_end_bidi_request
ioapic_read_entry __eoi_ioapic_pinioapic_mask_entry ioapic_write_entry
cpuset_nodemask_valid_mems_allowed
free_pcppages_bulk
__radix_tree_create
shmem_mapping__radix_tree_replace
security_capable_noaudit
event_sched_in.isra.113.part.114perf_mux_hrtimer_restart
radix_tree_node_alloc.constprop.25
radix_tree_extend
node_tag_get.constprop.24
__zone_watermark_ok
node_dirty_ok
__inc_numa_state clear_page_orig
get_pfnblock_flags_mask set_pageblock_migratetype move_freepages_block
__node_distancenode_reclaim
find_suitable_fallback
steal_suitable_fallback
check_new_page_bad
__cpuset_memory_pressure_bump
try_to_free_pages
unreserve_highatomic_pageblock
drain_all_pages
__alloc_pages_direct_compact
zone_reclaimable_pages
schedule_timeout_uninterruptible
wake_all_kswapds
out_of_memory
compaction_zonelist_suitable
warn_alloc
put_dec_full8
get_random_u32
perf_log_itrace_startperf_log_throttlehrtimer_forwardhrtimer_start_range_ns
free_pages_check_bad
page_mkcleandec_zone_page_stateset_page_dirtyradix_tree_iter_tag_set radix_tree_iter_resume
fmeter_update
throttle_direct_reclaim
do_try_to_free_pages
first_online_pgdatnext_zone
flush_work
try_to_compact_pages
compaction_defer_reset
wakeup_kswapd
task_will_free_mem
oom_kill_process
schedule_timeout_killableblocking_notifier_call_chain
mark_oom_victimwake_oom_reaper.part.28
has_intersects_mems_allowed
dump_header.isra.30oom_badness
__compaction_suitable.part.38
cpuset_print_current_mems_allowed
show_mem
get_work_pool
check_flush_dependency
__init_waitqueue_head
insert_work
wait_for_completion
node_page_state
node_pagecache_reclaimable
shrink_node
blk_attempt_req_merge elv_rqhash_find rpm_resumeblk_mq_insert_requestsblk_update_bidi_request blk_finish_request
blk_update_request add_disk_randomness blk_delete_timerblk_account_io_done
blk_queue_end_tag
blk_stat_addlaptop_io_completion
__blk_put_request
allow_direct_reclaim.part.78
compaction_suitable
__delayacct_freepages_start
zone_watermark_ok_safe
__delayacct_freepages_end
blk_account_io_completion
bio_advance
blk_dump_rq_flags
blk_recalc_rq_segments add_timer_randomness
__ioapic_read_entry
part_round_stats part_dec_in_flight
elv_completed_request freed_requestelv_put_request put_io_context
__wake_up
__pm_runtime_idle
del_timer
__rpm_get_callback
dev_pm_disable_wake_irq_checkrpm_callback dev_pm_enable_wake_irq_checkrpm_idle
_warn_unseeded_randomness_raw_read_lock_irqsaveextract_crng _raw_read_unlock_irqrestore
pr_cont_kernfs_name
show_free_areas
next_online_pgdat
page_mapped rmap_walk
pgdat_balanced
shrink_node_memcg.constprop.84
shrink_slab.part.69
part_in_flight
queued_read_lock_slowpath
sum_zone_node_page_state
inactive_list_is_low.constprop.87 shrink_active_list
shrink_inactive_list
down_read_trylock up_read
fragmentation_index
kernfs_name_locked
__freed_request
attempt_merge
lock_hrtimer_base.isra.20
enqueue_hrtimer__remove_hrtimertick_program_event wake_up_nohz_cpu
ktime_get_seconds
mix_pool_bytescredit_entropy_bits
show_mem_node_skip.part.76
hugetlb_show_meminfo
show_swap_cache_info
call_rwsem_wake
zone_watermark_ok
strlcpy
calc_wheel_index
try_to_del_timer_sync
housekeeping_test_cpu
compaction_deferredcompact_zone_order defer_compaction
put_dec_trunc8
perf_output_begin __perf_event_header__init_id
perf_output_copy
__perf_event__output_id_sample perf_output_end
perf_event_pid_type
cpumask_next_and
__blk_recalc_rq_segments
find_lock_task_mm
_raw_read_lock
put_task_struct
do_send_sig_info
down_read notifier_call_chain
__thaw_task
mempolicy_nodemask_intersects
cpuset_mems_allowed_intersects
dump_unreclaimable_slab
int_sqrt lru_add_drain
isolate_lru_pages.isra.79
page_evictable
page_referenced
move_active_pages_to_lru
putback_lru_page try_to_release_page
msleep shrink_page_list
putback_inactive_pages
current_may_throttle
wakeup_flusher_threads
wait_iff_congested
blk_mq_hctx_mark_pending.isra.43
get_slabinfo
blk_rq_set_mixed_mergeelv_merge_requests ioprio_best
rwsem_wake
perf_event_header__init_idperf_event__output_id_sample perf_output_put_handle
__rpm_callback rpm_check_suspend_allowed
total_swapcache_pages
clear_wb_congested
__msecs_to_jiffies
__isolate_lru_page
try_to_free_swap
try_to_unmap_flushadd_to_swap
__remove_mapping
try_to_unmap
try_to_unmap_flush_dirty
inc_node_page_state
__filemap_set_wb_err
__wakeup_flusher_threads_bdi.part.53
__fragmentation_index
timerqueue_addtimerqueue_del
clockevents_switch_state
__page_mapcount
page_rmapping
add_page_to_unevictable_list isolate_lru_page
__lock_task_sighand send_signal
__dev_pm_qos_read_valuedevice_links_read_lock device_links_read_unlock
rb_insert_color
wb_wakeup
__task_pid_nr_ns
elv_rqhash_repositionelv_rqhash_del.part.12
_raw_spin_lock_bh mod_delayed_work_on
__rwsem_mark_wake.constprop.5 wake_up_q
call_rwsem_down_read_failed
wake_up_process
page_swapcount
pm_suspended_storagedelete_from_swap_cache
arch_tlbbatch_flushget_swap_page
add_to_swap_cache
put_swap_page __delete_from_swap_cache
__delete_from_page_cache workingset_eviction
rmap_walk_locked
errseq_set
try_to_grab_pending__queue_delayed_work
task_active_pid_ns
rb_eraserb_next
cpumask_any_but
native_flush_tlb_others flush_tlb_func_common.constprop.11
rwsem_down_read_failed
_swap_info_get
try_to_wake_up
get_swap_pages
__drain_swap_slots_cache.constprop.1
radix_tree_maybe_preload_order__add_to_swap_cache
__swap_entry_free
__radix_tree_insert
radix_tree_delete
plist_requeue
__del_from_avail_list
scan_swap_map_slots
drain_slots_cache_cpu.constprop.2
swapcache_free_entries
kvfree
unaccount_page_cache_pageradix_tree_clear_tags
__swap_info_get
swap_count_continued
pwq_dec_nr_in_flight
pwq_activate_delayed_work
set_task_cpu
__task_rq_lock
ttwu_do_activate.isra.83 select_fallback_rq
dump_pagenode_tag_clear
set_task_rq_fair__printk_ratelimit do_set_cpus_allowedcpuset_cpus_allowed_fallback
plist_del
scan_swap_map_try_ssd_cluster inc_cluster_info_page
compact_zone
_mix_pool_bytes
kill_fasync
crng_reseed
swap_do_scheduled_discard
__free_cluster
migrate_prep_local isolate_migratepages_block drain_local_pages
putback_movable_pages __reset_isolation_suitable
__pageblock_pfn_to_pagemigrate_pages
release_freepages
__update_load_avg_blocked_se.isra.33
vmalloc_to_page
complete
sort
add_to_avail_list
send_sigio
extract_entropy.constprop.43 memzero_explicit
invalidate_batched_entropy
module_put _extract_crng _crng_backtrack_protect
compact_unlock_should_abort.isra.35
compact_trylock_irqsave.isra.20
update_pageblock_skipisolate_movable_page
putback_active_hugepage PageMovableputback_movable_page set_pfnblock_flags_mask
size_to_hstate
move_to_new_page
__put_anon_vma
page_get_anon_vma
remove_migration_ptes
xfer_secondary_poolaccount.constprop.44
_raw_write_lock_irqsave
chacha20_block
plist_add
__wake_up_locked
queued_write_lock_slowpath
decay_load
send_sigio_to_task
migrate_page
down_write
up_write
security_file_send_sigiotask
migrate_page_move_mapping migrate_page_copymigrate_page_states
call_rwsem_down_write_failed
rwsem_down_write_failed
radix_tree_replace_slot__dec_node_state __inc_node_statebuffer_migrate_lock_buffers __dec_zone_state __inc_zone_state copy_page
__lock_buffer
osq_lock rwsem_spin_on_owner osq_unlock__rwsem_mark_wake.part.2
wake_q_add
Our Solution
●Ignore kernel features that are orthogonal to correct module functionality.
●Implement “kernel gateway functions” that provide simple services, such as memory allocation.
●Provide a generic, simple API that developers can use to exercise their module functionality.
●Generate a user space application that is as fast and scalable as the underlying fuzzer.
mutex_lock
fat_dir_empty
fat_get_short_entry
__brelse
mutex_unlockmsdos_find
msdos_format_namefat_scan fat_remove_entries
current_time
mark_buffer_dirty_inode sync_dirty_buffer
fat_sync_inode__fat_remove_entries
fat_msg
__mark_inode_dirty
drop_nlink clear_nlink fat_detach
_raw_spin_lock
fat_flush_inodes
writeback_inode
msdos_rmdir
fat__get_entry
printk
__bread_gfp ___ratelimit fat_bmap __find_get_block __breadahead
strchrstrncmp
