tl;dr Over the course of 48 hours, AFLFast found 6 unique flaws in Perl, while AFL 2.32b found 4, all of which AFLFast failed to find.
Today I'm going to talk about my experiences fuzzing
Perl over the course of 48 hours; 24 of which were spent with the
AFLFast fork of AFL 2.30b, and 24 of which were spent with
AFL 2.32b, a new version of AFL that has tweaks to the
-d "quick & dirty mode" where it skips
deterministic steps in order to better emulate the behavior of AFLFast. Michal Zalewski calls this
AFLFast claimed in an email to the AFL-users mailing list:
to have exposed several previously unreported CVEs that could not be exposed by AFL in 24 hours and otherwise exposed vulnerabilities significantly faster than AFL while generating orders of magnitude more unique crashes.
I wanted to test these claims, so what follows is a highly unscientific comparison of the two. Your mileage may vary, but you can read more about AFLFast in their CCS'16 Paper entitled "Coverage-based Greybox Fuzzing as Markov Chain". You can also read up on AFL's technical details and historical notes.
I chose Perl for a variety of reasons, primarily:
I've been using AFL for several years, and in that time I've helped find, fix and close well over 100 bugs (mostly null pointer derefs, assertion failures, etc), so I know what to expect from AFL and Perl and their dev team.
The first 24 hours were spent fuzzing Perl, built from git source, with
AFLFast and a starting corpus of the usual 55 or so test cases I handpick for their ability to repeatedly trigger bugs in Perl.
A sample Perl test case:
This example is why I sometimes joke about my cat being able to create perfect Perl code by running across my keyboard. :joy:
$~='`';$_=$:=$~|'%';$;=$^='/'|$~;$;++;$\=$~|"'";$;++;$:.=++$;;$/=++$;;+$\++;$_.= '#'|$~;$,=++$/;$_.="$\$^$\"";++$,;$_.='@'|'*'&~'!';$_.="$,$;$/$\"";$_.+='!.'|$~. $~;$_.="$^$/$\$:$\"";$_.='@'|':'&~'*';$_.=$:;$_.=$^&'|';$_.=$".$\;$_.=+"$~$~$~"| '!#+';++$.;$.++;`$_$:,>&$.`;
I compile Perl the same way every time:
./Configure -des -Dusedevel -DDEBUGGING -Dcc=afl-clang-fast -Doptimize=-O2\ -g && AFL_USE_ASAN=1 make -j2
And I run Perl with AFL the same way every time:
AFL_PRELOAD=/home/geeknik/aflfast/libdislocator/libdislocator.so afl-fuzz -t50+ -m none -i in -o out ./perl @@
afl-plot after 24 hours:
Might be a bug in AFLFast, but when you first start it up, it has a very high number of pending favs, which throws this first graph out of wack.
start_time: 1471592218 last_update : 1471678622 fuzzer_pid: 1014 cycles_done : 8 execs_done: 14732904 execs_per_sec : 184.69 paths_total : 24147 paths_favored : 1870 paths_found : 24092 paths_imported: 0 max_depth : 38 cur_path : 23211 pending_favs : 1446 pending_total : 16332 variable_paths: 24142 stability : 86.95% bitmap_cvg: 58.14% unique_crashes: 331 unique_hangs : 364 last_path : 1471678615 last_crash: 1471678242 last_hang : 1471678601 execs_since_crash : 66182 exec_timeout : 50 afl_banner: perl afl_version : 2.30b command_line : afl-fuzz -t50+ -m none -i in -o out ./perl @@
After 24 hours, AFLFast declares we have
331 unique crashes. Upon triage, only
6 of these were unique and serious enough to warrant emails to the Perl security team:
This is a pretty good showing by AFLFast, especially with
8 full cycles being completed in just 24 hours. Past Perl fuzzing sessions have dragged on for weeks without completing a single cycle (or generating a single crash). Now lets move on to AFL 2.32b with the
Perl was recompiled with
AFL 2.32b (
FidgetyAFL) using the same source as before, no changes were pulled in, no files were edited, etc. The same starting corpus was used and the only other change was the AFL command line:
AFL_PRELOAD=/home/geeknik/aflfast/libdislocator/libdislocator.so afl-fuzz -d -t50+ -m none -i in -o out ./perl @@
afl-plot after 24 hours:
start_time: 1471680593 last_update : 1471766996 fuzzer_pid: 14641 cycles_done : 0 execs_done: 16938116 execs_per_sec : 60.64 paths_total : 24461 paths_favored : 2408 paths_found : 24406 paths_imported: 0 max_depth : 13 cur_path : 23749 pending_favs : 245 pending_total : 19469 variable_paths: 24456 stability : 84.09% bitmap_cvg: 59.79% unique_crashes: 662 unique_hangs : 500 last_path : 1471766988 last_crash: 1471763507 last_hang : 1471733050 execs_since_crash : 676444 exec_timeout : 50 afl_banner: perl afl_version : 2.32b command_line : afl-fuzz -d -t50+ -m none -i in -o afl232-out ./perl @@
After 24 hours, FidgetyAFL declares we have
662 unique crashes. Curiously we have exactly 2x more crashes than AFLFast. Upon triage, only
4 of these were unique:
I thought after finding the null ptr deref, we were on a new track and would start finding some more interesting things. I started noticing more and more of these
Bus error so I decide to look closer. Run it under gdb and I see
SIGBUS. I had to do some quick reading, ask a few questions. I'm pretty sure I dismissed those earlier as just extraneous DEBUG output, so I've made a note to go back and check for others.
While not able to complete a full cycle during the 24 hour period, this is not a bad showing by AFL 2.32b compared to prior fuzzing sessions without the
FidgetyAFL tweaks, which could go on for days or weeks before finding a single crash.
As you can see from this completely unscientific study, AFLFast is, in my opinion, neither better or worse at finding a wider variety of security flaws when compared to AFL 2.32b over the same period of time. And while AFL 2.32b managed to find more unique crashes overall with 662 compared to AFLFast's 331, I feel like they all kept pointing to the same line of code which means they aren't all that unique.
Perl has many places for bugs to hide, some have been lurking for many, many years. I'm going to keep fuzzing with a mixture of both AFLFast and AFL 2.32+ because 11 (see note) potential vulnerabilities in 48 hours is a mighty good number of holes to patch. A Perl developer I spoke with said they are interested if perl -D crashes, so that's a good place to start if you want to join in on the fun.
Note: During the same span of time (the 48 hours prior to this being written), AFLFast triggered a null ptr deref + segfault in PHP 5.6.25 using a random corpus from 3v4l.org (which in my opinion is a gold mine for anyone looking to fuzz PHP).
I did not count the many Assertion Failures which occurred during these 2 rounds of fuzzing, mainly because they have no impact on security, and since this is a
-DDEBUGGING build of Perl, asserts can frequent fail. I may go back and submit them to the dev team at a later point if I have time.
I am positive that I missed some SIGBUS faults in both test sessions, so I will be going back over the fuzzing output again in the coming days, and submitting any new flaws that I find.
All fuzzing was conducted on $5/mo DigitalOcean Debian 8.5 x64 VMs with the bare minimum software necessary to compile and fuzz Perl.
Without these people, this wouldn't have been possible:
MIT License - Copyright (c) 2016 Brian 'geeknik' Carpenter
Last updated: 8/21/2016 03:23:54