Repository navigation
Out of memory bugs no longer reproducable with the attached test case #977
Description
Quite often out-of-memory bugs are not reproducible with the reproducer attached to the bugs.chromium.org bug. The bug should be closed if not reproduced in 14 days, but there are likely other reproducers that trigger out-of-memory that prevent the bug to be closed, and those reproducers are not accessible through the bug entry. How to proceed ?
Example: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3703 (similar situations for GDAL too)
Activity
inferno-chromium commentedon Nov 11, 2017
So, we close bug in 14 days if it is not seen in crash stats for last 14 days. In this case, this out-of-memory crash is still happening, you can see that by clicking the stats button. We haven't hit a new reproducible crash testcase for this (https://oss-fuzz.com/v2/testcases?open=yes&q=Out-of-memory%20proj4), so that is why this bug will stay open. Unreproducible crashes are only filed if they are in top crashers. I don't have a solution for your request, do you have a suggestion ?
rouault commentedon Nov 11, 2017
As the out of memory occurs with other reproducers, couldn't they replace the one (or be added) that is attached to the ticket but is no longer reproducible ?
Dor1s commentedon Nov 12, 2017
That's the issue: we cannot find a single reproducer, but the same crash is occurring again and again. We file it automatically with the first "reproducer" found, but that testcase is not very useful as well as all the other "crash" inputs we see after that.
Technically, such crash can be "reproduced" in the following way:
-rss_limit_mb=2048This is what happens on CF. The fuzzer is crashing with the same OOM error again and again, but the error cannot be reproduced with a single input :(
There is a paragraph in the bold in the bug description (https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3703):
I wonder whether you've read that text and it's not clear, or you simply didn't pay attention to that block of text? Sorry for asking this, but we need that feedback as you are not the first who gets confused by "non reproducible" crashes. We might want to re-write the text if it's not clear.
rouault commentedon Nov 12, 2017
I did read the text, but it doesn't help me solving the issue in practice. Very often there's no useful stacktrace, so no possible speculative fix. Probably the way you describe would work, but it implies more work. I'm just lazy :-)
caolanm commentedon Nov 12, 2017
I run into this with LibreOffice too, and what would work better for me is the most recent input that triggered it rather than the initial one, or a link to all the inputs that triggered it.
Often enough I get something that I can reproduce locally with the original testcase, especially under valgrind, fix it, the stats jump downwards with my fix but not quite to zero and then grow again so it feels like another bug with input at some other location resulting in the same symptoms
Pro commentedon Nov 12, 2017
I have a similar problem for a timeout issue on open62541:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3713
It causes around 700 timeout crashes per day but I cannot reproduce the
Unreproduciblebug with the provided test case. And interestingly all the stack traces in the output show different lines in the code which cause the timeout.It would be helpful if you could update the page here:
https://github.com/google/oss-fuzz/blob/master/docs/reproducing.md
with some tips on how one can at least try to reproduce
unreproduciblebugs, e.g. the steps you (@Dor1s) mentioned above:Or are there some additional tools we could use? Some more test-cases we can try?
Dor1s commentedon Nov 14, 2017
Thanks all for your feedback! We'll think about storing multiple testcases OR at least storing the most recent input.
Good idea, I'll update that!
Valgrind (mentioned by @caolanm) can be helpful sometimes, but please note that it doesn't work with ASan (and MSan, I think?), so you need to use either a non sanitized build or a UBSan one.
titanous commentedon Jan 24, 2018
I've recently seen some unreproducible timeouts that apparently have multiple distinct causes, and it would be very useful if multiple testcases were stored when a crash is unreproducible.
inferno-chromium commentedon Feb 2, 2018
Now all testcases will be stored alongwith the crash input in cloud storage bucket.
Name will be like
.testcase for input file (this file will only exist if there is crash)
.log for its log file.
E.g.
gs://skia-logs.clusterfuzz-external.appspot.com/libFuzzer_skia_image_filter_deserialize/libfuzzer_asan_skia/2018-02-02
03:54:18:732351.log
03:54:18:732351.testcase
Thanks for @jonathanmetzman for implementing this.
@kcc as fyi.
Maybe someday we add prefix- to filename, then you can only pull those type of specific testcase, right now you need to pull directory for a day and then search for oom-, crash-, timeout-, etc