FullFuzz test exploits-minimized #176

guyzilberblum · 2024-05-10T14:42:32Z

guyzilberblum
May 10, 2024

While im watching the tutuorial about the full fuzz test patrick did forge test and broke the invariant at the do more math function
while i did as he showed and the test passed without finding any bug notice that i did try to switch the fail on revert to false this is my code
"""

    // SPDX-License-Identifier: MIT
        pragma solidity 0.8.20;
        import {Test} from "forge-std/Test.sol";
        import{StatefulFuzzCatches} from "../../src/invariant-break/StatefulFuzzCatches.sol";
        import {StdInvariant} from "forge-std/StdInvariant.sol";
        contract StateFUllfuzz is StdInvariant,Test {
       StatefulFuzzCatches sff;
       function setUp() public {
       sff = new StatefulFuzzCatches();
      targetContract(address(sff));
      }


    function testFuzzStatelessDoMoreMath (uint128 randNumber) public  {
    assert(sff.doMoreMathAgain(randNumber)!=0);

    }


   function testStatefulfuzz_CatchesInvariant() public view {
   
    assert(sff.storedValue()!= 0);
     
  }
  }

"""

EngrPips · 2024-05-10T15:55:29Z

EngrPips
May 10, 2024

 // SPDX-License-Identifier: MIT
        pragma solidity 0.8.20;
        import {Test} from "forge-std/Test.sol";
        import{StatefulFuzzCatches} from "../../src/invariant-break/StatefulFuzzCatches.sol";
        import {StdInvariant} from "forge-std/StdInvariant.sol";
        contract StateFUllfuzz is StdInvariant,Test {
       StatefulFuzzCatches sff;
       function setUp() public {
       sff = new StatefulFuzzCatches();
      targetContract(address(sff));
      }


    function testFuzzStatelessDoMoreMath (uint128 randNumber) public  {
    assert(sff.doMoreMathAgain(randNumber)!=0);

    }


   function testStatefulfuzz_CatchesInvariant() public view {
   
    assert(sff.storedValue()!= 0);
     
  }
  }

4 replies

EngrPips May 10, 2024

Hello @guyzilberblum, Fuzz testing strongly depends on the seed used and the number of runs made to determine the outcome. So, if you would like to have the same output as Patrick, you need to be running your fuzz testing with the same seed he is using and the same runs he has set in his foundry.toml

kapitanluffy5 May 11, 2024

This what I got

kapitanluffy5 May 11, 2024

This is what Patrick got

EngrPips May 11, 2024

You are running a unit test while Patrick ran a Fuzz test. That is because you named the function that is supposed to run the Fuzz test as testStatefulFuzz_CatchesInvariant when, in fact, you are supposed to name it statefulFuzz_CatchesInvariant.

guyzilberblum · 2024-05-10T16:03:07Z

guyzilberblum
May 10, 2024
Author

I used the same stats

…

On Fri, 10 May 2024 at 18:59 EngrPips ***@***.***> wrote: Hello @guyzilberblum <https://github.com/guyzilberblum>, Fuzz testing strongly depends on the seed used and the number of runs made to determine the outcome. So, if you would like to have the same output as Patrick, you need to be running your fuzz testing with the same seed he is using and the same runs he has set in his foundry.toml — Reply to this email directly, view it on GitHub <#176 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BHZOBIPOSGWLDN7MVWISBWDZBTVH3AVCNFSM6AAAAABHQYEKRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOBWGA3TG> . You are receiving this because you were mentioned.Message ID: <Cyfrin/security-and-auditing-full-course-s23/repo-discussions/176/comments/9386073 @github.com>

1 reply

EngrPips May 10, 2024

You are supposed to get the same output if you use the same stats. That's weird; you can share your test and output and Patrick's test and output so we can see what is different.

guyzilberblum · 2024-05-10T16:33:48Z

guyzilberblum
May 10, 2024
Author

Ok I will send tomorrow it’s the weekend here I’m going out thanks a lot talk soon

…

On Fri, 10 May 2024 at 19:26 EngrPips ***@***.***> wrote: You are supposed to get the same output if you use the same stats. That's weird; you can share your test and output and Patrick's test and output so we can see what is different. — Reply to this email directly, view it on GitHub <#176 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BHZOBILK2ELLR6PJXQRNIC3ZBTYMXAVCNFSM6AAAAABHQYEKRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOBWGY4TK> . You are receiving this because you were mentioned.Message ID: <Cyfrin/security-and-auditing-full-course-s23/repo-discussions/176/comments/9386695 @github.com>

1 reply

EngrPips May 10, 2024

Alright, friend. Enjoy your weekend, and I will be expecting those

guyzilberblum · 2024-05-10T16:51:11Z

guyzilberblum
May 10, 2024
Author

Thanks, you too!

…

On Fri, 10 May 2024 at 19:50 EngrPips ***@***.***> wrote: Alright, friend. Enjoy your weekend, and I will be expecting those — Reply to this email directly, view it on GitHub <#176 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BHZOBIL63WM2IP2Q6LCAKLTZBT3F5AVCNFSM6AAAAABHQYEKRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOBXGAYDS> . You are receiving this because you were mentioned.Message ID: <Cyfrin/security-and-auditing-full-course-s23/repo-discussions/176/comments/9387009 @github.com>

1 reply

EngrPips May 10, 2024

Yeah, sure I will, hahaha

kapitanluffy5 · 2024-05-11T13:31:20Z

kapitanluffy5
May 11, 2024

No it did more tests but still the test passed no errors

…

On Sat, 11 May 2024 at 16:26 EngrPips ***@***.***> wrote: You are running a unit test while Patrick ran a Fuzz test. That is because you named the function that is supposed to run the Fuzz test as testStatefulFuzz_CatchesInvariant when, in fact, you are supposed to name it statefulFuzz_CatchesInvariant. — Reply to this email directly, view it on GitHub <#176 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BEI52O75QCFL3OS54WYZIALZBYL7VAVCNFSM6AAAAABHQYEKRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOJVGUZDE> . You are receiving this because you commented.Message ID: <Cyfrin/security-and-auditing-full-course-s23/repo-discussions/176/comments/9395522 @github.com>

1 reply

EngrPips May 11, 2024

I am not sure how you mean.

guyzilberblum · 2024-05-11T15:27:44Z

guyzilberblum
May 11, 2024
Author

It still doesn't work

…

On Sat, 11 May 2024 at 17:19 EngrPips ***@***.***> wrote: I am not sure how you mean. — Reply to this email directly, view it on GitHub <#176 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BHZOBIKIUZ33O27OSKCOAX3ZBYSGRAVCNFSM6AAAAABHQYEKRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOJVHA2TQ> . You are receiving this because you were mentioned.Message ID: <Cyfrin/security-and-auditing-full-course-s23/repo-discussions/176/comments/9395858 @github.com>

3 replies

EngrPips May 11, 2024

Show your test and the output again.

kapitanluffy5 May 11, 2024

EngrPips May 11, 2024

I am trying to understand why your fuzz test doesn't show details like Patrick fuzz test did. I mean the (runs: , calls: , reverts: )

kapitanluffy5 · 2024-05-11T20:56:36Z

kapitanluffy5
May 11, 2024

I have no idea I will sit over the code again tomorrow gm

…

On Sat, 11 May 2024 at 23:55 EngrPips ***@***.***> wrote: I am trying to understand why your fuzz test doesn't show details like Patrick fuzz test did. I mean the (runs: , calls: , reverts: ) — Reply to this email directly, view it on GitHub <#176 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BEI52OZNX7DNRHU3D53AL2TZB2ASZAVCNFSM6AAAAABHQYEKRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOJZGQYTS> . You are receiving this because you commented.Message ID: <Cyfrin/security-and-auditing-full-course-s23/repo-discussions/176/comments/9399419 @github.com>

1 reply

EngrPips May 11, 2024

Alright, friend. Let me know whatever you find.

kapitanluffy5 · 2024-05-12T07:54:51Z

kapitanluffy5
May 12, 2024

Ok I changed the seed to 0x5 and it find an assertion failed I don’t know if it’s like assertion violation but it solved the problem

…

On Sun, 12 May 2024 at 0:07 EngrPips ***@***.***> wrote: Alright, friend. Let me know whatever you find. — Reply to this email directly, view it on GitHub <#176 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BEI52O6ZIYWFRLC7WJHHE73ZB2B7PAVCNFSM6AAAAABHQYEKRSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOJZGUZTA> . You are receiving this because you commented.Message ID: <Cyfrin/security-and-auditing-full-course-s23/repo-discussions/176/comments/9399530 @github.com>

6 replies

sharonphiliplima Jul 1, 2024

Help me out too please

[fuzz]
runs = 1000
seed = '0x5' 

[invariant]
runs = 256
depth = 15
fail_on_revert = false

and

contract StatefulFuzzCatchesTest is StdInvariant, Test {

    StatefulFuzzCatches statefulFuzzCatches;
    function setUp() public{
        statefulFuzzCatches = new StatefulFuzzCatches();
        targetContract(address(statefulFuzzCatches));
    }

    function testDoMoreMathAgain(uint128 randomNumber) public {
        assert(statefulFuzzCatches.doMoreMathAgain(randomNumber) != 0);
    }

    function statefulFuzz_catchesInvariant() public view{
        assert(statefulFuzzCatches.storedValue() != 0);

    }
}

forge test --mt statefulFuzz_catchesInvariant
[⠒] Compiling...
No files changed, compilation skipped

Ran 1 test for test/invariant-break/StatefulFuzzCatchesTest.t.sol:StatefulFuzzCatchesTest
[PASS] statefulFuzz_catchesInvariant() (runs: 256, calls: 3840, reverts: 49)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 490.99ms (482.83ms CPU time)

Ran 1 test suite in 494.56ms (490.99ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)

EngrPips Jul 1, 2024

Hey, You should increase the invariant runs, and maybe it will catch that scenario where storedValue is 0.

sharonphiliplima Jul 1, 2024

Got it. So, I had to actually make the runs = 1000 to get it work! Thanks.

sharonphiliplima Jul 1, 2024

Does that mean that we can't rely on it?

EngrPips Jul 1, 2024

Actually, fuzzing isn't 100% reliable because some edge cases can be too far away in the number of runs needed to catch them, and that is why we have formal verification, which does not throw random value to a contract to break it but instead prove that a condition holds mathematically. You know mathematics never lied, lol. When you get to the evm and formal verification course of the Cyfrin curriculum, you will learn in great detail formal-verification.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FullFuzz test exploits-minimized #176

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 8 comments 18 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

FullFuzz test exploits-minimized #176

Replies: 8 comments · 18 replies

guyzilberblum May 10, 2024 Author

guyzilberblum May 10, 2024 Author

guyzilberblum May 10, 2024 Author

guyzilberblum May 11, 2024 Author

Replies: 8 comments 18 replies

guyzilberblum
May 10, 2024
Author

guyzilberblum
May 10, 2024
Author

guyzilberblum
May 10, 2024
Author

guyzilberblum
May 11, 2024
Author