Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Include standard error in the result to make it easier to find errors. #56

Open
Tasteful opened this issue Dec 2, 2024 · 0 comments · May be fixed by #57
Open

Include standard error in the result to make it easier to find errors. #56

Tasteful opened this issue Dec 2, 2024 · 0 comments · May be fixed by #57
Labels
bug Something isn't working

Comments

@Tasteful
Copy link

Tasteful commented Dec 2, 2024 •
edited by devlooped-bot
Loading

Describe the Bug

When executing the retest command in the Azure DevOps pipeline we got unexpected exceptions when test was failing in the first attempt but succeeded after that.

Steps to Reproduce

Add a failing test, the error output is not included.

2024-12-02T09:32:39.6673615Z   Running tests: 0%
2024-12-02T09:32:40.4604022Z   Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Application.FunctionalTest\bin\Release\net8.0\company.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T09:32:40.6001361Z   VSTest version 17.12.0 (x64)
2024-12-02T09:32:40.6143386Z   
2024-12-02T09:32:40.8450984Z   Starting test execution, please wait...
2024-12-02T09:32:40.9109329Z   A total of 1 test files matched the specified pattern.
2024-12-02T09:33:09.9122186Z   Running tests: 0%
2024-12-02T09:33:40.1810138Z   Running tests: 0%
2024-12-02T09:34:10.4749776Z   Running tests: 0%
2024-12-02T09:34:40.7409708Z   Running tests: 0%
2024-12-02T09:35:10.9912717Z   Running tests: 0%
2024-12-02T09:35:41.2105825Z   Running tests: 0%
2024-12-02T09:36:11.5094507Z   Running tests: 0%
2024-12-02T09:36:41.6984378Z   Running tests: 0%
2024-12-02T09:36:45.6858614Z   [xUnit.net 00:04:03.78]     company.Application.Sales.DiscountScenariosTest_performance_productlists.Performance [SKIP]
2024-12-02T09:36:46.9030210Z     Skipped company.Application.Sales.DiscountScenariosTest_performance_productlists.Performance [1 ms]
2024-12-02T09:36:54.6798321Z   [xUnit.net 00:04:12.78]     company.Application.Customers.OrganizationLargeDataTest.Create_and_query_one_by_one [SKIP]
2024-12-02T09:36:54.6800777Z   [xUnit.net 00:04:12.78]     company.Application.Customers.OrganizationLargeDataTest.Create_and_query [SKIP]
2024-12-02T09:36:55.8419968Z     Skipped company.Application.Customers.OrganizationLargeDataTest.Create_and_query_one_by_one [1 ms]
2024-12-02T09:36:55.8422016Z     Skipped company.Application.Customers.OrganizationLargeDataTest.Create_and_query [1 ms]
2024-12-02T09:37:11.9675535Z   Running tests: 0%
2024-12-02T09:37:42.2254034Z   Running tests: 0%
2024-12-02T09:38:12.4954851Z   Running tests: 0%
2024-12-02T09:38:42.7164232Z   Running tests: 0%
2024-12-02T09:39:03.5622838Z   [xUnit.net 00:06:21.66]     company.Application.Products.PriceListItemTest.BatchCreate [SKIP]
2024-12-02T09:39:03.9796608Z     Skipped company.Application.Products.PriceListItemTest.BatchCreate [1 ms]
2024-12-02T09:39:04.1210644Z   [xUnit.net 00:06:22.22]     company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock [SKIP]
2024-12-02T09:39:05.4648231Z     Skipped company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock [1 ms]
2024-12-02T09:39:06.3747827Z   [xUnit.net 00:06:24.47]     company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock_async [SKIP]
2024-12-02T09:39:06.3984435Z     Skipped company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock_async [1 ms]
2024-12-02T09:39:12.9653416Z   Running tests: 0%
2024-12-02T09:39:43.1508352Z   Running tests: 0%
2024-12-02T09:39:49.0636899Z   [xUnit.net 00:07:07.16]     company.Application.FieldFramework.SystemObjectFieldTypeTest.ListSystemFields_NotHidden_And_Have_No_Translation [SKIP]
2024-12-02T09:39:50.1187020Z     Skipped company.Application.FieldFramework.SystemObjectFieldTypeTest.ListSystemFields_NotHidden_And_Have_No_Translation [1 ms]
2024-12-02T09:40:13.3679894Z   Running tests: 0%
2024-12-02T09:40:43.6224550Z   Running tests: 0%
2024-12-02T09:41:13.9013802Z   Running tests: 0%
2024-12-02T09:41:44.1897585Z   Running tests: 0%
2024-12-02T09:42:14.4554020Z   Running tests: 0%
2024-12-02T09:42:44.6970853Z   Running tests: 0%
2024-12-02T09:43:14.9453462Z   Running tests: 0%
2024-12-02T09:43:33.2749219Z   Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_10_36_09.trx
2024-12-02T09:43:33.2751626Z   
2024-12-02T09:43:33.3012504Z   Passed!  - Failed:     0, Passed:  4097, Skipped:     7, Total:  4104, Duration: 7 m 22 s - company.Application.FunctionalTest.dll (net8.0)
2024-12-02T09:43:34.8916534Z   Running tests: 0%
2024-12-02T09:43:35.3048860Z   Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Web.Application.FunctionalTest\bin\Release\net8.0\company.Web.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T09:43:35.4136135Z   VSTest version 17.12.0 (x64)
2024-12-02T09:43:35.4222197Z   
2024-12-02T09:43:35.6804642Z   Starting test execution, please wait...
2024-12-02T09:43:35.7521536Z   A total of 1 test files matched the specified pattern.
2024-12-02T09:44:05.0995094Z   Running tests: 0%
2024-12-02T09:44:35.3779195Z   Running tests: 0%
2024-12-02T09:45:05.5609960Z   Running tests: 0%
2024-12-02T09:45:35.7582249Z   Running tests: 0%
2024-12-02T09:46:05.5828177Z     Failed company.Web.Application.AdminWebApi.Controllers.Products.ProductsControllerTest.QueryDataAsync_WithSearchText_ForBaseProduct [561 ms]
2024-12-02T09:46:05.6230857Z ##[error]EXEC(0,0): Error Message: 
2024-12-02T09:46:05.6234280Z EXEC : error Message:  [C:\SysData\vsts-agent1\_work\1\s\company.Build.TestFunctional.target]
2024-12-02T09:46:05.6236142Z                Γåô (pos 0)
2024-12-02T09:46:05.6236726Z   Collection: [eafbcb81-21ce-4c7b-a9a0-84c92e9ffd21]
2024-12-02T09:46:05.6237242Z     Stack Trace:
2024-12-02T09:46:05.6238186Z        at company.Web.Application.AdminWebApi.Controllers.Products.ProductsControllerTest.QueryDataAsync_WithSearchText_ForBaseProduct() in /_/Test/company.Web.Application.FunctionalTest/AdminWebApi/Controllers/Products/ProductsControllerTest.cs:line 163
2024-12-02T09:46:05.6239130Z   --- End of stack trace from previous location ---
2024-12-02T09:46:05.6328427Z      Assert.DoesNotContain() Failure: Filter matched in collection
2024-12-02T09:46:05.9957743Z   Running tests: 0%
2024-12-02T09:46:36.2924005Z   Running tests: 0%
2024-12-02T09:47:01.6132997Z   Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_10_44_06.trx
2024-12-02T09:47:01.6139981Z   
2024-12-02T09:47:01.6243152Z   Failed!  - Failed:     1, Passed:  1266, Skipped:     0, Total:  1267, Duration: 2 m 57 s - company.Web.Application.FunctionalTest.dll (net8.0)
2024-12-02T09:47:06.5322474Z   Running tests: 1%
2024-12-02T09:47:36.7674647Z   Running tests: 1%
2024-12-02T09:47:40.8288727Z   Retrying 1 failed test: 0%
2024-12-02T09:47:42.0336194Z   Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Web.Application.FunctionalTest\bin\Release\net8.0\company.Web.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T09:47:42.1476790Z   VSTest version 17.12.0 (x64)
2024-12-02T09:47:42.1582980Z   
2024-12-02T09:47:42.3517951Z   Starting test execution, please wait...
2024-12-02T09:47:42.4160527Z   A total of 1 test files matched the specified pattern.
2024-12-02T09:48:10.0618569Z   Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_10_48_10.trx
2024-12-02T09:48:10.0619313Z   
2024-12-02T09:48:10.0681507Z   Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 4 s - company.Web.Application.FunctionalTest.dll (net8.0)
2024-12-02T09:48:11.1103609Z   Running tests: 1%
2024-12-02T09:48:11.1104083Z   Retrying 1 failed test: 0%

As in the log above the test is writing to the error stream and it's picked up as EXEC : error Message: and it's very hard to understand from where it come.

Expected Behavior

The failed test should be visible with the test data included.

2024-12-02T12:40:53.7207327Z   Running tests: 0%
2024-12-02T12:40:54.2566868Z   Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Application.FunctionalTest\bin\Release\net8.0\company.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T12:40:54.3654094Z   VSTest version 17.12.0 (x64)
2024-12-02T12:40:54.3765417Z   
2024-12-02T12:40:54.6078033Z   Starting test execution, please wait...
2024-12-02T12:40:54.6784033Z   A total of 1 test files matched the specified pattern.
2024-12-02T12:41:23.9736448Z   Running tests: 0%
2024-12-02T12:41:54.2555315Z   Running tests: 0%
2024-12-02T12:42:24.5091097Z   Running tests: 0%
2024-12-02T12:42:54.7434686Z   Running tests: 0%
2024-12-02T12:43:24.9961153Z   Running tests: 0%
2024-12-02T12:43:55.2438927Z   Running tests: 0%
2024-12-02T12:44:25.5122528Z   Running tests: 0%
2024-12-02T12:44:55.7770807Z   Running tests: 0%
2024-12-02T12:45:06.6002079Z   [xUnit.net 00:04:10.97]     company.Application.Sales.DiscountScenariosTest_performance_productlists.Performance [SKIP]
2024-12-02T12:45:07.9335906Z     Skipped company.Application.Sales.DiscountScenariosTest_performance_productlists.Performance [1 ms]
2024-12-02T12:45:26.0562787Z   Running tests: 0%
2024-12-02T12:45:28.4220239Z   [xUnit.net 00:04:32.80]     company.Application.FieldFramework.SystemObjectFieldTypeTest.ListSystemFields_NotHidden_And_Have_No_Translation [SKIP]
2024-12-02T12:45:28.8184575Z     Skipped company.Application.FieldFramework.SystemObjectFieldTypeTest.ListSystemFields_NotHidden_And_Have_No_Translation [1 ms]
2024-12-02T12:45:56.3438379Z   Running tests: 0%
2024-12-02T12:46:20.1952961Z   [xUnit.net 00:05:24.51]     company.Application.Sales.CartContextTest.Confirm_Order_With_Discount [FAIL]
2024-12-02T12:46:20.3442111Z     Failed company.Application.Sales.CartContextTest.Confirm_Order_With_Discount [789 ms]
2024-12-02T12:46:20.3476711Z ##[error]EXEC(0,0): Error Message: 
2024-12-02T12:46:20.3480502Z EXEC : error Message:  [C:\SysData\vsts-agent1\_work\1\s\company.Build.TestFunctional.target]
2024-12-02T12:46:20.3481472Z      System.NullReferenceException : Object reference not set to an instance of an object.
2024-12-02T12:46:20.3482152Z     Stack Trace:
2024-12-02T12:46:20.3482777Z   --- End of stack trace from previous location ---
2024-12-02T12:46:20.3483788Z        at company.Application.Sales.CartContextTest.Confirm_Order_With_Discount() in /_/Test/company.Application.FunctionalTest/Sales/CartContextTest.cs:line 486
2024-12-02T12:46:26.5785112Z   Running tests: 0%
2024-12-02T12:46:56.7782305Z   Running tests: 0%
2024-12-02T12:47:27.0432181Z   Running tests: 0%
2024-12-02T12:47:57.3380289Z   Running tests: 0%
2024-12-02T12:48:18.4290223Z   [xUnit.net 00:07:22.80]     company.Application.Products.PriceListItemTest.BatchCreate [SKIP]
2024-12-02T12:48:19.0115100Z     Skipped company.Application.Products.PriceListItemTest.BatchCreate [1 ms]
2024-12-02T12:48:27.5578756Z   Running tests: 0%
2024-12-02T12:48:57.8561192Z   Running tests: 0%
2024-12-02T12:49:19.4309411Z   [xUnit.net 00:08:23.80]     company.Application.Customers.OrganizationLargeDataTest.Create_and_query_one_by_one [SKIP]
2024-12-02T12:49:19.4312032Z   [xUnit.net 00:08:23.81]     company.Application.Customers.OrganizationLargeDataTest.Create_and_query [SKIP]
2024-12-02T12:49:19.8339408Z     Skipped company.Application.Customers.OrganizationLargeDataTest.Create_and_query [1 ms]
2024-12-02T12:49:19.8342823Z     Skipped company.Application.Customers.OrganizationLargeDataTest.Create_and_query_one_by_one [1 ms]
2024-12-02T12:49:28.0812795Z   Running tests: 0%
2024-12-02T12:49:58.3583202Z   Running tests: 0%
2024-12-02T12:50:28.6461864Z   Running tests: 0%
2024-12-02T12:50:58.8824052Z   Running tests: 0%
2024-12-02T12:51:29.0880513Z   Running tests: 0%
2024-12-02T12:51:29.4982802Z   [xUnit.net 00:10:33.87]     company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock [SKIP]
2024-12-02T12:51:30.3101589Z     Skipped company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock [1 ms]
2024-12-02T12:51:31.7634704Z   [xUnit.net 00:10:36.14]     company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock_async [SKIP]
2024-12-02T12:51:31.7686782Z     Skipped company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock_async [1 ms]
2024-12-02T12:51:49.9169877Z   Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_13_44_24.trx
2024-12-02T12:51:49.9175241Z   
2024-12-02T12:51:49.9457167Z   Failed!  - Failed:     1, Passed:  4096, Skipped:     7, Total:  4104, Duration: 7 m 24 s - company.Application.FunctionalTest.dll (net8.0)
2024-12-02T12:51:59.2905407Z   Running tests: 1%
2024-12-02T12:52:27.5741269Z   Retrying 1 failed test: 0%
2024-12-02T12:52:28.5668787Z   Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Application.FunctionalTest\bin\Release\net8.0\company.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T12:52:28.7469900Z   VSTest version 17.12.0 (x64)
2024-12-02T12:52:28.7695311Z   
2024-12-02T12:52:29.0461102Z   Starting test execution, please wait...
2024-12-02T12:52:29.1227452Z   A total of 1 test files matched the specified pattern.
2024-12-02T12:52:57.8723679Z   Running tests: 1%
2024-12-02T12:52:57.8724983Z   Retrying 1 failed test: 0%
2024-12-02T12:53:05.1956453Z   Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_13_53_03.trx
2024-12-02T12:53:05.1956994Z   
2024-12-02T12:53:05.2039329Z   Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 2 s - company.Application.FunctionalTest.dll (net8.0)

In the above log, if you searching for [FAIL] you will see one row with information that not are included in the first log, and I think it's good to make the visible as well.

Version Info

0.6.3

Back this issue
Back this issue
@Tasteful Tasteful added the bug Something isn't working label Dec 2, 2024
@Tasteful Tasteful linked a pull request Dec 2, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Include messages to standard error, this to make the log complete Tasteful/dotnet-retest
1 participant
@Tasteful