ydb/.github/config/mute_rules.md

How to Mute a test

• Through a PR Report

  • Open report in PR

  • In context menu of test select Crete mute issue

  • Through the Test history dashboard

  • Enter the test name or path in the full_name contain field, click Apply - the search is done by the occurrence.

  • Click the Mute link, which will create a draft issue in GitHub.

• Add the issue to the Mute and Un-mute project.

• Set the status to Mute

• Set the owner field to the team name (see the issue for the owner's name).

• Open muted_ya.txt in a new tab and edit it.

• Copy the line under Add line to muted_ya.txt (for example, like in the screenshot, ydb/core/kqp/ut/query KqpStats.SysViewClientLost) and add it to muted_ya.txt.

• Edit the branch for merging, for example, replace {username}-patch-1 with mute/{username}.

• Create a PR - copy the PR name from the issue name.

• Copy the issue description to the PR, keep the line Not for changelog (changelog entry is not required).

• Take "OK" from member of test owner team in PR

• Merge.

• Link Issue and Pr (field "Development" in issue and PR)

• Inform test owner team about new mutes - dm or in public chat (with mention of maintainer of team)

• You are awesome!

How to UnMute a test

--IN PROGRESS--

• Open muted_ya.txt
• Press "Edit file" and delete line of test
• Commit changes (Edit the branch for merging, for example, replace {username}-patch-1 with mute/{username})
• Edit PR name like "UnMute {testname}"
• Take "OK" from member of test owner team in PR
• Merge
• If test have an issue in Mute and Un-mute in status Muted - Move it to Unmuted
• Link Issue and Pr (field "Development" in issue and PR)
• Move issue to status Unmuted
• You are awesome!

How to manage muted tests by team

--IN PROGRESS--

Explore your tests stability

If you want to get more info about stability of your test visit dashboard (fill field owner={your_team_name})

Find your muted tests

Not all muted tests have issue in github project about this , we working on it

• Open project Mute and Un-mute
• click in label with name of your team, example link to qp muted tests (cgi ?filterQuery=owner%3Aqp)
• Open Mute {testname} issue
• Perform How to unmute

Flaky Tests

Who and When Monitors Flaky Tests

The CI duty engineer (in progress) checks flaky tests once a day (only working days).

• Open the Flaky dashboard.
• Perform the sections Mute Flaky Test and Test Flaps More - Need to Unmute once a day or ondemand

Mute Flaky Tests

Open the Flaky dashboard.

• Select today's date.
• Look at the tests in the Mute candidate table.

• Select today's date in the date_window.
• Select days_ago_window = 5 (how many days back from the selected day to calculate statistics). Currently, there are calculations for 1 day and 5 days ago.
  • If you want to understand how long ago and how often the test started failing, you can click the history link in the table (loading may take time) or select days_ago_window = 1.
• For days_ago_window = 5, set the values to filter out isolated failures and low run counts:
  • fail_count >= 3
  • run_count >= 10
• Click the Mute link, which will create a draft issue in GitHub.
• Perform steps from How to mute
• You are awesome!

Test is no longer flaky - Time to Unmute

• Open the Flaky dashboard.
• Look at the tests in the UNMute candidate table.

• If the summary: column shows mute <= 3 and success rate >= 98% - it's time to enable the test.
• Perform steps from How to Unmute
• You are awesome!