Browse Source

fix(releases): add time limit onto fetch_commits celery task (#76705)

add time limit onto this celery task of 15 minutes. the p100 in most
cases is far less than this. only a few times have we observed longer.
p99 is never above ~5 minutes. Deploys will usually kill long running
tasks anyways, so this is only important if there's a long running task
and we're not deploying. It is hypothesized the long running transaction
causing #inc-865 is coming from this task.

See
https://sentry.sentry.io/performance/summary/?project=1&query=&referrer=performance-transaction-summary&statsPeriod=14d&transaction=sentry.tasks.commits.fetch_commits&unselectedSeries=avg%28%29&unselectedSeries=p99%28%29&unselectedSeries=p95%28%29&unselectedSeries=p75%28%29&unselectedSeries=p50%28%29&unselectedSeries=Releases
Josh Ferge 6 months ago
parent
commit
15fcf7e673
1 changed files with 2 additions and 0 deletions
  1. 2 0
      src/sentry/tasks/commits.py

+ 2 - 0
src/sentry/tasks/commits.py

@@ -70,6 +70,8 @@ def handle_invalid_identity(identity, commit_failure=False):
     name="sentry.tasks.commits.fetch_commits",
     queue="commits",
     default_retry_delay=60 * 5,
+    soft_time_limit=60 * 15,
+    time_limit=60 * 15 + 5,
     max_retries=5,
     silo_mode=SiloMode.REGION,
 )