Browse Source

New Search API

- Filters are no longer extendable
- Initial draft ElasticSearch backend

Squashed commit of the following:

commit 8aa1b955ec44f578bf0e88926d647f20c6062197
Author: David Cramer <dcramer@gmail.com>
Date:   Tue May 20 20:58:44 2014 -0700

    Restore partial filter API (primarily for rendering sidebar)

commit 5f47a66881cdb2a3c720b72797c0358e2fa6a087
Merge: 3b65d9a 2a602c1
Author: David Cramer <dcramer@gmail.com>
Date:   Tue May 20 20:42:21 2014 -0700

    Merge branch 'master' into search-refactor

    Conflicts:
    	src/sentry/search/django/backend.py

commit 3b65d9aac197a37617fa0e28db098f512dd8f76c
Author: David Cramer <dcramer@gmail.com>
Date:   Mon May 19 20:19:18 2014 -0700

    A form

commit f7f3846d4cfba4691a121f0bb19aa3c01d7a223b
Author: David Cramer <dcramer@gmail.com>
Date:   Sun May 18 23:05:02 2014 -0700

    Kill sidebar on stream

commit caadfa7af4065490b7661b65ba8a23f22beffd4c
Author: David Cramer <dcramer@gmail.com>
Date:   Sun May 18 12:44:15 2014 -0700

    Move search constants into .constants

commit ca694a375c39add48c4b1e385beed6ce5a97846e
Merge: 92848be 245b39e
Author: David Cramer <dcramer@gmail.com>
Date:   Sun May 18 12:38:58 2014 -0700

    Merge branch 'master' into es

    Conflicts:
    	CHANGES
    	src/sentry/conf/server.py
    	src/sentry/filters/widgets.py
    	src/sentry/search/django/backend.py
    	src/sentry/testutils/fixtures.py
    	src/sentry/web/frontend/groups.py

commit 92848bea888f154483339b071f6b7953b49f13d6
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 22:32:42 2014 -0700

    Add missing params to base/elastic search

commit 657c679e079b78bc71ad42cb6f949114bd9e8f87
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 13:29:00 2014 -0700

    Correct behavior of Django search backend

commit 6c456c1b0aeac6b4df9e860e4cbc0c0d924167d7
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 13:23:39 2014 -0700

    Remove legacy search

commit 70f08401f114cda98dd3ac77f1ae4b69e17bef0b
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 13:15:52 2014 -0700

    Bring sort values back

commit 81a2b00788f135500e95c3bf7b074341c44d010b
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 12:57:43 2014 -0700

    First pass at removing filter related code

commit ee278b9ce5e60ebad85114d6dca947fe4bf290b2
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 12:47:54 2014 -0700

    Abstract search results

commit b3230e5ff27c31c5e0205b8ee20fdbff5b495908
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 12:31:09 2014 -0700

    search => query

commit a6885ff1b2cda751cc67ba8f9417f73f7712ae29
Merge: 04d6314 ab448ee
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 12:30:11 2014 -0700

    Merge branch 'master' into es

    Conflicts:
    	src/sentry/web/frontend/groups.py

commit 04d63140bcd9216077ecffb05fc6870ee7db973a
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 20 11:46:57 2014 -0700

    Initial work on abstracting stream fully into search

commit e4feaa00066bc4239ad89ca8a962f0cc205f9d30
Merge: 3ff6779 0584c8d
Author: David Cramer <dcramer@gmail.com>
Date:   Sat Apr 19 14:09:35 2014 -0700

    Merge branch 'master' into es

commit 3ff67797786bd9bc035ea88deb104ca305ab3e0f
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 22:14:03 2014 -0700

    Remove print

commit f8c1aedfef370084b644020b37990c5d52963f7a
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 22:12:45 2014 -0700

    Tag and status queries

commit db3623d37c61dbe8ada296ab74f40f52f9bf2236
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 21:39:16 2014 -0700

    Optional query param

commit e1ca10d656fd33ea7adcb28a8adfd17803159579
Merge: af8d390 453ea56
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 21:34:53 2014 -0700

    Merge branch 'master' into es

commit af8d390e48dda2e71781ff3c1f2ea4e7b437e9f6
Merge: 1a76e2c aadf4ee
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 21:34:34 2014 -0700

    Merge branch 'master' into es

commit aadf4eed2d8d3960286aedfa252fdb8027aec9f9
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 21:34:27 2014 -0700

    Auto slug in fixtures

commit 1a76e2cce5df9420ffa8abf5bbca492a3075728e
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 21:34:16 2014 -0700

    Fix test

commit 2e42bb9eb3667fb985183c7c4cb7be8b101840d5
Merge: e39eb66 0add3c9
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 21:31:53 2014 -0700

    Merge branch 'master' into es

commit 0add3c97359a8ac3c0e691ad4be64df351a993a3
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 21:31:45 2014 -0700

    Add create_team/create_project to fixtures

commit e39eb667c1d49b1f2225667bbff738001892b3ce
Author: David Cramer <dcramer@gmail.com>
Date:   Sun Apr 13 18:38:29 2014 -0400

    Initial draft code for new search backends
David Cramer 10 years ago
parent
commit
634594d241

+ 1 - 0
CHANGES

@@ -8,6 +8,7 @@ Backwards Incompatible Changes
 - The UDP server has been removed. Threaded/async models are the preferred replacement.
 - The ``is_rate_limited`` plugin hook has been removed in favor of singular quota managers.
 - The trends feature has been removed until it can be reimplemented in a more scalable way.
+- Filters have been removed. Integrations should use the tagging infrastructure instead.
 - NodeStore.generate_id() now returns a base64-encoded UUID.
 - The API for interfaces has been rewritten.
 - GroupMeta.objects.get_value no longer errors when a value is missing.

+ 1 - 0
setup.py

@@ -49,6 +49,7 @@ dev_requires = [
 tests_require = [
     'casscache',
     'cqlsh',
+    'elasticsearch',
     'exam>=0.5.1',
     'eventlet',
     'httpretty',

+ 38 - 87
src/sentry/api/endpoints/project_group_index.py

@@ -1,21 +1,14 @@
 from datetime import timedelta
-from django.http import HttpResponse
 from django.utils import timezone
+from rest_framework.response import Response
 
+from sentry.app import search
 from sentry.api.base import Endpoint
 from sentry.api.permissions import assert_perm
 from sentry.api.serializers import serialize
-from sentry.constants import (
-    SORT_OPTIONS, SORT_CLAUSES, SCORE_CLAUSES,
-    MYSQL_SORT_CLAUSES, MYSQL_SCORE_CLAUSES,
-    SQLITE_SORT_CLAUSES, SQLITE_SCORE_CLAUSES,
-    ORACLE_SORT_CLAUSES, ORACLE_SCORE_CLAUSES,
-    MSSQL_SORT_CLAUSES, MSSQL_SCORE_CLAUSES,
-    DEFAULT_SORT_OPTION,
-)
-from sentry.models import TagKey, Group, Project
+from sentry.constants import DEFAULT_SORT_OPTION
+from sentry.models import TagKey, Project
 from sentry.utils.dates import parse_date
-from sentry.utils.db import get_db_engine
 
 
 class ProjectGroupIndexEndpoint(Endpoint):
@@ -29,29 +22,32 @@ class ProjectGroupIndexEndpoint(Endpoint):
 
         assert_perm(project, request.user, request.auth)
 
-        group_list = Group.objects.all()
+        query_kwargs = {
+            'project': project,
+        }
+
+        if request.GET.get('status'):
+            query_kwargs['status'] = int(request.GET['status'])
 
         if request.user.is_authenticated() and request.GET.get('bookmarks'):
-            group_list = group_list.filter(
-                bookmark_set__project=project,
-                bookmark_set__user=request.user,
-            )
-        else:
-            group_list = group_list.filter(project=project)
-
-        status = request.GET.get('status')
-        if status:
-            group_list = group_list.filter(status=int(status))
-
-        tag_keys = TagKey.objects.all_keys(project)
-        for tag in tag_keys:
-            value = request.GET.get(tag)
-            if value:
-                group_list = group_list.filter(
-                    grouptag__project=project,
-                    grouptag__key=tag,
-                    grouptag__value=value,
-                )
+            query_kwargs['bookmarked_by'] = request.user
+
+        sort_by = request.GET.get('sort') or request.session.get('streamsort')
+        if sort_by is None:
+            sort_by = DEFAULT_SORT_OPTION
+
+        # Save last sort in session
+        if sort_by != request.session.get('streamsort'):
+            request.session['streamsort'] = sort_by
+
+        query_kwargs['sort_by'] = sort_by
+
+        tags = {}
+        for tag_key in TagKey.objects.all_keys(project):
+            if request.GET.get(tag_key):
+                tags[tag_key] = request.GET[tag_key]
+        if tags:
+            query_kwargs['tags'] = tags
 
         # TODO: dates should include timestamps
         date_from = request.GET.get('since')
@@ -62,7 +58,6 @@ class ProjectGroupIndexEndpoint(Endpoint):
         time_to = request.GET.get('tt')
 
         today = timezone.now()
-
         # date format is Y-m-d
         if any(x is not None for x in [date_from, time_from, date_to, time_to]):
             date_from, date_to = parse_date(date_from, time_from), parse_date(date_to, time_to)
@@ -70,60 +65,16 @@ class ProjectGroupIndexEndpoint(Endpoint):
             date_from = today - timedelta(days=5)
             date_to = None
 
-        if date_filter == 'first_seen':
-            if date_from:
-                group_list = group_list.filter(first_seen__gte=date_from)
-            elif date_to:
-                group_list = group_list.filter(first_seen__lte=date_to)
-        else:
-            # TODO(dcramer): a date_to no longer makes a lot of sense, and will
-            # need corrected when search lands
-            if date_from:
-                group_list = group_list.filter(last_seen__gte=date_from)
-            if date_to:
-                group_list = group_list.filter(last_seen__lte=date_to)
-
-        sort = request.GET.get('sort') or request.session.get('streamsort')
-        if sort is None:
-            sort = DEFAULT_SORT_OPTION
-        elif sort not in SORT_OPTIONS or sort.startswith('accel_'):
-            return HttpResponse(status=400)
-
-        # Save last sort in session
-        if sort != request.session.get('streamsort'):
-            request.session['streamsort'] = sort
-
-        engine = get_db_engine('default')
-        if engine.startswith('sqlite'):
-            score_clause = SQLITE_SORT_CLAUSES.get(sort)
-            filter_clause = SQLITE_SCORE_CLAUSES.get(sort)
-        elif engine.startswith('mysql'):
-            score_clause = MYSQL_SORT_CLAUSES.get(sort)
-            filter_clause = MYSQL_SCORE_CLAUSES.get(sort)
-        elif engine.startswith('oracle'):
-            score_clause = ORACLE_SORT_CLAUSES.get(sort)
-            filter_clause = ORACLE_SCORE_CLAUSES.get(sort)
-        elif engine in ('django_pytds', 'sqlserver_ado', 'sql_server.pyodbc'):
-            score_clause = MSSQL_SORT_CLAUSES.get(sort)
-            filter_clause = MSSQL_SCORE_CLAUSES.get(sort)
-        else:
-            score_clause = SORT_CLAUSES.get(sort)
-            filter_clause = SCORE_CLAUSES.get(sort)
+        query_kwargs['date_from'] = date_from
+        query_kwargs['date_to'] = date_to
+        if date_filter:
+            query_kwargs['date_filter'] = date_filter
 
-        assert score_clause
+        # TODO: proper pagination support
+        cursor = request.GET.get('cursor')
+        if cursor:
+            query_kwargs['cursor'] = cursor
 
-        if sort == 'tottime':
-            group_list = group_list.filter(time_spent_count__gt=0)
-        elif sort == 'avgtime':
-            group_list = group_list.filter(time_spent_count__gt=0)
+        results = search.query(**query_kwargs)
 
-        group_list = group_list.extra(
-            select={'sort_value': score_clause},
-        )
-
-        return self.paginate(
-            request=request,
-            queryset=group_list,
-            order_by='-sort_value',
-            on_results=lambda x: serialize(x, request.user),
-        )
+        return Response(serialize(list(results), request.user))

+ 0 - 1
src/sentry/conf/server.py

@@ -473,7 +473,6 @@ SENTRY_IGNORE_EXCEPTIONS = (
     'OperationalError',
 )
 
-
 SENTRY_KEY = None
 
 # Absolute URL to the sentry root directory. Should not include a trailing slash.

+ 0 - 41
src/sentry/constants.py

@@ -36,44 +36,6 @@ SORT_OPTIONS = SortedDict((
     ('avgtime', _('Average Time Spent')),
 ))
 
-SORT_CLAUSES = {
-    'priority': 'sentry_groupedmessage.score',
-    'date': 'EXTRACT(EPOCH FROM sentry_groupedmessage.last_seen)',
-    'new': 'EXTRACT(EPOCH FROM sentry_groupedmessage.first_seen)',
-    'freq': 'sentry_groupedmessage.times_seen',
-    'tottime': 'sentry_groupedmessage.time_spent_total',
-    'avgtime': '(sentry_groupedmessage.time_spent_total / sentry_groupedmessage.time_spent_count)',
-}
-SCORE_CLAUSES = SORT_CLAUSES.copy()
-
-SQLITE_SORT_CLAUSES = SORT_CLAUSES.copy()
-SQLITE_SORT_CLAUSES.update({
-    'date': "(julianday(sentry_groupedmessage.last_seen) - 2440587.5) * 86400.0",
-    'new': "(julianday(sentry_groupedmessage.first_seen) - 2440587.5) * 86400.0",
-})
-SQLITE_SCORE_CLAUSES = SQLITE_SORT_CLAUSES.copy()
-
-MYSQL_SORT_CLAUSES = SORT_CLAUSES.copy()
-MYSQL_SORT_CLAUSES.update({
-    'date': 'UNIX_TIMESTAMP(sentry_groupedmessage.last_seen)',
-    'new': 'UNIX_TIMESTAMP(sentry_groupedmessage.first_seen)',
-})
-MYSQL_SCORE_CLAUSES = MYSQL_SORT_CLAUSES.copy()
-
-ORACLE_SORT_CLAUSES = SCORE_CLAUSES.copy()
-ORACLE_SORT_CLAUSES.update({
-    'date': "(cast(sentry_groupedmessage.last_seen as date)-TO_DATE('01/01/1970 00:00:00', 'MM-DD-YYYY HH24:MI:SS')) * 24 * 60 * 60",
-    'new': "(cast(sentry_groupedmessage.first_seen as date)-TO_DATE('01/01/1970 00:00:00', 'MM-DD-YYYY HH24:MI:SS')) * 24 * 60 * 60",
-})
-ORACLE_SCORE_CLAUSES = ORACLE_SORT_CLAUSES.copy()
-
-MSSQL_SORT_CLAUSES = SCORE_CLAUSES.copy()
-MSSQL_SORT_CLAUSES.update({
-    'date': "DATEDIFF(s, '1970-01-01T00:00:00', sentry_groupedmessage.last_seen)",
-    'new': "DATEDIFF(s, '1970-01-01T00:00:00', sentry_groupedmessage.first_seen)",
-})
-MSSQL_SCORE_CLAUSES = MSSQL_SORT_CLAUSES.copy()
-
 SEARCH_SORT_OPTIONS = SortedDict((
     ('score', _('Score')),
     ('date', _('Last Seen')),
@@ -202,9 +164,6 @@ EVENTS_PER_PAGE = 15
 # Default sort option for the group stream
 DEFAULT_SORT_OPTION = 'date'
 
-# Default sort option for the search results
-SEARCH_DEFAULT_SORT_OPTION = 'date'
-
 # Setup languages for only available locales
 LANGUAGE_MAP = dict(settings.LANGUAGES)
 LANGUAGES = [(k, LANGUAGE_MAP[k]) for k in get_all_languages() if k in LANGUAGE_MAP]

+ 0 - 1
src/sentry/filters/__init__.py

@@ -8,7 +8,6 @@ sentry.filters
 
 from sentry.filters.base import *  # NOQA
 from sentry.filters.builtins import *  # NOQA
-from sentry.filters.helpers import *  # NOQA
 from sentry.filters.widgets import *  # NOQA
 
 # Backwards compatibility

+ 5 - 15
src/sentry/filters/base.py

@@ -29,9 +29,13 @@ class Filter(object):
     show_label = True
     max_choices = 50
 
-    def __init__(self, request, project):
+    def __init__(self, request, project, label=None, column=None):
         self.request = request
         self.project = project
+        if label is not None:
+            self.label = label
+        if column is not None:
+            self.column = column
 
     def is_set(self):
         return bool(self.get_value())
@@ -71,10 +75,6 @@ class Filter(object):
             cache.set(key, result, 60)
         return SortedDict((l, l) for l in result)
 
-    def get_query_set(self, queryset):
-        kwargs = {self.column: self.get_value()}
-        return queryset.filter(**kwargs)
-
     def process(self, data):
         """``self.request`` is not available within this method"""
         return data
@@ -82,13 +82,3 @@ class Filter(object):
     def render(self):
         widget = self.get_widget()
         return widget.render(self.get_value())
-
-
-class TagFilter(Filter):
-    def get_query_set(self, queryset):
-        col, val = self.get_column(), self.get_value()
-        queryset = queryset.filter(**dict(
-            grouptag__key=col,
-            grouptag__value=val,
-        ))
-        return queryset.distinct()

+ 0 - 66
src/sentry/filters/helpers.py

@@ -1,66 +0,0 @@
-"""
-sentry.filters.helpers
-~~~~~~~~~~~~~~~~~~~~~~
-
-:copyright: (c) 2010-2014 by the Sentry Team, see AUTHORS for more details.
-:license: BSD, see LICENSE for more details.
-"""
-
-# Widget api is pretty ugly
-from __future__ import absolute_import
-
-__all__ = ('get_filters',)
-
-import logging
-
-from django.conf import settings
-from sentry.constants import TAG_LABELS
-from sentry.filters.base import TagFilter
-from sentry.plugins import plugins
-from sentry.utils.safe import safe_execute
-
-
-FILTER_CACHE = {}
-TAG_FILTER_CACHE = {}
-
-
-def get_filters(model=None, project=None):
-    filter_list = []
-
-    # Add builtins (specified with the FILTERS setting)
-    for class_path in settings.SENTRY_FILTERS:
-        if class_path not in FILTER_CACHE:
-            module_name, class_name = class_path.rsplit('.', 1)
-            try:
-                module = __import__(module_name, {}, {}, class_name)
-                cls = getattr(module, class_name)
-            except Exception:
-                logger = logging.getLogger('sentry.errors.filters')
-                logger.exception('Unable to import %s', class_path)
-                continue
-            FILTER_CACHE[class_path] = cls
-        filter_list.append(FILTER_CACHE[class_path])
-
-    if project:
-        for tag in project.get_tags():
-            if tag not in TAG_FILTER_CACHE:
-                # Generate a new filter class because we are lazy and do
-                # not want to rewrite code
-                class new(TagFilter):
-                    label = TAG_LABELS.get(tag) or tag.replace('_', ' ').title()
-                    column = tag
-                new.__name__ = '__%sGeneratedFilter' % tag.encode('utf8')
-                TAG_FILTER_CACHE[tag] = new
-            filter_list.append(TAG_FILTER_CACHE[tag])
-
-    # Add plugin-provided filters
-    for plugin in plugins.for_project(project):
-        results = safe_execute(plugin.get_filters, project)
-        if results:
-            for filter_cls in results:
-                if filter_cls not in filter_list:
-                    filter_list.append(filter_cls)
-
-    # yield all filters which support ``model``
-    for filter_cls in filter_list:
-        yield filter_cls

+ 1 - 2
src/sentry/manager.py

@@ -310,8 +310,7 @@ class GroupManager(BaseManager):
                 is_regression=is_regression,
             )
 
-        if getattr(settings, 'SENTRY_INDEX_SEARCH', settings.SENTRY_USE_SEARCH):
-            index_event.delay(event)
+        index_event.delay(event)
 
         # TODO: move this to the queue
         if is_new and not raw:

+ 0 - 11
src/sentry/plugins/base.py

@@ -499,17 +499,6 @@ class IPlugin(local):
         >>>     return [('tag-name', 'tag-value')]
         """
 
-    def get_filters(self, project=None, **kwargs):
-        """
-        Provides additional filters to the builtins.
-
-        Must return an iterable.
-
-        >>> def get_filters(self, project, **kwargs):
-        >>>     return [MyFilterClass]
-        """
-        return []
-
     def get_notification_forms(self, **kwargs):
         """
         Provides additional UserOption forms for the Notification Settings page.

Some files were not shown because too many files changed in this diff