Merge pull request #284 from EverythingMe/feature_google_oauth

Feature: Google OAuth support (instead of deprecated OpenID)

README.md

@@ -1,18 +1,24 @@
-# [_re:dash_](https://github.com/everythingme/redash)
-![Build Status](https://circleci.com/gh/EverythingMe/redash.png?circle-token=8a695aa5ec2cbfa89b48c275aea298318016f040 "Build Status")
+<p align="center">
+  <img title="re:dash" src='https://raw.githubusercontent.com/EverythingMe/redash/screenshots/redash_logo.png' />
+
+</p>
+<p align="center">
+    <img title="Build Status" src='https://circleci.com/gh/EverythingMe/redash.png?circle-token=8a695aa5ec2cbfa89b48c275aea298318016f040'/>
+</p>
 
 **_re:dash_** is our take on freeing the data within our company in a way that will better fit our culture and usage patterns.
 
-Prior to **_re:dash_**, we tried to use tranditional BI suites and discovered a set of bloated, technically challenged and slow tools/flows. What we were looking for was a more hacker'ish way to look at data, so we built one.
+Prior to **_re:dash_**, we tried to use traditional BI suites and discovered a set of bloated, technically challenged and slow tools/flows. What we were looking for was a more hacker'ish way to look at data, so we built one.
 
 **_re:dash_** was built to allow fast and easy access to billions of records, that we process and collect using Amazon Redshift ("petabyte scale data warehouse" that "speaks" PostgreSQL).
+Today **_re:dash_** has support for querying multiple databases, including: Redshift, Google BigQuery, PostgreSQL, MySQL, Graphite and custom scripts.
 
 **_re:dash_** consists of two parts:
 
-1. **Query Editor**: think of [JS Fiddle](http://jsfiddle.net) for SQL queries. It's your way to share data in the organization in an open way, by sharing both the dataset and the query that generated it. This way everyone can peer review not only the resulting dataset but also the process that generated it. Also it's possible to fork it and generate new datasets and reach new insights. 
-2. **Dashboards/Visualizations**: once you have a dataset, you can create different visualizations out of it, and then combine several visualizations into a single dashboard. Currently it supports bar charts, pivot table and cohorts.
+1. **Query Editor**: think of [JS Fiddle](http://jsfiddle.net) for SQL queries. It's your way to share data in the organization in an open way, by sharing both the dataset and the query that generated it. This way everyone can peer review not only the resulting dataset but also the process that generated it. Also it's possible to fork it and generate new datasets and reach new insights.
+2. **Dashboards/Visualizations**: once you have a dataset, you can create different visualizations out of it, and then combine several visualizations into a single dashboard. Currently it supports charts, pivot table and cohorts.
 
-This is the first release, which is more than usable but still has its rough edges and way to go to fulfill its full potential. The Query Editor part is quite solid, but the visualizations need more work to enrich them and to make them more user friendly.
+**_re:dash_** is a work in progress and has its rough edges and way to go to fulfill its full potential. The Query Editor part is quite solid, but the visualizations need more work to enrich them and to make them more user friendly.
 
 ## Demo
 
@@ -22,51 +28,21 @@ You can try out the demo instance: http://rd-demo.herokuapp.com/ (login with any
 
 Due to Heroku dev plan limits, it has a small database of flights (see schema [here](http://rd-demo.herokuapp.com/dashboard/schema)). Also due to another Heroku limitation, it is running with the regular user, hence you can DELETE or INSERT data/tables. Please be nice and don't do this.
 
+## Getting Started
+
+* [Setting re:dash on your own server (Ubuntu)](https://github.com/EverythingMe/redash/wiki/Setting-re:dash-on-your-own-server-(for-version-0.4))
+* Additional documentation in the [Wiki](https://github.com/everythingme/redash/wiki).
+
+
 ## Getting help
 
 * [Google Group (mailing list)](https://groups.google.com/forum/#!forum/redash-users): the best place to get updates about new releases or ask general questions.
-* #redash IRC channel on [Freenode](http://www.freenode.net/).
-
-## Technology
-
-* Python
-* [AngularJS](http://angularjs.org/)
-* [PostgreSQL](http://www.postgresql.org/) / [AWS Redshift](http://aws.amazon.com/redshift/)
-* [Redis](http://redis.io)
-
-PostgreSQL is used both as the operatinal database for the system, but also as the data store that is being queried. To be exact, we built this system to use on top of Amazon's Redshift, which supports the PG driver. But it's quite simple to add support for other datastores, and we do plan to do so.
-
-This is our first large scale AngularJS project, and we learned a lot during the development of it. There are still things we need to iron out, and comments on the way we use AngularJS are more than welcome (and pull requests just as well).
-
-### HighCharts
-
-HighCharts is really great, but it's not free for commercial use. Please refer to their [licensing options](http://shop.highsoft.com/highcharts.html), to see what applies for your use. 
-
-It's very likely that in the future we will switch to [D3.js](http://d3js.org/) instead.
-
-## Getting Started
-
-* [Setting up re:dash on Heroku in 5 minutes](https://github.com/EverythingMe/redash/wiki/Setting-up-re:dash-on-Heroku-in-5-minutes)
-* [Setting re:dash on your own server (Ubuntu)](https://github.com/EverythingMe/redash/wiki/Setting-re:dash-on-your-own-server-(Ubuntu))
-
-**Need help setting re:dash or one of the dependencies up?** Ping @arikfr on the IRC #redash channel or send a message to the [mailing list](https://groups.google.com/forum/#!forum/redash-users), and he will gladly help.
+* Find us [on gitter](https://gitter.im/EverythingMe/redash#) (chat).
+* Contact Arik, the maintainer directly: arik@everything.me.
 
 ## Roadmap
 
-Below you can see the "big" features of the next 3 releases (for full list, click on the link):
-
-### [v0.3](https://github.com/EverythingMe/redash/issues?milestone=2&state=open)
-
-- Dashboard filters: ability to filter/slice the data you see in a single dashboard using filters (date or selectors).
-- Multiple databases support (including other database type than PostgreSQL).
-- Scheduled reports by email.
-- Comments on queries.
-
-### [v0.4](https://github.com/EverythingMe/redash/issues?milestone=3&state=open)
-
-- Query versioning. 
-- More "realtime" UI (using websockets).
-- More visualizations.
+TBD.
 
 ## Reporting Bugs and Contributing Code
 

circle.yml

@@ -23,3 +23,6 @@ deployment:
     branch: master
     commands:
       - make upload
+notify:
+  webhooks:
+    - url: https://webhooks.gitter.im/e/895d09c3165a0913ac2f

rd_ui/app/login.html

@@ -65,7 +65,7 @@
 
             <div class="row">
                 <div class="col-xs-6 col-sm-6 col-md-6">
-                    <a href="/google_auth/login?next={{next}}" class="btn btn-lg btn-info btn-block">Google</a>
+                    <a href="/oauth/google?next={{next}}" class="btn btn-lg btn-info btn-block">Google</a>
                 </div>
             </div>
 

rd_ui/app/scripts/visualizations/base.js

@@ -68,7 +68,6 @@
       template: '<filters></filters>\n' + Visualization.renderVisualizationsTemplate,
       replace: false,
       link: function (scope) {
-
         scope.select2Options = {
           width: '50%'
         };
@@ -103,7 +102,7 @@
           if (filters) {
             scope.filters = filters;
 
-            if (filters.length) {
+            if (filters.length && false) {
               readURL();
 
               // start watching for changes and update URL

rd_ui/app/scripts/visualizations/chart.js

@@ -33,7 +33,7 @@
         $scope.chartSeries = [];
         $scope.chartOptions = {};
 
-        var reloadData = _.throttle(function(data) {
+        var reloadData = function(data) {
           if (!data || ($scope.queryResult && $scope.queryResult.getData()) == null) {
             $scope.chartSeries.splice(0, $scope.chartSeries.length);
           } else {
@@ -49,8 +49,8 @@
               }
               $scope.chartSeries.push(_.extend(s, additional));
             });
-          }
-        }, 500);
+          };
+        };
 
         $scope.$watch('options', function (chartOptions) {
           if (chartOptions) {
@@ -87,6 +87,8 @@
           'Pie': 'pie'
         };
 
+        scope.globalSeriesType = 'column';
+
         scope.stackingOptions = {
           "None": "none",
           "Normal": "normal",
@@ -120,6 +122,13 @@
         var chartOptionsUnwatch = null,
             columnsWatch = null;
 
+        scope.$watch('globalSeriesType', function(type, old) {
+          if (type && old && type !== old && scope.visualization.options.seriesOptions) {
+            _.each(scope.visualization.options.seriesOptions, function(sOptions) {
+              sOptions.type = type;
+            });
+          }
+        });
         scope.$watch('visualization.type', function (visualizationType) {
           if (visualizationType == 'CHART') {
             if (scope.visualization.options.series.stacking === null) {
@@ -135,7 +144,9 @@
 
               // TODO: remove uneeded ones?
               if (scope.visualization.options.seriesOptions == undefined) {
-                scope.visualization.options.seriesOptions = {};
+                scope.visualization.options.seriesOptions = {
+                  type: scope.globalSeriesType
+                };
               };
 
               _.each(scope.series, function(s, i) {
@@ -230,4 +241,4 @@
       }
     }
   });
-}());
+}());

rd_ui/app/views/visualizations/chart_editor.html

@@ -18,6 +18,15 @@
                             class="form-control"></select>
                 </div>
             </div>
+
+            <div class="form-group">
+                <label class="control-label col-sm-2">Series Type</label>
+
+                <div class="col-sm-10">
+                    <select required ng-options="value as key for (key, value) in seriesTypes"
+                            ng-model="globalSeriesType" class="form-control"></select>
+                </div>
+            </div>
         </div>
     </div>
 
@@ -88,4 +97,4 @@
 
         </div>
     </div>
-</div>
+</div>

rd_ui/app/views/visualizations/edit_visualization.html

@@ -24,4 +24,4 @@
     </div>
 
   </form>
-</div>
+</div>

rd_ui/test/unit/test_visualization_renderer.js

@@ -54,7 +54,7 @@ describe('VisualizationRenderer', function() {
   });
 
 
-  describe('URL binding', function() {
+  /*describe('URL binding', function() {
 
     beforeEach(inject(function($rootScope, $compile, $location) {
       spyOn($location, 'search').andCallThrough();
@@ -85,5 +85,5 @@ describe('VisualizationRenderer', function() {
         var searchFilters = angular.fromJson($location.search().filters);
         expect(searchFilters[filters[0].friendlyName]).toEqual('newValue');
     }));
-  });
+  });*/
 });

redash/authentication.py

@@ -6,10 +6,8 @@ import logging
 
 from flask import request, make_response, redirect, url_for
 from flask.ext.login import LoginManager, login_user, current_user
-from flask.ext.googleauth import GoogleAuth, login
-from werkzeug.contrib.fixers import ProxyFix
 
-from redash import models, settings
+from redash import models, settings, google_oauth
 
 login_manager = LoginManager()
 logger = logging.getLogger('authentication')
@@ -57,48 +55,15 @@ class HMACAuthentication(object):
         return decorated
 
 
-def validate_email(email):
-    if not settings.GOOGLE_APPS_DOMAIN:
-        return True
-
-    return email in settings.ALLOWED_EXTERNAL_USERS or email.endswith("@%s" % settings.GOOGLE_APPS_DOMAIN)
-
-
-def create_and_login_user(app, user):
-    if not validate_email(user.email):
-        return
-
-    try:
-        user_object = models.User.get(models.User.email == user.email)
-        if user_object.name != user.name:
-            logger.debug("Updating user name (%r -> %r)", user_object.name, user.name)
-            user_object.name = user.name
-            user_object.save()
-    except models.User.DoesNotExist:
-        logger.debug("Creating user object (%r)", user.name)
-        user_object = models.User.create(name=user.name, email=user.email, groups = models.User.DEFAULT_GROUPS)
-
-    login_user(user_object, remember=True)
-
-login.connect(create_and_login_user)
-
-
 @login_manager.user_loader
 def load_user(user_id):
     return models.User.select().where(models.User.id == user_id).first()
 
 
 def setup_authentication(app):
-    if settings.GOOGLE_OPENID_ENABLED:
-        openid_auth = GoogleAuth(app, url_prefix="/google_auth")
-        # If we don't have a list of external users, we can use Google's federated login, which limits
-        # the domain with which you can sign in.
-        if not settings.ALLOWED_EXTERNAL_USERS and settings.GOOGLE_APPS_DOMAIN:
-            openid_auth._OPENID_ENDPOINT = "https://www.google.com/a/%s/o8/ud?be=o8" % settings.GOOGLE_APPS_DOMAIN
-
     login_manager.init_app(app)
     login_manager.anonymous_user = models.AnonymousUser
-    app.wsgi_app = ProxyFix(app.wsgi_app)
     app.secret_key = settings.COOKIE_SECRET
+    app.register_blueprint(google_oauth.blueprint)
 
     return HMACAuthentication()

redash/controllers.py

@@ -69,8 +69,7 @@ def login():
         return redirect(request.args.get('next') or '/')
 
     if not settings.PASSWORD_LOGIN_ENABLED:
-        blueprint = app.extensions['googleauth'].blueprint
-        return redirect(url_for("%s.login" % blueprint.name, next=request.args.get('next')))
+        return redirect(url_for("google_oauth.authorize", next=request.args.get('next')))
 
     if request.method == 'POST':
         user = models.User.select().where(models.User.email == request.form['username']).first()
@@ -84,7 +83,7 @@ def login():
                            analytics=settings.ANALYTICS,
                            next=request.args.get('next'),
                            username=request.form.get('username', ''),
-                           show_google_openid=settings.GOOGLE_OPENID_ENABLED)
+                           show_google_openid=settings.GOOGLE_OAUTH_ENABLED)
 
 
 @app.route('/logout')
@@ -111,7 +110,6 @@ def status_api():
 
     manager_status = redis_connection.hgetall('redash:status')
     status['manager'] = manager_status
-    status['manager']['queue_size'] = redis_connection.llen('queries') + redis_connection.llen('scheduled_queries')
     status['manager']['outdated_queries_count'] = models.Query.outdated_queries().count()
 
     queues = {}
@@ -414,48 +412,52 @@ class QueryResultListAPI(BaseResource):
 
 
 class QueryResultAPI(BaseResource):
-    @require_permission('view_query')
-    def get(self, query_result_id):
-        query_result = models.QueryResult.get_by_id(query_result_id)
-        if query_result:
-            data = json.dumps({'query_result': query_result.to_dict()}, cls=utils.JSONEncoder)
-            return make_response(data, 200, cache_headers)
-        else:
-            abort(404)
+    @staticmethod
+    def csv_response(query_result):
+        s = cStringIO.StringIO()
 
+        query_data = json.loads(query_result.data)
+        writer = csv.DictWriter(s, fieldnames=[col['name'] for col in query_data['columns']])
+        writer.writer = utils.UnicodeWriter(s)
+        writer.writeheader()
+        for row in query_data['rows']:
+            for k, v in row.iteritems():
+                if isinstance(v, numbers.Number) and (v > 1000 * 1000 * 1000 * 100):
+                    row[k] = datetime.datetime.fromtimestamp(v/1000.0)
+
+            writer.writerow(row)
+
+        headers = {'Content-Type': "text/csv; charset=UTF-8"}
+        headers.update(cache_headers)
+        return make_response(s.getvalue(), 200, headers)
 
-class CsvQueryResultsAPI(BaseResource):
     @require_permission('view_query')
-    def get(self, query_id, query_result_id=None):
-        if not query_result_id:
+    def get(self, query_id=None, query_result_id=None, filetype='json'):
+        if query_result_id is None and query_id is not None:
             query = models.Query.get(models.Query.id == query_id)
             if query:
                 query_result_id = query._data['latest_query_data']
 
-        query_result = query_result_id and models.QueryResult.get_by_id(query_result_id)
+        if query_result_id:
+            query_result = models.QueryResult.get_by_id(query_result_id)
+
         if query_result:
-            s = cStringIO.StringIO()
+            if filetype == 'json':
+                data = json.dumps({'query_result': query_result.to_dict()}, cls=utils.JSONEncoder)
+                return make_response(data, 200, cache_headers)
+            else:
+                return self.csv_response(query_result)
 
-            query_data = json.loads(query_result.data)
-            writer = csv.DictWriter(s, fieldnames=[col['name'] for col in query_data['columns']])
-            writer.writer = utils.UnicodeWriter(s)
-            writer.writeheader()
-            for row in query_data['rows']:
-                for k, v in row.iteritems():
-                    if isinstance(v, numbers.Number) and (v > 1000 * 1000 * 1000 * 100):
-                        row[k] = datetime.datetime.fromtimestamp(v/1000.0)
-
-                writer.writerow(row)
-
-            return make_response(s.getvalue(), 200, {'Content-Type': "text/csv; charset=UTF-8"})
         else:
             abort(404)
 
-api.add_resource(CsvQueryResultsAPI, '/api/queries/<query_id>/results/<query_result_id>.csv',
-                 '/api/queries/<query_id>/results.csv',
-                 endpoint='csv_query_results')
+
 api.add_resource(QueryResultListAPI, '/api/query_results', endpoint='query_results')
-api.add_resource(QueryResultAPI, '/api/query_results/<query_result_id>', endpoint='query_result')
+api.add_resource(QueryResultAPI,
+                 '/api/query_results/<query_result_id>',
+                 '/api/queries/<query_id>/results.<filetype>',
+                 '/api/queries/<query_id>/results/<query_result_id>.<filetype>',
+                 endpoint='query_result')
 
 
 class JobAPI(BaseResource):

redash/data/query_runner_mysql.py

@@ -18,7 +18,7 @@ def mysql(connection_string):
     
     def query_runner(query):
         connections_params = [entry.split('=')[1] for entry in connection_string.split(';')]
-        connection = MySQLdb.connect(*connections_params)
+        connection = MySQLdb.connect(*connections_params, charset="utf8", use_unicode=True)
         cursor = connection.cursor()
 
         logging.debug("mysql got query: %s", query)
@@ -61,4 +61,4 @@ def mysql(connection_string):
         return json_data, error
         
     
-    return query_runner
+    return query_runner

redash/google_oauth.py

@@ -0,0 +1,81 @@
+import logging
+from flask.ext.login import login_user
+import requests
+from flask import redirect, url_for, Blueprint
+from flask_oauth import OAuth
+from redash import models, settings
+
+logger = logging.getLogger('google_oauth')
+oauth = OAuth()
+
+request_token_params = {'scope': 'https://www.googleapis.com/auth/userinfo.email', 'response_type': 'code'}
+
+if settings.GOOGLE_APPS_DOMAIN:
+    request_token_params['hd'] = settings.GOOGLE_APPS_DOMAIN
+else:
+    logger.warning("No Google Apps domain defined, all Google accounts allowed.")
+
+google = oauth.remote_app('google',
+                          base_url='https://www.google.com/accounts/',
+                          authorize_url='https://accounts.google.com/o/oauth2/auth',
+                          request_token_url=None,
+                          request_token_params=request_token_params,
+                          access_token_url='https://accounts.google.com/o/oauth2/token',
+                          access_token_method='POST',
+                          access_token_params={'grant_type': 'authorization_code'},
+                          consumer_key=settings.GOOGLE_CLIENT_ID,
+                          consumer_secret=settings.GOOGLE_CLIENT_SECRET)
+
+
+blueprint = Blueprint('google_oauth', __name__)
+
+
+def get_user_profile(access_token):
+    headers = {'Authorization': 'OAuth '+access_token}
+    response = requests.get('https://www.googleapis.com/oauth2/v1/userinfo', headers=headers)
+
+    if response.status_code == 401:
+        logger.warning("Failed getting user profile (response code 401).")
+        return None
+
+    return response.json()
+
+
+def create_and_login_user(name, email):
+    try:
+        user_object = models.User.get(models.User.email == email)
+        if user_object.name != name:
+            logger.debug("Updating user name (%r -> %r)", user_object.name, name)
+            user_object.name = name
+            user_object.save()
+    except models.User.DoesNotExist:
+        logger.debug("Creating user object (%r)", name)
+        user_object = models.User.create(name=name, email=email, groups=models.User.DEFAULT_GROUPS)
+
+    login_user(user_object, remember=True)
+
+
+@blueprint.route('/oauth/google', endpoint="authorize")
+def login():
+    # TODO, suport next
+    callback=url_for('.callback', _external=True)
+    logger.debug("Callback url: %s", callback)
+    return google.authorize(callback=callback)
+
+
+@blueprint.route('/oauth/google_callback', endpoint="callback")
+@google.authorized_handler
+def authorized(resp):
+    access_token = resp['access_token']
+
+    if access_token is None:
+        logger.warning("Access token missing in call back request.")
+        return redirect(url_for('login'))
+
+    profile = get_user_profile(access_token)
+    if profile is None:
+        return redirect(url_for('login'))
+
+    create_and_login_user(profile['name'], profile['email'])
+
+    return redirect(url_for('index'))

redash/settings.py

@@ -59,12 +59,15 @@ CELERY_FLOWER_URL = os.environ.get("REDASH_CELERY_FLOWER_URL", "/flower")
 # Google Apps domain to allow access from; any user with email in this Google Apps will be allowed
 # access
 GOOGLE_APPS_DOMAIN = os.environ.get("REDASH_GOOGLE_APPS_DOMAIN", "")
-GOOGLE_OPENID_ENABLED = parse_boolean(os.environ.get("REDASH_GOOGLE_OPENID_ENABLED", "true"))
-PASSWORD_LOGIN_ENABLED = parse_boolean(os.environ.get("REDASH_PASSWORD_LOGIN_ENABLED", "false"))
-ALLOWED_EXTERNAL_USERS = array_from_string(os.environ.get("REDASH_ALLOWED_EXTERNAL_USERS", ''))
+
+GOOGLE_CLIENT_ID = os.environ.get("REDASH_GOOGLE_CLIENT_ID", "")
+GOOGLE_CLIENT_SECRET = os.environ.get("REDASH_GOOGLE_CLIENT_SECRET", "")
+GOOGLE_OAUTH_ENABLED = GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET
+
+PASSWORD_LOGIN_ENABLED = parse_boolean(os.environ.get("REDASH_PASSWORD_LOGIN_ENABLED", "true"))
 STATIC_ASSETS_PATH = fix_assets_path(os.environ.get("REDASH_STATIC_ASSETS_PATH", "../rd_ui/app/"))
 WORKERS_COUNT = int(os.environ.get("REDASH_WORKERS_COUNT", "2"))
-JOB_EXPIRY_TIME = int(os.environ.get("REDASH_JOB_EXPIRY_TIME", 3600*24))
+JOB_EXPIRY_TIME = int(os.environ.get("REDASH_JOB_EXPIRY_TIME", 3600*6))
 COOKIE_SECRET = os.environ.get("REDASH_COOKIE_SECRET", "c292a0a3aa32397cdb050e233733900f")
 LOG_LEVEL = os.environ.get("REDASH_LOG_LEVEL", "INFO")
 EVENTS_LOG_PATH = os.environ.get("REDASH_EVENTS_LOG_PATH", "")
@@ -73,4 +76,4 @@ CLIENT_SIDE_METRICS = parse_boolean(os.environ.get("REDASH_CLIENT_SIDE_METRICS",
 ANALYTICS = os.environ.get("REDASH_ANALYTICS", "")
 
 # Features:
-FEATURE_TABLES_PERMISSIONS = parse_boolean(os.environ.get("REDASH_FEATURE_TABLES_PERMISSIONS", "false"))
+FEATURE_TABLES_PERMISSIONS = parse_boolean(os.environ.get("REDASH_FEATURE_TABLES_PERMISSIONS", "false"))

redash/tasks.py

@@ -5,7 +5,7 @@ import redis
 from celery import Task
 from celery.result import AsyncResult
 from celery.utils.log import get_task_logger
-from redash import redis_connection, models, statsd_client
+from redash import redis_connection, models, statsd_client, settings
 from redash.utils import gen_query_hash
 from redash.worker import celery
 from redash.data.query_runner import get_query_runner
@@ -64,7 +64,12 @@ class QueryTask(object):
                     logging.info("[Manager][%s] Found existing job: %s", query_hash, job_id)
 
                     job = cls(job_id=job_id)
-                else:
+                    if job.ready():
+                        logging.info("[%s] job found is ready (%s), removing lock", query_hash, job.celery_status)
+                        redis_connection.delete(QueryTask._job_lock_id(query_hash, data_source.id))
+                        job = None
+
+                if not job:
                     pipe.multi()
 
                     if scheduled:
@@ -75,7 +80,7 @@ class QueryTask(object):
                     result = execute_query.apply_async(args=(query, data_source.id), queue=queue_name)
                     job = cls(async_result=result)
                     logging.info("[Manager][%s] Created new job: %s", query_hash, job.id)
-                    pipe.set(cls._job_lock_id(query_hash, data_source.id), job.id)
+                    pipe.set(cls._job_lock_id(query_hash, data_source.id), job.id, settings.JOB_EXPIRY_TIME)
                     pipe.execute()
                 break
 
@@ -113,6 +118,17 @@ class QueryTask(object):
             'query_result_id': query_result_id,
         }
 
+    @property
+    def is_cancelled(self):
+        return self._async_result.status == 'REVOKED'
+
+    @property
+    def celery_status(self):
+        return self._async_result.status
+
+    def ready(self):
+        return self._async_result.ready()
+
     def cancel(self):
         return self._async_result.revoke(terminate=True)
 
@@ -151,6 +167,41 @@ def refresh_queries():
     statsd_client.gauge('manager.seconds_since_refresh', now - float(status.get('last_refresh_at', now)))
 
 
+@celery.task(base=BaseTask)
+def cleanup_tasks():
+    # in case of cold restart of the workers, there might be jobs that still have their "lock" object, but aren't really
+    # going to run. this job removes them.
+
+    lock_keys = redis_connection.keys("query_hash_job:*") # TODO: use set instead of keys command
+    query_tasks = [QueryTask(job_id=j) for j in redis_connection.mget(lock_keys)]
+
+    logger.info("Found %d locks", len(query_tasks))
+
+    inspect = celery.control.inspect()
+    active_tasks = inspect.active()
+    if active_tasks is None:
+        active_tasks = []
+    else:
+        active_tasks = active_tasks.values()
+
+    all_tasks = set()
+    for task_list in active_tasks:
+        for task in task_list:
+            all_tasks.add(task['id'])
+
+    logger.info("Active jobs count: %d", len(all_tasks))
+
+    for i, t in enumerate(query_tasks):
+        if t.ready():
+            # if locked task is ready already (failed, finished, revoked), we don't need the lock anymore
+            logger.warning("%s is ready (%s), removing lock.", lock_keys[i], t.celery_status)
+            redis_connection.delete(lock_keys[i])
+
+        if t.celery_status == 'STARTED' and t.id not in all_tasks:
+            logger.warning("Couldn't find active job for: %s, removing lock.", lock_keys[i])
+            redis_connection.delete(lock_keys[i])
+
+
 @celery.task(bind=True, base=BaseTask, track_started=True)
 def execute_query(self, query, data_source_id):
     # TODO: maybe this should be a class?

redash/worker.py

@@ -13,6 +13,10 @@ celery.conf.update(CELERY_RESULT_BACKEND=settings.CELERY_BACKEND,
                            'task': 'redash.tasks.refresh_queries',
                            'schedule': timedelta(seconds=30)
                        },
+                       'cleanup_tasks': {
+                           'task': 'redash.tasks.cleanup_tasks',
+                           'schedule': timedelta(minutes=5)
+                       },
                    },
                    CELERY_TIMEZONE='UTC')
 

requirements.txt

@@ -1,7 +1,7 @@
 Flask==0.10.1
-Flask-GoogleAuth==0.4
 Flask-RESTful==0.2.10
 Flask-Login==0.2.9
+Flask-OAuth==0.12
 passlib==1.6.2
 Jinja2==2.7.2
 MarkupSafe==0.18
@@ -10,7 +10,7 @@ aniso8601==0.82
 blinker==1.3
 itsdangerous==0.23
 peewee==2.2.2
-psycopg2==2.5.1
+psycopg2==2.5.2
 python-dateutil==2.1
 pytz==2013.9
 redis==2.7.5

tests/test_authentication.py

@@ -1,52 +1,25 @@
-from unittest import TestCase
 from mock import patch
-from flask_googleauth import ObjectDict
 from tests import BaseTestCase
-from redash.authentication import validate_email, create_and_login_user
-from redash import settings, models
+from redash import models
+from redash.google_oauth import create_and_login_user
 from tests.factories import user_factory
 
 
-class TestEmailValidation(TestCase):
-    def test_accepts_address_with_correct_domain(self):
-        with patch.object(settings, 'GOOGLE_APPS_DOMAIN', 'example.com'):
-            self.assertTrue(validate_email('example@example.com'))
-
-    def test_accepts_address_from_exception_list(self):
-        with patch.multiple(settings, GOOGLE_APPS_DOMAIN='example.com', ALLOWED_EXTERNAL_USERS=['whatever@whatever.com']):
-            self.assertTrue(validate_email('whatever@whatever.com'))
-
-    def test_accept_any_address_when_domain_empty(self):
-        with patch.object(settings, 'GOOGLE_APPS_DOMAIN', None):
-            self.assertTrue(validate_email('whatever@whatever.com'))
-
-    def test_rejects_address_with_incorrect_domain(self):
-        with patch.object(settings, 'GOOGLE_APPS_DOMAIN', 'example.com'):
-            self.assertFalse(validate_email('whatever@whatever.com'))
-
-
 class TestCreateAndLoginUser(BaseTestCase):
     def test_logins_valid_user(self):
         user = user_factory.create(email='test@example.com')
 
-        with patch.object(settings, 'GOOGLE_APPS_DOMAIN', 'example.com'), patch('redash.authentication.login_user') as login_user_mock:
-            create_and_login_user(None, user)
+        with patch('redash.google_oauth.login_user') as login_user_mock:
+            create_and_login_user(user.name, user.email)
             login_user_mock.assert_called_once_with(user, remember=True)
 
     def test_creates_vaild_new_user(self):
-        openid_user = ObjectDict({'email': 'test@example.com', 'name': 'Test User'})
+        email = 'test@example.com'
+        name = 'Test User'
 
-        with patch.multiple(settings, GOOGLE_APPS_DOMAIN='example.com'), \
-             patch('redash.authentication.login_user') as login_user_mock:
+        with patch('redash.google_oauth.login_user') as login_user_mock:
 
-            create_and_login_user(None, openid_user)
+            create_and_login_user(name, email)
 
             self.assertTrue(login_user_mock.called)
-            user = models.User.get(models.User.email == openid_user.email)
-
-    def test_ignores_invliad_user(self):
-        user = ObjectDict({'email': 'test@whatever.com'})
-
-        with patch.object(settings, 'GOOGLE_APPS_DOMAIN', 'example.com'), patch('redash.authentication.login_user') as login_user_mock:
-            create_and_login_user(None, user)
-            self.assertFalse(login_user_mock.called)
+            user = models.User.get(models.User.email == email)

tests/test_controllers.py

@@ -380,7 +380,7 @@ class TestLogin(BaseTestCase):
         with app.test_client() as c, patch.object(settings, 'PASSWORD_LOGIN_ENABLED', False):
             rv = c.get('/login')
             self.assertEquals(rv.status_code, 302)
-            self.assertTrue(rv.location.endswith(url_for('GoogleAuth.login')))
+            self.assertTrue(rv.location.endswith(url_for('google_oauth.authorize')))
 
     def test_get_login_form(self):
         with app.test_client() as c: