Merge pull request #1167 from ariarijp/fix-bootstrap-script-for-ubuntu

Fix the version of setuptools on bootstrap script for Ubuntu

.bowerrc

@@ -0,0 +1,3 @@
+{
+    "directory": "rd_ui/app/bower_components"
+}

.coveragerc

@@ -0,0 +1,5 @@
+[report]
+omit =
+    */settings.py
+    */python?.?/*
+    */site-packages/nose/*

.dockerignore

@@ -0,0 +1,4 @@
+rd_ui/.tmp/
+rd_ui/node_modules/
+.git/
+.vagrant/

.env.example

@@ -0,0 +1,5 @@
+REDASH_STATIC_ASSETS_PATH="../rd_ui/app/"
+REDASH_LOG_LEVEL="INFO"
+REDASH_REDIS_URL=redis://localhost:6379/1
+REDASH_DATABASE_URL="postgresql://redash"
+REDASH_COOKIE_SECRET=veryverysecret

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,24 @@
+Welcome to Redash's GitHub repo! 👋🎉
+
+Do you need help or have a question? Checkout the Support category in our discussion forum: https://discuss.redash.io/c/support. 
+
+Got an idea for a new feature? Check if it isn't on the roadmap already: http://bit.ly/redash-roadmap and start a new discussion in the features category: https://discuss.redash.io/c/feature-requests 🌟.
+
+Found a bug? Please fill out the sections below... thank you 👍
+
+### Issue Summary
+
+A summary of the issue and the browser/OS environment in which it occurs.
+
+### Steps to Reproduce
+
+1. This is the first step
+2. This is the second step, etc.
+
+Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?
+
+### Technical details:
+
+* Redash Version:
+* Browser/OS:
+* How did you install Redash:

.gitignore

@@ -1,10 +1,29 @@
+.coveralls.yml
 .idea
 *.pyc
-rd_service/settings.py
+.coverage
 rd_ui/dist
 .DS_Store
+celerybeat-schedule*
+.#*
+\#*#
+*~
+_build
 
 # Vagrant related
 .vagrant
 Berksfile.lock
-rd_service/dump.rdb
+redash/dump.rdb
+.env
+.ruby-version
+venv
+
+dump.rdb
+
+# Docker related
+docker-compose.yml
+
+node_modules
+.tmp
+.sass-cache
+rd_ui/app/bower_components

.landscape.yaml

@@ -0,0 +1,2 @@
+ignore-paths:
+    - migrations

Berksfile

@@ -1,3 +0,0 @@
-cookbook 'apt'
-cookbook 'postgresql'
-cookbook 'redash', git: 'git@github.com:EverythingMe/chef-redash.git'

CONTRIBUTING.md

@@ -0,0 +1,79 @@
+# Contributing Guide
+
+Thank you for taking the time to contribute! :tada::+1:
+
+The following is a set of guidelines for contributing to Redash. These are guidelines, not rules, please use your best judgement and feel free to propose changes to this document in a pull request.
+
+## Quick Links:
+
+- [Feature Roadmap](https://trello.com/b/b2LUHU7A/re-dash-roadmap)
+- [Feature Requests](https://discuss.redash.io/c/feature-requests)
+- [Gitter Chat](https://gitter.im/getredash/redash) or [Slack](https://slack.redash.io)
+- [Documentation](http://docs.redash.io)
+- [Blog](http://blog.redash.io/)
+- [Twitter](https://twitter.com/getredash)
+
+---
+:star: If you already here and love the project, please make sure to press the Star button. :star:
+
+---
+
+ 
+## Table of Contents
+
+[How can I contribute?](#how-can-i-contribute)
+
+- [Reporting Bugs](#reporting-bugs)
+- [Suggesting Enhancements / Feature Requests](#suggesting-enhancements--feature-requests)
+- [Pull Requests](#pull-requests)
+- [Documentation](#documentation)
+- Design?
+ 
+[Addtional Notes](#additional-notes)
+
+- [Release Method](#release-method)
+- [Code of Conduct](#code-of-conduct)
+ 
+## How can I contribute?
+
+### Reporting Bugs
+
+When creating a new bug report, please make sure to:
+
+- Search for existing issues first. If you find a previous report of your issue, please update the existing issue with additional information instead of creating a new one.
+- If you are not sure if your issue is really a bug or just some configuration/setup problem, please start a discussion in [the support forum](https://discuss.redash.io/c/support) first. Unless you can provide clear steps to reproduce, it's probably better to start with a thread in the forum and later to open an issue.
+- If you still decide to open an issue, please review the template and guidelines and include as much details as possible.
+ 
+### Suggesting Enhancements / Feature Requests
+
+If you would like to suggest an enchancement or ask for a new feature:
+
+- Please check [the roadmap](https://trello.com/b/b2LUHU7A/re-dash-roadmap) for existing Trello card for what you want to suggest/ask. If there is, feel free to upvote it to signal interest or add your comments.
+- If there is no existing card, open a thread in [the forum](https://discuss.redash.io/c/feature-requests) to start a discussion about what you want to suggest. Try to provide as much details and context as possible and include information about *the problem you want to solve* rather only *your proposed solution*.
+ 
+### Pull Requests
+
+- **Code contributions are welcomed**. For big changes or significant features, it's usually better to reach out first and discuss what you want to implement and how (we recommend reading: [Pull Request First](https://medium.com/practical-blend/pull-request-first-f6bb667a9b6#.ozlqxvj36)). This to make sure that what you want to implement is aligned with our goals for the project and that no one else is already working on it.
+- Include screenshots and animated GIFs in your pull request whenever possible.
+- Please add [documentation](#documentation) for new features or changes in functionality along with the code.
+- Please follow existing code style. We use PEP8 for Python and sensible style for Javascript.
+ 
+### Documentation
+
+The project's documentation can be found at [docs.redash.io](http://docs.redash.io/). The [documentation sources](https://github.com/getredash/redash/tree/master/docs) are managed along with the code and to contribute edits / new pages, you can use GitHub's interface. Click the "Edit on GitHub" link on the documentation page to quickly open the edit interface.
+
+The pages are written in *reStructuredText* format, which is very similar to Markdown.
+
+## Additional Notes
+
+### Release Method
+
+We publish a stable release every ~2 months, although the goal is to get to a stable release every month. You can see the change log on [GitHub releases page](http://github.com/getredash/redash/releases).
+
+Every build of the master branch updates the latest *RC release*. These releases are usually stable, but might contain regressions and therefore recommended for "advanced users" only.
+
+When we release a new stable release, we also update the *latest* Docker image tag, the EC2 AMIs and GCE images.
+
+## Code of Conduct
+
+This project adheres to the Contributor Covenant [code of conduct](http://redash.io/community/code_of_conduct). By participating, you are expected to uphold this code. Please report unacceptable behavior to team@redash.io.

Dockerfile

@@ -0,0 +1,53 @@
+FROM ubuntu:trusty
+
+# Ubuntu packages
+RUN apt-get update && \
+  apt-get install -y python-pip python-dev curl build-essential pwgen libffi-dev sudo git-core wget \
+  # Postgres client
+  libpq-dev \
+  # Additional packages required for data sources:
+  libssl-dev libmysqlclient-dev freetds-dev libsasl2-dev && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/*
+
+# Users creation
+RUN useradd --system --comment " " --create-home redash
+
+# Pip requirements for all data source types
+RUN pip install -U setuptools==23.1.0 && \
+  pip install supervisor==3.1.2
+
+COPY . /opt/redash/current
+RUN chown -R redash /opt/redash/current
+
+# Setting working directory
+WORKDIR /opt/redash/current
+
+ENV REDASH_STATIC_ASSETS_PATH="../rd_ui/dist/"
+
+# Install project specific dependencies
+RUN pip install -r requirements_all_ds.txt && \
+  pip install -r requirements.txt
+
+RUN curl https://deb.nodesource.com/setup_4.x | bash - && \
+  apt-get install -y nodejs && \
+  sudo -u redash -H make deps && \
+  rm -rf node_modules rd_ui/node_modules /home/redash/.npm /home/redash/.cache && \
+  apt-get purge -y nodejs && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/*
+
+# Setup supervisord
+RUN mkdir -p /opt/redash/supervisord && \
+    mkdir -p /opt/redash/logs && \
+    cp /opt/redash/current/setup/docker/supervisord/supervisord.conf /opt/redash/supervisord/supervisord.conf
+
+# Fix permissions
+RUN chown -R redash /opt/redash
+
+# Expose ports
+EXPOSE 5000
+EXPOSE 9001
+
+# Startup script
+CMD ["supervisord", "-c", "/opt/redash/supervisord/supervisord.conf"]

LICENSE

@@ -1,4 +1,5 @@
-Copyright 2013 DoAT. All rights reserved.
+Copyright (c) 2013-2016, Arik Fraimovich. 
+All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification, 
 are permitted provided that the following conditions are met:
@@ -10,17 +11,13 @@ are permitted provided that the following conditions are met:
    this list of conditions and the following disclaimer in the documentation and/or
    other materials provided with the distribution.
 
-THIS SOFTWARE IS PROVIDED “AS IS” WITHOUT ANY WARRANTIES WHATSOEVER. 
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, 
-THE IMPLIED WARRANTIES OF NON INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A 
-PARTICULAR PURPOSE ARE HEREBY DISCLAIMED. IN NO EVENT SHALL DoAT OR CONTRIBUTORS 
-BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, 
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 
+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS 
+BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 
 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 
 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 
 NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, 
 EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-The views and conclusions contained in the software and documentation are those of 
-the authors and should not be interpreted as representing official policies, 
-either expressed or implied, of DoAT.

Makefile

@@ -1,16 +1,22 @@
 NAME=redash
-VERSION=0.2
-FULL_VERSION=$(VERSION).$(CIRCLE_BUILD_NUM)
-FILENAME=$(CIRCLE_ARTIFACTS)/$(NAME).$(FULL_VERSION).tar.gz
+VERSION=`python ./manage.py version`
+FULL_VERSION=$(VERSION)+b$(CIRCLE_BUILD_NUM)
+BASE_VERSION=$(shell python ./manage.py version | cut -d + -f 1)
+# VERSION gets evaluated every time it's referenced, therefore we need to use VERSION here instead of FULL_VERSION.
+FILENAME=$(CIRCLE_ARTIFACTS)/$(NAME).$(VERSION).tar.gz
 
 deps:
-	cd rd_ui && npm install
-	cd rd_ui && npm install grunt-cli bower
-	cd rd_ui && bower install
-	cd rd_ui && grunt build
+	if [ -d "./rd_ui/app" ]; then npm install; fi
+	if [ -d "./rd_ui/app" ]; then npm run bower install; fi
+	if [ -d "./rd_ui/app" ]; then npm run build; fi
 
 pack:
-	tar -zcv -f $(FILENAME) --exclude=".git*" --exclude="*.pyc" --exclude="*.pyo" --exclude="rd_ui/node_modules" --exclude="rd_ui/dist/bower_components" --exclude="rd_ui/app" *
+	sed -ri "s/^__version__ = '([0-9.]*)'/__version__ = '$(FULL_VERSION)'/" redash/__init__.py
+	tar -zcv -f $(FILENAME) --exclude="optipng*" --exclude=".git*" --exclude="*.pyc" --exclude="*.pyo" --exclude="venv" --exclude="node_modules" --exclude="rd_ui/dist/bower_components" --exclude="rd_ui/app" *
 
 upload:
-	python bin/upload_version.py $(FULL_VERSION) $(FILENAME)
+	python bin/release_manager.py $(CIRCLE_SHA1) $(BASE_VERSION) $(FILENAME)
+
+test:
+	nosetests --with-coverage --cover-package=redash tests/
+	#grunt test

Procfile.dev

@@ -0,0 +1,2 @@
+web: ./manage.py runserver -p $PORT --host 0.0.0.0
+worker: ./bin/run celery worker --app=redash.worker --beat -Qqueries,celery,scheduled_queries

Procfile.heroku

@@ -0,0 +1,2 @@
+web: ./manage.py runserver -d -r -p $PORT --host 0.0.0.0
+worker: celery worker --app=redash.worker -c2 --beat -Q queries,celery,scheduled_queries

README.md

@@ -1,118 +1,56 @@
-# [_re:dash_](https://github.com/everythingme/redash)
+More details about the future of re:dash : http://bit.ly/journey-first-step
+
+---
+
+<p align="center">
+  <img title="re:dash" src='http://redash.io/static/old_img/redash_logo.png' width="200px"/>
+</p>
+<p align="center">
+    <img title="Build Status" src='https://circleci.com/gh/getredash/redash.png?circle-token=8a695aa5ec2cbfa89b48c275aea298318016f040'/>
+</p>
+
+[![Join the chat at https://gitter.im/getredash/redash](https://badges.gitter.im/getredash/redash.svg)](https://gitter.im/getredash/redash?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![Documentation](https://img.shields.io/badge/docs-redash.io-brightgreen.svg)](http://docs.redash.io)
 
 **_re:dash_** is our take on freeing the data within our company in a way that will better fit our culture and usage patterns.
 
-Prior to **_re:dash_**, we tried to use tranditional BI suites and discovered a set of bloated, technically challenged and slow tools/flows. What we were looking for was a more hacker'ish way to look at data, so we built one.
+Prior to **_re:dash_**, we tried to use traditional BI suites and discovered a set of bloated, technically challenged and slow tools/flows. What we were looking for was a more hacker'ish way to look at data, so we built one.
 
 **_re:dash_** was built to allow fast and easy access to billions of records, that we process and collect using Amazon Redshift ("petabyte scale data warehouse" that "speaks" PostgreSQL).
+Today **_re:dash_** has support for querying multiple databases, including: Redshift, Google BigQuery, PostgreSQL, MySQL, Graphite,
+Presto, Google Spreadsheets, Cloudera Impala, Hive and custom scripts.
 
 **_re:dash_** consists of two parts:
 
-1. **Query Editor**: think of [JS Fiddle](http://jsfiddle.net) for SQL queries. It's your way to share data in the organization in an open way, by sharing both the dataset and the query that generated it. This way everyone can peer review not only the resulting dataset but also the process that generated it. Also it's possible to fork it and generate new datasets and reach new insights. 
-2. **Dashboards/Visualizations**: once you have a dataset, you can create different visualizations out of it, and then combine several visualizations into a single dashboard. Currently it supports bar charts, pivot table and cohorts.
+1. **Query Editor**: think of [JS Fiddle](http://jsfiddle.net) for SQL queries. It's your way to share data in the organization in an open way, by sharing both the dataset and the query that generated it. This way everyone can peer review not only the resulting dataset but also the process that generated it. Also it's possible to fork it and generate new datasets and reach new insights.
+2. **Dashboards/Visualizations**: once you have a dataset, you can create different visualizations out of it, and then combine several visualizations into a single dashboard. Currently it supports charts, pivot table and cohorts.
 
-This is the first release, which is more than usable but still has its rough edges and way to go to fulfill its full potential. The Query Editor part is quite solid, but the visualizations need more work to enrich them and to make them more user friendly.
+**_re:dash_** is a work in progress and has its rough edges and way to go to fulfill its full potential. The Query Editor part is quite solid, but the visualizations need more work to enrich them and to make them more user friendly.
 
 ## Demo
 
-![Screenshots](https://raw.github.com/EverythingMe/redash/screenshots/screenshots.gif)
+<img src="https://cloud.githubusercontent.com/assets/71468/12611424/1faf4d6a-c4f5-11e5-89b5-31efc1155d2c.gif" width="60%"/>
 
-You can try out the demo instance: http://rd-demo.herokuapp.com/ (login with any Google account).
-
-Due to Heroku dev plan limits, it has a small database of flights (see schema [here](http://rd-demo.herokuapp.com/dashboard/schema)). Also due to another Heroku limitation, it is running with the regular user, hence you can DELETE or INSERT data/tables. Please be nice and don't do this.
-
-## Getting help
-
-* [Google Group (mailing list)](https://groups.google.com/forum/#!forum/redash-users): the best place to get updates about new releases or ask general questions.
-* #redash IRC channel on [Freenode](http://www.freenode.net/).
-
-## Technology
-
-* [AngularJS](http://angularjs.org/)
-* [Tornado](http://tornadoweb.org)
-* [PostgreSQL](http://www.postgresql.org/) / [AWS Redshift](http://aws.amazon.com/redshift/)
-* [Redis](http://redis.io)
-
-PostgreSQL is used both as the operatinal database for the system, but also as the data store that is being queried. To be exact, we built this system to use on top of Amazon's Redshift, which supports the PG driver. But it's quite simple to add support for other datastores, and we do plan to do so.
-
-This is our first large scale AngularJS project, and we learned a lot during the development of it. There are still things we need to iron out, and comments on the way we use AngularJS are more than welcome (and pull requests just as well).
-
-### HighCharts
-
-HighCharts is really great, but it's not free for commercial use. Please refer to their [licensing options](http://shop.highsoft.com/highcharts.html), to see what applies for your use. 
-
-It's very likely that in the future we will switch to [D3.js](http://d3js.org/) instead.
+You can try out the demo instance: http://demo.redash.io/ (login with any Google account).
 
 ## Getting Started
 
-1. Clone the repo:
-```bash
-git clone git@github.com:EverythingMe/redash.git
-```
-2. Create settings file from the example one (& update relevant settings):
-```bash
-cp rd_service/settings_example.py rd_service/settings.py
-```
-It's highly recommended that the user you use to connect to the data database (the one you query) is read-only.
-3. Create the operational databases from rd_service/data/tables.sql
-3. Install `npm` packages (mainly: Bower & Grunt):
-```bash
-cd rd_ui
-npm install
-```
-4. Install `bower` packages:
-```bash
-bower install
-```
-5. Build the UI:
-```bash
-grunt build
-```
-6. Install PIP packages:
-```bash
-pip install -r ../rd_service/requirements.txt
-```
-6. Start the API server:
-```bash
-cd ../rd_service
-python server.py
-```
-7. Start the workers:
-```bash
-python cli.py worker
-```
-8. Open `http://localhost:8888/` and query away.
+* [Setting up re:dash instance](http://redash.io/deployment/setup.html) (includes links to ready made AWS/GCE images).
+* [Documentation](http://docs.redash.io).
 
-## Roadmap
 
-We plan to release new minor version every 2-3 weeks. Of course, if we get additional help from contributors it will help speed things up.
+## Getting Help
 
-Below you can see the "big" features of the next 3 releases (for full list, click on the link):
-
-### [v0.2](https://github.com/EverythingMe/redash/issues?milestone=1&state=open)
-
-- Ability to generate multiple visualizations for a single query (dataset) in a more flexible way than today. Also easier extensbility points to add additional visualizations.
-- Dashboard filters: ability to filter/slice the data you see in a single dashboard using filters (date or selectors).
-- UI Improvements (better notifications & flows, improved queries page) 
-- Comments on queries.
-
-### [v0.3](https://github.com/EverythingMe/redash/issues?milestone=2&state=open)
-
-- Support for API access using API keys, instead of Google Login.
-- Multiple databases support (including other database type than PostgreSQL).
-- Scheduled reports by email.
-
-### [v0.4](https://github.com/EverythingMe/redash/issues?milestone=3&state=open)
-
-- Query versioning. 
-- More "realtime" UI (using websockets).
-- More visualizations.
+* Issues: https://github.com/getredash/redash/issues
+* Discussion Forum: https://discuss.redash.io/
+* Slack: http://slack.redash.io/
+* Gitter (chat): https://gitter.im/getredash/redash
 
 ## Reporting Bugs and Contributing Code
 
-* Want to report a bug or request a feature? Please open [an issue](https://github.com/everythingme/redash/issues/new).
-* Want to help us build **_re:dash_**? Fork the project and make a pull request. We need all the help we can get!
+* Want to report a bug or request a feature? Please open [an issue](https://github.com/getredash/redash/issues/new).
+* Want to help us build **_re:dash_**? Fork the project, edit in a [dev environment](http://docs.redash.io/en/latest/dev/vagrant.html), and make a pull request. We need all the help we can get!
 
 ## License
 
-See [LICENSE](https://github.com/EverythingMe/redash/blob/master/LICENSE) file.
+See [LICENSE](https://github.com/getredash/redash/blob/master/LICENSE) file.

Vagrantfile

@@ -2,59 +2,14 @@
 # vi: set ft=ruby :
 
 # Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
-VAGRANTFILE_API_VERSION = '2'
-
-POSTGRES_PASSWORD     = 'securepass'
-
-# Currently, chef postgress cookbook works with cleartext paswords,
-# unless the password begins with 'md5'
-# See https://github.com/hw-cookbooks/postgresql/issues/95
-require "digest/md5"
-postgres_password_md5 = 'md5'+Digest::MD5.hexdigest(POSTGRES_PASSWORD+'postgres')
-
-# After starting the vagrant machine, the application is accessible via the URL
-# http://localhost:9999
-HOST_PORT_TO_FORWARD_TO_REDASH = 9999
-
-# Deploy direcly the code in parent dir; Don't download a release tarball 
-live_testing_deployment = true
+VAGRANTFILE_API_VERSION = "2"
 
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-  config.vm.box     = 'ubuntu-precise-cloudimg-amd64'
-  config.vm.box_url = 'http://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-amd64-vagrant-disk1.box'
-  
-  if config.respond_to? :cache
-    config.cache.auto_detect  = true
+  config.vm.box = "redash/dev"
+  config.vm.synced_folder "./", "/opt/redash/current"
+  config.vm.network "forwarded_port", guest: 5000, host: 9001
+  config.vm.provision "shell" do |s|
+    s.inline = "/opt/redash/current/setup/vagrant/provision.sh"
+    s.privileged = false
   end
-  
-  config.berkshelf.enabled    = true
-  config.omnibus.chef_version = :latest
-
-  config.vm.network 'forwarded_port', guest: 8888, host: HOST_PORT_TO_FORWARD_TO_REDASH
-  
-  if live_testing_deployment
-    config.vm.synced_folder "..", "/opt/redash"
-  end
-  
-  config.vm.provision :chef_solo do |chef|
-     # run apt-get update before anything else (specifically postgresql)..
-     chef.add_recipe 'apt'
-     chef.add_recipe 'redash::redis_for_redash'
-     chef.add_recipe 'postgresql::client'
-     chef.add_recipe 'postgresql::server'
-     chef.add_recipe 'redash::redash_pg_schema'
-     chef.add_recipe 'redash::redash'
-     # chef.log_level  = :debug
-     chef.json = {
-       'apt'        => { 'compiletime' => true },
-       'postgresql' => { 'password'    => {'postgres' => postgres_password_md5 } },
-       'redash'     => { 'db'          => {'host'     => 'localhost',
-                                           'user'     => 'postgres',
-                                           'password' => POSTGRES_PASSWORD },
-                         'allow'           => {'google_app_domain' => 'gmail.com',
-                                               'admins'            => ['joe@egmail.com','jack@gmail.com']},
-                         'install_tarball' => !live_testing_deployment,
-                         'user'            => live_testing_deployment ? 'vagrant' : 'redash'}
-     }
-   end
 end

bin/pre_compile

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Heroku pre_compile script
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+pushd $DIR/..
+
+# heroku requires cffi to be in requirements.txt in order for libffi to be installed.
+# https://github.com/heroku/heroku-buildpack-python/blob/master/bin/steps/cryptography
+# to avoid making it a requirement for other build systems, we'll inject it now
+# into the requirements.txt file
+
+# Remove Heroku unsupported Python packages:
+grep -v -E "^(pymssql|thrift|sasl|pyhive)" requirements_all_ds.txt >> requirements.txt
+
+# make the heroku Procfile the active one
+cp Procfile.heroku Procfile
+
+popd

bin/release_manager.py

@@ -0,0 +1,147 @@
+import os
+import sys
+import json
+import re
+import subprocess
+import requests
+
+github_token = os.environ['GITHUB_TOKEN']
+auth = (github_token, 'x-oauth-basic')
+repo = 'getredash/redash'
+
+def _github_request(method, path, params=None, headers={}):
+    if not path.startswith('https://api.github.com'):
+        url = "https://api.github.com/{}".format(path)
+    else:
+        url = path
+
+    if params is not None:
+        params = json.dumps(params)
+
+    response = requests.request(method, url, data=params, auth=auth)
+    return response
+
+def exception_from_error(message, response):
+    return Exception("({}) {}: {}".format(response.status_code, message, response.json().get('message', '?')))
+
+def rc_tag_name(version):
+    return "v{}-rc".format(version)
+
+def get_rc_release(version):
+    tag = rc_tag_name(version)
+    response = _github_request('get', 'repos/{}/releases/tags/{}'.format(repo, tag))
+
+    if response.status_code == 404:
+        return None
+    elif response.status_code == 200:
+        return response.json()
+
+    raise exception_from_error("Unknown error while looking RC release: ", response)
+
+def create_release(version, commit_sha):
+    tag = rc_tag_name(version)
+
+    params = {
+        'tag_name': tag,
+        'name': "{} - RC".format(version),
+        'target_commitish': commit_sha,
+        'prerelease': True
+    }
+
+    response = _github_request('post', 'repos/{}/releases'.format(repo), params)
+
+    if response.status_code != 201:
+        raise exception_from_error("Failed creating new release", response)
+
+    return response.json()
+
+def upload_asset(release, filepath):
+    upload_url = release['upload_url'].replace('{?name,label}', '')
+    filename = filepath.split('/')[-1]
+
+    with open(filepath) as file_content:
+        headers = {'Content-Type': 'application/gzip'}
+        response = requests.post(upload_url, file_content, params={'name': filename}, headers=headers, auth=auth, verify=False)
+
+    if response.status_code != 201:  # not 200/201/...
+        raise exception_from_error('Failed uploading asset', response)
+
+    return response
+
+def remove_previous_builds(release):
+    for asset in release['assets']:
+        response = _github_request('delete', asset['url'])
+        if response.status_code != 204:
+            raise exception_from_error("Failed deleting asset", response)
+
+def get_changelog(commit_sha):
+    latest_release = _github_request('get', 'repos/{}/releases/latest'.format(repo))
+    if latest_release.status_code != 200:
+        raise exception_from_error('Failed getting latest release', latest_release)
+
+    latest_release = latest_release.json()
+    previous_sha = latest_release['target_commitish']
+
+    args = ['git', '--no-pager', 'log', '--merges', '--grep', 'Merge pull request', '--pretty=format:"%h|%s|%b|%p"', '{}...{}'.format(previous_sha, commit_sha)]
+    log = subprocess.check_output(args)
+    changes = ["Changes since {}:".format(latest_release['name'])]
+
+    for line in log.split('\n'):
+        try:
+            sha, subject, body, parents = line[1:-1].split('|')
+        except ValueError:
+            continue
+
+        try:
+            pull_request = re.match("Merge pull request #(\d+)", subject).groups()[0]
+            pull_request = " #{}".format(pull_request)
+        except Exception, ex:
+            pull_request = ""
+
+        author = subprocess.check_output(['git', 'log', '-1', '--pretty=format:"%an"', parents.split(' ')[-1]])[1:-1]
+
+        changes.append("{}{}: {} ({})".format(sha, pull_request, body.strip(), author))
+
+    return "\n".join(changes)
+
+def update_release_commit_sha(release, commit_sha):
+    params = {
+        'target_commitish': commit_sha,
+    }
+
+    response = _github_request('patch', 'repos/{}/releases/{}'.format(repo, release['id']), params)
+
+    if response.status_code != 200:
+        raise exception_from_error("Failed updating commit sha for existing release", response)
+
+    return response.json()
+
+def update_release(version, build_filepath, commit_sha):
+    try:
+        release = get_rc_release(version)
+        if release:
+            release = update_release_commit_sha(release, commit_sha)
+        else:
+            release = create_release(version, commit_sha)
+
+        print "Using release id: {}".format(release['id'])
+
+        remove_previous_builds(release)
+        response = upload_asset(release, build_filepath)
+
+        changelog = get_changelog(commit_sha)
+
+        response = _github_request('patch', release['url'], {'body': changelog})
+        if response.status_code != 200:
+            raise exception_from_error("Failed updating release description", response)
+
+    except Exception, ex:
+        print ex
+
+if __name__ == '__main__':
+    commit_sha = sys.argv[1]
+    version = sys.argv[2]
+    filepath = sys.argv[3]
+
+    # TODO: make sure running from git directory & remote = repo
+    update_release(version, filepath, commit_sha)

bin/run

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+# Ideally I would use stdin with source, but in older bash versions this
+# wasn't supported properly.
+TEMP_ENV_FILE=`mktemp /tmp/redash_env.XXXXXX`
+sed 's/^REDASH/export REDASH/' .env > $TEMP_ENV_FILE
+source $TEMP_ENV_FILE
+rm $TEMP_ENV_FILE
+
+exec "$@"

bin/upload_version.py

@@ -1,31 +0,0 @@
-#!python
-import os
-import sys
-import json
-import requests
-
-if __name__ == '__main__':
-  version = sys.argv[1]
-  filepath = sys.argv[2]
-  filename = filepath.split('/')[-1]
-  github_token = os.environ['GITHUB_TOKEN']
-  auth = (github_token, 'x-oauth-basic')
-  commit_sha = os.environ['CIRCLE_SHA1']
-
-  params = json.dumps({
-    'tag_name': 'v{0}'.format(version),
-    'name': 're:dash v{0}'.format(version),
-    'target_commitish': commit_sha
-  })
-
-  response = requests.post('https://api.github.com/repos/everythingme/redash/releases',
-                            data=params,
-                            auth=auth)
-
-  upload_url = response.json()['upload_url']
-  upload_url = upload_url.replace('{?name}', '')
-
-  with open(filepath) as file_content:
-    headers = {'Content-Type': 'application/gzip'}
-    response = requests.post(upload_url, file_content, params={'name': filename}, auth=auth, headers=headers, verify=False)
-

bin/vagrant_ctl.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+set -e
+
+help() {
+  echo "Usage: "
+  echo "`basename "$0"` {start, test}"
+}
+
+case "$1" in
+  start)
+    vagrant up
+    vagrant ssh -c "cd /opt/redash/current; bin/run honcho start -f Procfile.dev;"
+    ;;
+  test)
+    vagrant up
+    vagrant ssh -c "cd /opt/redash/current; make test"
+    ;;
+  *)
+    help
+    ;;
+esac

bower.json

@@ -0,0 +1,47 @@
+{
+  "name": "redash",
+  "version": "0.11.0",
+  "dependencies": {
+    "angular": "1.2.18",
+    "angular-resource": "1.2.18",
+    "angular-route": "1.2.18",
+    "angular-growl": "0.4.0",
+    "json3": "3.2.4",
+    "jquery": "1.9.1",
+    "bootstrap": "3.3.6",
+    "es5-shim": "2.0.8",
+    "angular-moment": "0.10.3",
+    "moment": "~2.8.0",
+    "codemirror": "4.8.0",
+    "underscore": "1.5.1",
+    "pivottable": "2.0.2",
+    "cornelius": "https://github.com/restorando/cornelius.git",
+    "gridster": "0.2.0",
+    "mousetrap": "~1.4.6",
+    "jquery-ui": "~1.10.4",
+    "underscore.string": "~2.3.3",
+    "marked": "~0.3.2",
+    "pace": "~0.5.1",
+    "font-awesome": "~4.2.0",
+    "mustache": "~1.0.0",
+    "canvg": "gabelerner/canvg",
+    "angular-ui-bootstrap-bower": "~0.12.1",
+    "leaflet": "~0.7.3",
+    "angular-base64-upload": "~0.1.11",
+    "angular-ui-select": "~0.13.2",
+    "angular-bootstrap-show-errors": "~2.3.0",
+    "angular-sanitize": "1.2.18",
+    "d3": "3.5.6",
+    "angular-ui-sortable": "~0.13.4",
+    "angular-resizable": "^1.2.0",
+    "material-design-iconic-font": "^2.2.0",
+    "plotly.js": "^1.9.0"
+  },
+  "devDependencies": {
+    "angular-mocks": "1.2.18",
+    "angular-scenario": "1.2.18"
+  },
+  "resolutions": {
+    "angular": "1.2.18"
+  }
+}

circle.yml

@@ -1,19 +1,40 @@
 machine:
+  services:
+    - docker
   node:
     version:
-      0.10.22
+      0.12.4
   python:
     version:
       2.7.3
 dependencies:
   pre:
-    - make deps
-    - pip install requests
+    - pip install -r requirements_dev.txt
+    - pip install -r requirements.txt
+    - pip install pymongo==3.2.1
+    - if [ "$CIRCLE_BRANCH" = "master" ]; then make deps; fi
+  cache_directories:
+    - node_modules/
+    - rd_ui/node_modules/
+    - rd_ui/app/bower_components/
 test:
-  post:
-    - make pack 
+  override:
+    - nosetests --with-xunit --xunit-file=$CIRCLE_TEST_REPORTS/junit.xml --with-coverage --cover-package=redash tests/
 deployment:
-  github:
+  github_and_docker:
     branch: master
     commands:
+      - make pack
       - make upload
+      - echo "rd_ui/app" >> .dockerignore
+      - docker pull redash/redash:latest
+      - docker build -t redash/redash:$(./manage.py version | sed -e "s/\+/./") .
+      - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS
+      - docker push redash/redash:$(./manage.py version | sed -e "s/\+/./")
+notify:
+  webhooks:
+    - url: https://webhooks.gitter.im/e/895d09c3165a0913ac2f
+general:
+  branches:
+    ignore:
+      - gh-pages

docker-compose-example.yml

@@ -0,0 +1,25 @@
+redash:
+  image: redash/redash:latest
+  ports:
+    - "5000:5000"
+  links:
+    - redis
+    - postgres
+  environment:
+    REDASH_STATIC_ASSETS_PATH: "../rd_ui/dist/"
+    REDASH_LOG_LEVEL: "INFO"
+    REDASH_REDIS_URL: "redis://redis:6379/0"
+    REDASH_DATABASE_URL: "postgresql://postgres@postgres/postgres"
+    REDASH_COOKIE_SECRET: veryverysecret
+redis:
+  image: redis:2.8
+postgres:
+  image: postgres:9.3
+  volumes:
+    - /opt/postgres-data:/var/lib/postgresql/data
+redash-nginx:
+  image: redash/nginx:latest
+  ports:
+    - "80:80"
+  links:
+    - redash

docs/Makefile

@@ -0,0 +1,192 @@
+# Makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+PAPER         =
+BUILDDIR      = _build
+
+# User-friendly check for sphinx-build
+ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
+$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
+endif
+
+# Internal variables.
+PAPEROPT_a4     = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+
+.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
+
+help:
+	@echo "Please use \`make <target>' where <target> is one of"
+	@echo "  html       to make standalone HTML files"
+	@echo "  dirhtml    to make HTML files named index.html in directories"
+	@echo "  singlehtml to make a single large HTML file"
+	@echo "  pickle     to make pickle files"
+	@echo "  json       to make JSON files"
+	@echo "  htmlhelp   to make HTML files and a HTML help project"
+	@echo "  qthelp     to make HTML files and a qthelp project"
+	@echo "  applehelp  to make an Apple Help Book"
+	@echo "  devhelp    to make HTML files and a Devhelp project"
+	@echo "  epub       to make an epub"
+	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
+	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
+	@echo "  latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
+	@echo "  text       to make text files"
+	@echo "  man        to make manual pages"
+	@echo "  texinfo    to make Texinfo files"
+	@echo "  info       to make Texinfo files and run them through makeinfo"
+	@echo "  gettext    to make PO message catalogs"
+	@echo "  changes    to make an overview of all changed/added/deprecated items"
+	@echo "  xml        to make Docutils-native XML files"
+	@echo "  pseudoxml  to make pseudoxml-XML files for display purposes"
+	@echo "  linkcheck  to check all external links for integrity"
+	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
+	@echo "  coverage   to run coverage check of the documentation (if enabled)"
+
+clean:
+	rm -rf $(BUILDDIR)/*
+
+html:
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
+dirhtml:
+	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
+
+singlehtml:
+	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
+	@echo
+	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
+
+pickle:
+	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
+	@echo
+	@echo "Build finished; now you can process the pickle files."
+
+json:
+	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
+	@echo
+	@echo "Build finished; now you can process the JSON files."
+
+htmlhelp:
+	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
+	@echo
+	@echo "Build finished; now you can run HTML Help Workshop with the" \
+	      ".hhp project file in $(BUILDDIR)/htmlhelp."
+
+qthelp:
+	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
+	@echo
+	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
+	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
+	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/redash.qhcp"
+	@echo "To view the help file:"
+	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/redash.qhc"
+
+applehelp:
+	$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
+	@echo
+	@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
+	@echo "N.B. You won't be able to view it unless you put it in" \
+	      "~/Library/Documentation/Help or install it in your application" \
+	      "bundle."
+
+devhelp:
+	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
+	@echo
+	@echo "Build finished."
+	@echo "To view the help file:"
+	@echo "# mkdir -p $$HOME/.local/share/devhelp/redash"
+	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/redash"
+	@echo "# devhelp"
+
+epub:
+	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
+	@echo
+	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
+
+latex:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo
+	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
+	@echo "Run \`make' in that directory to run these through (pdf)latex" \
+	      "(use \`make latexpdf' here to do that automatically)."
+
+latexpdf:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through pdflatex..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+latexpdfja:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through platex and dvipdfmx..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+text:
+	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
+	@echo
+	@echo "Build finished. The text files are in $(BUILDDIR)/text."
+
+man:
+	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
+	@echo
+	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
+
+texinfo:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo
+	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
+	@echo "Run \`make' in that directory to run these through makeinfo" \
+	      "(use \`make info' here to do that automatically)."
+
+info:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo "Running Texinfo files through makeinfo..."
+	make -C $(BUILDDIR)/texinfo info
+	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
+
+gettext:
+	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
+	@echo
+	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
+
+changes:
+	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
+	@echo
+	@echo "The overview file is in $(BUILDDIR)/changes."
+
+linkcheck:
+	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	@echo
+	@echo "Link check complete; look for any errors in the above output " \
+	      "or in $(BUILDDIR)/linkcheck/output.txt."
+
+doctest:
+	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
+	@echo "Testing of doctests in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/doctest/output.txt."
+
+coverage:
+	$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
+	@echo "Testing of coverage in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/coverage/python.txt."
+
+xml:
+	$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
+	@echo
+	@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
+
+pseudoxml:
+	$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
+	@echo
+	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."

docs/conf.py

@@ -0,0 +1,110 @@
+# -*- coding: utf-8 -*-
+#
+# re:dash documentation build configuration file, created by
+# sphinx-quickstart on Mon Jul 20 22:40:24 2015.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = []
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'Re:dash'
+copyright = u'2013-2016, Arik Fraimovich'
+author = u'Arik Fraimovich'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+exclude_patterns = ['_build']
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = False
+
+# -- Options for HTML output ----------------------------------------------
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+import sphinx_rtd_theme
+html_theme = "sphinx_rtd_theme"
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+# The name for this set of Sphinx documents.  If None, it defaults to
+# "<project> v<release> documentation".
+#html_title = None
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+#html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# If true, links to the reST sources are added to the pages.
+html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+html_show_sphinx = False
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+html_show_copyright = False
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'redashdoc'
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'redash', u're:dash Documentation',
+     [author], 1)
+]
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+  (master_doc, 'redash', u're:dash Documentation',
+   author, 'redash', 'One line description of project.',
+   'Miscellaneous'),
+]

docs/datasources.rst

@@ -0,0 +1,261 @@
+Supported Data Sources
+######################
+
+Re:dash supports several types of data sources, and if you set it up using the provided images, it should already have
+the needed dependencies to use them all. Starting from version 0.7 and newer, you can manage data sources from the UI
+by browsing to ``/data_sources`` on your instance.
+
+If one of the listed data source types isn't available when trying to create a new data source, make sure that:
+
+1. You installed required dependencies.
+2. If you've set custom value for the ``REDASH_ENABLED_QUERY_RUNNERS`` setting, it's included in the list.
+
+PostgreSQL / Redshift / Greenplum
+---------------------------------
+
+-  **Options**:
+
+   -  Database name (mandatory)
+   -  User
+   -  Password
+   -  Host
+   -  Port
+   
+-  **Additional requirements**:
+
+   - None
+
+
+MySQL
+-----
+
+-  **Options**:
+
+   -  Database name (mandatory)
+   -  User
+   -  Password
+   -  Host
+   -  Port
+
+-  **Additional requirements**:
+
+   - ``MySQL-python`` python package
+
+
+Google BigQuery
+---------------
+
+-  **Options**:
+
+   -  Project ID (mandatory)
+   -  JSON key file, generated when creating a service account (see `instructions <https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount>`__).
+
+
+-  **Additional requirements**:
+
+   - ``google-api-python-client``, ``oauth2client`` and ``pyopenssl`` python packages (on Ubuntu it might require installing ``libffi-dev`` and ``libssl-dev`` as well).
+
+
+Graphite
+--------
+
+-  **Options**:
+
+   -  Url (mandatory)
+   -  User
+   -  Password
+   -  Verify SSL certificate
+
+
+MongoDB
+-------
+
+-  **Options**:
+
+   -  Connection String (mandatory)
+   -  Database name
+   -  Replica set name
+
+-  **Additional requirements**:
+
+   - ``pymongo`` python package.
+
+For information on how to write MongoDB queries, see :doc:`documentation </usage/mongodb_querying>`.
+
+
+ElasticSearch
+-------------
+
+...
+
+InfluxDB
+--------
+
+...
+
+Presto
+------
+
+-  **Options**:
+
+    - Host (mandatory)
+        - Address to a Presto coordinator.
+    - Port
+        - Port to a Presto coordinator. `8080` is the default port.
+    - Schema
+        - Default schema name of Presto. You can read other schemas by qualified name like `FROM myschema.table1`.
+    - Catalog
+        - Catalog (connector) name of Presto such as `hive-cdh4`, `hive-hadoop1`, etc.
+    - Username
+        - User name to connect to a Presto.
+
+-  **Additional requirements**:
+
+   - ``pyhive`` python package.
+
+Hive
+----
+
+...
+
+Impala
+------
+
+...
+
+URL
+---
+
+A URL based data source which requests URLs that return the :doc:`results JSON
+format </dev/results_format>`.
+
+Very useful in situations where you want to expose the data without
+connecting directly to the database.
+
+The query itself inside Re:dash will simply contain the URL to be
+executed (i.e. http://myserver/path/myquery)
+
+-  **Options**:
+
+   -  Url - set this if you want to limit queries to certain base path.
+
+
+Google Spreadsheets
+-------------------
+
+-  **Options**:
+
+   -  JSON key file, generated when creating a service account (see `instructions <https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount>`__).
+
+-  **Additional requirements**:
+
+   -  ``gspread`` and ``oauth2client`` python packages.
+
+Notes:
+
+1. To be able to load the spreadsheet in Re:dash - share your it with
+   your ServiceAccount's email (it can be found in the credentials json
+   file, for example
+   43242343247-fjdfakljr3r2@developer.gserviceaccount.com).
+2. The query format is "DOC\_UUID\|SHEET\_NUM" (for example
+   "kjsdfhkjh4rsEFSDFEWR232jkddsfh\|0")
+
+
+Python
+------
+
+**Execute other queries, manipulate and compute with Python code**
+
+This is a special query runner, that will execute provided Python code as the query. Useful for various scenarios such as
+merging data from different data sources, doing data transformation/manipulation that isn't trivial with SQL, merging
+with remote data or using data analysis libraries such as Pandas (see `example query <https://gist.github.com/arikfr/be7c2888520c44cf4f0f>`__).
+
+While the Python query runner uses a sandbox (RestrictedPython), it's not 100% secure and the security depends on the
+modules you allow to import. We recommend enabling the Python query runner only in a trusted environment (meaning: behind
+VPN and with users you trust).
+
+-  **Options**:
+
+   -  Allowed Modules in a comma separated list (optional). **NOTE:**
+      You MUST make sure these modules are installed on the machine
+      running the Celery workers.
+
+Notes:
+
+- For security, the python query runner is disabled by default.
+  To enable, add ``redash.query_runner.python`` to the ``REDASH_ADDITIONAL_QUERY_RUNNERS`` environmental variable. If you used
+  the bootstrap script, or one of the provided images, add to ``/opt/redash/.env`` file the line: ``export REDASH_ADDITIONAL_QUERY_RUNNERS=redash.query_runner.python``.
+
+
+Vertica
+-----
+
+-  **Options**:
+
+   -  Database (mandatory)
+   -  User
+   -  Password
+   -  Host
+   -  Port
+
+-  **Additional requirements**:
+
+   - ``vertica-python`` python package
+
+Oracle
+------
+
+- **Options**
+
+   -  DSN Service name
+   -  User
+   -  Password
+   -  Host
+   -  Port
+
+- **Additional requirements**
+
+  - ``cx_Oracle`` python package. This requires the installation of the Oracle `instant client <http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html>`__.
+
+Treasure Data
+------
+
+- **Options**
+
+   -  Type (TreasureData)
+   -  API Key
+   -  Database Name
+   -  Type (Presto/Hive[default])
+   
+- **Additional requirements**
+   - Must have account on https://console.treasuredata.com
+   
+Documentation: https://docs.treasuredata.com/articles/redash
+
+
+
+Microsoft SQL Server
+-----
+
+-  **Options**:
+
+   -  Database (mandatory)
+   -  User  #TODO: DB users only? What about domain users? 
+   -  Password
+   -  Server
+   -  Port
+
+-  **Notes**:
+
+   - Data type support is currently quite limited.
+   - Complex and new types are converted to strings in ``Re:dash``
+      - Coerce into simpler types if needed using ``CAST()``
+   - Known conversion issues for: 
+      - DATE
+      - TIME
+      - DATETIMEOFFSET
+ 
+-  **Additional requirements**:
+
+   - ``freetds-dev`` C library
+   - ``pymsssql`` python package, requires FreeTDS to be installed first

docs/dev.rst

@@ -0,0 +1,11 @@
+Developer Information
+=====================
+
+.. toctree::
+    :maxdepth: 2
+    :glob:
+
+    dev/vagrant
+    dev/*
+
+

docs/dev/query_execution.rst

@@ -0,0 +1,94 @@
+Query Execution Model
+#####################
+
+Introduction
+============
+
+The first datasource which was used with Re:dash was Redshift. Because
+we had billions of records in Redshift, and some queries were costly to
+re-run, from the get go there was the idea of caching query results in
+Re:dash.
+
+This was to relieve stress from the Redshift cluster and also to improve
+user experience.
+
+How queries get executed and cached in Re:dash?
+===============================================
+
+Server
+------
+
+To make sure each query is executed only once at any giving time, we
+translate the query to a ``query hash``, using the following code:
+
+.. code:: python
+
+    COMMENTS_REGEX = re.compile("/\*.*?\*/")
+
+    def gen_query_hash(sql):
+        sql = COMMENTS_REGEX.sub("", sql)
+        sql = "".join(sql.split()).lower()
+        return hashlib.md5(sql.encode('utf-8')).hexdigest()
+
+When query execution is done, the result gets stored to
+``query_results`` table. Also we check for all queries in the
+``queries`` table that have the same query hash and update their
+reference to the query result we just saved
+(`code <https://github.com/getredash/redash/blob/master/redash/models.py#L235>`__).
+
+Client
+------
+
+The client (UI) will execute queries in two scenarios:
+
+1. (automatically) When opening a query page of a query that doesn't
+   have a result yet.
+2. (manually) When the user clicks on "Execute".
+
+In each case the client does a POST request to ``/api/query_results``
+with the following parameters: ``query`` (the query text),
+``data_source_id`` (data source to execute the query with) and ``ttl``.
+
+When loading a cached result, ``ttl`` will be the one set to the query
+(if it was set). This is a relic from previous versions, and I'm not
+sure if it's really used anymore, as usually we will fetch query result
+using its id.
+
+When loading a non cached result, ``ttl`` will be 0 which will "force"
+the server to execute the query.
+
+As a response to ``/api/query_results`` the server will send either the
+query results (in case of a cached query) or job id of the currently
+executing query. When job id received the client will start polling on
+this id, until a query result received (this is encapsulated in
+``Query`` and ``QueryResult`` services).
+
+Ideas on how to implement query parameters
+==========================================
+
+Client side only implementation
+-------------------------------
+
+(This was actually implemented in. See pull request `#363 <https://github.com/getredash/redash/pull/363>`__ for details.)
+
+The basic idea of how to implement parametized queries is to treat the
+query as a template and merge it with parameters taken from query string
+or UI (or both).
+
+When the caching facility isn't required (with queries that return in a
+reasonable time frame) the implementation can be completely client side
+and the backend can be "blind" to the parameters - it just receives the
+final query to execute and returns result.
+
+As one improvement over this, we can let the UI/user specify the TTL
+value when making the request to ``/api/query_results``, in which case
+caching will be availble too, while not having to make the server aware
+of the parameters.
+
+Hybrid
+------
+
+Another option, will be to store the list of possible parameters for a
+query, with their default/optional values. In such case, the server can
+prefetch all the options and cache them to provide faster results to the
+client.

docs/dev/results_format.rst

@@ -0,0 +1,30 @@
+Data Source Results Format
+==========================
+
+All data sources in Re:dash return the following results in JSON format:
+
+.. code:: javascript
+
+    {
+        "columns" : [
+            {
+               // Required: a unique identifier of the column name in this result
+               "name" : "COLUMN_NAME",
+               // Required: friendly name of the column that will appear in the results
+               "friendly_name" : "FRIENDLY_NAME",
+               // Optional: If not specified sort might not work well.
+               // Supported types: integer, float, boolean, string (default), datetime (ISO-8601 text format)
+               "type" : "VALUE_TYPE"
+            },
+            ...
+        ],
+        "rows" : [
+            {
+                // name is the column name as it appears in the columns above.
+                // VALUE is a valid JSON value. For dates its an ISO-8601 string.
+                "name" : VALUE,
+                "name2" : VALUE2
+            },
+            ...
+        ]
+    }

docs/dev/saml.rst

@@ -0,0 +1,34 @@
+SAML Authentication and Authorization
+#####################################
+
+Authentication
+==============
+
+Add to your .env file REDASH_SAML_METADATA_URL config value which
+needs to point to the SAML provider metadata url, eg https://app.onelogin.com/saml/metadata/
+
+And an optional REDASH_SAML_CALLBACK_SERVER_NAME which contains the
+ server name of the redash server for the callbacks from the SAML provider (eg demo.redash.io)
+
+On the SAML provider side, example configuration for OneLogin is:
+SAML Consumer URL: http://demo.redash.io/saml/login
+SAML Audience: http://demo.redash.io/saml/callback
+SAML Recipient: http://demo.redash.io/saml/callback
+
+Example configuration for Okta is:
+Single Sign On URL: http://demo.redash.io/saml/callback
+Recipient URL: http://demo.redash.io/saml/callback
+Destination URL: http://demo.redash.io/saml/callback
+
+with parameters 'FirstName' and 'LastName', both configured to be included in the SAML assertion.
+
+
+Authorization
+=============
+To manage group assignments in Redash using your SAML provider, configure SAML response to include
+attribute with key 'RedashGroups', and value as names of groups in Redash.
+
+Example configuration for Okta is:
+In the Group Attribute Statements -
+Name: RedashGroups
+Filter: Starts with: this-is-a-group-in-redash

docs/dev/vagrant.rst

@@ -0,0 +1,23 @@
+Setting up development environment (using Vagrant)
+==================================================
+
+To simplify contribution there is a `Vagrant
+box <https://vagrantcloud.com/redash/boxes/dev>`__ available with all
+the needed software to run Re:dash for development (use it only for
+development, for demo purposes there is
+`redash/demo <https://vagrantcloud.com/redash/boxes/demo>`__ box and the
+AWS/GCE images).
+
+To get started with this box:
+
+1.  Make sure you have recent version of
+    `Vagrant <https://www.vagrantup.com/>`__ installed.
+2.  Clone the Re:dash repository:
+    ``git clone https://github.com/getredash/redash.git``.
+3.  Change dir into the repository (``cd redash``)
+4a. To execute tests, run ``./bin/vagrant_ctl.sh test``
+4b. To run the app, run ``./bin/vagrant_ctl.sh start``.
+    This might take some time the first time you run it,
+    as it downloads the Vagrant virtual box.
+    Now the server should be available on your host on port 9001 and you
+    can login with username admin and password admin.

docs/index.rst

@@ -0,0 +1,57 @@
+.. image:: http://redash.io/static/old_img/redash_logo.png
+   :width: 200px
+
+Open Source Data Collaboration and Visualization Platform
+===================================
+
+**Re:dash** is our take on freeing the data within our company in a way that will better fit our culture and usage patterns.
+
+Prior to **Re:dash**, we tried to use traditional BI suites and discovered a set of bloated, technically challenged and slow tools/flows. What we were looking for was a more hacker'ish way to look at data, so we built one.
+
+**Re:dash** was built to allow fast and easy access to billions of records, that we process and collect using Amazon Redshift ("petabyte scale data warehouse" that "speaks" PostgreSQL).
+Today **_re:dash_** has support for querying multiple databases, including: Redshift, Google BigQuery,Google Spreadsheets, PostgreSQL, MySQL, Graphite and custom scripts.
+
+Features
+########
+
+1. **Query Editor**: think of `JS Fiddle`_ for SQL queries. It's your way to share data in the organization in an open way, by sharing both the dataset and the query that generated it. This way everyone can peer review not only the resulting dataset but also the process that generated it.
+2. **Visualizations**: once you have a dataset, you can create different visualizations out of it. Currently it supports charts, pivot table and cohorts.
+3. **Dashboards**: combine several visualizations into a single dashboard.
+
+Demo
+####
+
+.. figure:: https://cloud.githubusercontent.com/assets/71468/12611424/1faf4d6a-c4f5-11e5-89b5-31efc1155d2c.gif
+   :alt: Screenshots
+
+You can try out the demo instance: `http://demo.redash.io`_ (login with any Google account).
+
+.. _http://demo.redash.io: http://demo.redash.io
+.. _JS Fiddle: http://jsfiddle.net
+
+Getting Started
+###############
+
+:doc:`Setting up Re:dash instance </setup>` (includes links to ready made AWS/GCE images).
+
+Getting Help
+############
+
+* Source: https://github.com/getredash/redash
+* Issues: https://github.com/getredash/redash/issues
+* Discussion Forum: https://discuss.redash.io/
+* Slack: http://slack.redash.io/
+* Gitter (chat): https://gitter.im/getredash/redash
+
+TOC
+###
+
+.. toctree::
+   :maxdepth: 2
+
+   setup
+   upgrade
+   datasources
+   usage
+   dev
+   misc

docs/misc.rst

@@ -0,0 +1,10 @@
+Miscellaneous
+=============
+
+.. toctree::
+    :maxdepth: 2
+    :glob:
+
+    misc/*
+
+

docs/misc/backup_restore.rst

@@ -0,0 +1,74 @@
+How To: Backup your Re:dash database and restore it on a different server
+=================
+
+**Note:** This guide assumes that the default database name (redash) has not been changed.
+
+1. Check the size of your redash database. This can be done by creating a query within redash itself against the 'Re:dash metadata' data source.
+
+.. code::
+
+    select t1.datname AS db_name, pg_size_pretty(pg_database_size(t1.datname)) as db_size
+    from pg_database t1
+    where t1.datname = 'redash'
+
+
+2. Check the amount of available disk space on your existing server.
+
+.. code::
+
+    df -hT
+
+
+3. Backup the existing redash database.
+
+.. code::
+
+    sudo -u redash pg_dump redash | gzip > redash_backup.gz
+
+
+4. Transfer the backup to the new server.
+
+5. `Perform a clean install of Re:dash <http://docs.redash.io/en/latest/setup.html>`__ on the new server.
+
+6. Check the amount of available disk space on the new server.
+
+.. code::
+
+    df -hT
+
+
+7. Login as postgres user on the new server.
+
+.. code::
+
+    sudo -u postgres -i
+
+
+8. drop the current redash database, create a new database named redash, and then restore the backup into the new database.
+
+.. code::
+
+    dropdb redash
+    createdb -T template0 redash
+    gunzip -c redash_backup.gz | psql redash
+
+
+9. Set a new password of your choosing for the 'redash_reader' user (since the new installation generated a random password).
+
+.. code::
+
+    psql -c "ALTER ROLE redash_reader WITH PASSWORD 'yourpasswordgoeshere';"
+
+
+**Note:** Then you must navigate to the 'Re:dash metadata' data source (/data_sources/1) in the new Re:dash installation and change the password to match the one entered above.
+
+10. Grant permissions on the redash database to the redash_reader user.
+
+.. code::
+
+    psql -c "grant select(id,name,type) ON data_sources to redash_reader;" redash
+    psql -c "grant select(id,name) ON users to redash_reader;" redash
+    psql -c "grant select on events, queries, dashboards, widgets, visualizations, query_results to redash_reader;" redash
+
+
+Create a new query in redash (using Re:dash metadata as the data source) to test that everything is working as expected.

docs/misc/google_developers_project.rst

@@ -0,0 +1,50 @@
+How To: Create a Google Developers Project
+==========================================
+
+1. Go to the `Google Developers
+   Console <https://console.developers.google.com/>`__.
+2. Select a project, or create a new one by clicking Create Project:
+
+   1. In the Project name field, type in a name for your project.
+   2. In the Project ID field, optionally type in a project ID for your
+      project or use the one that the console has created for you. This
+      ID must be unique world-wide.
+   3. Click the **Create** button and wait for the project to be
+      created.
+   4. Click on the new project name in the list to start editing the
+      project.
+
+3. In the left sidebar, select the **APIs** item below "APIs & auth". A
+   list of Google web services appears.
+4. Find the **Google+ API** service and set its status to **ON**—notice
+   that this action moves the service to the top of the list.
+5. In the sidebar under "APIs & auth", select **Credentials** and in that screen choose the **OAuth consent screen** tab
+
+   -  Choose an Email Address and specify a Product Name.
+
+6. In the sidebar under "APIs & auth", select **Credentials**.
+7. Click **Add Credentials** button and choose **OAuth 20 Client ID**. 
+
+   -  In the **Application type** section of the dialog, select **Web
+      application**.
+   -  In the **Authorized JavaScript origins** field, enter the origin
+      for your app. You can enter multiple origins to use with multiple
+      Re:dash instance. Wildcards are not allowed. In the example below,
+      we assume your Re:dash instance address is *redash.example.com*:
+
+   ::
+
+       http://redash.example.com
+       https://redash.example.com
+
+   -  In the Authorized redirect URI field, enter the redirect URI
+      callback:
+
+   ::
+
+       http://redash.example.com/oauth/google_callback
+
+   -  Click the ``Create`` button.
+
+8. In the resulting **Client ID for web application** section, copy the
+   **Client ID** and **Client secret** to your ``.env`` file.

docs/misc/letsencrypt.rst

@@ -0,0 +1,141 @@
+How To: Encrypt your Re:dash installation with a free SSL certificate from Let's Encrypt
+=================
+
+**Note:** This below steps were tested on Ubuntu 14.04, but *should* work with any Debian-based distro.
+
+`Let's Encrypt <https://letsencrypt.org/>`__ is a new certificate authority sponsored by major tech companies including Mozilla, Google, Cisco, and Facebook. Unlike traditional CA authorities, Let's Encrypt allows you to generate and renew an SSL certificate quickly and **at no cost**.
+
+1. Open port 443 in your security group (if using AWS or GCE).
+
+2. Update package lists, install git, and clone the letsencrypt repository.
+
+.. code::
+
+    sudo apt-get update
+    sudo apt-get install git
+    sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
+
+
+3. Stop nginx and redash, then ensure that no processes are still listening on port 80.
+
+.. code::
+
+    sudo supervisorctl stop redash_server
+    sudo service nginx stop
+    netstat -na | grep ':80.*LISTEN'
+
+
+4. Generate your letsencrypt certificate.
+
+.. code::
+
+      cd /opt/letsencrypt
+      sudo pip install urllib3[secure] --upgrade
+      ./letsencrypt-auto certonly --standalone
+
+
+In most cases you'll want to enter 'example.com www.example.com' when prompted for your domain so that you can use the certificate on http://example.com and http://www.example.com.
+
+5. Optionally generate a stronger Diffie-Hellman ephemeral parameter. Without this step, you will not achieve higher than a B score on `SSLLabs <https://www.ssllabs.com/ssltest/>`__. Please note that on a low-end server (VPS or micro/small GCE instance) this step can take approximately 20-30 minutes.
+
+.. code::
+
+      cd /etc/ssl/certs
+      sudo openssl dhparam -out dhparam.pem 3072
+
+
+6. Backup the existing nginx redash config, delete it, and then create a new version with the code supplied below.
+
+.. code::
+
+      sudo cp /etc/nginx/sites-available/redash /etc/nginx/sites-available/redash.bak
+      sudo rm /etc/nginx/sites-available/redash
+      sudo nano /etc/nginx/sites-available/redash
+
+
+.. code:: nginx
+
+      upstream redash_servers {
+         server 127.0.0.1:5000;
+      }
+
+      server {
+         listen 80;
+
+      # Allow accessing /ping without https. Useful when placing behind load balancer.
+      location /ping {
+         proxy_set_header Host $http_host;
+         proxy_set_header X-Real-IP $remote_addr;
+         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+         proxy_pass       http://redash_servers;
+      }
+
+      location / {
+         # Enforce SSL.
+         return 301 https://$host$request_uri;
+         }
+      }
+
+      server {
+         listen 443 ssl;
+         ssl on;
+
+         # Make sure to set paths to your certificate .pem and .key files.
+         ssl_certificate /etc/letsencrypt/live/YOURDOMAIN.TLD/fullchain.pem;
+         ssl_certificate_key /etc/letsencrypt/live/YOURDOMAIN.TLD/privkey.pem;
+         ssl_dhparam /etc/ssl/certs/dhparam.pem;
+
+         # Use secure protocols and ciphers which are compatible with modern browsers
+         ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
+         ssl_prefer_server_ciphers on;
+         ssl_ciphers AES256+EECDH:AES256+EDH;
+         ssl_session_cache shared:SSL:20m;
+      
+         # Enforce strict transport security
+         add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
+
+         access_log /var/log/nginx/redash.access.log;
+
+         gzip on;
+         gzip_types *;
+         gzip_proxied any;
+
+         location / {
+         proxy_set_header Host $http_host;
+         proxy_set_header X-Real-IP $remote_addr;
+         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+         proxy_set_header X-Forwarded-Proto $scheme;
+         proxy_pass       http://redash_servers;
+         proxy_redirect   off;
+         }
+      }
+
+
+7. Start the nginx and redash servers again.
+
+.. code::
+
+  sudo service nginx start
+  sudo supervisorctl start redash_server
+
+
+8. Verify the installation by running a `SSLLabs test <https://www.ssllabs.com/ssltest/>`__. This guide *should* yield an A+ score. If everything is working as expected, optionally delete the old redash nginx config:
+
+.. code::
+
+  sudo rm /etc/nginx/sites-available/redash.bak
+
+
+**Important Note:** letsencrypt certificates only remain valid for 90 days. To renew your certificate, simply follow steps 3 and 4 again:
+
+.. code::
+
+  sudo supervisorctl stop redash_server
+  sudo service nginx stop
+  netstat -na | grep ':80.*LISTEN'
+
+  cd /opt/letsencrypt
+  ./letsencrypt-auto certonly --standalone
+  
+  sudo service nginx start
+  sudo supervisorctl start redash_server

docs/misc/ssl.rst

@@ -0,0 +1,66 @@
+SSL (HTTPS) Setup
+=================
+
+If you used the provided images or the bootstrap script, to start using
+SSL with your instance you need to:
+
+1. Update the nginx config file (``/etc/nginx/sites-available/redash``)
+   with SSL configuration (see below an example). Make sure to upload
+   the certificate to the server, and set the paths correctly in the new
+   config.
+
+2. Open port 443 in your security group (if using AWS or GCE).
+
+.. code:: nginx
+
+    upstream redash_servers {
+      server 127.0.0.1:5000;
+    }
+
+    server {
+      listen 80;
+
+      # Allow accessing /ping without https. Useful when placing behind load balancer.
+      location /ping {
+        proxy_set_header Host $http_host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_pass       http://redash_servers;
+      }
+
+      location / {
+        # Enforce SSL.
+        return 301 https://$host$request_uri;
+      }
+    }
+
+    server {
+      listen 443 ssl;
+
+      # Make sure to set paths to your certificate .pem and .key files.
+      ssl on;
+      ssl_certificate /path-to/cert.pem; # or crt
+      ssl_certificate_key /path-to/cert.key;
+
+      # Specifies that we don't want to use SSLv2 (insecure) or SSLv3 (exploitable)
+      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
+      # Uses the server's ciphers rather than the client's
+      ssl_prefer_server_ciphers on;
+      # Specifies which ciphers are okay and which are not okay. List taken from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
+      ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
+      
+      access_log /var/log/nginx/redash.access.log;
+
+      gzip on;
+      gzip_types *;
+      gzip_proxied any;
+
+      location / {
+        proxy_set_header Host $http_host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_pass       http://redash_servers;
+        proxy_redirect   off;
+      }
+    }

docs/requirements.txt

@@ -0,0 +1,3 @@
+sphinx
+sphinx-autobuild
+sphinx_rtd_theme

docs/settings.rst

@@ -0,0 +1,60 @@
+Settings
+########
+
+Much of the functionality of Re:dash can be changes with settings. Settings are read by `/redash/settings.py` from environment variables which (for most installs) can be set in `/opt/redash/current/.env`
+
+The follow is a list of settings and what they control:
+
+- **REDASH_NAME**: name of the site, used in page titles, *default "Re:dash"*
+- **REDASH_REDIS_URL**: *default "redis://localhost:6379/0"*
+- **REDASH_PROXIES_COUNT**: *default "1"*
+- **REDASH_STATSD_HOST**: *default "127.0.0.1"*
+- **REDASH_STATSD_PORT**: *default "8125"*
+- **REDASH_STATSD_PREFIX**: *default "redash"*
+- **REDASH_DATABASE_URL**: *default "postgresql://postgres"*
+- **REDASH_CELERY_BROKER**: *default REDIS_URL*
+- **REDASH_CELERY_BACKEND**: *default CELERY_BROKER*
+- **REDASH_QUERY_RESULTS_CLEANUP_ENABLED**: *default "true"*
+- **REDASH_QUERY_RESULTS_CLEANUP_COUNT**: *default "100"*
+- **REDASH_QUERY_RESULTS_CLEANUP_MAX_AGE**: *default "7"*
+- **REDASH_AUTH_TYPE**: *default "api_key"*
+- **REDASH_PASSWORD_LOGIN_ENABLED**: *default "true"*
+- **REDASH_ENFORCE_HTTPS**: *default "false"*
+- **REDASH_MULTI_ORG**: *default "false"*
+- **REDASH_GOOGLE_CLIENT_ID**: *default ""*
+- **REDASH_GOOGLE_CLIENT_SECRET**: *default ""*
+- **REDASH_SAML_METADATA_URL**: *default ""*
+- **REDASH_SAML_CALLBACK_SERVER_NAME**: *default ""*
+- **REDASH_STATIC_ASSETS_PATH**: *default "../rd_ui/app/"*
+- **REDASH_JOB_EXPIRY_TIME**: *default 3600 * 6*
+- **REDASH_COOKIE_SECRET**: *default "c292a0a3aa32397cdb050e233733900f"*
+- **REDASH_LOG_LEVEL**: *default "INFO"*
+- **REDASH_MAIL_SERVER**: *default "localhost"*
+- **REDASH_MAIL_PORT**: *default 25*
+- **REDASH_MAIL_USE_TLS**: *default "false"*
+- **REDASH_MAIL_USE_SSL**: *default "false"*
+- **REDASH_MAIL_USERNAME**: *default None*
+- **REDASH_MAIL_PASSWORD**: *default None*
+- **REDASH_MAIL_DEFAULT_SENDER**: *default None*
+- **REDASH_MAIL_MAX_EMAILS**: *default None*
+- **REDASH_MAIL_ASCII_ATTACHMENTS**: *default "false"*
+- **REDASH_HOST**: *default ""*
+- **REDASH_HIPCHAT_API_TOKEN**: *default None*
+- **REDASH_HIPCHAT_API_URL**: *default None*
+- **REDASH_HIPCHAT_ROOM_ID**: *default None*
+- **REDASH_WEBHOOK_ENDPOINT**: *default None*
+- **REDASH_WEBHOOK_USERNAME**: *default None*
+- **REDASH_CORS_ACCESS_CONTROL_ALLOW_ORIGIN**: *default ""*
+- **REDASH_CORS_ACCESS_CONTROL_ALLOW_CREDENTIALS**: *default "false"*
+- **REDASH_CORS_ACCESS_CONTROL_REQUEST_METHOD**: *default GET, POST, PUT""*
+- **REDASH_CORS_ACCESS_CONTROL_ALLOW_HEADERS**: *default "Content-Type"*
+- **REDASH_ENABLED_QUERY_RUNNERS**: *default ",".join(default_query_runners)*
+- **REDASH_ADDITIONAL_QUERY_RUNNERS**: *default ""*
+- **REDASH_SENTRY_DSN**: *default ""*
+- **REDASH_ALLOW_SCRIPTS_IN_USER_INPUT**: disable sanitization of text input, allowing full HTML, *default "true"*
+- **REDASH_DATE_FORMAT**: *default "DD/MM/YY"*
+- **REDASH_FEATURE_ALLOW_ALL_TO_EDIT**: *default "true"*
+- **REDASH_FEATURE_TABLES_PERMISSIONS**: *default "false"*
+- **REDASH_VERSION_CEHCK**: *default "true"*
+- **REDASH_BIGQUERY_HTTP_TIMEOUT**: *default "600"*
+- **REDASH_SCHEMA_RUN_TABLE_SIZE_CALCULATIONS**: *default "false"*

docs/setup.rst

@@ -0,0 +1,244 @@
+Setting up Re:dash instance
+###########################
+
+The `provisioning
+script <https://raw.githubusercontent.com/getredash/redash/master/setup/ubuntu/bootstrap.sh>`__
+works on Ubuntu 12.04, Ubuntu 14.04 and Debian Wheezy. This script
+installs all needed dependencies and creates basic setup.
+
+To ease the process, there are also images for AWS, Google Compute
+Cloud and Docker. These images created with the same provision script using Packer.
+
+Create an instance
+==================
+
+AWS
+---
+
+Launch the instance with from the pre-baked AMI (for small deployments
+t2.micro should be enough):
+
+-  us-east-1: `ami-52c3373f <https://console.aws.amazon.com/ec2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-52c3373f>`__
+-  us-west-1: `ami-c6c5bda6 <https://console.aws.amazon.com/ec2/home?region=us-west-1#LaunchInstanceWizard:ami=ami-c6c5bda6>`__
+-  us-west-2: `ami-f0b04e90 <https://console.aws.amazon.com/ec2/home?region=us-west-2#LaunchInstanceWizard:ami=ami-f0b04e90>`__
+-  eu-west-1: `ami-f3910780 <https://console.aws.amazon.com/ec2/home?region=eu-west-1#LaunchInstanceWizard:ami=ami-f3910780>`__
+-  eu-central-1: `ami-00719d6f <https://console.aws.amazon.com/ec2/home?region=eu-central-1#LaunchInstanceWizard:ami=ami-00719d6f>`__
+-  sa-east-1: `ami-af2fa7c3 <https://console.aws.amazon.com/ec2/home?region=sa-east-1#LaunchInstanceWizard:ami=ami-af2fa7c3>`__
+-  ap-northeast-1: `ami-78967519 <https://console.aws.amazon.com/ec2/home?region=ap-northeast-1#LaunchInstanceWizard:ami=ami-78967519>`__
+-  ap-southeast-1: `ami-bdbb6ade <https://console.aws.amazon.com/ec2/home?region=ap-southeast-1#LaunchInstanceWizard:ami=ami-bdbb6ade>`__
+-  ap-southeast-2: `ami-8edbf4ed <https://console.aws.amazon.com/ec2/home?region=ap-southeast-2#LaunchInstanceWizard:ami=ami-8edbf4ed>`__
+
+(the above AMIs are of version: 0.10.1)
+
+When launching the instance make sure to use a security group, that **only** allows incoming traffic on: port 22 (SSH), 80 (HTTP) and 443 (HTTPS). These AMIs are based on Ubuntu so you will need to use the user ``ubuntu`` when connecting to the instance via SSH.
+
+Now proceed to `"Setup" <#setup>`__.
+
+Google Compute Engine
+---------------------
+
+First, you need to add the images to your account:
+
+.. code:: bash
+
+    $ gcloud compute images create "redash-091-b1377" --source-uri gs://redash-images/redash.0.9.1.b1377.tar.gz
+
+Next you need to launch an instance using this image (n1-standard-1
+instance type is recommended).
+
+If you plan using Re:dash with BigQuery, you can use a dedicated image which comes with BigQuery preconfigured
+(using instance permissions):
+
+.. code:: bash
+
+    $ gcloud compute images create "redash-091-b1377-bq" --source-uri gs://redash-images/redash.0.9.1.b1377-bq.tar.gz
+
+Note that you need to launch this instance with BigQuery access:
+
+.. code:: bash
+
+    $ gcloud compute instances create <your_instance_name> --image redash-091-b1377-bq --scopes storage-ro,bigquery
+
+(the same can be done from the web interface, just make sure to enable
+BigQuery access)
+
+Please note that currently the Google Compute Engine images are for version 0.9.1. After creating the instance, please
+run the :doc:`upgrade process <upgrade>` and then proceed to `"Setup" <#setup>`__.
+
+Docker Compose
+------
+
+1. Make sure you have a Docker machine up and running.
+2. Make sure your current working directory is the root of this GitHub repository.
+3. Run ``docker-compose up postgres``.
+4. Run ``./setup/docker/create_database.sh``. This will access the postgres container and set up the database.
+5. Run ``docker-compose up``
+6. Run ``docker-machine ls``, take note of the ip for the Docker machine you are using, and open the web browser.
+7. Visit that Docker machine IP at port 80, and you should see a Re:dash login screen.
+
+Now proceed to `"Setup" <#setup>`__.
+
+
+Heroku
+------
+
+Due to the nature of Heroku deployments, upgrading to a newer version of Redash
+requires performing the steps outlined on the `"How to Upgrade" <http://docs.redash.io/en/latest/upgrade.html>`__ page.
+
+1. Install `Heroku CLI <https://toolbelt.heroku.com/>`__.
+
+2. Create Heroku App::
+
+    $ heroku apps:create <app name>
+
+2. Set application buildpacks::
+
+    $ heroku buildpacks:set heroku/python
+    $ heroku buildpacks:add --index 1 heroku/nodejs
+
+3. Add Postgres and Redis addons::
+
+    $ heroku addons:create heroku-postgresql:hobby-dev
+    $ heroku addons:create heroku-redis:hobby-dev
+
+4. Update the cookie secret (**Important** otherwise anyone can sign new cookies and impersonate users. You may be able to run the command ``pwgen 32 -1`` to generate a random string)::
+
+    $ heroku config:set REDASH_COOKIE_SECRET='<create a secret token and put here>'
+
+5. Push the repository to Heroku::
+
+    $ git push heroku master
+
+6. Create database tables::
+
+    $ heroku run ./manage.py database create_tables
+
+7. Create admin user::
+
+    $ heroku run ./manage.py users create --admin "Admin" admin
+
+7. Start worker process::
+
+    $ heroku ps:scale worker=1
+
+
+Other
+-----
+
+Download the provision script and run it on your machine. Note that:
+
+1. You need to run the script as root.
+2. It was tested only on Ubuntu 12.04, Ubuntu 14.04 and Debian Wheezy.
+3. It's designed to run on a "clean" machine. If you're running this script on a machine that is used for other purposes, you might want to tweak it to your needs (like removing the ``apt-get dist-upgrade`` call at the beginning of it).
+
+Setup
+=====
+
+Once you created the instance with either the image or the script, you
+should have a running Re:dash instance with everything you need to get
+started . Re:dash should be available using the server IP or DNS name
+you assigned to it. You can point your browser to this address, and login
+with the user "admin" (password: "admin"). But to make it useful, there are
+a few more steps that you need to manually do to complete the setup:
+
+First ssh to your instance and change directory to ``/opt/redash``. If
+you're using the GCE image, switch to root (``sudo su``).
+
+Users & Google Authentication setup
+-----------------------------------
+
+Most of the settings you need to edit are in the ``/opt/redash/.env``
+file.
+
+1. Update the cookie secret (important! otherwise anyone can sign new
+   cookies and impersonate users): change "veryverysecret" in the line:
+   ``export REDASH_COOKIE_SECRET=veryverysecret`` to something else (you
+   can run the command ``pwgen 32 -1`` to generate a random string).
+
+2. By default we create an admin user with the password "admin". You
+   can change this password opening the: ``/users/me#password`` page after
+   logging in as admin.
+
+3. If you want to use Google OAuth to authenticate users, you need to
+   create a Google Developers project (see :doc:`instructions </misc/google_developers_project>`)
+   and then add the needed configuration in the ``.env`` file:
+
+.. code::
+
+   export REDASH_GOOGLE_CLIENT_ID=""
+   export REDASH_GOOGLE_CLIENT_SECRET=""
+
+
+4. Configure the domain(s) you want to allow to use with Google Apps, by running the command:
+
+.. code::
+
+   cd /opt/redash/current
+   sudo -u redash bin/run ./manage.py org set_google_apps_domains {{domains}}
+
+
+If you're passing multiple domains, separate them with commas.
+
+
+5. Restart the web server to apply the configuration changes:
+   ``sudo supervisorctl restart redash_server``.
+
+6. Once you have Google OAuth enabled, you can login using your Google
+   Apps account. If you want to grant admin permissions to some users,
+   you can do this by adding them to the admin group (from ``/groups`` page).
+
+7. If you don't use Google OAuth or just need username/password logins,
+   you can create additional users by opening the ``/users/new`` page.
+
+Datasources
+-----------
+
+To make Re:dash truly useful, you need to setup your data sources in it. Browse to ``/data_sources`` on your instance,
+to create new data source connection.
+
+See :doc:`documentation </datasources>` for the different options.
+Your instance comes ready with dependencies needed to setup supported sources.
+
+Mail Configuration
+------------------
+
+For the system to be able to send emails (for example when alerts trigger), you need to set the mail server to use and the
+host name of your Re:dash server. If you're using one of our images, you can do this by editing the `.env` file:
+
+.. code::
+
+   # Note that not all values are required, as they have default values.
+
+   export REDASH_MAIL_SERVER="" # default: localhost
+   export REDASH_MAIL_PORT="" # default: 25
+   export REDASH_MAIL_USE_TLS="" # default: False
+   export REDASH_MAIL_USE_SSL="" # default: False
+   export REDASH_MAIL_USERNAME="" # default: None
+   export REDASH_MAIL_PASSWORD="" # default: None
+   export REDASH_MAIL_DEFAULT_SENDER="" # Email address to send from
+
+   export REDASH_HOST="" # base address of your Re:dash instance, for example: "https://demo.redash.io"
+
+- Note that not all values are required, as there are default values.
+- It's recommended to use some mail service, like `Amazon SES <https://aws.amazon.com/ses/>`__, `Mailgun <http://www.mailgun.com/>`__
+  or `Mandrill <http://mandrillapp.com>`__ to send emails to ensure deliverability.
+
+To test email configuration, you can run `bin/run ./manage.py send_test_mail` (from `/opt/redash/current`).
+
+How to upgrade?
+---------------
+
+It's recommended to upgrade once in a while your Re:dash instance to
+benefit from bug fixes and new features. See :doc:`here </upgrade>` for full upgrade
+instructions (including Fabric script).
+
+Configuration
+-------------
+
+For a full list of environment variables, see :doc:`the settings page </settings>`.
+
+Notes
+=====
+
+-  If this is a production setup, you should enforce HTTPS and make sure
+   you set the cookie secret (see :doc:`instructions </misc/ssl>`).

docs/upgrade.rst

@@ -0,0 +1,37 @@
+How to Upgrade
+##############
+
+It's recommended to upgrade your Re:dash instance once there are new
+releases, to benefit from new features and bug fixes. The upgrade
+process is relatively simple, and assuming you used one of the base
+images we provide, you can just use the
+`Fabric <http://www.fabfile.org/>`__ script provided here:
+https://gist.github.com/arikfr/440d1403b4aeb76ebaf8.
+
+How to run the Fabric script
+============================
+
+1. Install Fabric: ``pip install fabric requests`` (needed only once)
+2. Download the ``fabfile.py`` from the gist.
+3. Run the script:
+   ``fab -H{your Re:dash host} -u{the ssh user for this host} -i{path to key file for passwordless login} deploy_latest_release``
+
+   ``-i`` is optional and it is only needed in case you're using private-key based authentication (and didn't add the key file to your authentication agent or set its path in your SSH config).
+
+What the Fabric script does
+===========================
+
+Even if you didn't use the image, it's very likely you can reuse most of
+this script with small modifications. What this script does is:
+
+1. Find the URL of the latest release tarball (from `GitHub releases
+   page <http://github.com/getredash/redash/releases>`__).
+2. Download it.
+3. Create new directory for this version (for example:
+   ``/opt/redash/redash.0.5.0.b685``).
+4. Unpack that (``tar -C {dir} -xvf {tarball path}``).
+5. Link ``/opt/redash/.env`` file into this directory.
+6. Apply any new migrations.
+7. Link ``/opt/redash/current`` to new version.
+8. Install any new requirements - ``sudo pip install -r requirements.txt`` 
+9. Restart web server and celery workers.

docs/usage.rst

@@ -0,0 +1,10 @@
+Usage
+=====
+
+.. toctree::
+    :maxdepth: 2
+    :glob:
+
+    usage/*
+
+

docs/usage/elasticsearch_querying.rst

@@ -0,0 +1,72 @@
+ElasticSearch: Querying
+#######################
+
+ElasticSearch currently supports only simple Lucene style queries (like
+Kibana but without the aggregation).
+
+Full blown JSON based ElasticSearch queries (including aggregations)
+will be added later.
+
+Simple query example:
+=====================
+
+-  Query the index named "twitter"
+-  Filter by "user:kimchy"
+-  Return the fields: "@timestamp", "tweet" and "user"
+-  Return up to 15 results
+-  Sort by @timestamp ascending
+
+.. code:: json
+
+    {
+        "index" : "twitter",
+        "query" : "user:kimchy",
+        "fields" : ["@timestamp", "tweet", "user"],
+        "size" : 15,
+        "sort" : "@timestamp:asc"
+    }
+
+Simple query on a logstash ElasticSearch instance:
+==================================================
+
+-  Query the index named "logstash-2015.04.\*" (in this case its all of
+   April 2015)
+-  Filter by type:events AND eventName:UserUpgrade AND channel:selfserve
+-  Return fields: "@timestamp", "userId", "channel", "utm\_source",
+   "utm\_medium", "utm\_campaign", "utm\_content"
+-  Return up to 250 results
+-  Sort by @timestamp ascending
+
+.. code:: json
+
+    {
+        "index" : "logstash-2015.04.*",
+        "query" : "type:events AND eventName:UserUpgrade AND channel:selfserve",
+        "fields" : ["@timestamp", "userId", "channel", "utm_source", "utm_medium", "utm_campaign", "utm_content"],
+        "size" : 250,
+        "sort" : "@timestamp:asc"
+    }
+
+Simple query on a ElasticSearch instance:
+==================================================
+
+
+- Query the index named "twitter"
+- Filter by user equal "kimchy"
+- Return the fields: "@timestamp", "tweet" and "user"
+- Return up to 15 results
+- Sort by @timestamp ascending
+
+.. code:: json
+
+    {
+        "index" : "twitter",
+        "query" : {
+        "match": {
+            "user" : "kimchy"
+            }
+        },
+        "fields" : ["@timestamp", "tweet", "user"],
+        "size" : 15,
+        "sort" : "@timestamp:asc"
+    }

docs/usage/maintenance.rst

@@ -0,0 +1,112 @@
+Ongoing Maintanence and Basic Operations
+########################################
+
+Configuration and logs
+======================
+
+The supervisor config can be found in
+``/opt/redash/supervisord/supervisord.conf``.
+
+There you can see the names of its programs (``redash_celery``,
+``redash_server``) and the location of their logs.
+
+Restart
+=======
+
+Restarting the Web Server
+-------------------------
+
+``sudo supervisorctl restart redash_server``
+
+Restarting Celery Workers
+-------------------------
+
+``sudo supervisorctl restart redash_celery``
+
+Restarting Celery Workers & the Queries Queue
+---------------------------------------------
+
+In case you are handling a problem, and you need to stop the currently
+running queries and reset the queue, follow the steps below.
+
+1. Stop celery: ``sudo supervisorctl stop redash_celery`` (celery might
+   take some time to stop, if it's in the middle of running a query)
+
+2. Flush redis: ``redis-cli flushall``.
+
+3. Start celery: ``sudo supervisorctl start redash_celery``
+
+Changing the Number of Workers
+==============================
+
+By default, Celery will start a worker per CPU core. Because most of
+Re:dash's tasks are IO bound, the real limit for number of workers you
+can use depends on the amount of memory your machine has. It's
+recommended to increase number of workers, to support more concurrent
+queries.
+
+1. Open the supervisord configuration file:
+   ``/opt/redash/supervisord/supervisord.conf``
+
+2. Edit the ``[program:redash_celery]`` section and add to the *command*
+   value, the param "-c" with the number of concurrent workers you need.
+
+3. Restart supervisord to apply new configuration:
+   ``sudo /etc/init.d/redash_supervisord restart``.
+
+DB
+==
+
+Backup Re:dash's DB:
+--------------------
+
+Uncompressed backup: ``sudo -u redash pg_dump > backup_filename.sql``
+
+Compressed backup: ``sudo -u redash pg_dump redash | gzip > backup_filename.gz``
+
+Version
+=======
+
+See current version:
+
+``bin/run ./manage.py version``
+
+Monitoring
+==========
+Re:dash ships by default with a HTTP handler that gives you useful information about the
+health of your application. The endpoint is ``/status.json`` and requires a super admin
+API key to be given if you're not already logged in. This API key can be obtained from 
+the dedicated tab in your profile.
+
+You'll find below an example output of this endpoint:
+
+.. code-block:: json 
+
+    {
+      "dashboards_count": 30,
+      "manager": {
+        "last_refresh_at": "1465392784.433638",
+        "outdated_queries_count": 1,
+        "query_ids": "[34]",
+        "queues": {
+          "queries": {
+            "data_sources": "Redshift data, re:dash metadata, MySQL data, MySQL read-only, Redshift read-only",
+            "size": 1
+          },
+          "scheduled_queries": {
+            "data_sources": "Redshift data, re:dash metadata, MySQL data, MySQL read-only, Redshift read-only",
+            "size": 0
+          }
+        }
+      },
+      "queries_count": 204,
+      "query_results_count": 11161,
+      "redis_used_memory": "6.09M",
+      "unused_query_results_count": 32,
+      "version": "0.10.0+b1774",
+      "widgets_count": 176,
+      "workers": []
+    }
+
+    
+If you plan to hit this endpoint without being logged in, you'll need to provide your API key as a query parameter. Example endpoint with an API key: ``/status.json?api_key=fooBarqsLlGJQIs3maPErUxKuxwWGIpDXoSzQsx7xdv``

docs/usage/mongodb_querying.rst

@@ -0,0 +1,74 @@
+MongoDB: Querying
+#################
+
+Simple query example:
+=====================
+
+.. code:: json
+
+     {
+         "collection" : "my_collection",
+         "query" : {
+             "date" : {
+                 "$gt" : "ISODate(\"2015-01-15 11:41\")",
+             },
+             "type" : 1
+         },
+         "fields" : {
+             "_id" : 1,
+             "name" : 2
+         },
+         "sort" : [
+            {
+                 "name" : "date",
+                 "direction" : -1
+            }
+         ]
+     }
+
+Live example on the demo instance:
+http://demo.redash.io/queries/394/source.
+
+Aggregation
+===========
+
+Uses a syntax similar to the one used in PyMongo, however to support the
+correct order of sorting, it uses a regular list for the "$sort"
+operation that converts into a SON (sorted dictionary) object before
+execution.
+
+Aggregation query example:
+
+.. code:: json
+
+     {
+         "collection" : "things",
+         "aggregate" : [
+             {
+                 "$unwind" : "$tags"
+             },
+             {
+                 "$group" : {
+                     "_id" : "$tags",
+                     "count" : { "$sum" : 1 }
+                 }
+             },
+             {
+                 "$sort" : [
+                     {
+                         "name" : "count",
+                         "direction" : -1
+                     },
+                     {
+                         "name" : "_id",
+                         "direction" : -1
+                     }
+                 ]
+             }
+         ]
+     }
+
+Live examples on the demo instance:
+
+1. http://demo.redash.io/queries/393/source
+2. http://demo.redash.io/queries/387/source

docs/usage/permissions.rst

@@ -0,0 +1,34 @@
+Permissions Model
+#################
+
+In version 0.9.0 we introduced a new permissions model based on groups. Each user by default joins the ``Default`` group, but
+can be a member of any number of groups.
+
+Group membership defines the actions you're allowed to take (although currently there is no UI to edit group action permissions),
+but also what data sources you have access to (for this we have UI).
+
+How does it work?
+=================
+
+* Each user belongs to one or more groups. By default each user joins the ``Default`` group. So the common
+  data sources, should be associated with this group.
+* Each data source will be associated with one or more groups. Each connection to a group will define,
+  whether this group has full access to this data source (view existing queries and run new ones) or view only access,
+  which allows only viewing existing queries and results.
+* Any dashboard can contain visualizations from any data source (as long as the creating user has access to them). When
+  a user who doesn't have access to some visualization (because he doesn't have access to the data source) opens a dashboard,
+  he will see that there is a visualization there but won't see the details.
+
+.. figure:: https://cloud.githubusercontent.com/assets/71468/12002946/dc5032ca-ab16-11e5-90e7-aae9234a596b.png
+
+   Dashboard widget with a visualization the user doesn't have access to.
+
+If a user has access to at least one widget on a dashboard, they can see this dashboard in the list of all dashboards.
+
+
+What if I want to limit the user to only some tables?
+=====================================================
+
+The idea is to leverage your database's security model, and hence create a user with access to the tables/columns you
+want to give access to. Create a data source that is using this user and then associate it with a group of users who need
+this level of access.

docs/usage/special_features.rst

@@ -0,0 +1,44 @@
+Special Features
+#################
+
+Re:dash has a lot of very useful features and most of them can be found easily when using the UI. This page features the less well-known ones.
+
+Queries
+========
+It is possible to have filters for query results and visualizations. Thanks to filters, you can restrain the result to a certain or multiple values. Filters are enabled by following a naming convention for columns.
+
+If you want to focus only on a specific value, you will need to alias your column to ``<columnName>::filter``. Here is an example:
+
+.. code:: sql
+
+    select action as "action::filter", count (0) as "actions count"
+    from events
+    group by action
+
+You can see this query and the rendered UI `here <http://demo.redash.io/queries/143/source#table>`_.
+
+If you are interested in multi filters (meaning that you can select multiple values), you will need to alias your column to ``<columnName>::multi-filter``. Here is an example:
+
+.. code:: sql
+
+    select action as "action::multi-filter", count (0) as "actions count"
+    from events
+    group by action
+
+You can see this query and the rendered UI `here <http://demo.redash.io/queries/144/source#table>`_.
+
+Note that you can use ``__filter`` or ``__multiFilter`` if your database doesn't support ``::`` in column names (such as BigQuery).
+
+Dashboards
+==========
+It is possible to group multiple dashboards in the dashboards menu. To do this, you need to follow a naming convention by using a column (``:``) to separate the dashboard group and the actual dashboard name. For example, if you name 2 dashboards ``Foo: Bar`` and ``Foo: Baz``, they will be grouped under the ``Foo`` namespace in the dropdown menu.
+
+If you've got queries that have some filters and you want to apply filters at the dashboard level (that apply to all queries), you need to set a flag. Can you do it through the admin interface at ``/admin/dashboard`` or you can do it by manually setting the column ``dashboard_filters_enabled`` of the table ``dashboards`` to ``TRUE`` in the Re:dash database.
+
+Exporting query results to CSV or JSON
+======================================
+Query results can be automatically exported to CSV or JSON by using your API key. Your API key can be found when viewing your profile, from the top right menu in the navigation bar.
+
+The format of the URL is the following: ``https://<redash_domain>/api/queries/<query_id>/results.(csv|json)?api_key=<your_api_key>``. Here is a working example: `<http://demo.redash.io/api/queries/63/results.json?api_key=874fcd93ce4b6ef87a9aad41c712bcd5d17cdc8f>`_.
+
+Using this URL you can easily import query results directly into Google Spreadsheets, using the ``importdata`` function. For example: ``=importdata("...")``.

gulpfile.js

@@ -0,0 +1,131 @@
+// Generated on 2016-02-09 using generator-angular 0.15.1
+'use strict';
+
+var gulp = require('gulp');
+var $ = require('gulp-load-plugins')();
+var lazypipe = require('lazypipe');
+var rimraf = require('rimraf');
+var wiredep = require('wiredep').stream;
+var runSequence = require('run-sequence');
+var map = require('lodash.map');
+
+var yeoman = {
+  app: 'rd_ui/app',
+  dist: 'rd_ui/dist'
+};
+
+function applyAppPath(p) {
+  if (typeof p === 'string') {
+    return yeoman.app + p;
+  } else {
+    return map(p, function (path) {
+      return applyAppPath(path);
+    });
+  }
+}
+
+var paths = {
+  scripts: [yeoman.app + '/scripts/**/*.js'],
+  styles: [yeoman.app + '/styles/**/*.css'],
+  views: {
+    main: applyAppPath(['/index.html', '/vendor_scripts.html', '/login.html', '/embed.html', '/public.html', '/app_layout.html', '/signed_out_layout.html']),
+    files: [yeoman.app + '/views/**/*.html']
+  }
+};
+
+////////////////////////
+// Reusable pipelines //
+////////////////////////
+
+var lintScripts = lazypipe()
+  .pipe($.jshint, '.jshintrc')
+  .pipe($.jshint.reporter, 'jshint-stylish');
+
+var styles = lazypipe()
+  .pipe($.autoprefixer, 'last 1 version')
+  .pipe(gulp.dest, '.tmp/styles');
+
+///////////
+// Tasks //
+///////////
+
+gulp.task('styles', function () {
+  return gulp.src(paths.styles)
+    .pipe(styles());
+});
+
+gulp.task('lint:scripts', function () {
+  return gulp.src(paths.scripts)
+    .pipe(lintScripts());
+});
+
+gulp.task('clean:tmp', function (cb) {
+  rimraf('./.tmp', cb);
+});
+
+// inject bower components
+gulp.task('bower', function () {
+  return gulp.src(paths.views.main)
+    .pipe(wiredep({
+      directory: yeoman.app + '/bower_components',
+      ignorePath: '..'
+    }))
+  .pipe(gulp.dest(yeoman.app + '/views'));
+});
+
+///////////
+// Build //
+///////////
+
+gulp.task('clean:dist', function (cb) {
+  rimraf('./dist', cb);
+});
+
+gulp.task('client:build', ['html', 'styles'], function () {
+  var jsFilter = $.filter('**/*.js');
+  var cssFilter = $.filter('**/*.css');
+
+  return gulp.src(paths.views.main)
+    .pipe($.useref({searchPath: [yeoman.app, '.tmp']}))
+    .pipe(jsFilter)
+    .pipe($.ngAnnotate())
+    .pipe($.uglify())
+    .pipe(jsFilter.restore())
+    .pipe($.print())
+    .pipe(cssFilter)
+    .pipe($.minifyCss({cache: true}))
+    .pipe(cssFilter.restore())
+    .pipe(new $.revAll({dontRenameFile: ['.html'], dontUpdateReference: ['vendor_scripts.html', 'app_layout.html', 'signed_out_layout.html']}).revision())
+    .pipe(gulp.dest(yeoman.dist));
+});
+
+gulp.task('html', function () {
+  return gulp.src(yeoman.app + '/views/**/*')
+    .pipe(gulp.dest(yeoman.dist + '/views'));
+});
+
+gulp.task('images', function () {
+  return gulp.src(yeoman.app + '/images/**/*')
+    .pipe($.cache($.imagemin({
+        optimizationLevel: 5,
+        progressive: true,
+        interlaced: true
+    })))
+    .pipe(gulp.dest(yeoman.dist + '/images'));
+});
+
+gulp.task('copy:extras', function () {
+  return gulp.src(applyAppPath(['/*/.*', '/google_login.png', '/favicon.ico', '/robots.txt']), { dot: true })
+    .pipe(gulp.dest(yeoman.dist));
+});
+
+gulp.task('copy:fonts', function () {
+  return gulp.src(applyAppPath(['/fonts/**/*', '/bower_components/font-awesome/fonts/*', '/bower_components/material-design-iconic-font/dist/fonts/*']))
+    .pipe(gulp.dest(yeoman.dist + '/fonts'));
+});
+
+gulp.task('build', ['clean:dist'], function () {
+  runSequence(['images', 'copy:extras', 'copy:fonts', 'client:build']);
+});
+
+gulp.task('default', ['build']);

manage.py

@@ -0,0 +1,59 @@
+#!/usr/bin/env python
+"""
+CLI to manage redash.
+"""
+import json
+
+from flask_script import Manager
+
+from redash import settings, models, __version__
+from redash.wsgi import app
+from redash.cli import users, groups, database, data_sources, organization
+from redash.monitor import get_status
+
+manager = Manager(app)
+manager.add_command("database", database.manager)
+manager.add_command("users", users.manager)
+manager.add_command("groups", groups.manager)
+manager.add_command("ds", data_sources.manager)
+manager.add_command("org", organization.manager)
+
+
+
+@manager.command
+def version():
+    """Displays re:dash version."""
+    print __version__
+
+@manager.command
+def status():
+    print json.dumps(get_status(), indent=2)
+
+@manager.command
+def runworkers():
+    """Start workers (deprecated)."""
+    print "** This command is deprecated. Please use Celery's CLI to control the workers. **"
+
+
+@manager.shell
+def make_shell_context():
+    from redash.models import db
+    return dict(app=app, db=db, models=models)
+
+
+@manager.command
+def check_settings():
+    """Show the settings as re:dash sees them (useful for debugging)."""
+    for name, item in settings.all_settings().iteritems():
+        print "{} = {}".format(name, item)
+
+@manager.command
+def send_test_mail():
+    from redash import mail
+    from flask_mail import Message
+
+    mail.send(Message(subject="Test Message from re:dash", recipients=[settings.MAIL_DEFAULT_SENDER], body="Test message."))
+
+
+if __name__ == '__main__':
+    manager.run()

migrations/0001_allow_delete_query.py

@@ -0,0 +1,15 @@
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+from redash import models
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        migrate(
+            migrator.add_column('queries', 'is_archived', models.Query.is_archived)
+        )
+
+    db.close_db(None)

migrations/0002_fix_timestamp_fields.py

@@ -0,0 +1,20 @@
+from redash.models import db
+
+if __name__ == '__main__':
+    db.connect_db()
+    columns = (
+        ('activity_log', 'created_at'),
+        ('dashboards', 'created_at'),
+        ('data_sources', 'created_at'),
+        ('events', 'created_at'),
+        ('groups', 'created_at'),
+        ('queries', 'created_at'),
+        ('widgets', 'created_at'),
+        ('query_results', 'retrieved_at')
+    )
+
+    with db.database.transaction():
+        for column in columns:
+            db.database.execute_sql("ALTER TABLE {} ALTER COLUMN {} TYPE timestamp with time zone;".format(*column))
+
+    db.close_db(None)

migrations/0003_update_data_source_config.py

@@ -0,0 +1,91 @@
+import json
+import jsonschema
+from jsonschema import ValidationError
+
+from redash import query_runner
+from redash.models import DataSource
+
+
+def validate_configuration(query_runner_type, configuration_json):
+    query_runner_class = query_runner.query_runners.get(query_runner_type, None)
+    if query_runner_class is None:
+        return False
+
+    try:
+        if isinstance(configuration_json, basestring):
+            configuration = json.loads(configuration_json)
+        else:
+            configuration = configuration_json
+        jsonschema.validate(configuration, query_runner_class.configuration_schema())
+    except (ValidationError, ValueError):
+        return False
+
+    return True
+
+def update(data_source):
+    print "[%s] Old options: %s" % (data_source.name, data_source.options)
+
+    if validate_configuration(data_source.type, data_source.options):
+        print "[%s] configuration already valid. skipping." % data_source.name
+        return
+
+    if data_source.type == 'pg':
+        values = data_source.options.split(" ")
+        configuration = {}
+        for value in values:
+            k, v = value.split("=", 1)
+            configuration[k] = v
+            if k == 'port':
+                configuration[k] = int(v)
+
+        data_source.options = json.dumps(configuration)
+
+    elif data_source.type == 'mysql':
+        mapping = {
+            'Server': 'host',
+            'User': 'user',
+            'Pwd': 'passwd',
+            'Database': 'db'
+        }
+
+        values = data_source.options.split(";")
+        configuration = {}
+        for value in values:
+            k, v = value.split("=", 1)
+            configuration[mapping[k]] = v
+        data_source.options = json.dumps(configuration)
+
+    elif data_source.type == 'graphite':
+        old_config = json.loads(data_source.options)
+
+        configuration = {
+            "url": old_config["url"]
+        }
+
+        if "verify" in old_config:
+            configuration['verify'] = old_config['verify']
+
+        if "auth" in old_config:
+            configuration['username'], configuration['password'] = old_config["auth"]
+
+        data_source.options = json.dumps(configuration)
+
+    elif data_source.type == 'url':
+        data_source.options = json.dumps({"url": data_source.options})
+
+    elif data_source.type == 'script':
+        data_source.options = json.dumps({"path": data_source.options})
+
+    elif data_source.type == 'mongo':
+        data_source.type = 'mongodb'
+
+    else:
+        print "[%s] No need to convert type of: %s" % (data_source.name, data_source.type)
+
+    print "[%s] New options: %s" % (data_source.name, data_source.options)
+    data_source.save(only=data_source.dirty_fields)
+
+
+if __name__ == '__main__':
+    for data_source in DataSource.select(DataSource.id, DataSource.name, DataSource.type, DataSource.options):
+        update(data_source)

migrations/0004_allow_null_in_event_user.py

@@ -0,0 +1,12 @@
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        migrate(
+            migrator.drop_not_null('events', 'user_id')
+        )

migrations/0005_add_updated_at.py

@@ -0,0 +1,25 @@
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+from redash import models
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        migrate(
+            migrator.add_column('queries', 'updated_at', models.Query.updated_at),
+            migrator.add_column('dashboards', 'updated_at', models.Dashboard.updated_at),
+            migrator.add_column('widgets', 'updated_at', models.Widget.updated_at),
+            migrator.add_column('users', 'created_at', models.User.created_at),
+            migrator.add_column('users', 'updated_at', models.User.updated_at),
+            migrator.add_column('visualizations', 'created_at', models.Visualization.created_at),
+            migrator.add_column('visualizations', 'updated_at', models.Visualization.updated_at)
+        )
+
+        db.database.execute_sql("UPDATE queries SET updated_at = created_at;")
+        db.database.execute_sql("UPDATE dashboards SET updated_at = created_at;")
+        db.database.execute_sql("UPDATE widgets SET updated_at = created_at;")
+
+    db.close_db(None)

migrations/0006_queries_last_edit_by.py

@@ -0,0 +1,17 @@
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+from redash import models
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        migrate(
+            migrator.add_column('queries', 'last_modified_by_id', models.Query.last_modified_by)
+        )
+
+        db.database.execute_sql("UPDATE queries SET last_modified_by_id = user_id;")
+
+    db.close_db(None)

migrations/0007_add_schedule_to_queries.py

@@ -0,0 +1,21 @@
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+from redash import models
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        migrate(
+            migrator.add_column('queries', 'schedule', models.Query.schedule),
+        )
+
+        db.database.execute_sql("UPDATE queries SET schedule = ttl WHERE ttl > 0;")
+
+        migrate(
+            migrator.drop_column('queries', 'ttl')
+        )
+
+    db.close_db(None)

migrations/0008_make_ds_name_unique.py

@@ -0,0 +1,20 @@
+from redash.models import db
+
+if __name__ == '__main__':
+    db.connect_db()
+
+    with db.database.transaction():
+        # Make sure all data sources names are unique.
+        db.database.execute_sql("""
+        UPDATE data_sources
+        SET name = new_names.name
+        FROM (
+            SELECT id, name || ' ' || id as name
+            FROM (SELECT id, name, rank() OVER (PARTITION BY name ORDER BY created_at ASC) FROM data_sources) ds WHERE rank > 1
+        ) AS new_names
+        WHERE data_sources.id = new_names.id;
+        """)
+        # Add unique constraint on data_sources.name.
+        db.database.execute_sql("ALTER TABLE data_sources ADD CONSTRAINT unique_name UNIQUE (name);")
+
+    db.close_db(None)

migrations/0009_add_api_key_to_user.py

@@ -0,0 +1,24 @@
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+from redash import models
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        column = models.User.api_key
+        column.null = True
+        migrate(
+            migrator.add_column('users', 'api_key', models.User.api_key),
+        )
+
+        for user in models.User.select(models.User.id, models.User.api_key):
+            user.save(only=user.dirty_fields)
+
+        migrate(
+            migrator.add_not_null('users', 'api_key')
+        )
+
+    db.close_db(None)

migrations/0010_allow_deleting_datasources.py

@@ -0,0 +1,14 @@
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        migrate(
+            migrator.drop_not_null('queries', 'data_source_id'),
+        )
+
+    db.close_db(None)

migrations/0010_create_alerts.py

@@ -0,0 +1,8 @@
+from redash.models import db, Alert, AlertSubscription
+
+if __name__ == '__main__':
+    with db.database.transaction():
+        Alert.create_table()
+        AlertSubscription.create_table()
+
+    db.close_db(None)

migrations/0011_migrate_bigquery_to_json.py

@@ -0,0 +1,44 @@
+from base64 import b64encode
+import json
+from redash.models import DataSource
+
+
+def convert_p12_to_pem(p12file):
+    from OpenSSL import crypto
+    with open(p12file, 'rb') as f:
+        p12 = crypto.load_pkcs12(f.read(), "notasecret")
+
+    return crypto.dump_privatekey(crypto.FILETYPE_PEM, p12.get_privatekey())
+
+if __name__ == '__main__':
+
+    for ds in DataSource.select(DataSource.id, DataSource.type, DataSource.options):
+
+        if ds.type == 'bigquery':
+            options = json.loads(ds.options)
+
+            if 'jsonKeyFile' in options:
+                continue
+
+            new_options = {
+                'projectId': options['projectId'],
+                'jsonKeyFile': b64encode(json.dumps({
+                    'client_email': options['serviceAccount'],
+                    'private_key': convert_p12_to_pem(options['privateKey'])
+                }))
+            }
+
+            ds.options = json.dumps(new_options)
+            ds.save(only=ds.dirty_fields)
+        elif ds.type == 'google_spreadsheets':
+            options = json.loads(ds.options)
+            if 'jsonKeyFile' in options:
+                continue
+
+            with open(options['credentialsFilePath']) as f:
+                new_options = {
+                    'jsonKeyFile': b64encode(f.read())
+                }
+
+            ds.options = json.dumps(new_options)
+            ds.save(only=ds.dirty_fields)

migrations/0012_add_list_users_permission.py

@@ -0,0 +1,7 @@
+from redash import models
+
+if __name__ == '__main__':
+
+    default_group = models.Group.select(models.Group.id, models.Group.permissions).where(models.Group.name=='default').first()
+    default_group.permissions.append('list_users')
+    default_group.save(only=[models.Group.permissions])

migrations/0013_update_counter_options.py

@@ -0,0 +1,23 @@
+import json
+from redash import models
+
+if __name__ == '__main__':
+    for vis in models.Visualization.select():
+        if vis.type == 'COUNTER':
+            options = json.loads(vis.options)
+            print "Before: ", options
+            if 'rowNumber' in options and options['rowNumber'] is not None:
+                options['rowNumber'] += 1
+            else:
+                options['rowNumber'] = 1
+
+            if 'counterColName' not in options:
+                options['counterColName'] = 'counter'
+
+            if 'targetColName' not in options:
+                options['targetColName'] = 'target'
+            options['targetRowNumber'] = options['rowNumber']
+
+            print "After: ", options
+            vis.options = json.dumps(options)
+            vis.save()

migrations/0014_add_alert_rearm_seconds.py

@@ -0,0 +1,21 @@
+import peewee
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+from redash import models
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    cursor = db.database.execute_sql("SELECT column_name FROM information_schema.columns WHERE table_name='alerts' and column_name='rearm';")
+    if cursor.rowcount > 0:
+        print "Column exists. Skipping."
+        exit()
+
+    with db.database.transaction():
+        migrate(
+            migrator.add_column('alerts', 'rearm', models.Alert.rearm),
+        )
+
+    db.close_db(None)

migrations/0014_migrate_existing_es_to_kibana.py

@@ -0,0 +1,10 @@
+__author__ = 'lior'
+
+from redash.models import DataSource
+
+if __name__ == '__main__':
+
+    for ds in DataSource.select(DataSource.id, DataSource.type):
+        if ds.type == 'elasticsearch':
+            ds.type = 'kibana'
+            ds.save(only=ds.dirty_fields)

migrations/0015_add_schedule_query_permission.py

@@ -0,0 +1,6 @@
+from redash import models
+
+if __name__ == '__main__':
+    default_group = models.Group.select(models.Group.id, models.Group.permissions).where(models.Group.name=='default').first()
+    default_group.permissions.append('schedule_query')
+    default_group.save(only=[models.Group.permissions])

migrations/0016_add_alert_subscriber.py

@@ -0,0 +1,9 @@
+from redash.models import db, Alert, AlertSubscription
+
+if __name__ == '__main__':
+    with db.database.transaction():
+        # There was an AWS/GCE image created without this table, to make sure this exists we run this migration.
+        if not AlertSubscription.table_exists():
+            AlertSubscription.create_table()
+
+    db.close_db(None)

migrations/0016_drop_tables_from_group.py

@@ -0,0 +1,14 @@
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+from redash.models import db
+
+if __name__ == '__main__':
+    db.connect_db()
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        migrate(
+            migrator.drop_column('groups', 'tables')
+        )
+
+    db.close_db(None)

migrations/0017_add_organization.py

@@ -0,0 +1,34 @@
+from redash.models import db, Organization, Group
+from redash import settings
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+if __name__ == '__main__':
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        Organization.create_table()
+
+        default_org = Organization.create(name="Default", slug='default', settings={
+            Organization.SETTING_GOOGLE_APPS_DOMAINS: list(settings.GOOGLE_APPS_DOMAIN)
+        })
+
+        column = Group.org
+        column.default = default_org
+
+        migrate(
+            migrator.add_column('groups', 'org_id', column),
+            migrator.add_column('events', 'org_id', column),
+            migrator.add_column('data_sources', 'org_id', column),
+            migrator.add_column('users', 'org_id', column),
+            migrator.add_column('dashboards', 'org_id', column),
+            migrator.add_column('queries', 'org_id', column),
+            migrator.add_column('query_results', 'org_id', column),
+        )
+
+        # Change the uniqueness constraint on user email to be (org, email):
+        migrate(
+            migrator.drop_index('users', 'users_email'),
+            migrator.add_index('users', ('org_id', 'email'), unique=True)
+        )
+
+    db.close_db(None)

migrations/0018_add_groups_refs.py

@@ -0,0 +1,44 @@
+from collections import defaultdict
+from redash.models import db, DataSourceGroup, DataSource, Group, Organization, User
+from playhouse.migrate import PostgresqlMigrator, migrate
+import peewee
+
+if __name__ == '__main__':
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        # Add type to groups
+        migrate(
+            migrator.add_column('groups', 'type', Group.type)
+        )
+
+        for name in ['default', 'admin']:
+            group = Group.get(Group.name==name)
+            group.type = Group.BUILTIN_GROUP
+            group.save()
+
+        # Create association table between data sources and groups
+        DataSourceGroup.create_table()
+
+        # add default to existing data source:
+        default_org = Organization.get_by_id(1)
+        default_group = Group.get(Group.name=="default")
+        for ds in DataSource.all(default_org):
+            DataSourceGroup.create(data_source=ds, group=default_group)
+
+        # change the groups list on a user object to be an ids list
+        migrate(
+            migrator.rename_column('users', 'groups', 'old_groups'),
+        )
+
+        migrate(migrator.add_column('users', 'groups', User.groups))
+
+        group_map = dict(map(lambda g: (g.name, g.id), Group.select()))
+        user_map = defaultdict(list)
+        for user in User.select(User, peewee.SQL('old_groups')):
+            group_ids = [group_map[group] for group in user.old_groups]
+            user.update_instance(groups=group_ids)
+
+        migrate(migrator.drop_column('users', 'old_groups'))
+
+    db.close_db(None)

migrations/0019_add_super_admin_permission.py

@@ -0,0 +1,6 @@
+from redash import models
+
+if __name__ == '__main__':
+    admin_group = models.Group.get(models.Group.name=='admin')
+    admin_group.permissions.append('super_admin')
+    admin_group.save()

migrations/0020_change_ds_name_to_non_uniqe.py

@@ -0,0 +1,19 @@
+from redash.models import db
+import peewee
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+if __name__ == '__main__':
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        # Change the uniqueness constraint on data source name to be (org, name):
+        # In some cases it's a constraint:
+        db.database.execute_sql('ALTER TABLE data_sources DROP CONSTRAINT IF EXISTS unique_name')
+        # In others only an index:
+        db.database.execute_sql('DROP INDEX IF EXISTS data_sources_name')
+
+        migrate(
+            migrator.add_index('data_sources', ('org_id', 'name'), unique=True)
+        )
+
+    db.close_db(None)

migrations/0021_create_api_keys_table.py

@@ -0,0 +1,5 @@
+from redash.models import ApiKey
+
+
+if __name__ == '__main__':
+    ApiKey.create_table()

migrations/0022_add_new_permissions.py

@@ -0,0 +1,10 @@
+from redash import models
+
+if __name__ == '__main__':
+    with models.db.database.transaction():
+        groups = models.Group.select(models.Group.id, models.Group.permissions).where(models.Group.name=='default')
+        for group in groups:
+            group.permissions.append('list_dashboards')
+            group.permissions.append('list_alerts')
+            group.permissions.append('list_data_sources')
+            group.save(only=[models.Group.permissions])

migrations/0023_add_notification_destination.py

@@ -0,0 +1,81 @@
+import peewee
+from redash import settings
+from redash.models import db, NotificationDestination, AlertSubscription, Alert, Organization, User
+from redash.destinations import get_configuration_schema_for_destination_type
+from redash.utils.configuration import ConfigurationContainer
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+if __name__ == '__main__':
+    migrator = PostgresqlMigrator(db.database)
+    with db.database.transaction():
+
+        if not NotificationDestination.table_exists():
+            NotificationDestination.create_table()
+            
+            # Update alert subscription fields
+            migrate(
+                migrator.add_column('alert_subscriptions', 'destination_id', AlertSubscription.destination)
+            )
+
+            try:
+                org = Organization.get_by_slug('default')
+                user = User.select().where(User.org==org, peewee.SQL("%s = ANY(groups)", org.admin_group.id)).get()
+            except Exception:
+                print "!!! Warning: failed finding default organization or admin user, won't migrate Webhook/HipChat alert subscriptions."
+                exit()
+
+            if settings.WEBHOOK_ENDPOINT:
+                # Have all existing alerts send to webhook if already configured
+                schema = get_configuration_schema_for_destination_type('webhook')
+                conf = {'url': settings.WEBHOOK_ENDPOINT}
+                if settings.WEBHOOK_USERNAME:
+                    conf['username'] = settings.WEBHOOK_USERNAME
+                    conf['password'] = settings.WEBHOOK_PASSWORD
+                options = ConfigurationContainer(conf, schema)
+
+                webhook = NotificationDestination.create(
+                    org=org,
+                    user=user,
+                    name="Webhook",
+                    type="webhook",
+                    options=options
+                )
+
+                for alert in Alert.select():
+                    AlertSubscription.create(
+                        user=user,
+                        destination=webhook,
+                        alert=alert
+                    )
+
+            if settings.HIPCHAT_API_TOKEN:
+                # Have all existing alerts send to HipChat if already configured
+                schema = get_configuration_schema_for_destination_type('hipchat')
+
+                conf = {}
+
+                if settings.HIPCHAT_API_URL:
+                    conf['url'] = '{url}/room/{room_id}/notification?auth_token={token}'.format(
+                        url=settings.HIPCHAT_API_URL, room_id=settings.HIPCHAT_ROOM_ID, token=settings.HIPCHAT_API_TOKEN)
+                else:
+                    conf['url'] = 'https://hipchat.com/v2/room/{room_id}/notification?auth_token={token}'.format(
+                        room_id=settings.HIPCHAT_ROOM_ID, token=settings.HIPCHAT_API_TOKEN)
+
+                options = ConfigurationContainer(conf, schema)
+
+                hipchat = NotificationDestination.create(
+                    org=org,
+                    user=user,
+                    name="HipChat",
+                    type="hipchat",
+                    options=options
+                )
+
+                for alert in Alert.select():
+                    AlertSubscription.create(
+                        user=user,
+                        destination=hipchat,
+                        alert=alert
+                    )
+
+    db.close_db(None)

migrations/0023_make_sure_correct_group_type.py

@@ -0,0 +1,8 @@
+from redash import models
+
+if __name__ == '__main__':
+    with models.db.database.transaction():
+        groups = models.Group.select(models.Group.id, models.Group.type).where(models.Group.name=='default')
+        for group in groups:
+            group.type = models.Group.BUILTIN_GROUP
+            group.save(only=[models.Group.type])

migrations/0024_add_options_to_query.py

@@ -0,0 +1,10 @@
+from redash.models import db, Query
+from playhouse.migrate import PostgresqlMigrator, migrate
+
+if __name__ == '__main__':
+    migrator = PostgresqlMigrator(db.database)
+
+    with db.database.transaction():
+        migrate(
+            migrator.add_column('queries', 'options', Query.options),
+        )

package.json

@@ -0,0 +1,43 @@
+{
+  "name": "redash",
+  "devDependencies": {
+    "gulp": "^3.9.0",
+    "gulp-connect": "^2.2.0",
+    "gulp-autoprefixer": "2.3.1",
+    "gulp-cache": "^0.2.10",
+    "rimraf": "^2.4.0",
+    "gulp-filter": "^2.0.2",
+    "gulp-imagemin": "^2.3.0",
+    "gulp-jshint": "^1.11.1",
+    "gulp-karma": "0.0.4",
+    "gulp-load-plugins": "^0.10.0",
+    "gulp-plumber": "^1.0.1",
+    "gulp-minify-css": "^1.2.0",
+    "gulp-uglify": "^1.2.0",
+    "gulp-useref": "^3.0.0",
+    "gulp-util": "^3.0.6",
+    "gulp-watch": "^4.2.4",
+    "run-sequence": "^1.1.1",
+    "wiredep": "^2.2.2",
+    "lazypipe": "^0.2.4",
+    "gulp-ng-annotate": "^1.0.0",
+    "open": "0.0.5",
+    "jshint-stylish": "^1.0.0",
+    "gulp-print": "^2.0.1",
+    "gulp-rev-all": "^0.8.22",
+    "bower": "~1.7.1",
+    "gulp-cli": "~1.2.0",
+    "lodash.map": "^4.4.0"
+  },
+  "engines": {
+    "node": ">=0.10.0"
+  },
+  "scripts": {
+    "test": "echo 'No tests.'",
+    "build": "gulp build",
+    "bower": "bower",
+    "heroku-postbuild": "npm install --dev && npm run bower install && npm run build && npm prune --production"
+  },
+  "dependencies": {
+  }
+}

rd_service/cli.py

@@ -1,59 +0,0 @@
-"""
-CLI to start the workers.
-
-TODO: move API server startup here.
-"""
-import atfork
-atfork.monkeypatch_os_fork_functions()
-import atfork.stdlib_fixer
-atfork.stdlib_fixer.fix_logging_module()
-
-import argparse
-import logging
-import urlparse
-import redis
-import time
-import settings
-import data
-
-
-def start_workers(data_manager):
-    try:
-        old_workers = data_manager.redis_connection.smembers('workers')
-        data_manager.redis_connection.delete('workers')
-
-        logging.info("Cleaning old workers: %s", old_workers)
-
-        data_manager.start_workers(settings.WORKERS_COUNT, settings.CONNECTION_STRING)
-        logging.info("Workers started.")
-
-        while True:
-            try:
-                data_manager.refresh_queries()
-            except Exception as e:
-                logging.error("Something went wrong with refreshing queries...")
-                logging.exception(e)
-            time.sleep(60)
-    except KeyboardInterrupt:
-        logging.warning("Exiting; waiting for threads")
-        data_manager.stop_workers()
-
-
-if __name__ == '__main__':
-    channel = logging.StreamHandler()
-    logging.getLogger().addHandler(channel)
-    logging.getLogger().setLevel(settings.LOG_LEVEL)
-
-    parser = argparse.ArgumentParser()
-    parser.add_argument("command")
-    args = parser.parse_args()
-
-    url = urlparse.urlparse(settings.REDIS_URL)
-    redis_connection = redis.StrictRedis(host=url.hostname, port=url.port, db=0, password=url.password)
-    data_manager = data.Manager(redis_connection, settings.INTERNAL_DB_CONNECTION_STRING, settings.MAX_CONNECTIONS)
-
-    if args.command == "worker":
-        start_workers(data_manager)
-    else:
-        print "Unknown command"
-

rd_service/data/__init__.py

@@ -1,4 +0,0 @@
-from manager import Manager
-from worker import Job
-import models
-import utils

rd_service/data/manager.py

@@ -1,192 +0,0 @@
-"""
-Data manager. Used to manage and coordinate execution of queries.
-"""
-import collections
-from contextlib import contextmanager
-import json
-import logging
-import psycopg2
-import qr
-import redis
-import time
-import worker
-import settings
-from utils import gen_query_hash
-
-
-class QueryResult(collections.namedtuple('QueryData', 'id query data runtime retrieved_at query_hash')):
-    def to_dict(self, parse_data=False):
-        d = self._asdict()
-
-        if parse_data and d['data']:
-            d['data'] = json.loads(d['data'])
-
-        return d
-
-
-class Manager(object):
-    def __init__(self, redis_connection, db_connection_string, db_max_connections):
-        self.redis_connection = redis_connection
-        self.workers = []
-        self.db_connection_string = db_connection_string
-        self.queue = qr.PriorityQueue("jobs", **self.redis_connection.connection_pool.connection_kwargs)
-        self.max_retries = 5
-        self.status = {
-            'last_refresh_at': 0,
-            'started_at': time.time()
-        }
-
-        self._save_status()
-
-    # TODO: Use our Django Models
-    def get_query_result_by_id(self, query_result_id):
-        with self.db_transaction() as cursor:
-            sql = "SELECT id, query, data, runtime, retrieved_at, query_hash FROM query_results " \
-                  "WHERE id=%s LIMIT 1"
-            cursor.execute(sql, (query_result_id,))
-            query_result = cursor.fetchone()
-
-        if query_result:
-            query_result = QueryResult(*query_result)
-
-        return query_result
-
-    def get_query_result(self, query, ttl=0):
-        query_hash = gen_query_hash(query)
-
-        with self.db_transaction() as cursor:
-            sql = "SELECT id, query, data, runtime, retrieved_at, query_hash FROM query_results " \
-                  "WHERE query_hash=%s " \
-                  "AND retrieved_at < now() at time zone 'utc' - interval '%s second'" \
-                  "ORDER BY retrieved_at DESC LIMIT 1"
-            cursor.execute(sql, (query_hash, psycopg2.extensions.AsIs(ttl)))
-            query_result = cursor.fetchone()
-
-        if query_result:
-            query_result = QueryResult(*query_result)
-
-        return query_result
-
-    def add_job(self, query, priority):
-        query_hash = gen_query_hash(query)
-        logging.info("[Manager][%s] Inserting job with priority=%s", query_hash, priority)
-        try_count = 0
-        job = None
-
-        while try_count < self.max_retries:
-            try_count += 1
-
-            pipe = self.redis_connection.pipeline()
-            try:
-                pipe.watch('query_hash_job:%s' % query_hash)
-                job_id = pipe.get('query_hash_job:%s' % query_hash)
-                if job_id:
-                    logging.info("[Manager][%s] Found existing job: %s", query_hash, job_id)
-                    job = worker.Job.load(self.redis_connection, job_id)
-                else:
-                    job = worker.Job(self.redis_connection, query, priority)
-                    pipe.multi()
-                    job.save(pipe)
-                    logging.info("[Manager][%s] Created new job: %s", query_hash, job.id)
-                    self.queue.push(job.id, job.priority)
-                break
-
-            except redis.WatchError:
-                continue
-
-        if not job:
-            logging.error("[Manager][%s] Failed adding job for query.", query_hash)
-
-        return job
-
-    def refresh_queries(self):
-        sql = """SELECT queries.query, queries.ttl, retrieved_at
-        FROM (SELECT query, min(ttl) as ttl FROM queries  WHERE ttl > 0 GROUP by query) queries
-        JOIN (SELECT query, max(retrieved_at) as retrieved_at
-        FROM query_results
-        GROUP BY query) query_results on query_results.query=queries.query
-        WHERE queries.ttl > 0
-          AND query_results.retrieved_at + ttl * interval '1 second' < now() at time zone 'utc';"""
-
-        self.status['last_refresh_at'] = time.time()
-        self._save_status()
-
-        logging.info("Refreshing queries...")
-        queries = self.run_query(sql)
-        for query, ttl, retrieved_at in queries:
-            self.add_job(query, worker.Job.LOW_PRIORITY)
-
-        logging.info("Done refreshing queries... %d" % len(queries))
-
-    def store_query_result(self, query, data, run_time, retrieved_at):
-        query_result_id = None
-        query_hash = gen_query_hash(query)
-        sql = "INSERT INTO query_results (query_hash, query, data, runtime, retrieved_at) " \
-              "VALUES (%s, %s, %s, %s, %s) RETURNING id"
-        with self.db_transaction() as cursor:
-            cursor.execute(sql, (query_hash, query, data, run_time, retrieved_at))
-            if cursor.rowcount == 1:
-                query_result_id = cursor.fetchone()[0]
-                logging.info("[Manager][%s] Inserted query data; id=%s", query_hash, query_result_id)
-
-                sql = "UPDATE queries SET latest_query_data_id=%s WHERE query_hash=%s"
-                cursor.execute(sql, (query_result_id, query_hash))
-
-                logging.info("[Manager][%s] Updated %s queries.", query_hash, cursor.rowcount)
-            else:
-                logging.error("[Manager][%s] Failed inserting query data.", query_hash)
-        return query_result_id
-
-    def run_query(self, *args):
-        sql = args[0]
-        logging.debug("running query: %s %s", sql, args[1:])
-
-        with self.db_transaction() as cursor:
-            cursor.execute(sql, args[1:])
-            if cursor.description:
-                data = list(cursor)
-            else:
-                data = cursor.rowcount
-
-        return data
-
-    def start_workers(self, workers_count, connection_string):
-        if self.workers:
-            return self.workers
-        
-        if settings.CONNECTION_ADAPTER == "mysql":
-            import query_runner_mysql            
-            runner = query_runner_mysql.mysql(connection_string)
-        else:
-            import query_runner            
-            runner = query_runner.redshift(connection_string)
-
-        redis_connection_params = self.redis_connection.connection_pool.connection_kwargs
-        self.workers = [worker.Worker(self, redis_connection_params, runner)
-                        for _ in range(workers_count)]
-        for w in self.workers:
-            w.start()
-
-        return self.workers
-
-    def stop_workers(self):
-        for w in self.workers:
-            w.continue_working = False
-            w.join()
-
-    @contextmanager
-    def db_transaction(self):
-        connection = psycopg2.connect(self.db_connection_string)
-        cursor = connection.cursor()
-        try:
-            yield cursor
-        except:
-            connection.rollback()
-            raise
-        else:
-            connection.commit()
-        finally:
-            connection.close()
-
-    def _save_status(self):
-        self.redis_connection.hmset('manager:status', self.status)

rd_service/data/models.py

@@ -1,210 +0,0 @@
-"""
-Django ORM based models to describe the data model of re:dash.
-"""
-import hashlib
-import json
-import time
-from django.db import models
-from django.template.defaultfilters import slugify
-import utils
-
-
-class QueryResult(models.Model):
-    id = models.AutoField(primary_key=True)
-    query_hash = models.CharField(max_length=32)
-    query = models.TextField()
-    data = models.TextField()
-    runtime = models.FloatField()
-    retrieved_at = models.DateTimeField()
-
-    class Meta:
-        app_label = 'redash'
-        db_table = 'query_results'
-
-    def to_dict(self):
-        return {
-            'id': self.id,
-            'query_hash': self.query_hash,
-            'query': self.query,
-            'data': json.loads(self.data),
-            'runtime': self.runtime,
-            'retrieved_at': self.retrieved_at
-        }
-
-    def __unicode__(self):
-        return u"%d | %s | %s" % (self.id, self.query_hash, self.retrieved_at)
-
-
-class Query(models.Model):
-    id = models.AutoField(primary_key=True)
-    latest_query_data = models.ForeignKey(QueryResult)
-    name = models.CharField(max_length=255)
-    description = models.CharField(max_length=4096)
-    query = models.TextField()
-    query_hash = models.CharField(max_length=32)
-    api_key = models.CharField(max_length=40)
-    ttl = models.IntegerField()
-    user = models.CharField(max_length=360)
-    created_at = models.DateTimeField(auto_now_add=True)
-
-    class Meta:
-        app_label = 'redash'
-        db_table = 'queries'
-
-    def to_dict(self, with_result=True, with_stats=False,
-                with_visualizations=False):
-        d = {
-            'id': self.id,
-            'latest_query_data_id': self.latest_query_data_id,
-            'name': self.name,
-            'description': self.description,
-            'query': self.query,
-            'query_hash': self.query_hash,
-            'ttl': self.ttl,
-            'user': self.user,
-            'api_key': self.api_key,
-            'created_at': self.created_at,
-        }
-
-        if with_stats:
-            d['avg_runtime'] = self.avg_runtime
-            d['min_runtime'] = self.min_runtime
-            d['max_runtime'] = self.max_runtime
-            d['last_retrieved_at'] = self.last_retrieved_at
-            d['times_retrieved'] = self.times_retrieved
-
-        if with_result and self.latest_query_data_id:
-            d['latest_query_data'] = self.latest_query_data.to_dict()
-
-        if with_visualizations:
-            d['visualizations'] = [vis.to_dict(with_query=False)
-                                   for vis in self.visualizations.all()]
-
-        return d
-
-    @classmethod
-    def all_queries(cls):
-        query = """SELECT queries.*, query_stats.*
-FROM queries
-LEFT OUTER JOIN
-  (SELECT qu.query_hash,
-          count(0) AS "times_retrieved",
-          avg(runtime) AS "avg_runtime",
-          min(runtime) AS "min_runtime",
-          max(runtime) AS "max_runtime",
-          max(retrieved_at) AS "last_retrieved_at"
-   FROM queries qu
-   JOIN query_results qr ON qu.query_hash=qr.query_hash
-   GROUP BY qu.query_hash) query_stats ON query_stats.query_hash = queries.query_hash
-        """
-        return cls.objects.raw(query)
-
-    def save(self, *args, **kwargs):
-        self.query_hash = utils.gen_query_hash(self.query)
-        self._set_api_key()
-        super(Query, self).save(*args, **kwargs)
-
-    def _set_api_key(self):
-        if not self.api_key:
-            self.api_key = hashlib.sha1(
-                u''.join([str(time.time()), self.query, self.user, self.name])).hexdigest()
-
-    def __unicode__(self):
-        return unicode(self.id)
-
-
-class Dashboard(models.Model):
-    id = models.AutoField(primary_key=True)
-    slug = models.CharField(max_length=140)
-    name = models.CharField(max_length=100)
-    user = models.CharField(max_length=360)
-    layout = models.TextField()
-    is_archived = models.BooleanField(default=False)
-
-    class Meta:
-        app_label = 'redash'
-        db_table = 'dashboards'
-
-    def to_dict(self, with_widgets=False):
-        layout = json.loads(self.layout)
-
-        if with_widgets:
-            widgets = {w.id: w.to_dict() for w in self.widgets.all()}
-            widgets_layout = map(lambda row: map(lambda widget_id: widgets.get(widget_id, None), row), layout)
-        else:
-            widgets_layout = None
-
-        return {
-            'id': self.id,
-            'slug': self.slug,
-            'name': self.name,
-            'user': self.user,
-            'layout': layout,
-            'widgets': widgets_layout
-        }
-
-    def save(self, *args, **kwargs):
-        # TODO: make sure slug is unique
-        if not self.slug:
-            self.slug = slugify(self.name)
-        super(Dashboard, self).save(*args, **kwargs)
-
-    def __unicode__(self):
-        return u"%s=%s" % (self.id, self.name)
-
-
-class Visualization(models.Model):
-    id = models.AutoField(primary_key=True)
-    type = models.CharField(max_length=100)
-    query = models.ForeignKey(Query, related_name='visualizations')
-    name = models.CharField(max_length=255)
-    description = models.CharField(max_length=4096)
-    options = models.TextField()
-
-    class Meta:
-        app_label = 'redash'
-        db_table = 'visualizations'
-
-    def to_dict(self, with_query=True):
-        d = {
-            'id': self.id,
-            'type': self.type,
-            'name': self.name,
-            'description': self.description,
-            'options': json.loads(self.options),
-        }
-
-        if with_query:
-            d['query'] = self.query.to_dict()
-
-        return d
-
-    def __unicode__(self):
-        return u"%s=>%s" % (self.id, self.query_id)
-
-
-class Widget(models.Model):
-    id = models.AutoField(primary_key=True)
-    type = models.CharField(max_length=100)
-    query = models.ForeignKey(Query, related_name='widgets')
-    visualization = models.ForeignKey(Visualization, related_name='widgets')
-    width = models.IntegerField()
-    options = models.TextField()
-    dashboard = models.ForeignKey(Dashboard, related_name='widgets')
-
-    class Meta:
-        app_label = 'redash'
-        db_table = 'widgets'
-
-    def to_dict(self):
-        return {
-            'id': self.id,
-            'type': self.type,
-            'width': self.width,
-            'options': json.loads(self.options),
-            'visualization': self.visualization.to_dict(),
-            'dashboard_id': self.dashboard_id
-        }
-
-    def __unicode__(self):
-        return u"%s=>%s" % (self.id, self.dashboard_id)

rd_service/data/query_runner.py

@@ -1,67 +0,0 @@
-"""
-QueryRunner is the function that the workers use, to execute queries. This is the Redshift
-(PostgreSQL in fact) version, but easily we can write another to support additional databases
-(MySQL and others).
-
-Because the worker just pass the query, this can be used with any data store that has some sort of
-query language (for example: HiveQL).
-"""
-import logging
-import json
-import psycopg2
-import sys
-import select
-from .utils import JSONEncoder
-
-def redshift(connection_string):
-    def column_friendly_name(column_name):
-        return column_name
- 
-    def wait(conn):
-        while 1:
-            state = conn.poll()
-            if state == psycopg2.extensions.POLL_OK:
-                break
-            elif state == psycopg2.extensions.POLL_WRITE:
-                select.select([], [conn.fileno()], [])
-            elif state == psycopg2.extensions.POLL_READ:
-                select.select([conn.fileno()], [], [])
-            else:
-                raise psycopg2.OperationalError("poll() returned %s" % state)
- 
-    def query_runner(query):
-        connection = psycopg2.connect(connection_string, async=True)
-        wait(connection)
- 
-        cursor = connection.cursor()
- 
-        try:
-            cursor.execute(query)
-            wait(connection)
- 
-            column_names = [col.name for col in cursor.description]
- 
-            rows = [dict(zip(column_names, row)) for row in cursor]
-            columns = [{'name': col.name,
-                        'friendly_name': column_friendly_name(col.name),
-                        'type': None} for col in cursor.description]
- 
-            data = {'columns': columns, 'rows': rows}
-            json_data = json.dumps(data, cls=JSONEncoder)
-            error = None
-            cursor.close()
-        except psycopg2.DatabaseError as e:
-            json_data = None
-            error = e.message
-        except KeyboardInterrupt:
-            connection.cancel()
-            error = "Query cancelled by user."
-            json_data = None
-        except Exception as e:
-            raise sys.exc_info()[1], None, sys.exc_info()[2]
-        finally:
-            connection.close()
- 
-        return json_data, error
- 
-    return query_runner

rd_service/data/query_runner_mysql.py

@@ -1,56 +0,0 @@
-"""
-QueryRunner is the function that the workers use, to execute queries. This is the Redshift
-(PostgreSQL in fact) version, but easily we can write another to support additional databases
-(MySQL and others).
-
-Because the worker just pass the query, this can be used with any data store that has some sort of
-query language (for example: HiveQL).
-"""
-import logging
-import json
-import MySQLdb
-import sys
-import select
-from .utils import JSONEncoder
-
-def mysql(connection_string):
-    if connection_string.endswith(';'):
-        connection_string = connection_string[0:-1]
-    
-    def query_runner(query):
-        connections_params = [entry.split('=')[1] for entry in connection_string.split(';')]
-        connection = MySQLdb.connect(*connections_params)
-        cursor = connection.cursor()
-
-        logging.debug("mysql got query: %s", query)
-        
-        try:
-            cursor.execute(query)
-            
-            data = cursor.fetchall()
-            
-            num_fields = len(cursor.description)
-            column_names = [i[0] for i in cursor.description]
-            
-            rows = [dict(zip(column_names, row)) for row in data]
-
-            columns = [{'name': col_name,
-                        'friendly_name': col_name,
-                        'type': None} for col_name in column_names]
-            
-            data = {'columns': columns, 'rows': rows}
-            json_data = json.dumps(data, cls=JSONEncoder)
-            error = None
-            cursor.close()
-        except MySQLdb.Error, e:
-            json_data = None
-            error = e.message
-        except Exception as e:
-            raise sys.exc_info()[1], None, sys.exc_info()[2]
-        finally:
-            connection.close()
-        
-        return json_data, error
-        
-    
-    return query_runner

rd_service/data/tables.sql

@@ -1,55 +0,0 @@
-BEGIN;
-CREATE TABLE "query_results" (
-    "id" serial NOT NULL PRIMARY KEY,
-    "query_hash" varchar(32) NOT NULL,
-    "query" text NOT NULL,
-    "data" text NOT NULL,
-    "runtime" double precision NOT NULL,
-    "retrieved_at" timestamp with time zone NOT NULL
-)
-;
-CREATE TABLE "queries" (
-    "id" serial NOT NULL PRIMARY KEY,
-    "latest_query_data_id" integer REFERENCES "query_results" ("id") DEFERRABLE INITIALLY DEFERRED,
-    "name" varchar(255) NOT NULL,
-    "description" varchar(4096),
-    "query" text NOT NULL,
-    "query_hash" varchar(32) NOT NULL,
-    "api_key" varchar(40),
-    "ttl" integer NOT NULL,
-    "user" varchar(360) NOT NULL,
-    "created_at" timestamp with time zone NOT NULL
-)
-;
-CREATE TABLE "dashboards" (
-    "id" serial NOT NULL PRIMARY KEY,
-    "slug" varchar(140) NOT NULL,
-    "name" varchar(100) NOT NULL,
-    "user" varchar(360) NOT NULL,
-    "layout" text NOT NULL,
-    "is_archived" boolean NOT NULL
-)
-;
-CREATE TABLE "visualizations" (
-    "id" serial NOT NULL PRIMARY KEY,
-    "type" varchar(100) NOT NULL,
-    "query_id" integer NOT NULL REFERENCES "queries" ("id") DEFERRABLE INITIALLY DEFERRED,
-    "name" varchar(255) NOT NULL,
-    "description" varchar(4096),
-    "options" text NOT NULL
-)
-;
-CREATE TABLE "widgets" (
-    "id" serial NOT NULL PRIMARY KEY,
-    "type" varchar(100) NOT NULL,
-    "width" integer NOT NULL,
-    "options" text NOT NULL,
-    "visualization_id" integer NOT NULL REFERENCES "visualizations" ("id") DEFERRABLE INITIALLY DEFERRED,
-    "dashboard_id" integer NOT NULL REFERENCES "dashboards" ("id") DEFERRABLE INITIALLY DEFERRED
-)
-;
-CREATE INDEX "queries_latest_query_data_id" ON "queries" ("latest_query_data_id");
-CREATE INDEX "widgets_query_id" ON "widgets" ("query_id");
-CREATE INDEX "widgets_dashboard_id" ON "widgets" ("dashboard_id");
-
-COMMIT;

rd_service/data/utils.py

@@ -1,71 +0,0 @@
-import cStringIO
-import csv
-import codecs
-import decimal
-import datetime
-import json
-import re
-import hashlib
-
-COMMENTS_REGEX = re.compile("/\*.*?\*/")
-
-
-def gen_query_hash(sql):
-    """Returns hash of the given query after stripping all comments, line breaks and multiple
-    spaces, and lower casing all text.
-
-    TODO: possible issue - the following queries will get the same id:
-        1. SELECT 1 FROM table WHERE column='Value';
-        2. SELECT 1 FROM table where column='value';
-    """
-    sql = COMMENTS_REGEX.sub("", sql)
-    sql = "".join(sql.split()).lower()
-    return hashlib.md5(sql.encode('utf-8')).hexdigest()
-
-
-class JSONEncoder(json.JSONEncoder):
-    """Custom JSON encoding class, to handle Decimal and datetime.date instances.
-    """
-    def default(self, o):
-        if isinstance(o, decimal.Decimal):
-            return float(o)
-
-        if isinstance(o, datetime.date):
-            return o.isoformat()
-
-        super(JSONEncoder, self).default(o)
-
-
-class UnicodeWriter:
-    """
-    A CSV writer which will write rows to CSV file "f",
-    which is encoded in the given encoding.
-    """
-    def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
-        # Redirect output to a queue
-        self.queue = cStringIO.StringIO()
-        self.writer = csv.writer(self.queue, dialect=dialect, **kwds)
-        self.stream = f
-        self.encoder = codecs.getincrementalencoder(encoding)()
-
-    def _encode_utf8(self, val):
-        if isinstance(val, (unicode, str)):
-            return val.encode('utf-8')
-
-        return val
-
-    def writerow(self, row):
-        self.writer.writerow([self._encode_utf8(s) for s in row])
-        # Fetch UTF-8 output from the queue ...
-        data = self.queue.getvalue()
-        data = data.decode("utf-8")
-        # ... and reencode it into the target encoding
-        data = self.encoder.encode(data)
-        # write to the target stream
-        self.stream.write(data)
-        # empty queue
-        self.queue.truncate(0)
-
-    def writerows(self, rows):
-        for row in rows:
-            self.writerow(row)

rd_service/data/worker.py

@@ -1,255 +0,0 @@
-"""
-Worker implementation to execute incoming queries.
-"""
-import json
-import logging
-import os
-import threading
-import uuid
-import datetime
-import time
-import signal
-import setproctitle
-import redis
-from utils import gen_query_hash
-
-
-class Job(object):
-    HIGH_PRIORITY = 1
-    LOW_PRIORITY = 2
-
-    WAITING = 1
-    PROCESSING = 2
-    DONE = 3
-    FAILED = 4
-
-    def __init__(self, redis_connection, query, priority,
-                 job_id=None,
-                 wait_time=None, query_time=None,
-                 updated_at=None, status=None, error=None, query_result_id=None,
-                 process_id=0):
-        self.redis_connection = redis_connection
-        self.query = query
-        self.priority = priority
-        self.query_hash = gen_query_hash(self.query)
-        self.query_result_id = query_result_id
-        if process_id == 'None':
-            self.process_id = None
-        else:
-            self.process_id = int(process_id)
-
-        if job_id is None:
-            self.id = str(uuid.uuid1())
-            self.new_job = True
-            self.wait_time = 0
-            self.query_time = 0
-            self.error = None
-            self.updated_at = time.time() # job_dict.get('updated_at', time.time())
-            self.status = self.WAITING # int(job_dict.get('status', self.WAITING))
-        else:
-            self.id = job_id
-            self.new_job = False
-            self.error = error
-            self.wait_time = wait_time
-            self.query_time = query_time
-            self.updated_at = updated_at
-            self.status = status
-
-    def to_dict(self):
-        return {
-            'query': self.query,
-            'priority': self.priority,
-            'id': self.id,
-            'wait_time': self.wait_time,
-            'query_time': self.query_time,
-            'updated_at': self.updated_at,
-            'status': self.status,
-            'error': self.error,
-            'query_result_id': self.query_result_id,
-            'process_id': self.process_id
-        }
-
-    @staticmethod
-    def _redis_key(job_id):
-        return 'job:%s' % job_id
-
-    def cancel(self):
-        # TODO: Race condition:
-        # it's possible that it will be picked up by worker while processing the cancel order
-        if self.is_finished():
-            return
-
-        if self.status == self.PROCESSING:
-            os.kill(self.process_id, signal.SIGINT)
-        else:
-            self.done(None, "Interrupted/Cancelled while running.")
-
-    def save(self, pipe=None):
-        if not pipe:
-            pipe = self.redis_connection.pipeline()
-
-        if self.new_job:
-            pipe.set('query_hash_job:%s' % self.query_hash, self.id)
-
-        if self.is_finished():
-            pipe.delete('query_hash_job:%s' % self.query_hash)
-
-        pipe.sadd('jobs_set', self.id)
-        pipe.hmset(self._redis_key(self.id), self.to_dict())
-        pipe.publish(self._redis_key(self.id), json.dumps(self.to_dict()))
-        pipe.execute()
-
-    def processing(self, process_id):
-        self.status = self.PROCESSING
-        self.process_id = process_id
-        self.wait_time = time.time() - self.updated_at
-        self.updated_at = time.time()
-        self.save()
-
-    def is_finished(self):
-        return self.status in (self.FAILED, self.DONE)
-
-    def done(self, query_result_id, error):
-        if error:
-            self.status = self.FAILED
-        else:
-            self.status = self.DONE
-
-        self.query_result_id = query_result_id
-        self.error = error
-        self.query_time = time.time() - self.updated_at
-        self.updated_at = time.time()
-        self.save()
-
-    def __str__(self):
-        return "<Job:%s,priority:%d,status:%d>" % (self.id, self.priority, self.status)
-
-    @classmethod
-    def _load(cls, redis_connection, job_id):
-        return redis_connection.hgetall(cls._redis_key(job_id))
-
-    @classmethod
-    def load(cls, redis_connection, job_id):
-        job_dict = cls._load(redis_connection, job_id)
-        job = None
-        if job_dict:
-            job = Job(redis_connection, job_id=job_dict['id'], query=job_dict['query'].decode('utf-8'),
-                      priority=int(job_dict['priority']), updated_at=float(job_dict['updated_at']),
-                      status=int(job_dict['status']), wait_time=float(job_dict['wait_time']),
-                      query_time=float(job_dict['query_time']), error=job_dict['error'],
-                      query_result_id=job_dict['query_result_id'],
-                      process_id=job_dict['process_id'])
-
-        return job
-
-
-class Worker(threading.Thread):
-    def __init__(self, manager, redis_connection_params, query_runner, sleep_time=0.1):
-        self.manager = manager
-
-        self.redis_connection_params = {k: v for k, v in redis_connection_params.iteritems()
-                                        if k in ('host', 'db', 'password', 'port')}
-        self.continue_working = True
-        self.query_runner = query_runner
-        self.sleep_time = sleep_time
-        self.child_pid = None
-        self.worker_id = uuid.uuid1()
-        self.status = {
-            'jobs_count': 0,
-            'cancelled_jobs_count': 0,
-            'done_jobs_count': 0,
-            'updated_at': time.time(),
-            'started_at': time.time()
-        }
-        self._save_status()
-        self.manager.redis_connection.sadd('workers', self._key)
-
-        super(Worker, self).__init__(name="Worker-%s" % self.worker_id)
-
-    def set_title(self, title=None):
-        base_title = "redash worker:%s" % self.worker_id
-        if title:
-            full_title = "%s - %s" % (base_title, title)
-        else:
-            full_title = base_title
-
-        setproctitle.setproctitle(full_title)
-
-    def run(self):
-        logging.info("[%s] started.", self.name)
-        while self.continue_working:
-            job_id = self.manager.queue.pop()
-            if job_id:
-                self._update_status('jobs_count')
-                logging.info("[%s] Processing %s", self.name, job_id)
-                self._fork_and_process(job_id)
-                if self.child_pid == 0:
-                    return
-            else:
-                time.sleep(self.sleep_time)
-
-    def _update_status(self, counter):
-        self.status['updated_at'] = time.time()
-        self.status[counter] += 1
-        self._save_status()
-
-    @property
-    def _key(self):
-        return 'worker:%s' % self.worker_id
-
-    def _save_status(self):
-        self.manager.redis_connection.hmset(self._key, self.status)
-
-    def _fork_and_process(self, job_id):
-        self.child_pid = os.fork()
-        if self.child_pid == 0:
-            self.set_title("processing %s" % job_id)
-            self._process(job_id)
-        else:
-            logging.info("[%s] Waiting for pid: %d", self.name, self.child_pid)
-            _, status = os.waitpid(self.child_pid, 0)
-            self._update_status('done_jobs_count')
-            if status > 0:
-                job = Job.load(self.manager.redis_connection, job_id)
-                if not job.is_finished():
-                    self._update_status('cancelled_jobs_count')
-                    logging.info("[%s] process interrupted and job %s hasn't finished; registering interruption in job",
-                                 self.name, job_id)
-                    job.done(None, "Interrupted/Cancelled while running.")
-
-            logging.info("[%s] Finished Processing %s (pid: %d status: %d)",
-                         self.name, job_id, self.child_pid, status)
-
-    def _process(self, job_id):
-        redis_connection = redis.StrictRedis(**self.redis_connection_params)
-        job = Job.load(redis_connection, job_id)
-        if job.is_finished():
-            logging.warning("[%s][%s] tried to process finished job.", self.name, job)
-            return
-
-        pid = os.getpid()
-        job.processing(pid)
-
-        logging.info("[%s][%s] running query...", self.name, job.id)
-        start_time = time.time()
-        self.set_title("running query %s" % job_id)
-
-        annotated_query = "/* Pid: %s, Job Id: %s, Query hash: %s, Priority: %s */ %s" % \
-                          (pid, job.id, job.query_hash, job.priority, job.query)
-
-        # TODO: here's the part that needs to be forked, not all of the worker process...
-        data, error = self.query_runner(annotated_query)
-        run_time = time.time() - start_time
-        logging.info("[%s][%s] query finished... data length=%s, error=%s",
-                     self.name, job.id, data and len(data), error)
-
-        # TODO: it is possible that storing the data will fail, and we will need to retry
-        # while we already marked the job as done
-        query_result_id = None
-        if not error:
-            self.set_title("storing results %s" % job_id)
-            query_result_id = self.manager.store_query_result(job.query, data, run_time,
-                                                              datetime.datetime.utcnow())
-
-        self.set_title("marking job as done %s" % job_id)
-        job.done(query_result_id, error)

rd_service/migrate.py

@@ -1,50 +0,0 @@
-import json
-import settings
-from data.models import *
-
-# first run:
-
-# CREATE TABLE "visualizations" (
-#     "id" serial NOT NULL PRIMARY KEY,
-#     "type" varchar(100) NOT NULL,
-#     "query_id" integer NOT NULL REFERENCES "queries" ("id") DEFERRABLE INITIALLY DEFERRED,
-#     "name" varchar(255) NOT NULL,
-#     "description" varchar(4096),
-#     "options" text NOT NULL
-# )
-# ;
-
-# ALTER TABLE widgets ADD COLUMN "visualization_id" integer REFERENCES "visualizations" ("id") DEFERRABLE INITIALLY DEFERRED;
-
-if __name__ == '__main__':
-    print 'migrating Widgets -> Visualizations ...'
-
-    for query in Query.objects.filter(name__icontains="cohort"):
-        vis = Visualization(query=query, name=query.name,
-                            description=query.description,
-                            type="COHORT", options="{}")
-        vis.save()
-
-
-    for widget in Widget.objects.all():
-        print 'processing widget %d' % widget.id
-        query = widget.query
-        vis_type = widget.type.upper()
-
-        vis = query.visualizations.filter(type=vis_type)
-        if vis:
-            print 'found'
-            widget.visualization = vis[0]
-            widget.save()
-
-        else:
-            options = json.loads(widget.options)
-            vis_options = {"series": options} if options else {}
-            vis_options = json.dumps(vis_options)
-
-            vis = Visualization(query=query, name=query.name,
-                                            description=query.description,
-                                            type=vis_type, options=vis_options)
-            vis.save()
-            widget.visualization = vis
-            widget.save()

rd_service/requirements.txt

@@ -1,10 +0,0 @@
-psycopg2==2.5.1
-redis==2.7.5
-tornado==3.0.2
-sqlparse==0.1.8
-Django==1.5.4
-django-db-pool==0.0.10
-qr==0.6.0
-python-dateutil==2.1
-setproctitle==1.1.8
-atfork==0.1.2

rd_service/server.py

@@ -1,381 +0,0 @@
-"""
-Tornado based API implementation for re:dash.
-
-Also at the moment the Tornado server is used to serve the static assets (and the Angular.js app),
-but this is only due to configuration issues and temporary.
-
-Usage:
-    python server.py [--port=8888] [--debug] [--static=..]
-
-    port - port to listen to
-    debug - enable debug mode (extensive logging, restart on code change)
-    static - static assets path
-
-If static option isn't specified it will be taken from settings.py.
-"""
-import csv
-import hashlib
-import json
-import numbers
-import os
-import urlparse
-import logging
-import cStringIO
-import datetime
-import dateutil.parser
-import redis
-import sqlparse
-import tornado.ioloop
-import tornado.web
-import tornado.auth
-import tornado.options
-import settings
-import time
-from data import utils
-import data
-
-
-class BaseHandler(tornado.web.RequestHandler):
-    def initialize(self):
-        self.data_manager = self.application.settings.get('data_manager', None)
-        self.redis_connection = self.application.settings['redis_connection']
-
-    def get_current_user(self):
-        user = self.get_secure_cookie("user")
-        return user
-
-    def write_json(self, response, encode=True):
-        if encode:
-            response = json.dumps(response, cls=utils.JSONEncoder)
-        self.set_header("Content-Type", "application/json; charset=UTF-8")
-        self.write(response)
-
-
-class BaseAuthenticatedHandler(BaseHandler):
-    @tornado.web.authenticated
-    def prepare(self):
-        pass
-
-
-class PingHandler(tornado.web.RequestHandler):
-    def get(self):
-        self.write("PONG")
-
-
-class GoogleLoginHandler(tornado.web.RequestHandler,
-                         tornado.auth.GoogleMixin):
-    @tornado.web.asynchronous
-    @tornado.gen.coroutine
-    def get(self):
-        if self.get_argument("openid.mode", None):
-            user = yield self.get_authenticated_user()
-
-            if user['email'] in settings.ALLOWED_USERS or user['email'].endswith("@%s" % settings.GOOGLE_APPS_DOMAIN):
-                logging.info("Authenticated: %s", user['email'])
-                self.set_secure_cookie("user", user['email'])
-                self.redirect("/")
-            else:
-                logging.error("Failed logging in with: %s", user)
-                self.authenticate_redirect()
-        else:
-            self.authenticate_redirect()
-
-
-class MainHandler(BaseAuthenticatedHandler):
-    def get(self, *args):
-        email_md5 = hashlib.md5(self.current_user.lower()).hexdigest()
-        gravatar_url = "https://www.gravatar.com/avatar/%s?s=40" % email_md5
-
-        user = {
-            'gravatar_url': gravatar_url,
-            'is_admin': self.current_user in settings.ADMINS,
-            'name': self.current_user
-        }
-
-        self.render("index.html", user=json.dumps(user), analytics=settings.ANALYTICS)
-
-
-class QueryFormatHandler(BaseAuthenticatedHandler):
-    def post(self):
-        arguments = json.loads(self.request.body)
-        query = arguments.get("query", "")
-
-        self.write(sqlparse.format(query, reindent=True, keyword_case='upper'))
-
-
-class StatusHandler(BaseAuthenticatedHandler):
-    def get(self):
-        status = {}
-        info = self.redis_connection.info()
-        status['redis_used_memory'] = info['used_memory_human']
-
-        status['queries_count'] = data.models.Query.objects.count()
-        status['query_results_count'] = data.models.QueryResult.objects.count()
-        status['dashboards_count'] = data.models.Dashboard.objects.count()
-        status['widgets_count'] = data.models.Widget.objects.count()
-
-        status['workers'] = [self.redis_connection.hgetall(w)
-                             for w in self.redis_connection.smembers('workers')]
-
-        manager_status = self.redis_connection.hgetall('manager:status')
-        status['manager'] = manager_status
-        status['manager']['queue_size'] = self.redis_connection.zcard('jobs')
-
-        self.write_json(status)
-
-
-class WidgetsHandler(BaseAuthenticatedHandler):
-    def post(self, widget_id=None):
-        widget_properties = json.loads(self.request.body)
-        widget_properties['options'] = json.dumps(widget_properties['options'])
-        widget = data.models.Widget(**widget_properties)
-        widget.save()
-
-        layout = json.loads(widget.dashboard.layout)
-        new_row = True
-
-        if len(layout) == 0 or widget.width == 2:
-            layout.append([widget.id])
-        elif len(layout[-1]) == 1:
-            neighbour_widget = data.models.Widget.objects.get(pk=layout[-1][0])
-            if neighbour_widget.width == 1:
-                layout[-1].append(widget.id)
-                new_row = False
-            else:
-                layout.append([widget.id])
-        else:
-            layout.append([widget.id])
-
-        widget.dashboard.layout = json.dumps(layout)
-        widget.dashboard.save()
-
-        self.write_json({'widget': widget.to_dict(), 'layout': layout, 'new_row': new_row})
-
-    def delete(self, widget_id):
-        widget_id = int(widget_id)
-        widget = data.models.Widget.objects.get(pk=widget_id)
-        # TODO: reposition existing ones
-        layout = json.loads(widget.dashboard.layout)
-        layout = map(lambda row: filter(lambda w: w != widget_id, row), layout)
-        layout = filter(lambda row: len(row) > 0, layout)
-        widget.dashboard.layout = json.dumps(layout)
-        widget.dashboard.save()
-
-        widget.delete()
-
-
-class DashboardHandler(BaseAuthenticatedHandler):
-    def get(self, dashboard_slug=None):
-        if dashboard_slug:
-            dashboard = data.models.Dashboard.objects.prefetch_related('widgets__visualization__query__latest_query_data').get(slug=dashboard_slug)
-            self.write_json(dashboard.to_dict(with_widgets=True))
-        else:
-            dashboards = [d.to_dict() for d in
-                          data.models.Dashboard.objects.filter(is_archived=False)]
-            self.write_json(dashboards)
-
-    def post(self, dashboard_id):
-        if dashboard_id:
-            dashboard_properties = json.loads(self.request.body)
-            dashboard = data.models.Dashboard.objects.get(pk=dashboard_id)
-            dashboard.layout = dashboard_properties['layout']
-            dashboard.name = dashboard_properties['name']
-            dashboard.save()
-
-            self.write_json(dashboard.to_dict(with_widgets=True))
-        else:
-            dashboard_properties = json.loads(self.request.body)
-            dashboard = data.models.Dashboard(name=dashboard_properties['name'],
-                                                    user=self.current_user,
-                                                    layout='[]')
-            dashboard.save()
-            self.write_json(dashboard.to_dict())
-
-    def delete(self, dashboard_slug):
-        dashboard = data.models.Dashboard.objects.get(slug=dashboard_slug)
-        dashboard.is_archived = True
-        dashboard.save()
-
-
-class QueriesHandler(BaseAuthenticatedHandler):
-    def post(self, id=None):
-        query_def = json.loads(self.request.body)
-        if 'created_at' in query_def:
-            query_def['created_at'] = dateutil.parser.parse(query_def['created_at'])
-
-        query_def.pop('latest_query_data', None)
-        query_def.pop('visualizations', None)
-
-        if id:
-            query = data.models.Query(**query_def)
-            fields = query_def.keys()
-            fields.remove('id')
-            query.save(update_fields=fields)
-        else:
-            query_def['user'] = self.current_user
-            query = data.models.Query(**query_def)
-            query.save()
-
-        self.write_json(query.to_dict(with_result=False))
-
-    def get(self, id=None):
-        if id:
-            q = data.models.Query.objects.get(pk=id)
-            if q:
-                self.write_json(q.to_dict(with_visualizations=True))
-            else:
-                self.send_error(404)
-        else:
-            self.write_json([q.to_dict(with_result=False, with_stats=True) for q in data.models.Query.all_queries()])
-
-
-class QueryResultsHandler(BaseAuthenticatedHandler):
-    def get(self, query_result_id):
-        query_result = self.data_manager.get_query_result_by_id(query_result_id)
-        if query_result:
-            self.write_json({'query_result': query_result.to_dict(parse_data=True)})
-        else:
-            self.send_error(404)
-
-    def post(self, _):
-        params = json.loads(self.request.body)
-
-        if params['ttl'] == 0:
-            query_result = None
-        else:
-            query_result = self.data_manager.get_query_result(params['query'], int(params['ttl']))
-
-        if query_result:
-            self.write_json({'query_result': query_result.to_dict(parse_data=True)})
-        else:
-            job = self.data_manager.add_job(params['query'], data.Job.HIGH_PRIORITY)
-            self.write({'job': job.to_dict()})
-
-
-class VisualizationHandler(BaseAuthenticatedHandler):
-    def get(self, id):
-        pass
-
-    def post(self, id=None):
-        kwargs = json.loads(self.request.body)
-        kwargs['options'] = json.dumps(kwargs['options'])
-
-        if id:
-            vis = data.models.Visualization(**kwargs)
-            fields = kwargs.keys()
-            fields.remove('id')
-            vis.save(update_fields=fields)
-        else:
-            vis = data.models.Visualization(**kwargs)
-            vis.save()
-
-        self.write_json(vis.to_dict(with_query=False))
-
-    def delete(self, id):
-        vis = data.models.Visualization.objects.get(pk=id)
-        vis.delete()
-
-
-class CsvQueryResultsHandler(BaseAuthenticatedHandler):
-    def get_current_user(self):
-        user = super(CsvQueryResultsHandler, self).get_current_user()
-        if not user:
-            api_key = self.get_argument("api_key", None)
-            query = data.models.Query.objects.get(pk=self.path_args[0])
-
-            if query.api_key and query.api_key == api_key:
-                user = "API-Key=%s" % api_key
-
-        return user
-
-    def get(self, query_id, result_id=None):
-        if not result_id:
-            query = data.models.Query.objects.get(pk=query_id)
-            if query:
-                result_id = query.latest_query_data_id
-
-        query_result = result_id and self.data_manager.get_query_result_by_id(result_id)
-        if query_result:
-            self.set_header("Content-Type", "text/csv; charset=UTF-8")
-            s = cStringIO.StringIO()
-
-            query_data = json.loads(query_result.data)
-            writer = csv.DictWriter(s, fieldnames=[col['name'] for col in query_data['columns']])
-            writer.writer = utils.UnicodeWriter(s)
-            writer.writeheader()
-            for row in query_data['rows']:
-                for k, v in row.iteritems():
-                    if isinstance(v, numbers.Number) and (v > 1000 * 1000 * 1000 * 100):
-                        row[k] = datetime.datetime.fromtimestamp(v/1000.0)
-
-                writer.writerow(row)
-
-            self.write(s.getvalue())
-        else:
-            self.send_error(404)
-
-
-class JobsHandler(BaseAuthenticatedHandler):
-    def get(self, job_id=None):
-        if job_id:
-            # TODO: if finished, include the query result
-            job = data.Job.load(self.data_manager.redis_connection, job_id)
-            self.write({'job': job.to_dict()})
-        else:
-            raise NotImplemented
-
-    def delete(self, job_id):
-        job = data.Job.load(self.data_manager.redis_connection, job_id)
-        job.cancel()
-
-
-def get_application(static_path, is_debug, redis_connection, data_manager):
-    return tornado.web.Application([(r"/", MainHandler),
-                                    (r"/ping", PingHandler),
-                                    (r"/api/queries/([0-9]*)/results(?:/([0-9]*))?.csv", CsvQueryResultsHandler),
-                                    (r"/api/queries/format", QueryFormatHandler),
-                                    (r"/api/queries(?:/([0-9]*))?", QueriesHandler),
-                                    (r"/api/query_results(?:/([0-9]*))?", QueryResultsHandler),
-                                    (r"/api/jobs/(.*)", JobsHandler),
-                                    (r"/api/visualizations(?:/([0-9]*))?", VisualizationHandler),
-                                    (r"/api/widgets(?:/([0-9]*))?", WidgetsHandler),
-                                    (r"/api/dashboards(?:/(.*))?", DashboardHandler),
-                                    (r"/admin/(.*)", MainHandler),
-                                    (r"/dashboard/(.*)", MainHandler),
-                                    (r"/queries(.*)", MainHandler),
-                                    (r"/login", GoogleLoginHandler),
-                                    (r"/status.json", StatusHandler),
-                                    (r"/(.*)", tornado.web.StaticFileHandler,
-                                     {"path": static_path})],
-                                   template_path=static_path,
-                                   static_path=static_path,
-                                   debug=is_debug,
-                                   login_url="/login",
-                                   cookie_secret=settings.COOKIE_SECRET,
-                                   redis_connection=redis_connection,
-                                   data_manager=data_manager)
-
-
-if __name__ == '__main__':
-    tornado.options.define("port", default=8888, type=int)
-    tornado.options.define("debug", default=False, type=bool)
-    tornado.options.define("static", default=settings.STATIC_ASSETS_PATH, type=str)
-
-    tornado.options.parse_command_line()
-
-    root_path = os.path.dirname(__file__)
-    static_path = os.path.abspath(os.path.join(root_path, tornado.options.options.static))
-
-    url = urlparse.urlparse(settings.REDIS_URL)
-    redis_connection = redis.StrictRedis(host=url.hostname, port=url.port, db=0, password=url.password)
-    data_manager = data.Manager(redis_connection, settings.INTERNAL_DB_CONNECTION_STRING,
-                                      settings.MAX_CONNECTIONS)
-
-    logging.info("re:dash web server stating on port: %d...", tornado.options.options.port)
-    logging.info("UI assets path: %s...", static_path)
-
-    application = get_application(static_path, tornado.options.options.debug,
-                                  redis_connection, data_manager)
-
-    application.listen(tornado.options.options.port)
-    tornado.ioloop.IOLoop.instance().start()

rd_service/settings_example.py

@@ -1,41 +0,0 @@
-"""
-Example settings module. You should make your own copy as settings.py and enter the real settings.
-"""
-
-import django.conf
-
-REDIS_URL = "redis://localhost:6379"
-
-# Either "pg" or "mysql"
-CONNECTION_ADAPTER = "mysql"
-# Connection string for the database that is used to run queries against
-# -- example mysql CONNECTION_STRING = "Server=;User=;Pwd=;Database="
-# -- example pg    CONNECTION_STRING = "user= password= host= port=5439 dbname="
-CONNECTION_STRING = "user= password= host= port=5439 dbname="
-# Connection string for the operational databases (where we store the queries, results, etc)
-INTERNAL_DB_CONNECTION_STRING = "dbname=postgres"
-# Google Apps domain to allow access from; any user with email in this Google Apps will be allowed
-# access
-GOOGLE_APPS_DOMAIN = ""
-# Email addresses of specific users not from the above set Google Apps Domain, that you want to
-# allow access to re:dash
-ALLOWED_USERS = []
-# Email addresses of admin users
-ADMINS = []
-STATIC_ASSETS_PATH = "../rd_ui/dist/"
-WORKERS_COUNT = 2
-MAX_CONNECTIONS = 3
-COOKIE_SECRET = "c292a0a3aa32397cdb050e233733900f"
-LOG_LEVEL = "INFO"
-ANALYTICS = ""
-
-# Configuration of the operational database for the Django models
-django.conf.settings.configure(DATABASES = { 'default': {
-        'ENGINE': 'dbpool.db.backends.postgresql_psycopg2',
-        'OPTIONS': {'MAX_CONNS': 10, 'MIN_CONNS': 1},
-        'NAME': 'postgres',
-        'USER': '',
-        'PASSWORD': '',
-        'HOST': '',
-        'PORT': '',
-        },}, TIME_ZONE = 'UTC')

rd_ui/.bowerrc

@@ -1,3 +0,0 @@
-{
-    "directory": "app/bower_components"
-}

rd_ui/.gitattributes

@@ -1 +0,0 @@
-* text=auto

rd_ui/.gitignore

@@ -1,4 +0,0 @@
-node_modules
-.tmp
-.sass-cache
-app/bower_components

rd_ui/.jshintrc

@@ -11,14 +11,17 @@
   "latedef": true,
   "newcap": true,
   "noarg": true,
-  "quotmark": "single",
+  "quotmark": false,
   "regexp": true,
   "undef": true,
   "unused": true,
-  "strict": true,
+  "strict": false,
   "trailing": true,
   "smarttabs": true,
   "globals": {
-    "angular": false
+    "angular": false,
+    "_": false,
+    "$": false,
+    "currentUser": false
   }
 }

rd_ui/.travis.yml

@@ -1,7 +0,0 @@
-language: node_js
-node_js:
-  - '0.8'
-  - '0.10'
-before_script:
-  - 'npm install -g bower grunt-cli'
-  - 'bower install'

rd_ui/Gruntfile.js

@@ -1,362 +0,0 @@
-// Generated on 2013-08-25 using generator-angular 0.4.0
-'use strict';
-var LIVERELOAD_PORT = 35729;
-var lrSnippet = require('connect-livereload')({ port: LIVERELOAD_PORT });
-var mountFolder = function (connect, dir) {
-  return connect.static(require('path').resolve(dir));
-};
-
-// # Globbing
-// for performance reasons we're only matching one level down:
-// 'test/spec/{,*/}*.js'
-// use this if you want to recursively match all subfolders:
-// 'test/spec/**/*.js'
-
-module.exports = function (grunt) {
-  require('load-grunt-tasks')(grunt);
-  require('time-grunt')(grunt);
-
-  // configurable paths
-  var yeomanConfig = {
-    app: 'app',
-    dist: 'dist'
-  };
-
-  try {
-    yeomanConfig.app = require('./bower.json').appPath || yeomanConfig.app;
-  } catch (e) {}
-
-  grunt.initConfig({
-    yeoman: yeomanConfig,
-    watch: {
-      coffee: {
-        files: ['<%= yeoman.app %>/scripts/{,*/}*.coffee'],
-        tasks: ['coffee:dist']
-      },
-      coffeeTest: {
-        files: ['test/spec/{,*/}*.coffee'],
-        tasks: ['coffee:test']
-      },
-      styles: {
-        files: ['<%= yeoman.app %>/styles/{,*/}*.css'],
-        tasks: ['copy:styles', 'autoprefixer']
-      },
-      livereload: {
-        options: {
-          livereload: LIVERELOAD_PORT
-        },
-        files: [
-          '<%= yeoman.app %>/{,*/}*.html',
-          '.tmp/styles/{,*/}*.css',
-          '{.tmp,<%= yeoman.app %>}/scripts/{,*/}*.js',
-          '<%= yeoman.app %>/images/{,*/}*.{png,jpg,jpeg,gif,webp,svg}'
-        ]
-      }
-    },
-    autoprefixer: {
-      options: ['last 1 version'],
-      dist: {
-        files: [{
-          expand: true,
-          cwd: '.tmp/styles/',
-          src: '{,*/}*.css',
-          dest: '.tmp/styles/'
-        }]
-      }
-    },
-    connect: {
-      options: {
-        port: 9000,
-        // Change this to '0.0.0.0' to access the server from outside.
-        hostname: 'localhost'
-      },
-      livereload: {
-        options: {
-          middleware: function (connect) {
-            return [
-              lrSnippet,
-              mountFolder(connect, '.tmp'),
-              mountFolder(connect, yeomanConfig.app)
-            ];
-          }
-        }
-      },
-      test: {
-        options: {
-          middleware: function (connect) {
-            return [
-              mountFolder(connect, '.tmp'),
-              mountFolder(connect, 'test')
-            ];
-          }
-        }
-      },
-      dist: {
-        options: {
-          middleware: function (connect) {
-            return [
-              mountFolder(connect, yeomanConfig.dist)
-            ];
-          }
-        }
-      }
-    },
-    open: {
-      server: {
-        url: 'http://localhost:<%= connect.options.port %>'
-      }
-    },
-    clean: {
-      dist: {
-        files: [{
-          dot: true,
-          src: [
-            '.tmp',
-            '<%= yeoman.dist %>/*',
-            '!<%= yeoman.dist %>/.git*'
-          ]
-        }]
-      },
-      server: '.tmp'
-    },
-    jshint: {
-      options: {
-        jshintrc: '.jshintrc'
-      },
-      all: [
-        'Gruntfile.js',
-        '<%= yeoman.app %>/scripts/{,*/}*.js'
-      ]
-    },
-    coffee: {
-      options: {
-        sourceMap: true,
-        sourceRoot: ''
-      },
-      dist: {
-        files: [{
-          expand: true,
-          cwd: '<%= yeoman.app %>/scripts',
-          src: '{,*/}*.coffee',
-          dest: '.tmp/scripts',
-          ext: '.js'
-        }]
-      },
-      test: {
-        files: [{
-          expand: true,
-          cwd: 'test/spec',
-          src: '{,*/}*.coffee',
-          dest: '.tmp/spec',
-          ext: '.js'
-        }]
-      }
-    },
-    // not used since Uglify task does concat,
-    // but still available if needed
-    /*concat: {
-      dist: {}
-    },*/
-    rev: {
-      dist: {
-        files: {
-          src: [
-            '<%= yeoman.dist %>/scripts/{,*/}*.js',
-            '<%= yeoman.dist %>/styles/{,*/}*.css',
-            '<%= yeoman.dist %>/images/{,*/}*.{png,jpg,jpeg,gif,webp,svg}',
-            '<%= yeoman.dist %>/styles/fonts/*'
-          ]
-        }
-      }
-    },
-    useminPrepare: {
-      html: '<%= yeoman.app %>/index.html',
-      options: {
-        dest: '<%= yeoman.dist %>'
-      }
-    },
-    usemin: {
-      html: ['<%= yeoman.dist %>/{,*/}*.html'],
-      css: ['<%= yeoman.dist %>/styles/{,*/}*.css'],
-      options: {
-        dirs: ['<%= yeoman.dist %>']
-      }
-    },
-    imagemin: {
-      dist: {
-        files: [{
-          expand: true,
-          cwd: '<%= yeoman.app %>/images',
-          src: '{,*/}*.{png,jpg,jpeg}',
-          dest: '<%= yeoman.dist %>/images'
-        }]
-      }
-    },
-    svgmin: {
-      dist: {
-        files: [{
-          expand: true,
-          cwd: '<%= yeoman.app %>/images',
-          src: '{,*/}*.svg',
-          dest: '<%= yeoman.dist %>/images'
-        }]
-      }
-    },
-    cssmin: {
-      // By default, your `index.html` <!-- Usemin Block --> will take care of
-      // minification. This option is pre-configured if you do not wish to use
-      // Usemin blocks.
-      // dist: {
-      //   files: {
-      //     '<%= yeoman.dist %>/styles/main.css': [
-      //       '.tmp/styles/{,*/}*.css',
-      //       '<%= yeoman.app %>/styles/{,*/}*.css'
-      //     ]
-      //   }
-      // }
-    },
-    htmlmin: {
-      dist: {
-        options: {
-          /*removeCommentsFromCDATA: true,
-          // https://github.com/yeoman/grunt-usemin/issues/44
-          //collapseWhitespace: true,
-          collapseBooleanAttributes: true,
-          removeAttributeQuotes: true,
-          removeRedundantAttributes: true,
-          useShortDoctype: true,
-          removeEmptyAttributes: true,
-          removeOptionalTags: true*/
-        },
-        files: [{
-          expand: true,
-          cwd: '<%= yeoman.app %>',
-          src: ['*.html', 'views/*.html'],
-          dest: '<%= yeoman.dist %>'
-        }]
-      }
-    },
-    // Put files not handled in other tasks here
-    copy: {
-      dist: {
-        files: [{
-          expand: true,
-          dot: true,
-          cwd: '<%= yeoman.app %>',
-          dest: '<%= yeoman.dist %>',
-          src: [
-            '*.{ico,png,txt}',
-            '.htaccess',
-            'bower_components/**/*',
-            'images/{,*/}*.{gif,webp}',
-            'fonts/*'
-          ]
-        }, {
-          expand: true,
-          cwd: '.tmp/images',
-          dest: '<%= yeoman.dist %>/images',
-          src: [
-            'generated/*'
-          ]
-        }]
-      },
-      styles: {
-        expand: true,
-        cwd: '<%= yeoman.app %>/styles',
-        dest: '.tmp/styles/',
-        src: '{,*/}*.css'
-      }
-    },
-    concurrent: {
-      server: [
-        'coffee:dist',
-        'copy:styles'
-      ],
-      test: [
-        'coffee',
-        'copy:styles'
-      ],
-      dist: [
-        'coffee',
-        'copy:styles',
-        'imagemin',
-        'svgmin',
-        'htmlmin'
-      ]
-    },
-    karma: {
-      unit: {
-        configFile: 'karma.conf.js',
-        singleRun: true
-      }
-    },
-    cdnify: {
-      dist: {
-        html: ['<%= yeoman.dist %>/*.html']
-      }
-    },
-    ngmin: {
-      dist: {
-        files: [{
-          expand: true,
-          cwd: '<%= yeoman.dist %>/scripts',
-          src: '*.js',
-          dest: '<%= yeoman.dist %>/scripts'
-        }]
-      }
-    },
-    uglify: {
-      dist: {
-        files: {
-          '<%= yeoman.dist %>/scripts/scripts.js': [
-            '<%= yeoman.dist %>/scripts/scripts.js'
-          ]
-        }
-      }
-    }
-  });
-
-  grunt.registerTask('server', function (target) {
-    if (target === 'dist') {
-      return grunt.task.run(['build', 'open', 'connect:dist:keepalive']);
-    }
-
-    grunt.task.run([
-      'clean:server',
-      'concurrent:server',
-      'autoprefixer',
-      'connect:livereload',
-      'open',
-      'watch'
-    ]);
-  });
-
-  grunt.registerTask('test', [
-    'clean:server',
-    'concurrent:test',
-    'autoprefixer',
-    'connect:test',
-    'karma'
-  ]);
-
-  grunt.registerTask('build', [
-    'clean:dist',
-    'useminPrepare',
-    'concurrent:dist',
-    'autoprefixer',
-    'concat',
-    'copy:dist',
-    'cdnify',
-    'ngmin',
-    'cssmin',
-    'uglify',
-    'rev',
-    'usemin'
-  ]);
-
-  grunt.registerTask('default', [
-    'jshint',
-    'test',
-    'build'
-  ]);
-};

rd_ui/app/app_layout.html

@@ -0,0 +1,119 @@
+<!DOCTYPE html>
+<!--[if lt IE 7]>      <html class="no-js lt-ie9 lt-ie8 lt-ie7" ng-app="redash" ng-controller='MainCtrl'> <![endif]-->
+<!--[if IE 7]>         <html class="no-js lt-ie9 lt-ie8" ng-app="redash" ng-controller='MainCtrl'> <![endif]-->
+<!--[if IE 8]>         <html class="no-js lt-ie9" ng-app="redash" ng-controller='MainCtrl'> <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" ng-app="redash" ng-controller='MainCtrl'> <!--<![endif]-->
+<head>
+    <base href="{{base_href}}">
+    <title ng-bind="pageTitle + '| {{name}}'"></title>
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+
+    <!-- build:css /styles/main.css -->
+    <link rel="stylesheet" href="/styles/superflat_redash.css">
+    <link rel="stylesheet" href="/bower_components/material-design-iconic-font/dist/css/material-design-iconic-font.css">
+    <link rel="stylesheet" href="/bower_components/codemirror/lib/codemirror.css">
+    <link rel="stylesheet" href="/bower_components/gridster/dist/jquery.gridster.css">
+    <link rel="stylesheet" href="/bower_components/pivottable/dist/pivot.css">
+    <link rel="stylesheet" href="/bower_components/cornelius/src/cornelius.css">
+    <link rel="stylesheet" href="/bower_components/angular-ui-select/dist/select.css">
+    <link rel="stylesheet" href="/bower_components/pace/themes/pace-theme-minimal.css">
+    <link rel="stylesheet" href="/bower_components/font-awesome/css/font-awesome.css">
+    <link rel="stylesheet" href="/bower_components/codemirror/addon/hint/show-hint.css">
+    <link rel="stylesheet" href="/bower_components/leaflet/dist/leaflet.css">
+    <link rel="stylesheet" href="/bower_components/angular-resizable/src/angular-resizable.css">
+    <link rel="stylesheet" href="/styles/redash.css">
+    <!-- endbuild -->
+
+    <link rel="icon" type="image/png" sizes="32x32" href="/images/favicon-32x32.png">
+    <link rel="icon" type="image/png" sizes="96x96" href="/images/favicon-96x96.png">
+    <link rel="icon" type="image/png" sizes="16x16" href="/images/favicon-16x16.png">
+
+</head>
+<body{% if headless %} class="headless"{% endif %}>
+<div growl></div>
+
+{% block content %}
+{% endblock %}
+
+<section id="1">
+  <div ng-view></div>
+  <div ng-if="showPermissionError" class="ng-cloak container" ng-cloak>
+    <div class="row">
+      <div class="text-center">
+        <h1><span class="zmdi zmdi-lock"></span></h1>
+        <p class="text-muted">
+          You do not have permission to view the requested page.
+        </p>
+      </div>
+    </div>
+  </div>
+
+  {% if not headless %}
+  {% raw %}
+  <div class="visible-print">
+    <hr>
+    Source: {{location}}
+  </div>
+</section>
+
+
+{% endraw %}
+{% include 'footer.html' %}
+{% endif %}
+
+{% include 'vendor_scripts.html' %}
+
+<!-- build:js({.tmp,rd_ui/app}) /scripts/scripts.js -->
+<script src="/scripts/app.js"></script>
+<script src="/scripts/services/services.js"></script>
+<script src="/scripts/services/resources.js"></script>
+<script src="/scripts/services/notifications.js"></script>
+<script src="/scripts/services/dashboards.js"></script>
+<script src="/scripts/controllers/controllers.js"></script>
+<script src="/scripts/controllers/dashboard.js"></script>
+<script src="/scripts/controllers/admin_controllers.js"></script>
+<script src="/scripts/controllers/data_sources.js"></script>
+<script src="/scripts/controllers/destinations.js"></script>
+<script src="/scripts/controllers/query_view.js"></script>
+<script src="/scripts/controllers/query_source.js"></script>
+<script src="/scripts/controllers/users.js"></script>
+<script src="/scripts/visualizations/base.js"></script>
+<script src="/scripts/visualizations/chart.js"></script>
+<script src="/scripts/visualizations/cohort.js"></script>
+<script src="/scripts/visualizations/map.js"></script>
+<script src="/scripts/visualizations/counter.js"></script>
+<script src="/scripts/visualizations/boxplot.js"></script>
+<script src="/scripts/visualizations/box.js"></script>
+<script src="/scripts/visualizations/table.js"></script>
+<script src="/scripts/visualizations/pivot.js"></script>
+<script src="/scripts/visualizations/wordcloud.js"></script>
+<script src="/scripts/vendor/cloud.js"></script>
+<script src="/scripts/directives/directives.js"></script>
+<script src="/scripts/directives/query_directives.js"></script>
+<script src="/scripts/directives/dashboard_directives.js"></script>
+<script src="/scripts/filters.js"></script>
+<script src="/scripts/controllers/alerts.js"></script>
+<!-- endbuild -->
+
+<script>
+    // TODO: move currentUser & features to be an Angular service
+    var clientConfig = {{ client_config|safe }};
+    var basePath = "{{base_href}}";
+    var currentUser = {{ user|safe }};
+
+    currentUser.canEdit = function(object) {
+        var user_id = object.user_id || (object.user && object.user.id);
+        return this.hasPermission('admin') || (user_id && (user_id == currentUser.id));
+    };
+
+    currentUser.hasPermission = function(permission) {
+        return this.permissions.indexOf(permission) != -1;
+    };
+
+    currentUser.isAdmin = currentUser.hasPermission('admin');
+</script>
+{% include '_includes/tail.html' %}
+
+</body>
+</html>

rd_ui/app/embed.html

@@ -0,0 +1,78 @@
+<!DOCTYPE html>
+<!--[if lt IE 7]>      <html class="no-js lt-ie9 lt-ie8 lt-ie7" ng-app="redash" ng-controller='MainCtrl'> <![endif]-->
+<!--[if IE 7]>         <html class="no-js lt-ie9 lt-ie8" ng-app="redash" ng-controller='MainCtrl'> <![endif]-->
+<!--[if IE 8]>         <html class="no-js lt-ie9" ng-app="redash" ng-controller='MainCtrl'> <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" ng-app="redash" ng-controller='EmbedCtrl'> <!--<![endif]-->
+<head>
+    <base href="{{base_href}}">
+    <title ng-bind="'{{name}} | ' + pageTitle"></title>
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+
+    <!-- build:css /styles/embed.css -->
+    <link rel="stylesheet" href="/styles/superflat_redash.css">
+    <link rel="stylesheet" href="/bower_components/codemirror/lib/codemirror.css">
+    <link rel="stylesheet" href="/bower_components/gridster/dist/jquery.gridster.css">
+    <link rel="stylesheet" href="/bower_components/pivottable/dist/pivot.css">
+    <link rel="stylesheet" href="/bower_components/cornelius/src/cornelius.css">
+    <link rel="stylesheet" href="/bower_components/angular-ui-select/dist/select.css">
+    <link rel="stylesheet" href="/bower_components/pace/themes/pace-theme-minimal.css">
+    <link rel="stylesheet" href="/bower_components/font-awesome/css/font-awesome.css">
+    <link rel="stylesheet" href="/bower_components/codemirror/addon/hint/show-hint.css">
+    <link rel="stylesheet" href="/bower_components/leaflet/dist/leaflet.css">
+    <link rel="stylesheet" href="/styles/redash.css">
+    <!-- endbuild -->
+
+    <link rel="icon" type="image/png" sizes="32x32" href="/images/favicon-32x32.png">
+    <link rel="icon" type="image/png" sizes="96x96" href="/images/favicon-96x96.png">
+    <link rel="icon" type="image/png" sizes="16x16" href="/images/favicon-16x16.png">
+
+    <style>
+        body { padding: 0; }
+    </style>
+</head>
+<body>
+<div growl></div>
+<div ng-view></div>
+
+{% include 'vendor_scripts.html' %}
+
+<!-- build:js({.tmp,rd_ui/app}) /scripts/embed-scripts.js -->
+<script src="/scripts/embed.js"></script>
+<script src="/scripts/services/services.js"></script>
+<script src="/scripts/services/resources.js"></script>
+<script src="/scripts/services/notifications.js"></script>
+<script src="/scripts/services/dashboards.js"></script>
+<script src="/scripts/controllers/controllers.js"></script>
+<script src="/scripts/controllers/dashboard.js"></script>
+<script src="/scripts/controllers/admin_controllers.js"></script>
+<script src="/scripts/controllers/data_sources.js"></script>
+<script src="/scripts/controllers/query_view.js"></script>
+<script src="/scripts/controllers/query_source.js"></script>
+<script src="/scripts/controllers/users.js"></script>
+<script src="/scripts/visualizations/base.js"></script>
+<script src="/scripts/visualizations/chart.js"></script>
+<script src="/scripts/visualizations/cohort.js"></script>
+<script src="/scripts/visualizations/map.js"></script>
+<script src="/scripts/visualizations/counter.js"></script>
+<script src="/scripts/visualizations/boxplot.js"></script>
+<script src="/scripts/visualizations/box.js"></script>
+<script src="/scripts/visualizations/table.js"></script>
+<script src="/scripts/visualizations/pivot.js"></script>
+<script src="/scripts/visualizations/wordcloud.js"></script>
+<script src="/scripts/vendor/cloud.js"></script>
+<script src="/scripts/directives/directives.js"></script>
+<script src="/scripts/directives/query_directives.js"></script>
+<script src="/scripts/directives/dashboard_directives.js"></script>
+<script src="/scripts/filters.js"></script>
+<script src="/scripts/controllers/alerts.js"></script>
+<!-- endbuild -->
+
+<script>
+    var clientConfig = {{ client_config|safe }};
+    var visualization = {{ visualization|safe }};
+    var query_result = {{ query_result|safe }};
+</script>
+
+</body>
+</html>