Update CHANGELOG

* Sankey: Make sure last stage has "Exit"

* Sankey: Use 2 as min stage width size to render

* Use null instead of "Exit"

* Add comment about corresponding exit node

* Add multiple stages on Cypress test

CHANGELOG.md

@@ -1,5 +1,149 @@
 # Change Log
 
+## v9.0.0-beta - 2020-06-11
+
+This release was long time in the making and has several major changes:
+
+- Our backend code was updated to support Python 3 and we no longer support Python 2. If you're using our Docker images, this should be a transparent change for you.
+- We replaced Celery with RQ for background jobs processing. This will require some setup updates -- see instructions below.
+- The frontend code is now 100% React and we removed all the Angular dependencies.
+
+This release was made possible by contributions from over 50 people: @ari-e, @ariarijp, @arihantsurana, @arikfr, @atharvai, @cemremengu, @chulucninh09, @citrin, @daniellangnet, @DavidHernandez, @deecay, @dmudro, @erans, @erels, @ezkl, @gabrieldutra, @gstaykov, @ialeinikov, @ikenji, @Jakdaw, @jezdez, @juanvasquezreyes, @koooge, @kravets-levko, @kykrueger, @leibowitz, @leosunmo, @lihan, @loganprice, @mickeey2525, @mnoorenberghe, @monicagangwar, @NicolasLM, @p-yang, @Ralnoc, @ranbena, @randyzwitch, @rauchy, @rxin, @saravananselvamohan, @satyamkrishna, @shinsuke-nara, @stefan-mees, @stevebuckingham, @susodapop, @taminif, @thewarpaint, @tsuyoshizawa, @uncletimmy3, @wengkham.
+
+### Upgrading
+
+Typically, if you are running your own instance of Redash and wish to upgrade, you would simply modify the Docker tag in your `docker-compose.yml` file. Since RQ has replaced Celery in this version, there are a couple extra modifications that need to be done in your `docker-compose.yml`:
+
+1. Under `services/scheduler/environment`, omit `QUEUES` and `WORKERS_COUNT` (and omit `environment` altogether if it is empty).
+2. Under `services`, add a new service for general RQ jobs:
+
+```yaml
+worker:
+  <<: *redash-service
+  command: worker
+  environment:
+    QUEUES: "periodic emails default"
+    WORKERS_COUNT: 1
+```
+
+Following that, force a recreation of your containers with `docker-compose up --force-recreate --build` and you should be good to go.
+
+### UX
+
+- Redesigned Query Results page:
+  - Completely new layout is easier to read for non-technical Redash users.
+  - Empty query results are clearly displayed. User is now prompted to edit or execute the query.
+- Mobile Experience Improvements:
+  - UI element spacing has been redesigned for clarity
+  - Admin pages now honor max-width. Tables scroll independent of the top menu.
+  - Large legends no longer shrink the visualization on small screens.
+  - Fix: it was sometimes impossible to scroll pages with dashboards because the visualizations captured every touch event.
+  - Fix: Visualizations on small screens would not always show horizontal scroll bars.
+- Dashboards can now be un-archived using the API.
+- Dashboard UI performance was improved.
+- List pages were changed to show a user's name instead of avatar.
+- Search-enabled tables now show a prompt for which columns will be searched.
+- In the visualization editor, the settings pane now scrolls independent of the visualization preview.
+- Tokens in the schema viewer now sort alphabetically.
+- Links to settings panes that require Admin privileges are now hidden from non-Admins.
+- The Admin page now remembers which tab you were viewing after a page reload.
+
+### Visualizations
+
+- Feature: Allow bubble size control with either coefficient or sizemode.
+- Feature: Table visualization now treats Unix timestamps in query results as timestamps.
+- Feature: It's now possible to provide a description to each Table column, appearing in UI as a tooltip.
+- Feature: Added tooltip and popover templating to the map with markers visualization.
+- Feature: Added an organization setting to hide the Plotly mode bar on all visualizations.
+- Feature: Cohort visualization now has appearance settings.
+- Feature: Add option to explicitly set Chart legend position.
+- Change: Deprecated visualizations are now hidden.
+- Change: Table settings editor now extends vertically instead of horizontally.
+- Change: The maximum table pagination is now 500.
+- Change: Pie chart labels maintain contrast against lighter slices.
+- Fix: Chart series switched places when picking Y axis.
+- Fix: Third column was not selectable for Bubble and Heatmap charts.
+- Fix: On the counter visualizations, the “count rows” option showed an empty string instead of 0.
+- Fix: Table visualization with column named "children" rendered +/- buttons.
+- Fix: Sankey visualization now correctly occupies all available area even with fewer stages.
+- Fix: Pie chart ignores series labels.
+
+### Data Sources
+
+- New Data Sources: Amazon Cloudwatch, Amazon CloudWatch Logs Insights, Azure Kusto, Exasol.
+- Athena:
+  - Added the option to specify a base cost in settings, displaying a price for each query when executed.
+- BigQuery:
+  - Fix: large jobs continued running after the user clicked “Cancel” query execution.
+- Cassandra:
+  - Updated driver to 3.21.0 which dramatically reduces Docker build times.
+  - SSL options are now available.
+- Clickhouse:
+  - You can now choose whether to verify the SSL certificate.
+- Databricks:
+  - Databricks now use an ODBC-based connector.
+  - Fix: Date column was coerced to DateTime in the front-end.
+- Druid:
+  - Added username and password authentication option.
+- Microsoft SQL Server
+  - Added support for ODBC connections via pyodbc. There are now two MSSQL data source types. One using TDS. The other is using ODBC.
+- MongoDB:
+  - Added support for running queries on secondary in replicaset mode.
+  - Fix: Connection test always succeeded.
+- Oracle:
+  - Fix: Connection would fail if username or password contained special characters.
+  - Fix: Comparisons would fail if scale was None.
+- RDS:
+  - Updated rds-combined-ca-bundle.pem to the latest CA.
+- Redshift:
+  - Added the ability to use IAM Roles and Users.
+  - Fix: Redshift was unable to have its schema refreshed.
+- Rockset:
+  - Fix: Allow Redash to load collections in all workspaces.
+- Snowflake:
+  - You can now refresh the snowflake schema without waking the cluster.
+  - Added support for all of Snowflake’s datetime types. Otherwise certain timestamps would only appear as strings in the front-end.
+- TreasureData:
+  - Fix: API calls would fail when setting a non-default region.
+
+### Alerts
+
+- Feature: Added ability to mute alerts without deleting them.
+- Fix: numerical comparisons failed if value from query was a string.
+
+### Parameters
+
+- Added Last x Days options for date range parameters.
+- Fix: Parameters added in empty queries were always added as text parameters
+
+### Bug Fixes
+
+- Fix: Alembic migration schema was preventing v4 users from upgrading. In v5 we started encrypting data source credentials in the database.
+- Fix: System admin dashboard would not show correct database size if non-default name was used.
+- Fix: refresh_queries job would break if any query had a bad schedule object.
+- Fix: Orgs with LDAP enabled couldn’t disable password login.
+- Fix: SSL mode was sometimes sent as an empty string to the database instead of omitted entirely.
+- Fix: When creating new Map visualization with clustering disabled, map would crash on save.
+- Fix: It was possible on the New Query page to click “Save” multiple times, causing multiple new query records to be created.
+- Fix: Visualization render errors on a dashboard would crash the entire page.
+- Fix: A scheduled execution failure would modify the query’s “updated_at” timestamp.
+- Fix: Parameter UI would wrap awkwardly during some drag operations.
+- Fix: In dashboard edit mode, users couldn’t modify widgets.
+- Fix: Frontend error when parsing a NaN float.
+
+### Other
+
+- Added TSV as a download format (in addition to CSV and Excel).
+- Added maildev settings (helps with automated settings).
+- Refine permissions usage in Redash to allow for guest users
+- The query results API now explicitly handles 404 errors.
+- Forked queries now retain the tags of the original query.
+- We now allow setting custom Sentry environments.
+- Started using Black linter for our Python source code
+- Added CLI command to re-encrypt data source details with new secret key.
+- Favorites list is now loaded on menu click instead of on page load.
+- Administrators can now allow connections to private IP addresses.
+
 ## v8.0.0 - 2019-10-27
 
 There were no changes in this release since `v8.0.0-beta.2`. This is just to mark a stable release.
@@ -8,24 +152,23 @@ There were no changes in this release since `v8.0.0-beta.2`. This is just to mar
 
 This is an update to the previous beta release, which includes:
 
-* Add options for users to share anonymous usage information with us (see [docs](https://redash.io/help/open-source/admin-guide/usage-data) for details).
+- Add options for users to share anonymous usage information with us (see [docs](https://redash.io/help/open-source/admin-guide/usage-data) for details).
-* Visualizations:
+- Visualizations:
   - Allow the user to decide how to handle null values in charts.
-* Upgrade Sentry-SDK to latest version.
+- Upgrade Sentry-SDK to latest version.
-* Make horizontal table scroll visible in dashboard widgets without scrolling.
+- Make horizontal table scroll visible in dashboard widgets without scrolling.
-* Data Sources:
+- Data Sources:
-    * Add support for Azure Data Explorer (Kusto).
+  - Add support for Azure Data Explorer (Kusto).
-    * MySQL: fix connections without SSL configuration failing.
+  - MySQL: fix connections without SSL configuration failing.
-    * Amazon Redshift: option to set query group for adhoc/scheduled queries.
+  - Amazon Redshift: option to set query group for adhoc/scheduled queries.
-    * Hive: make error message more friendly.
+  - Hive: make error message more friendly.
-    * Qubole: add support to run Quantum queries.
+  - Qubole: add support to run Quantum queries.
-* Display data source icon in query editor.
+- Display data source icon in query editor.
-* Fix: allow users with view only acces to use the queries in Query Results
+- Fix: allow users with view only acces to use the queries in Query Results
-* Dashboard: when updating parameters refersh only widgets that use those parameters.
+- Dashboard: when updating parameters refersh only widgets that use those parameters.
 
 This release had contributions from 12 people: @arikfr, @cclauss, @gabrieldutra, @justinclift, @kravets-levko, @ranbena, @rauchy, @sandeepV2, @shinsuke-nara, @spacentropy, @sphenlee, @swfz.
 
-
 ## v8.0.0-beta - 2019-08-18
 
 After months of being heads down with hard work, it's finally time to wrap up the V8 release 🤩 This release includes many long awaited improvements to parameters, UX improvements, further React migration and other changes, fixes and improvements.

Dockerfile

@@ -10,8 +10,7 @@ RUN if [ "x$skip_frontend_build" = "x" ] ; then npm ci --unsafe-perm; fi
 
 COPY client /frontend/client
 COPY webpack.config.js /frontend/
-RUN if [ "x$skip_frontend_build" = "x" ] ; then npm run build; else mkdir /frontend/client/dist && touch /frontend/client/dist/multi_org.html && touch /frontend/client/dist/index.html; fi
+RUN if [ "x$skip_frontend_build" = "x" ] ; then npm run build; else mkdir -p /frontend/client/dist && touch /frontend/client/dist/multi_org.html && touch /frontend/client/dist/index.html; fi
-
 FROM python:3.7-slim
 
 EXPOSE 5000
@@ -58,7 +57,7 @@ RUN apt-get update && \
 ARG databricks_odbc_driver_url=https://databricks.com/wp-content/uploads/2.6.10.1010-2/SimbaSparkODBC-2.6.10.1010-2-Debian-64bit.zip
 ADD $databricks_odbc_driver_url /tmp/simba_odbc.zip
 RUN unzip /tmp/simba_odbc.zip -d /tmp/ \
-  && dpkg -i /tmp/SimbaSparkODBC-2.6.10.1010-2-Debian-64bit/simbaspark_2.6.10.1010-2_amd64.deb \
+  && dpkg -i /tmp/SimbaSparkODBC-*/*.deb \
   && echo "[Simba]\nDriver = /opt/simba/spark/lib/64/libsparkodbc_sb64.so" >> /etc/odbcinst.ini \
   && rm /tmp/simba_odbc.zip \
   && rm -rf /tmp/SimbaSparkODBC*

client/cypress/integration/visualizations/sankey_sunburst_spec.js

@@ -1,26 +1,29 @@
 /* global cy */
 
-import { createQuery } from "../../support/redash-api";
+import { createQuery, createDashboard, createVisualization, addWidget } from "../../support/redash-api";
+import { getWidgetTestId } from "../../support/dashboard";
 
 const SQL = `
-  SELECT 'a' AS stage1, 'a1' AS stage2, 11 AS value UNION ALL
+  SELECT 'a' AS s1, 'a1' AS s2, 'a2' AS s3, null AS s4, null AS s5, 11 AS value UNION ALL
-  SELECT 'a' AS stage1, 'a2' AS stage2, 12 AS value UNION ALL
+  SELECT 'a' AS s1, 'a2' AS s2, null AS s3, null AS s4, null AS s5, 12 AS value UNION ALL
-  SELECT 'a' AS stage1, 'a3' AS stage2, 45 AS value UNION ALL
+  SELECT 'a' AS s1, 'a3' AS s2, null AS s3, null AS s4, null AS s5, 45 AS value UNION ALL
-  SELECT 'a' AS stage1, 'a4' AS stage2, 54 AS value UNION ALL
+  SELECT 'a' AS s1, 'a4' AS s2, null AS s3, null AS s4, null AS s5, 54 AS value UNION ALL
-  SELECT 'b' AS stage1, 'b1' AS stage2, 33 AS value UNION ALL
+  SELECT 'b' AS s1, 'b1' AS s2, 'a2' AS s3, 'c1' AS s4, null AS s5, 33 AS value UNION ALL
-  SELECT 'b' AS stage1, 'b2' AS stage2, 73 AS value UNION ALL
+  SELECT 'b' AS s1, 'b2' AS s2, 'a4' AS s3, 'c3' AS s4, null AS s5, 73 AS value UNION ALL
-  SELECT 'b' AS stage1, 'b3' AS stage2, 90 AS value UNION ALL
+  SELECT 'b' AS s1, 'b3' AS s2, null AS s3, null AS s4, null AS s5, 90 AS value UNION ALL
-  SELECT 'c' AS stage1, 'c1' AS stage2, 19 AS value UNION ALL
+  SELECT 'c' AS s1, 'c1' AS s2, null AS s3, null AS s4, null AS s5, 19 AS value UNION ALL
-  SELECT 'c' AS stage1, 'c2' AS stage2, 92 AS value UNION ALL
+  SELECT 'c' AS s1, 'c2' AS s2, 'b2' AS s3, 'a2' AS s4, 'a3' AS s5, 92 AS value UNION ALL
-  SELECT 'c' AS stage1, 'c3' AS stage2, 63 AS value UNION ALL
+  SELECT 'c' AS s1, 'c3' AS s2, 'c4' AS s3, null AS s4, null AS s5, 63 AS value UNION ALL
-  SELECT 'c' AS stage1, 'c4' AS stage2, 44 AS v
+  SELECT 'c' AS s1, 'c4' AS s2, null AS s3, null AS s4, null AS s5, 44 AS value
 `;
 
 describe("Sankey and Sunburst", () => {
-  const viewportWidth = Cypress.config("viewportWidth");
-
   beforeEach(() => {
     cy.login();
+  });
+
+  describe("Creation through UI", () => {
+    beforeEach(() => {
       createQuery({ query: SQL }).then(({ id }) => {
         cy.visit(`queries/${id}/source`);
         cy.getByTestId("ExecuteButton").click();
@@ -40,10 +43,6 @@ describe("Sankey and Sunburst", () => {
         .find("svg")
         .should("exist");
 
-    // wait a bit before taking snapshot
-    cy.wait(500); // eslint-disable-line cypress/no-unnecessary-waiting
-    cy.percySnapshot("Visualizations - Sunburst", { widths: [viewportWidth] });
-
       cy.getByTestId("EditVisualizationDialog")
         .contains("button", "Save")
         .click();
@@ -65,10 +64,6 @@ describe("Sankey and Sunburst", () => {
         .find("svg")
         .should("exist");
 
-    // wait a bit before taking snapshot
-    cy.wait(500); // eslint-disable-line cypress/no-unnecessary-waiting
-    cy.percySnapshot("Visualizations - Sankey", { widths: [viewportWidth] });
-
       cy.getByTestId("EditVisualizationDialog")
         .contains("button", "Save")
         .click();
@@ -76,4 +71,77 @@ describe("Sankey and Sunburst", () => {
         .contains("span", visualizationName)
         .should("exist");
     });
+  });
+
+  const STAGES_WIDGETS = [
+    { name: "1 stage", query: `SELECT s1,value FROM (${SQL}) q`, position: { autoHeight: false, sizeY: 10, sizeX: 2 } },
+    {
+      name: "2 stages",
+      query: `SELECT s1,s2,value FROM (${SQL}) q`,
+      position: { autoHeight: false, col: 2, sizeY: 10, sizeX: 2 },
+    },
+    {
+      name: "3 stages",
+      query: `SELECT s1,s2,s3,value FROM (${SQL}) q`,
+      position: { autoHeight: false, col: 4, sizeY: 10, sizeX: 2 },
+    },
+    {
+      name: "4 stages",
+      query: `SELECT s1,s2,s3,s4,value FROM (${SQL}) q`,
+      position: { autoHeight: false, row: 9, sizeY: 10, sizeX: 2 },
+    },
+    {
+      name: "5 stages",
+      query: `SELECT s1,s2,s3,s4,s5,value FROM (${SQL}) q`,
+      position: { autoHeight: false, row: 9, col: 2, sizeY: 10, sizeX: 2 },
+    },
+  ];
+
+  it("takes a snapshot with Sunburst (1 - 5 stages)", function() {
+    createDashboard("Sunburst Visualization").then(dashboard => {
+      this.dashboardUrl = `/dashboard/${dashboard.slug}`;
+      return cy
+        .all(
+          STAGES_WIDGETS.map(sunburst => () =>
+            createQuery({ name: `Sunburst with ${sunburst.name}`, query: sunburst.query })
+              .then(queryData => createVisualization(queryData.id, "SUNBURST_SEQUENCE", "Sunburst", {}))
+              .then(visualization => addWidget(dashboard.id, visualization.id, { position: sunburst.position }))
+          )
+        )
+        .then(widgets => {
+          cy.visit(this.dashboardUrl);
+          widgets.forEach(widget => {
+            cy.getByTestId(getWidgetTestId(widget)).within(() => cy.get("svg").should("exist"));
+          });
+
+          // wait a bit before taking snapshot
+          cy.wait(500); // eslint-disable-line cypress/no-unnecessary-waiting
+          cy.percySnapshot("Visualizations - Sunburst");
+        });
+    });
+  });
+
+  it("takes a snapshot with Sankey (1 - 5 stages)", function() {
+    createDashboard("Sankey Visualization").then(dashboard => {
+      this.dashboardUrl = `/dashboard/${dashboard.slug}`;
+      return cy
+        .all(
+          STAGES_WIDGETS.map(sankey => () =>
+            createQuery({ name: `Sankey with ${sankey.name}`, query: sankey.query })
+              .then(queryData => createVisualization(queryData.id, "SANKEY", "Sankey", {}))
+              .then(visualization => addWidget(dashboard.id, visualization.id, { position: sankey.position }))
+          )
+        )
+        .then(widgets => {
+          cy.visit(this.dashboardUrl);
+          widgets.forEach(widget => {
+            cy.getByTestId(getWidgetTestId(widget)).within(() => cy.get("svg").should("exist"));
+          });
+
+          // wait a bit before taking snapshot
+          cy.wait(500); // eslint-disable-line cypress/no-unnecessary-waiting
+          cy.percySnapshot("Visualizations - Sankey");
+        });
+    });
+  });
 });

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "redash-client",
-  "version": "9.0.0-alpha",
+  "version": "9.0.0-beta",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {

package.json

@@ -1,6 +1,6 @@
 {
   "name": "redash-client",
-  "version": "9.0.0-alpha",
+  "version": "9.0.0-beta",
   "description": "The frontend part of Redash.",
   "main": "index.js",
   "scripts": {

redash/__init__.py

@@ -15,7 +15,7 @@ from .app import create_app  # noqa
 from .query_runner import import_query_runners
 from .destinations import import_destinations
 
-__version__ = "9.0.0-alpha"
+__version__ = "9.0.0-beta"
 
 
 if os.environ.get("REMOTE_DEBUG"):

viz-lib/src/visualizations/sankey/d3sankey.js

@@ -82,9 +82,11 @@ function Sankey() {
       x += 1;
     }
 
-    //
     moveSinksRight(x);
-    x = d3.max(nodes, n => n.x); // get new maximum x value
+    x = Math.max(
+      d3.max(nodes, n => n.x),
+      2
+    ); // get new maximum x value (min 2)
     scaleNodeBreadths((size[0] - nodeWidth) / (x - 1));
   }
 

viz-lib/src/visualizations/sankey/initSankey.js

@@ -69,6 +69,7 @@ function graph(data) {
     addLink(row[dataKeys[1]], row[dataKeys[2]], row.value || 0, 2);
     addLink(row[dataKeys[2]], row[dataKeys[3]], row.value || 0, 3);
     addLink(row[dataKeys[3]], row[dataKeys[4]], row.value || 0, 4);
+    addLink(row[dataKeys[4]], null, row.value || 0, 5); // this line ensures that the last stage has a corresponding exit node
   });
 
   const color = d3.scale.category20();