jprdonnelly/butler-sos
1
0
Fork 0
mirror of https://github.com/ptarmiganlabs/butler-sos.git synced 2025-12-19 17:58:18 -05:00
Code Issues Projects Releases Wiki Activity

5.0.0RC1

Browse Source
This commit is contained in:
Göran Sander
2019-09-12 20:37:56 +02:00
parent 77cc07b853
commit b1a6d17088
16 changed files with 400 additions and 389 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
README.md
View File

@@ -4,20 +4,34 @@
![Butler SOS](img/butler-sos-small.png)
Butler SenseOps Stats ("Butler SOS") is a DevOps monitoring tool for [Qlik Sense](https://www.qlik.com/us/products/qlik-sense).
It publishes operational Qlik Sense Enterprise metrics to [MQTT](https://en.wikipedia.org/wiki/MQTT) and [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/), from where it can be charted using tools like Grafana or acted on by downstream systems that listen to the MQTT topics used by Butler SOS.
Butler SOS uses the [Sense healthcheck API](http://help.qlik.com/en-US/sense-developer/November2017/Subsystems/EngineAPI/Content/GettingSystemInformation/HealthCheckStatus.htm) to gather operational metrics for the Sense servers specified in the YAML config file.
It also pulls log events from [Sense's Postgres logging database](http://help.qlik.com/en-US/sense/November2017/Subsystems/PlanningQlikSenseDeployments/Content/Deployment/Qlik-Logging-Service.htm), and forwards these to Influx and MQTT.
# === THIS IS A RELEASE CANDIDATE FOR v5.0. Do not use for production ===
**While mainly feature complete, documentation for v5.0 is still missing. Docs below are for previous versions of Butler SOS.**
**The final 5.0 version will introduce a separate doc site, with better structure etc. Stay tuned!**
--------------------
Butler SenseOps Stats ("Butler SOS") is a DevOps monitoring tool for [Qlik Sense](https://www.qlik.com/us/products/qlik-sense).
It publishes operational Qlik Sense Enterprise metrics to [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/) and [MQTT](https://en.wikipedia.org/wiki/MQTT), from where it can be charted using tools like Grafana or acted on by downstream systems that listen to the MQTT topics used by Butler SOS.
Butler SOS gathers operational metrics from several sources, including the [Sense healthcheck API](https://help.qlik.com/en-US/sense-developer/June2019/Subsystems/EngineAPI/Content/Sense_EngineAPI/GettingSystemInformation/HealthCheckStatus.htm) and [Session API](https://help.qlik.com/en-US/sense-developer/June2019/Subsystems/ProxyServiceAPI/Content/Sense_ProxyServiceAPI/ProxyServiceAPI-Session-Module-API.htm).
It also pulls log events from [Sense's Postgres logging database](https://help.qlik.com/en-US/sense/June2019/Subsystems/PlanningQlikSenseDeployments/Content/Sense_Deployment/Qlik-Logging-Service.htm), and forwards these to InfluxDB and MQTT.
**Why a separate tool for this?**
Good question. While Qlik Sense ships with a great Operations Monitor application, it is not useful or intended for real-time operational monitoring.
It is great for retrospective analysis of what happened in a Qlik Sense environment, but for a real-time understanding of what's going on in a Sense environment something else is needed - enter Butler SOS.
The Ops Monitor app is great for retrospective analysis of what happened in a Qlik Sense environment, but for a real-time understanding of what's going on in a Sense environment something else is needed - enter Butler SOS.
The most interesting use of Butler SOS is probably to create real-time dashboards based on the data in the Influx database, showing operational metrics for a Qlik Sense Enterprise environment.
A fully interactive demo dashboard is available [here](https://snapshot.raintank.io/dashboard/snapshot/1hNwAmi50lykKYXr6mswhKmll9myrH20?orgId=2).
The most common way of using Butler SOS is for creating real-time dashboards based on the data in the InfluxDB database, showing operational metrics for a Qlik Sense Enterprise environment.
A basic but fully interactive demo dashboard is available [here](https://snapshot.raintank.io/dashboard/snapshot/1hNwAmi50lykKYXr6mswhKmll9myrH20?orgId=2).
Sample screen shots:
Sample screen shots of [Grafana](https://grafana.com/) dashboards created using data extracted by Butler SOS:
![Grafana dashboard](img/SenseOps_dashboard_3.png "SenseOps dashboard showing errors and warnings, using Grafana")
@@ -31,48 +45,52 @@ Please see the [change log](https://github.com/ptarmiganlabs/butler-sos/blob/mas
Highlights in the most recent release are
### v4.0
### v5.0
Butler SOS is going through very active development, with significant new features added.
Once again, the format of the both the config file and the Influxdb schema has changed, which means that the SenseOps database in Influxdb has to be recreated.
This version adds a set of features that have been requested by quite a few people:
The upside is that version 4.0 adds several features that make Butler SOS easier to use in large Qlik Sense Enterprise environments with separated development, QA/acceptance, and production environments.
* Extract detailed user session data for specific virtual proxies. Previously it was only possible to see how many users/sessions were using Sense in total - no info on what specific users or what virtual proxies they use were extracted.
The new features make it possible to see exactly what users are connected right now, how many sessions each user has open, and what virtual proxies they are connected via.
* Data extracted by Butler SOS can now be stored in password protected InfluxDB databases.
* All data stored in InfluxDB is accompanied by a InfluxDB retention policy. This means there is now a way to make sure that the InfluxDB database does not grow beyond reasonable limits. Put differently: You can save detailed, fine-grained Sense metrics and specify that it should only be kept for (for example) 4 weeks. Any data older than the threshold is automatically purged from InfluxDB.
* Improved logging throughout the app makes it easier to debug and solve configuration issues that may arise.
Due to several new settings in the config file, it is recommended to completely review and update the file before deploying v4.0.
The new features in v5.0 means that Butler SOS' configuration file has a slightly new format. When upgrading to v5.0 from earlier versions you must ensure that your YAML config file meets the v5.0 format (see below for details).
* Added optional logging to disk file. If enabled, log files are rotated daily and stored for 30 days, after which they are automatically deleted.
* Improved tagging of data logged in Influxdb. Data can now be tagged with any number of user defined tags. This makes it possible to create much more refined dashboards in Grafana.
NOTE: these configurable tags are not compatible with previous Influx database schemas. The SenseOps database in Influxdb must be deleted before deploying Butler SOS v3.2. Next time Butler SOS is started a new SenseOps database in Influxdb will be created.
* Let the user control (by means of properties in the config file) which entries are extracted from Qlik Sense log db. This is configured on a per log level basis, for example "extract warning and errors, but not info messages".
## Install and setup
* Butler SOS has been tested with Qlik Sense Enterprise up until and including February 2019. Butler SOS uses core Sense APIs that are unlikely to change in future Sense versions. For that reasons Butler SOS is likely to work also with future Sense versions.
* Butler SOS has been tested with Qlik Sense Enterprise up until and including June 2019. Butler SOS uses core Sense APIs that are unlikely to change in future Sense versions. For that reasons Butler SOS is likely to work also with future Sense versions.
### Upgrading to v4
### Upgrading to v5
Version 4.0 introduces a slightly different schema for the InfluxDB database.
Version 5.0 introduces a slightly different schema for the InfluxDB database.
While it certainly is possible to migrate existing data, that will not be covered here. Let's instead drop the old InfluxDB database and start over with an empty one.
The steps to achieve this differ slightly depending on how you run Butler SOS. Conceptually they are:
*Running Butler SOS as a native Node.js app:*
* Stop Butler SOS if it is running
* From command line, run `influxdb -host <localhost or IP of InfluxDB server>`
* `show databases` to list InfluxDB databases on your Influxdb server
* `use SenseOps` within influx to select the Butler SOS database
* `drop database SenseOps` to delete the existing database. **WARNING! THERE IS NO WAY OF UN-DOING THIS!**
* `exit` will close the influx client
* Next time Butler SOS is started, an empty Influx database with the correct schema will be created
* Stop Butler SOS if it is running.
* From command line, run `influxdb --host <localhost or IP of InfluxDB server>`.
* `show databases` to list InfluxDB databases on your Influxdb server.
* `drop database <myDatabase>` to delete the existing database. **WARNING! THERE IS NO WAY OF UN-DOING THIS!**
* `create database <myDatabase>` to create a new, empty database.
* `create retention policy "14days" on "<myDatabase>" duration 14d replication 1` to create a new retention policy called "14days" that will keep data for 14 days before purging it from the SenseOps database. Note that you must later use the same retention policy in the YAML config file. Replace `<myDatabase>` with the name of your Butler SOS database in InfluxDB.
* `exit` will close the influx client.
* Use the database and retention policy/policies created above in your YAML config file.
* Next time Butler SOS starts it will use the newly created database and retention pocliy/policies.
*Running Butler SOS as a Docker container:*
* From command line, connect to the Docker container: `docker exec -it <container-name> /bin/bash`. <container-name> is the name given in the `docker-compose.yml` file, usually butler-sos.
* From command line, connect to the Docker container: `docker exec -it <container-name> /bin/bash`.
`<container-name>` is the name used in the `docker-compose.yml` file, usually butler-sos.
* From within the container, run `influxdb`.
* Follow the same steps as above ("use SenseOps", "drop database SenseOps", "exit"
* Follow the same steps as above ("show databases", "drop database", etc
* Exit the container by running `exit`
### Configuration file properties
Make a copy of ```./config/default-template.yaml```, rename the new file production.yaml. Edit as needed to match your Qlik Sense Enterprise environment.

18
changelog.md
View File

@@ -2,16 +2,22 @@
## v5.0
This release focuses on features requested by various people over the past years.
This release focuses on features requested by various people over the past couple of years.
They thus have their origins in real-life scenarios at various organisations around the world - hopefully they will also find wider use.
* More flexible use of InfluxDB, including authenticated (using InfluxDB's standard username/password authentication) access, and configurable port InfluxDB listens on.
* Extract data on what users have open sessions, broken down by virtual proxies.
This information is quite useful, it for example makes it easier to understand what users are affected by ongoing issues with a particular server, or for notifying all users connected to a particular virtual proxy about a pending server reboot etc.
* **FEATURE:** Extract data on what users have open sessions, broken down by virtual proxies.
This information is quite useful, it for example makes it easier to understand what users are affected by ongoing issues with a particular server, or for notifying all users connected to a particular virtual proxy about a pending server reboot etc. Another use case is to quickly identify when users have unreasonably many open sessions - which may be indicative of a Sense proxy that needs a restart.
The session information is stored in InfluxDB and/or sent as MQTT messages.
* General cleanup of the source code to make it easier to add new features in the future.
* **FEATURE**: More flexible use of InfluxDB, including authenticated (using InfluxDB's standard username/password authentication) access, and configurable port InfluxDB listens on.
* **FEATURE**: When starting Butler SOS for the first time, it will create a new database in InfluxDB. A new, default retention policy will also be created, based on info in Butler SOS' config file.
* **IMPROVEMENT**: When running in a Docker container, there is now a configurable limit to how many and large log files Docker will keep for the Butler SOS container.
* **BUG**: Fixed a couple of minor bugs around how tags are associated with Sense servers. The tagging feature is now pretty robust, and makes it possible to categorise Sense servers in a very flexible way. Those tags can then be used when creatign Grafana dashboards, making it easy to create and/or filter dashboards for all finance servers, all servers in Asia, all development servers etc. Very useful if you have many Sense servers!
* **MISC**: General cleanup of the source code to make it easier to add new features in the future. Docker image is now based on Node 12 (vs Node 8 previously).
## v4.0

BIN
img/SenseOps_dashboard_3.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 336 KiB

BIN
img/SenseOps_dashboard_4.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 414 KiB

BIN
img/user-sessions-1-large.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 192 KiB

BIN
img/user-sessions-1-small.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 153 KiB

4
src/Dockerfile
View File

@@ -1,5 +1,5 @@
# Use Node 8 LTS
FROM node:8
FROM node:12-stretch
# Add metadata about the image
LABEL maintainer="Göran Sander mountaindude@ptarmiganlabs.com"
@@ -22,7 +22,7 @@ RUN groupadd -r nodejs \
USER nodejs
# Set up Docker healthcheck
HEALTHCHECK --interval=12s --timeout=12s --start-period=30s CMD ["node", "healthcheck.js"]
HEALTHCHECK --interval=12s --timeout=12s --start-period=30s CMD ["node", "docker-healthcheck.js"]
CMD ["node", "butler-sos.js"]

112
src/butler-sos.js
View File

@@ -7,66 +7,72 @@ const healthMetrics = require('./lib/healthmetrics');
const logDb = require('./lib/logdb');
const sessionMetrics = require('./lib/sessionmetrics');
// Load certificates to use when connecting to healthcheck API
var path = require('path'),
certFile = path.resolve(__dirname, globals.config.get('Butler-SOS.cert.clientCert')),
keyFile = path.resolve(__dirname, globals.config.get('Butler-SOS.cert.clientCertKey')),
caFile = path.resolve(__dirname, globals.config.get('Butler-SOS.cert.clientCertCA'));
globals.initInfluxDB();
// ---------------------------------------------------
// Create restServer object
var restServer = restify.createServer({
name: 'Docker healthcheck for Butler-SOS',
version: globals.appVersion,
});
mainScript();
// Enable parsing of http parameters
restServer.use(restify.plugins.queryParser());
function mainScript() {
// Load certificates to use when connecting to healthcheck API
var path = require('path'),
certFile = path.resolve(__dirname, globals.config.get('Butler-SOS.cert.clientCert')),
keyFile = path.resolve(__dirname, globals.config.get('Butler-SOS.cert.clientCertKey')),
caFile = path.resolve(__dirname, globals.config.get('Butler-SOS.cert.clientCertCA'));
// Set up endpoint for Docker healthcheck REST server
restServer.get(
{
path: '/',
flags: 'i',
},
(req, res, next) => {
globals.logger.verbose(`MAIN: Docker healthcheck API endpoint called.`);
// ---------------------------------------------------
// Create restServer object
var restServer = restify.createServer({
name: 'Docker healthcheck for Butler-SOS',
version: globals.appVersion,
});
res.send(0);
next();
},
);
// Enable parsing of http parameters
restServer.use(restify.plugins.queryParser());
// Set specific log level (if/when needed to override the config file setting)
// Possible values are { error: 0, warn: 1, info: 2, verbose: 3, debug: 4, silly: 5 }
// Default is to use log level defined in config file
globals.logger.info('--------------------------------------');
globals.logger.info('Starting Butler SOS');
globals.logger.info(`Log level is: ${globals.getLoggingLevel()}`);
globals.logger.info(`App version is: ${globals.appVersion}`);
globals.logger.info('--------------------------------------');
// Set up endpoint for Docker healthcheck REST server
restServer.get(
{
path: '/',
flags: 'i',
},
(req, res, next) => {
globals.logger.verbose(`MAIN: Docker healthcheck API endpoint called.`);
// Log info about what Qlik Sense certificates are being used
globals.logger.debug(`Client cert: ${certFile}`);
globals.logger.debug(`Client cert key: ${keyFile}`);
globals.logger.debug(`CA cert: ${caFile}`);
res.send(0);
next();
},
);
// ---------------------------------------------------
// Set specific log level (if/when needed to override the config file setting)
// Possible values are { error: 0, warn: 1, info: 2, verbose: 3, debug: 4, silly: 5 }
// Default is to use log level defined in config file
globals.logger.info('--------------------------------------');
globals.logger.info('Starting Butler SOS');
globals.logger.info(`Log level is: ${globals.getLoggingLevel()}`);
globals.logger.info(`App version is: ${globals.appVersion}`);
globals.logger.info('--------------------------------------');
// Start Docker healthcheck REST server on port 12398
restServer.listen(12398, function() {
globals.logger.info('MAIN: Docker healthcheck server now listening');
});
// Log info about what Qlik Sense certificates are being used
globals.logger.debug(`Client cert: ${certFile}`);
globals.logger.debug(`Client cert key: ${keyFile}`);
globals.logger.debug(`CA cert: ${caFile}`);
// Set up extraction of data from log db
if (globals.config.get('Butler-SOS.logdb.enableLogDb') == true) {
logDb.setupLogDbTimer();
// ---------------------------------------------------
// Start Docker healthcheck REST server on port 12398
restServer.listen(12398, function() {
globals.logger.info('MAIN: Docker healthcheck server now listening');
});
// Set up extraction of data from log db
if (globals.config.get('Butler-SOS.logdb.enableLogDb') == true) {
logDb.setupLogDbTimer();
}
// Set up extraction of sessions data
if (globals.config.get('Butler-SOS.userSessions.enableSessionExtract') == true) {
sessionMetrics.setupUserSessionsTimer();
}
// Set up extraction on main metrics data (i.e. the Sense healthcheck API)
healthMetrics.setupHealthMetricsTimer();
}
// Set up extraction of sessions data
if (globals.config.get('Butler-SOS.userSessions.enableSessionExtract') == true) {
sessionMetrics.setupUserSessionsTimer();
}
// Set up extraction on main metrics data (i.e. the Sense healthcheck API)
healthMetrics.setupHealthMetricsTimer();

76
src/config/production_template.yaml
View File

@@ -9,7 +9,7 @@ Butler-SOS:
# Qlik Sense logging db config parameters
logdb:
enableLogDb: true
# Items below are mandatory if enabledLogDb=true
# Items below are mandatory if enableLogDb=true
pollingInterval: 60000 # How often (milliseconds) should Postgres log db be queried for warnings and errors?
queryPeriod: 5 minutes # How far back should Butler SOS query for log entries? Default is 5 min
host: <IP or FQDN of Qlik Sense logging db>
@@ -18,14 +18,14 @@ Butler-SOS:
qlogsReaderPwd: <pwd>
extractErrors: true # Should error level entries be extracted from log db into Influxdb?
extractWarnings: true # Should warn level entries be extracted from log db into Influxdb?
extractInfo: true # Should info level entries be extracted from log db into Influxdb?
extractInfo: true # Should info level entries be extracted from log db into Influxdb? Warning! Seting this to true will result in LOTS of log messages being retrrieved by Butler SOS!
# Certificates to use when querying Sense for healthcheck data. Get these from the Certificate Export in QMC.
cert:
clientCert: <path/to/cert/client.pem>
clientCertKey: <path/to/cert/client_key.pem>
clientCertCA: <path/to/cert/root.pem>
# If running Butler in a Docker container, the cert paths MUST be the following
# If running Butler SOS in a Docker container, the cert paths MUST be the following
# clientCert: /nodeapp/config/certificate/client.pem
# clientCertKey: /nodeapp/config/certificate/client_key.pem
# clientCertCA: /nodeapp/config/certificate/root.pem
@@ -49,6 +49,11 @@ Butler-SOS:
username: <username> # Username for Influxdb authentication. Mandatory if auth.enable=true
password: <password> # Password for Influxdb authentication. Mandatory if auth.enable=true
dbName: SenseOps
# Default retention policy that should be created in InfluxDB when Butler SOS creates a new database there.
# Any data older than retention policy threshold will be purged from InfluxDB.
retentionPolicy:
name: 4weeks
duration: 4w
# Control whether certain fields are stored in InfluxDB or not
# Use with caution! Enabling activeDocs, loadedDocs or inMemoryDocs may result in lots of data sent to InfluxDB.
includeFields:
@@ -58,20 +63,12 @@ Butler-SOS:
# Sessions per virtual proxy
userSessions:
enableSessionExtract: true # Query unique user IDs of what users have sessions open (true/false)?
enableSessionExtract: true # Query unique user IDs of what users have sessions open (true/false)?
# Items below are mandatory if enableSessionExtract=true
pollingInterval: 10000 # How often (milliseconds) should session data be polled?
influxDbRetentionPolicy: 7days
servers: # What hosts and including virtual proxies, should we get sessions for?
- host: sense1.mydomain.com:4243
virtualProxy: # Default virtual proxy
- host: sense2.mydomain.com:4243
virtualProxy: /finance # "finance" virtual proxy
- host: sense2.mydomain.com:4243
virtualProxy: /loadtest # "loadtest" virtual proxy
pollingInterval: 15000 # How often (milliseconds) should detailed session data be polled?
serversToMonitor:
pollingInterval: 30000 # How often (milliseconds) should the healthcheck API be polled?
pollingInterval: 30000 # How often (milliseconds) should the healthcheck API be polled?
# List of extra tags for each server. Useful for creating more advanced Grafana dashboards.
# Each server below MUST include these tags in its serverTags property.
@@ -84,22 +81,35 @@ Butler-SOS:
# Sense Servers that should be queried for healthcheck data
servers:
- host: <server1.my.domain>
serverName: <server1>
serverDescription: <description>
logDbHost: <host name as used in QLogs db>
serverTags:
server_group: DEV
serverLocation: Asia
server-type: virtual
serverBrand: Dell
- host: <server2.my.domain>
serverName: <server2>
serverDescription: <description>
logDbHost: <host name as used in QLogs db>
serverTags:
server_group: PROD
serverLocation: Europe
server-type: physical
serverBrand: HP
- host: <server1.my.domain>
serverName: <server1>
serverDescription: <description>
logDbHost: <host name as used in QLogs db>
userSessions:
enable: true
# Items below are mandatory if userSessions.enable=true
host: <server1.my.domain>:4243
virtualProxies:
- virtualProxy: / # Default virtual proxy
- virtualProxy: /hdr # "hdr" virtual proxy
serverTags:
server_group: DEV
serverLocation: Asia
server-type: virtual
serverBrand: Dell
- host: <server2.my.domain>
serverName: <server2>
serverDescription: <description>
logDbHost: <host name as used in QLogs db>
userSessions:
enable: true
# Items below are mandatory if userSessions.enable=true
host: <server2.my.domain>:4243
virtualProxies:
- virtualProxy: /finance # "finance" virtual proxy
serverTags:
server_group: PROD
serverLocation: Europe
server-type: physical
serverBrand: HP

9
src/docker-compose.yml
View File

@@ -2,7 +2,7 @@
version: '3.3'
services:
butler-sos:
image: ptarmiganlabs/butler-sos:latest
image: ptarmiganlabs/butler-sos:5.0.0
container_name: butler-sos
restart: always
volumes:
@@ -10,6 +10,9 @@ services:
- "./config:/nodeapp/config"
- "./logs:/nodeapp/logs"
environment:
- "NODE_ENV=production"
- "NODE_ENV=production" # Means that Butler SOS will read config data from production.yaml
logging:
driver: json-file
driver: "json-file"
options:
max-file: "5"
max-size: "5m"

114
src/globals.js
View File

@@ -59,8 +59,6 @@ getLoggingLevel = () => {
// Get info on what servers to monitor
const serverList = config.get('Butler-SOS.serversToMonitor.servers');
// Get info on what virtual proxies to get session data for
// Set up connection pool for accessing Qlik Sense log db
const pgPool = new Pool({
host: config.get('Butler-SOS.logdb.host'),
@@ -192,81 +190,52 @@ const influx = new Influx.InfluxDB({
],
});
if (config.get('Butler-SOS.influxdbConfig.enableInfluxdb')) {
influx
.getDatabaseNames()
.then(names => {
if (!names.includes(config.get('Butler-SOS.influxdbConfig.dbName'))) {
logger.info(`CONFIG: Creating Influx database.`);
return influx.createDatabase(config.get('Butler-SOS.influxdbConfig.dbName'));
}
})
.then(() => {
logger.info(`CONFIG: Connected to Influx database.`);
return;
})
function initInfluxDB() {
const dbName = config.get('Butler-SOS.influxdbConfig.dbName');
const enableInfluxdb = config.get('Butler-SOS.influxdbConfig.enableInfluxdb');
// Verify existance of retention policies
.then(() => {
logger.info(`CONFIG: Making sure Influxdb retention policies exist...`);
if (enableInfluxdb) {
influx
.getDatabaseNames()
.then(names => {
if (!names.includes(dbName)) {
influx
.createDatabase(dbName)
.then(() => {
logger.info(`CONFIG: Created new InfluxDB database: ${dbName}`);
influx
.showRetentionPolicies()
.then(retentionPolicies => {
// Make sure InfluxDB retention policy for main health metrics exists (if specified)
// If it needs to be created, something like 'create retention policy "14days" on "SenseOps" duration 14d replication 1' can be used from within Influxdb command line client.
var retentionPolicyMatch = retentionPolicies.filter(
retentionPolicy =>
retentionPolicy.name ===
config.get('Butler-SOS.serversToMonitor.influxDbRetentionPolicy'),
);
const newPolicy = config.get('Butler-SOS.influxdbConfig.retentionPolicy');
if (config.has('Butler-SOS.serversToMonitor.influxDbRetentionPolicy')) {
if (retentionPolicyMatch.length == 0) {
// Create new default retention policy
influx
.createRetentionPolicy(newPolicy.name, {
database: dbName,
duration: newPolicy.duration,
replication: 1,
isDefault: true,
})
.then(() => {
logger.info(`CONFIG: Created new InfluxDB retention policy: ${newPolicy.name}`);
})
.catch(err => {
logger.error(
`CONFIG: Error creating new InfluxDB retention policy "${newPolicy.name}"! ${err.stack}`,
);
});
})
.catch(err => {
logger.error(
`CONFIG: Retention policy ${config.get(
'Butler-SOS.serversToMonitor.influxDbRetentionPolicy',
)} does not exist in InfluxDB. Exiting.`,
`CONFIG: Error creating new InfluxDB database "${dbName}"! ${err.stack}`,
);
process.exit(1);
}
}
// Make sure InfluxDB retention policy for user sessions exists
// If it needs to be created, something like 'create retention policy "7days" on "SenseOps" duration 7d replication 1' can be used from within Influxdb command line client.
retentionPolicyMatch = retentionPolicies.filter(
retentionPolicy =>
retentionPolicy.name ===
config.get('Butler-SOS.userSessions.influxDbRetentionPolicy'),
);
if (config.get('Butler-SOS.userSessions.enableSessionExtract')) {
if (config.has('Butler-SOS.userSessions.influxDbRetentionPolicy')) {
if (retentionPolicyMatch.length == 0) {
logger.error(
`CONFIG: Retention policy ${config.get(
'Butler-SOS.userSessions.influxDbRetentionPolicy',
)} does not exist in InfluxDB. Exiting.`,
);
process.exit(1);
}
}
}
})
.catch(err => {
logger.error(
`CONFIG: Error getting list of existing retention policies in InfluxDB. Make sure the retention policies used in YAML config really exist in Influxdb. Exiting.`,
);
logger.error(`CONFIG: ${JSON.stringify(err, null, 2)}`);
process.exit(1);
});
})
.catch(err => {
logger.error(`CONFIG: Error creating/connecting to/verifying Influx database:`);
logger.error(`CONFIG: ${err}`);
});
});
} else {
logger.info(`CONFIG: Found InfluxDB database: ${dbName}`);
}
})
.catch(err => {
logger.error(`CONFIG: Error getting list of InfuxDB databases! ${err.stack}`);
});
}
}
// ------------------------------------
@@ -293,4 +262,5 @@ module.exports = {
pgPool,
appVersion,
serverList,
initInfluxDB,
};

27
src/lib/logdb.js
View File

@@ -13,10 +13,6 @@ function setupLogDbTimer() {
globals.logger.verbose('LOGDB: Event started: Query log db');
// Create list of logging levels to include in query
// extractErrors: true
// extractWarnings: true
// extractInfo: false
let arrayincludeLogLevels = [];
if (globals.config.get('Butler-SOS.logdb.extractErrors')) {
arrayincludeLogLevels.push("'ERROR'");
@@ -117,23 +113,16 @@ function setupLogDbTimer() {
// Write the whole reading to Influxdb
globals.influx
.writePoints(
[
{
measurement: 'log_event',
tags: tagsForDbEntry,
fields: {
message: row.payload.Message,
},
timestamp: row.timestamp,
},
],
.writePoints([
{
retentionPolicy: globals.config.get(
'Butler-SOS.logdb.influxDbRetentionPolicy',
),
measurement: 'log_event',
tags: tagsForDbEntry,
fields: {
message: row.payload.Message,
},
timestamp: row.timestamp,
},
)
])
.then(err => {
globals.logger.silly('LOGDB: Sent log db event to Influxdb');
})

229
src/lib/post-to-influxdb.js
View File

@@ -1,4 +1,5 @@
const globals = require('../globals');
var _ = require('lodash');
function postHealthMetricsToInfluxdb(host, body, influxTags) {
// Calculate server uptime
@@ -38,101 +39,98 @@ function postHealthMetricsToInfluxdb(host, body, influxTags) {
days + ' days, ' + hours + 'h ' + minutes.substr(-2) + 'm ' + seconds.substr(-2) + 's';
// Build tags structure that will be passed to InfluxDB
globals.logger.debug(`HEALTH METRICS: Health data: Tags sent to InfluxDB: ${JSON.stringify(influxTags)}`);
globals.logger.debug(
`HEALTH METRICS: Health data: Tags sent to InfluxDB: ${JSON.stringify(influxTags)}`,
);
// Write the whole reading to Influxdb
globals.influx
.writePoints(
[
{
measurement: 'sense_server',
tags: influxTags,
fields: {
version: body.version,
started: body.started,
uptime: formattedTime,
},
},
{
measurement: 'mem',
tags: influxTags,
fields: {
comitted: body.mem.comitted,
allocated: body.mem.allocated,
free: body.mem.free,
},
},
{
measurement: 'apps',
tags: influxTags,
fields: {
active_docs_count: body.apps.active_docs.length,
loaded_docs_count: body.apps.loaded_docs.length,
in_memory_docs_count: body.apps.in_memory_docs.length,
active_docs: globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
? body.apps.active_docs
: '',
loaded_docs: globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
? body.apps.loaded_docs
: '',
in_memory_docs: globals.config.get(
'Butler-SOS.influxdbConfig.includeFields.inMemoryDocs',
)
? body.apps.in_memory_docs
: '',
calls: body.apps.calls,
selections: body.apps.selections,
},
},
{
measurement: 'cpu',
tags: influxTags,
fields: {
total: body.cpu.total,
},
},
{
measurement: 'session',
tags: influxTags,
fields: {
active: body.session.active,
total: body.session.total,
},
},
{
measurement: 'users',
tags: influxTags,
fields: {
active: body.users.active,
total: body.users.total,
},
},
{
measurement: 'cache',
tags: influxTags,
fields: {
hits: body.cache.hits,
lookups: body.cache.lookups,
added: body.cache.added,
replaced: body.cache.replaced,
bytes_added: body.cache.bytes_added,
},
},
{
measurement: 'saturated',
tags: influxTags,
fields: {
saturated: body.saturated,
},
},
],
.writePoints([
{
retentionPolicy: globals.config.get('Butler-SOS.serversToMonitor.influxDbRetentionPolicy'),
measurement: 'sense_server',
tags: influxTags,
fields: {
version: body.version,
started: body.started,
uptime: formattedTime,
},
},
)
{
measurement: 'mem',
tags: influxTags,
fields: {
comitted: body.mem.comitted,
allocated: body.mem.allocated,
free: body.mem.free,
},
},
{
measurement: 'apps',
tags: influxTags,
fields: {
active_docs_count: body.apps.active_docs.length,
loaded_docs_count: body.apps.loaded_docs.length,
in_memory_docs_count: body.apps.in_memory_docs.length,
active_docs: globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
? body.apps.active_docs
: '',
loaded_docs: globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
? body.apps.loaded_docs
: '',
in_memory_docs: globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
? body.apps.in_memory_docs
: '',
calls: body.apps.calls,
selections: body.apps.selections,
},
},
{
measurement: 'cpu',
tags: influxTags,
fields: {
total: body.cpu.total,
},
},
{
measurement: 'session',
tags: influxTags,
fields: {
active: body.session.active,
total: body.session.total,
},
},
{
measurement: 'users',
tags: influxTags,
fields: {
active: body.users.active,
total: body.users.total,
},
},
{
measurement: 'cache',
tags: influxTags,
fields: {
hits: body.cache.hits,
lookups: body.cache.lookups,
added: body.cache.added,
replaced: body.cache.replaced,
bytes_added: body.cache.bytes_added,
},
},
{
measurement: 'saturated',
tags: influxTags,
fields: {
saturated: body.saturated,
},
},
])
.then(() => {
globals.logger.verbose(`HEALTH METRICS: Sent health data to Influxdb for server ${influxTags.server_name}`);
globals.logger.verbose(
`HEALTH METRICS: Sent health data to Influxdb for server ${influxTags.server_name}`,
);
})
.catch(err => {
@@ -141,7 +139,9 @@ function postHealthMetricsToInfluxdb(host, body, influxTags) {
}
function postUserSessionsToInfluxdb(host, virtualProxy, body, influxTags) {
globals.logger.debug(`USER SESSIONS: User sessions body received (VP=${virtualProxy}): ${JSON.stringify(body)}`);
globals.logger.debug(
`USER SESSIONS: User sessions body received (VP=${virtualProxy}): ${JSON.stringify(body)}`,
);
globals.logger.debug(
`USER SESSIONS: User session: Shared tags sent to InfluxDB (VP=${virtualProxy}): ${JSON.stringify(
influxTags,
@@ -150,7 +150,7 @@ function postUserSessionsToInfluxdb(host, virtualProxy, body, influxTags) {
// Build tags structure that will be passed to InfluxDB
// Get local copy of tags, then add user session specific tags
let tmpTags = influxTags;
let tmpTags = _.cloneDeep(influxTags);
tmpTags.user_session_virtual_proxy = virtualProxy;
tmpTags.user_session_host = host;
@@ -180,11 +180,12 @@ function postUserSessionsToInfluxdb(host, virtualProxy, body, influxTags) {
globals.logger.debug(`USER SESSIONS: User session: Body item: ${JSON.stringify(bodyItem)}`);
// Start over with fresh copy of shared tags
tmpTags = influxTags;
tmpTags = _.cloneDeep(influxTags);
tmpTags.user_session_virtual_proxy = virtualProxy;
tmpTags.user_session_host = host;
// Add extra tags for this body item
tmpTags.user_session_id = bodyItem.SessionId;
tmpTags.user_session_user_directory = bodyItem.UserDirectory;
tmpTags.user_session_user_id = bodyItem.UserId;
@@ -200,33 +201,35 @@ function postUserSessionsToInfluxdb(host, virtualProxy, body, influxTags) {
};
datapoint.push(sessionDatapoint);
});
globals.influx
.writePoints(datapoint, {
retentionPolicy: globals.config.get('Butler-SOS.userSessions.influxDbRetentionPolicy'),
})
.then(() => {
globals.logger.silly(
`USER SESSIONS: Influxdb measurements for server "${host}", virtual proxy "${virtualProxy}"": ${JSON.stringify(
datapoint,
null,
2,
)}`,
);
globals.logger.debug(`USER SESSIONS: Session count for server "${host}", virtual proxy "${virtualProxy}"": ${body.length}`);
globals.logger.debug(`USER SESSIONS: User list for server "${host}", virtual proxy "${virtualProxy}"": ${uniqueUserList}`);
globals.logger.verbose(`USER SESSIONS: Sent user session data to InfluxDB for server "${host}", virtual proxy "${virtualProxy}"`);
})
.catch(err => {
globals.logger.error(`USER SESSIONS: Error saving user session data to InfluxDB! ${err.stack}`);
});
.writePoints(datapoint)
.then(() => {
globals.logger.silly(
`USER SESSIONS: Influxdb datapoint for server "${host}", virtual proxy "${virtualProxy}"": ${JSON.stringify(
datapoint,
null,
2,
)}`,
);
globals.logger.debug(
`USER SESSIONS: Session count for server "${host}", virtual proxy "${virtualProxy}"": ${body.length}`,
);
globals.logger.debug(
`USER SESSIONS: User list for server "${host}", virtual proxy "${virtualProxy}"": ${uniqueUserList}`,
);
globals.logger.verbose(
`USER SESSIONS: Sent user session data to InfluxDB for server "${host}", virtual proxy "${virtualProxy}"`,
);
})
.catch(err => {
globals.logger.error(
`USER SESSIONS: Error saving user session data to InfluxDB! ${err.stack}`,
);
});
}
module.exports = {

105
src/lib/sessionmetrics.js
View File

@@ -56,62 +56,67 @@ function getSessionStatsFromSense(host, virtualProxy, influxTags) {
);
globals.logger.debug(`USER SESSIONS: Querying user sessions from ${fullUrl}`);
request(
{
followRedirect: true,
url: fullUrl,
method: 'GET',
headers: {
'Cache-Control': 'no-cache',
'Content-Type': 'application/json',
'X-Qlik-Xrfkey': 'abcdefghij987654',
XVirtualProxy: virtualProxy,
try {
request(
{
followRedirect: true,
url: fullUrl,
method: 'GET',
headers: {
'Cache-Control': 'no-cache',
'Content-Type': 'application/json',
'X-Qlik-Xrfkey': 'abcdefghij987654',
XVirtualProxy: virtualProxy,
},
json: true,
cert: fs.readFileSync(certFile),
key: fs.readFileSync(keyFile),
ca: fs.readFileSync(caFile),
rejectUnauthorized: false,
requestCert: true,
agent: false,
},
json: true,
cert: fs.readFileSync(certFile),
key: fs.readFileSync(keyFile),
ca: fs.readFileSync(caFile),
rejectUnauthorized: false,
requestCert: true,
agent: false,
},
function(error, response, body) {
// Check for error
globals.logger.debug(`USER SESSIONS: User session response from: ${response.request.href}`);
function(error, response, body) {
// Check for error
if (error) {
globals.logger.error(`USER SESSIONS: Error when calling proxy session API: ${error}`);
globals.logger.error(`USER SESSIONS: Response: ${response}`);
globals.logger.error(`USER SESSIONS: Body: ${body}`);
return;
}
globals.logger.debug(`USER SESSIONS: User session response from: ${response.request.href}`);
if (error) {
globals.logger.error(`USER SESSIONS: Error when calling proxy session API: ${error}`);
globals.logger.error(`USER SESSIONS: Response: ${response}`);
globals.logger.error(`USER SESSIONS: Body: ${body}`);
return;
}
if (!error && response.statusCode === 200) {
// globals.logger.verbose('Received ok response from ' + influxTags.host);
globals.logger.debug(
`USER SESSIONS: Body from ${response.request.href}: ${JSON.stringify(body, null, 2)}`,
);
// Post to MQTT (if enabled)
if (globals.config.get('Butler-SOS.mqttConfig.enableMQTT')) {
globals.logger.debug('USER SESSIONS: Calling user sessions MQTT posting method');
postToMQTT.postUserSessionsToMQTT(
response.request.uri.hostname,
response.request.headers.XVirtualProxy,
JSON.stringify(body, null, 2),
if (!error && response.statusCode === 200) {
// globals.logger.verbose('Received ok response from ' + influxTags.host);
globals.logger.debug(
`USER SESSIONS: Body from ${response.request.href}: ${JSON.stringify(body, null, 2)}`,
);
}
// Post to Influxdb (if enabled)
if (globals.config.get('Butler-SOS.influxdbConfig.enableInfluxdb')) {
globals.logger.debug('USER SESSIONS: Calling user sessions Influxdb posting method');
// Post to MQTT (if enabled)
if (globals.config.get('Butler-SOS.mqttConfig.enableMQTT')) {
globals.logger.debug('USER SESSIONS: Calling user sessions MQTT posting method');
postToInfluxdb.postUserSessionsToInfluxdb(host, virtualProxy, body, influxTags);
postToMQTT.postUserSessionsToMQTT(
response.request.uri.hostname,
response.request.headers.XVirtualProxy,
JSON.stringify(body, null, 2),
);
}
// Post to Influxdb (if enabled)
if (globals.config.get('Butler-SOS.influxdbConfig.enableInfluxdb')) {
globals.logger.debug('USER SESSIONS: Calling user sessions Influxdb posting method');
postToInfluxdb.postUserSessionsToInfluxdb(host, virtualProxy, body, influxTags);
}
}
}
},
);
},
);
} catch (err) {
globals.logger.error(
`USER SESSIONS: Error reading user sessions from host:${host}, virtual proxy:${virtualProxy}: ${err}`,
);
}
}
module.exports = {

14
src/package-lock.json generated
View File

@@ -677,9 +677,9 @@
"integrity": "sha512-lUGBnIamTAwk4znq5BcqsDaxSmZ9nDVJaij6NvRt/Tg4R69gERA+otPKbS86ROw9nxVMw2/mp1fnaiWqbs6Sdg=="
},
"file-stream-rotator": {
"version": "0.5.4",
"resolved": "https://registry.npmjs.org/file-stream-rotator/-/file-stream-rotator-0.5.4.tgz",
"integrity": "sha512-V/OHy0VhdDMEhvnlWqiRUYjO4JbIVNjKQ9lPlasJkR+bHWJdjbeVdFObdfme1C+7wfEJDjdLZfdzE+GMctOFAw==",
"version": "0.5.5",
"resolved": "https://registry.npmjs.org/file-stream-rotator/-/file-stream-rotator-0.5.5.tgz",
"integrity": "sha512-XzvE1ogpxUbARtZPZLICaDRAeWxoQLFMKS3ZwADoCQmurKEwuDD2jEfDVPm/R1HeKYsRYEl9PzVIezjQ3VTTPQ==",
"requires": {
"moment": "^2.11.2"
}
@@ -2143,11 +2143,11 @@
}
},
"winston-daily-rotate-file": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/winston-daily-rotate-file/-/winston-daily-rotate-file-4.0.0.tgz",
"integrity": "sha512-JWoYu+2Z9mlqRpeZu+CZ47hnYfmo+QjxdAfHjSJpJumqtu0k4bdoNe2W3XsPRFe5M4gb5jKOobTZ/OK7oCdhKg==",
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/winston-daily-rotate-file/-/winston-daily-rotate-file-4.1.0.tgz",
"integrity": "sha512-Y3wcdrgNwC75Lc2BiFDZUyfxNjyhqVzEou8uOyZMUkDJ23/UN+rq8arpJDaXwX512GIM7g6NpTDPac29fsCvcQ==",
"requires": {
"file-stream-rotator": "^0.5.4",
"file-stream-rotator": "^0.5.5",
"object-hash": "^1.3.0",
"triple-beam": "^1.3.0",
"winston-transport": "^4.2.0"

5
src/package.json
View File

@@ -1,6 +1,6 @@
{
"name": "butler-sos",
"version": "5.0.0",
"version": "5.0.0RC1",
"description": "Butler SenseOps Stats (\"Butler SOS\") is a Node.js service publishing operational Qlik Sense metrics to MQTT and Influxdb.",
"main": "butler-sos.js",
"scripts": {
@@ -32,13 +32,14 @@
"influx": "^5.4.1",
"js-yaml": "^3.13.1",
"jshint": "^2.10.2",
"lodash": "^4.17.15",
"mqtt": "^3.0.0",
"pg": "^7.12.1",
"request": "^2.88.0",
"restify": "^8.4.0",
"url-join": "^4.0.1",
"winston": "^3.2.1",
"winston-daily-rotate-file": "^4.0.0"
"winston-daily-rotate-file": "^4.1.0"
},
"devDependencies": {
"prettier": "1.18.2"