mirror of
https://github.com/ptarmiganlabs/butler-sos.git
synced 2025-12-19 09:47:53 -05:00
Merge upstream/master into copilot/review-qlick-sense-api-code and resolve conflicts
This commit is contained in:
Göran Sander
@@ -1,731 +0,0 @@
|
||||
# Butler SOS Build Process Analysis & Improvement Recommendations
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The Butler SOS project has a reasonably comprehensive build process but has significant opportunities for improvement in security, efficiency, and modern development practices. This analysis identifies 15 key areas for enhancement across build automation, security, testing, and deployment.
|
||||
|
||||
## Current Build Process Assessment
|
||||
|
||||
### Strengths
|
||||
|
||||
- ✅ **Comprehensive CI/CD Pipeline**: Well-structured GitHub Actions workflows for different platforms
|
||||
- ✅ **Multiple Target Platforms**: Supports macOS (x64, ARM64), Linux, and Docker
|
||||
- ✅ **Code Signing & Notarization**: Proper Apple code signing and notarization for macOS builds
|
||||
- ✅ **Release Automation**: Uses release-please for automated versioning and releases
|
||||
- ✅ **Security Scanning**: CodeQL active, Snyk implemented in insiders-build workflow, SBOM generation active in ci.yaml, and basic dependency checks
|
||||
- ✅ **Code Quality**: ESLint, Prettier, and CodeClimate integration
|
||||
- ✅ **Testing Framework**: Jest setup with coverage reporting
|
||||
- ✅ **Dependency Management**: Dependabot for automated dependency updates
|
||||
|
||||
### Critical Issues Identified
|
||||
|
||||
- 🔴 **Security vulnerabilities** in build process
|
||||
- 🔴 **Inefficient workflows** causing unnecessary resource usage
|
||||
- 🔴 **Missing modern build optimizations**
|
||||
- 🔴 **Incomplete testing coverage**
|
||||
- 🔴 **Outdated tooling and practices**
|
||||
|
||||
---
|
||||
|
||||
## Detailed Improvement Recommendations
|
||||
|
||||
### 1. Security Enhancements (HIGH PRIORITY)
|
||||
|
||||
#### 1.1 Consolidate and Enhance Snyk Security Scanning
|
||||
|
||||
**Current State**:
|
||||
|
||||
- ✅ Snyk is actively implemented in `insiders-build.yaml` workflow with SARIF upload
|
||||
- ✅ Snyk security scripts are configured in `package.json`
|
||||
- ✅ Snyk scanning is intentionally limited to insiders builds only (by design)
|
||||
- ✅ Previous separate `snyk-security._yml` workflow has been removed
|
||||
|
||||
**Analysis**:
|
||||
|
||||
- ✅ Snyk scanning is working properly in insiders build workflow with SARIF integration
|
||||
- ✅ Local Snyk testing available via `npm run security:full`
|
||||
- ✅ Snyk scanning scope is appropriately limited to development/insider builds
|
||||
- ✅ Clean workflow structure with no duplicate or unused Snyk configurations
|
||||
|
||||
**Current Implementation Status**:
|
||||
|
||||
- Snyk security scanning is properly implemented and working as intended
|
||||
- No additional Snyk workflow changes needed - current setup is optimal
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```bash
|
||||
# Add to package.json scripts
|
||||
"security:audit": "npm audit --audit-level=high",
|
||||
"security:full": "npm run security:audit && snyk test --severity-threshold=high"
|
||||
```
|
||||
|
||||
#### 1.2 Implement Supply Chain Security
|
||||
|
||||
**Missing**: Software Bill of Materials (SBOM) generation, dependency validation, and license compliance
|
||||
|
||||
**Current State**: Basic dependency management with Dependabot, but no comprehensive supply chain security
|
||||
|
||||
**Free Tools & Implementation Options**:
|
||||
|
||||
**A. Software Bill of Materials (SBOM) Generation**
|
||||
|
||||
**Current Implementation**: Using Microsoft SBOM Tool in CI/CD workflows
|
||||
|
||||
```bash
|
||||
# Microsoft SBOM Tool (already implemented in ci.yaml)
|
||||
# Downloads and uses: https://github.com/microsoft/sbom-tool/releases/latest/download/sbom-tool-linux-x64
|
||||
|
||||
# Alternative: CycloneDX (if you want local generation)
|
||||
npm install --save-dev @cyclonedx/cyclonedx-npm
|
||||
```
|
||||
|
||||
**Add to package.json scripts** (optional for local development):
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"sbom:generate": "cyclonedx-npm --output-file sbom.json",
|
||||
"sbom:validate": "cyclonedx-npm --validate",
|
||||
"security:sbom": "npm run sbom:generate && npm run sbom:validate"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: Microsoft SBOM Tool is already configured in your `ci.yaml` workflow and generates SPDX 2.2 format SBOMs that are automatically uploaded to GitHub releases.
|
||||
|
||||
**B. Dependency Pinning & Validation (FREE)**
|
||||
|
||||
```bash
|
||||
# Install dependency validation tools
|
||||
npm install --save-dev npm-check-updates
|
||||
npm install --save-dev audit-ci
|
||||
npm install --save-dev lockfile-lint
|
||||
```
|
||||
|
||||
**Add to package.json scripts**:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"deps:check": "ncu --doctor",
|
||||
"deps:audit": "audit-ci --config .audit-ci.json",
|
||||
"deps:lockfile": "lockfile-lint --path package-lock.json --validate-https --validate-integrity",
|
||||
"security:deps": "npm run deps:lockfile && npm run deps:audit"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**C. License Compliance Checking (FREE)**
|
||||
|
||||
**Current Implementation**: ✅ **Active** - `license-checker-rseidelsohn` is installed and configured with comprehensive npm scripts
|
||||
|
||||
**Current Scripts** (already implemented in package.json):
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"license:check": "license-checker-rseidelsohn --onlyAllow 'MIT;Apache-2.0;BSD-2-Clause;BSD-3-Clause;ISC;0BSD'",
|
||||
"license:report": "license-checker-rseidelsohn --csv --out licenses.csv",
|
||||
"license:summary": "license-checker-rseidelsohn --summary",
|
||||
"license:json": "license-checker-rseidelsohn --json --out licenses.json",
|
||||
"license:full": "npm run license:summary && npm run license:check && npm run license:report"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Available Commands**:
|
||||
|
||||
- `npm run license:check` - Validates only approved licenses (fails on non-compliant licenses)
|
||||
- `npm run license:report` - Generates CSV report (`licenses.csv`)
|
||||
- `npm run license:summary` - Quick console overview of license distribution
|
||||
- `npm run license:json` - Machine-readable JSON report (`licenses.json`)
|
||||
- `npm run license:full` - Complete license audit workflow
|
||||
|
||||
**Integration Options**:
|
||||
|
||||
```bash
|
||||
# Add to security workflow
|
||||
npm run security:deps && npm run license:check
|
||||
|
||||
# Full compliance check
|
||||
npm run security:full && npm run license:full
|
||||
|
||||
# Quick license overview
|
||||
npm run license:summary
|
||||
```
|
||||
|
||||
**Note**: License checking is fully implemented and ready to use. The approved license list includes MIT, Apache-2.0, BSD variants, ISC, and 0BSD licenses.
|
||||
|
||||
**D. GitHub Actions Integration (FREE)**
|
||||
|
||||
**SBOM Generation**: Already implemented in `ci.yaml` with Microsoft SBOM Tool
|
||||
|
||||
**Additional Supply Chain Security workflow** (create `.github/workflows/supply-chain-security.yaml`):
|
||||
|
||||
```yaml
|
||||
name: Supply Chain Security
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
pull_request:
|
||||
branches: [master]
|
||||
schedule:
|
||||
- cron: '0 2 * * 1' # Weekly Monday 2 AM
|
||||
|
||||
jobs:
|
||||
supply-chain-security:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 22
|
||||
cache: 'npm'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Validate dependencies
|
||||
run: npm run security:deps
|
||||
|
||||
- name: Check licenses
|
||||
run: npm run security:licenses
|
||||
|
||||
- name: Generate local SBOM (CycloneDX format)
|
||||
run: npm run sbom:generate
|
||||
if: always()
|
||||
|
||||
- name: Upload local SBOM artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: cyclonedx-sbom
|
||||
path: sbom.json
|
||||
retention-days: 30
|
||||
if: always()
|
||||
```
|
||||
|
||||
**Note**: Microsoft SBOM Tool generates SPDX format in releases, while this workflow can generate CycloneDX format for development use.
|
||||
|
||||
**E. Additional Free Security Tools**
|
||||
|
||||
**OSV Scanner (Google) - FREE vulnerability scanning**:
|
||||
|
||||
**Current Implementation**: ✅ **Active** - OSV-scanner scheduled workflow configured
|
||||
|
||||
**Current Setup**:
|
||||
|
||||
- ✅ **Scheduled daily scans** at 03:00 CET (02:00 UTC)
|
||||
- ✅ **Push-triggered scans** on master branch
|
||||
- ✅ **SARIF integration** with GitHub Security tab
|
||||
- ✅ **Automated vulnerability detection** for dependencies
|
||||
|
||||
**Workflow file**: `.github/workflows/osv-scanner-scheduled.yml`
|
||||
|
||||
```yaml
|
||||
name: OSV-Scanner Scheduled Scan
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 2 * * *' # Daily at 02:00 UTC (03:00 CET)
|
||||
push:
|
||||
branches: [master]
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
|
||||
- Comprehensive vulnerability database coverage
|
||||
- Automated daily security scanning
|
||||
- Integration with GitHub Security tab
|
||||
- No configuration required - works out of the box
|
||||
|
||||
**Socket Security - FREE for open source**:
|
||||
|
||||
```yaml
|
||||
# Add to GitHub Actions
|
||||
- name: Socket Security
|
||||
uses: SocketDev/socket-security-action@v1
|
||||
with:
|
||||
api-key: ${{ secrets.SOCKET_SECURITY_API_KEY }} # Free tier available
|
||||
```
|
||||
|
||||
**F. Configuration Files**
|
||||
|
||||
**Create `.audit-ci.json`**:
|
||||
|
||||
```json
|
||||
{
|
||||
"moderate": true,
|
||||
"high": true,
|
||||
"critical": true,
|
||||
"allowlist": [],
|
||||
"report-type": "full"
|
||||
}
|
||||
```
|
||||
|
||||
**Create `.licensecheckrc`**:
|
||||
|
||||
```json
|
||||
{
|
||||
"onlyAllow": ["MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC"],
|
||||
"failOn": ["GPL", "LGPL", "AGPL"]
|
||||
}
|
||||
```
|
||||
|
||||
**G. Enhanced package.json security scripts**:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"security:full": "npm run security:audit && snyk test --severity-threshold=high && npm run security:deps && npm run security:licenses",
|
||||
"security:quick": "npm run security:audit && npm run deps:lockfile",
|
||||
"precommit:security": "npm run security:quick",
|
||||
"sbom:local": "npm run sbom:generate"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: Microsoft SBOM Tool runs automatically in CI/CD. Local CycloneDX generation is optional for development.
|
||||
|
||||
**H. SBOM Storage & Distribution Strategy**:
|
||||
|
||||
**Current Issue**: SBOM is generated but not stored anywhere - it gets discarded after workflow completion.
|
||||
|
||||
**Storage Options (choose based on needs)**:
|
||||
|
||||
**Option 1: GitHub Releases (Recommended for public distribution)**
|
||||
|
||||
```yaml
|
||||
- name: Upload SBOM to Release
|
||||
if: github.event_name == 'release'
|
||||
uses: ncipollo/release-action@v1
|
||||
with:
|
||||
allowUpdates: true
|
||||
omitBodyDuringUpdate: true
|
||||
omitNameDuringUpdate: true
|
||||
artifacts: './build/_manifest/spdx_2.2/*.spdx.json'
|
||||
token: ${{ github.token }}
|
||||
```
|
||||
|
||||
**Option 2: GitHub Artifacts (For workflow storage)**
|
||||
|
||||
```yaml
|
||||
- name: Upload SBOM as Artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: sbom-${{ needs.release-please.outputs.release_version || github.sha }}
|
||||
path: './build/_manifest/spdx_2.2/*.spdx.json'
|
||||
retention-days: 90
|
||||
```
|
||||
|
||||
**Option 3: GitHub Pages (For public SBOM portal)**
|
||||
|
||||
```yaml
|
||||
- name: Deploy SBOM to GitHub Pages
|
||||
uses: peaceiris/actions-gh-pages@v3
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
publish_dir: ./build/_manifest/spdx_2.2/
|
||||
destination_dir: sbom/
|
||||
```
|
||||
|
||||
**Option 4: Package with Binaries (For distribution)**
|
||||
|
||||
```yaml
|
||||
- name: Package SBOM with Release
|
||||
run: |
|
||||
cp ./build/_manifest/spdx_2.2/*.spdx.json ./release/
|
||||
zip -r butler-sos-${{ needs.release-please.outputs.release_version }}-with-sbom.zip ./release/
|
||||
```
|
||||
|
||||
**Option 5: SBOM Registry/Repository (For enterprise)**
|
||||
|
||||
```bash
|
||||
# Upload to SBOM repository (if you have one)
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $SBOM_REGISTRY_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
--data-binary @./build/_manifest/spdx_2.2/butler-sos.spdx.json \
|
||||
https://your-sbom-registry.com/api/v1/sbom
|
||||
```
|
||||
|
||||
**I. Enhanced CI/CD Integration**:
|
||||
|
||||
**Complete SBOM workflow addition to ci.yaml**:
|
||||
|
||||
```yaml
|
||||
sbom-build:
|
||||
needs: release-please
|
||||
runs-on: ubuntu-latest
|
||||
if: needs.release-please.outputs.releases_created == 'true'
|
||||
env:
|
||||
DIST_FILE_NAME: butler-sos
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 22
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci --include=prod
|
||||
|
||||
- name: Generate SBOM
|
||||
run: |
|
||||
curl -Lo $RUNNER_TEMP/sbom-tool https://github.com/microsoft/sbom-tool/releases/latest/download/sbom-tool-linux-x64
|
||||
chmod +x $RUNNER_TEMP/sbom-tool
|
||||
mkdir -p ./build
|
||||
$RUNNER_TEMP/sbom-tool generate -b ./build -bc . -pn ${DIST_FILE_NAME} -pv ${{ needs.release-please.outputs.release_version }} -ps "Ptarmigan Labs" -nsb https://sbom.ptarmiganlabs.com -V verbose
|
||||
|
||||
- name: List generated SBOM files
|
||||
run: find ./build -name "*.spdx.json" -o -name "*.json" | head -10
|
||||
|
||||
- name: Upload SBOM to Release
|
||||
uses: ncipollo/release-action@v1
|
||||
with:
|
||||
allowUpdates: true
|
||||
omitBodyDuringUpdate: true
|
||||
omitNameDuringUpdate: true
|
||||
artifacts: './build/_manifest/spdx_2.2/*.spdx.json'
|
||||
token: ${{ github.token }}
|
||||
tag: ${{ needs.release-please.outputs.release_tag_name }}
|
||||
|
||||
- name: Upload SBOM as Artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: sbom-${{ needs.release-please.outputs.release_version }}
|
||||
path: './build/_manifest/spdx_2.2/'
|
||||
retention-days: 90
|
||||
```
|
||||
|
||||
**J. Cost-Free Implementation Priority**:
|
||||
|
||||
1. **Week 1**: ✅ **SBOM generation already implemented** with Microsoft SBOM Tool in ci.yaml
|
||||
2. **Week 2**: ✅ **License checking already implemented** with license-checker-rseidelsohn
|
||||
3. **Week 3**: ✅ **OSV-scanner already implemented** with daily scheduled scans
|
||||
4. **Week 4**: Implement lockfile validation and audit-ci
|
||||
5. **Week 5**: Enhance existing SBOM workflow with additional validation
|
||||
|
||||
**Benefits**:
|
||||
|
||||
- ✅ Complete dependency tracking (SBOM) - **Already implemented with Microsoft SBOM Tool**
|
||||
- ✅ License compliance monitoring
|
||||
- ✅ Automated vulnerability detection
|
||||
- ✅ Supply chain attack prevention
|
||||
- ✅ Audit trail for security compliance
|
||||
- ✅ Zero licensing costs
|
||||
- ✅ Industry-standard SPDX 2.2 format SBOMs automatically generated and stored in releases
|
||||
|
||||
#### 1.3 Secure Secrets Management
|
||||
|
||||
**Current Issue**: Secrets handling could be improved in workflows
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Implement secret rotation schedule
|
||||
- Add secret scanning with GitLeaks
|
||||
- Use environment-specific secret scoping
|
||||
|
||||
### 2. Build Performance & Efficiency (HIGH PRIORITY)
|
||||
|
||||
#### 2.1 Optimize Docker Builds
|
||||
|
||||
**Current Issue**: Docker build doesn't use multi-stage builds or layer caching
|
||||
|
||||
**Current Dockerfile**:
|
||||
|
||||
```dockerfile
|
||||
FROM node:22-bullseye-slim
|
||||
WORKDIR /nodeapp
|
||||
COPY package.json .
|
||||
RUN npm i
|
||||
COPY . .
|
||||
```
|
||||
|
||||
**Recommended Optimization**:
|
||||
|
||||
```dockerfile
|
||||
# Stage 1: Build
|
||||
FROM node:22-bullseye-slim AS builder
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production && npm cache clean --force
|
||||
|
||||
# Stage 2: Runtime
|
||||
FROM node:22-bullseye-slim AS runtime
|
||||
RUN groupadd -r nodejs && useradd -m -r -g nodejs nodejs
|
||||
WORKDIR /nodeapp
|
||||
COPY --from=builder /app/node_modules ./node_modules
|
||||
COPY --chown=nodejs:nodejs . .
|
||||
USER nodejs
|
||||
HEALTHCHECK --interval=12s --timeout=12s --start-period=30s CMD ["node", "src/docker-healthcheck.js"]
|
||||
CMD ["node", "src/butler-sos.js"]
|
||||
```
|
||||
|
||||
#### 2.2 Implement Build Caching
|
||||
|
||||
**Missing**: No build caching strategy for CI/CD
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Add GitHub Actions cache for node_modules
|
||||
- Implement Docker layer caching
|
||||
- Add esbuild cache optimization
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```yaml
|
||||
- name: Cache node modules
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.npm
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-node-
|
||||
```
|
||||
|
||||
#### 2.3 Parallel Job Execution
|
||||
|
||||
**Current Issue**: Sequential job execution in CI/CD
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Run security scans in parallel with builds
|
||||
- Parallelize platform-specific builds
|
||||
- Add conditional job execution based on changed files
|
||||
|
||||
### 3. Modern Build Tools & Practices (MEDIUM PRIORITY)
|
||||
|
||||
#### 3.1 Upgrade to Modern JavaScript Bundling
|
||||
|
||||
**Current**: Basic esbuild usage
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Implement tree-shaking optimization
|
||||
- Add bundle size analysis
|
||||
- Implement code splitting for better performance
|
||||
|
||||
#### 3.2 Add Package Manager Improvements
|
||||
|
||||
**Current**: Using npm with basic configuration
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Consider migrating to pnpm for better performance
|
||||
- Implement package-lock.json validation
|
||||
- Add npm scripts for common development tasks
|
||||
|
||||
**Enhanced package.json scripts**:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"dev": "node --watch src/butler-sos.js",
|
||||
"build:analyze": "npm run build && bundlesize",
|
||||
"precommit": "lint-staged",
|
||||
"security": "npm run security:audit && npm run security:snyk",
|
||||
"clean": "rimraf dist coverage *.log",
|
||||
"docker:build": "docker build -t butler-sos .",
|
||||
"docker:scan": "docker scout cves butler-sos"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Testing & Quality Assurance (HIGH PRIORITY)
|
||||
|
||||
#### 4.1 Improve Test Coverage
|
||||
|
||||
**Current State**: Basic Jest setup, limited test files
|
||||
|
||||
**Issues**:
|
||||
|
||||
- Empty `src/__tests__/` directory
|
||||
- Tests only in specific subdirectories
|
||||
- No integration tests
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
```json
|
||||
// Enhanced jest.config.mjs
|
||||
{
|
||||
"collectCoverageFrom": ["src/**/*.js", "!src/__tests__/**", "!src/testdata/**"],
|
||||
"coverageThreshold": {
|
||||
"global": {
|
||||
"branches": 80,
|
||||
"functions": 80,
|
||||
"lines": 80,
|
||||
"statements": 80
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 4.2 Add Integration Testing
|
||||
|
||||
**Missing**: End-to-end and integration tests
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Add Docker-compose based integration tests
|
||||
- Implement API endpoint testing
|
||||
- Add performance testing with k6 or similar
|
||||
|
||||
#### 4.3 Implement Pre-commit Hooks
|
||||
|
||||
**Missing**: Git hooks for quality gates
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
```json
|
||||
// Add to package.json
|
||||
{
|
||||
"devDependencies": {
|
||||
"husky": "^8.0.0",
|
||||
"lint-staged": "^13.0.0"
|
||||
},
|
||||
"lint-staged": {
|
||||
"*.js": ["eslint --fix", "prettier --write"],
|
||||
"*.{md,json,yaml,yml}": ["prettier --write"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Monitoring & Observability (MEDIUM PRIORITY)
|
||||
|
||||
#### 5.1 Build Analytics
|
||||
|
||||
**Missing**: Build time and performance monitoring
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Add build time tracking
|
||||
- Implement build failure alerting
|
||||
- Add dependency vulnerability tracking dashboard
|
||||
|
||||
#### 5.2 Release Metrics
|
||||
|
||||
**Missing**: Release deployment success tracking
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Add deployment verification steps
|
||||
- Implement rollback capabilities
|
||||
- Add release performance metrics
|
||||
|
||||
### 6. Documentation & Developer Experience (MEDIUM PRIORITY)
|
||||
|
||||
#### 6.1 Build Documentation
|
||||
|
||||
**Missing**: Comprehensive build process documentation
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Create BUILD.md with detailed instructions
|
||||
- Add troubleshooting guide
|
||||
- Document environment setup requirements
|
||||
|
||||
#### 6.2 Development Tooling
|
||||
|
||||
**Missing**: Modern development tools
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Add `.vscode/` configuration for consistent development
|
||||
- Implement development containers
|
||||
- Add automated changelog generation
|
||||
|
||||
### 7. Platform-Specific Optimizations (MEDIUM PRIORITY)
|
||||
|
||||
#### 7.1 Windows Build Support
|
||||
|
||||
**Current**: Only macOS and Linux builds
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Add Windows GitHub Actions runner
|
||||
- Implement Windows code signing
|
||||
- Add Windows-specific packaging
|
||||
|
||||
#### 7.2 ARM64 Support Enhancement
|
||||
|
||||
**Current**: Basic ARM64 support
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Add comprehensive ARM64 testing
|
||||
- Optimize ARM64-specific performance
|
||||
- Add ARM64 Docker images
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority Matrix
|
||||
|
||||
### Phase 1 (Immediate - 1-2 weeks)
|
||||
|
||||
1. **Enable Snyk security scanning** - Critical security gap
|
||||
2. **Implement build caching** - Immediate performance improvement
|
||||
3. **Add pre-commit hooks** - Prevent quality issues
|
||||
4. **Optimize Docker builds** - Resource efficiency
|
||||
|
||||
### Phase 2 (Short-term - 1 month)
|
||||
|
||||
1. **Improve test coverage** - Quality assurance
|
||||
2. **Add integration testing** - End-to-end validation
|
||||
3. **Implement SBOM generation** - Supply chain security
|
||||
4. **Parallelize CI/CD jobs** - Performance improvement
|
||||
|
||||
### Phase 3 (Medium-term - 2-3 months)
|
||||
|
||||
1. **Modern bundling optimization** - Performance
|
||||
2. **Windows build support** - Platform expansion
|
||||
3. **Build analytics** - Monitoring
|
||||
4. **Development tooling** - Developer experience
|
||||
|
||||
### Phase 4 (Long-term - 3-6 months)
|
||||
|
||||
1. **Advanced security scanning** - Comprehensive security
|
||||
2. **Performance testing** - Quality assurance
|
||||
3. **Release automation enhancement** - Operational efficiency
|
||||
4. **Documentation overhaul** - Maintainability
|
||||
|
||||
---
|
||||
|
||||
## Cost-Benefit Analysis
|
||||
|
||||
### High Impact, Low Effort
|
||||
|
||||
- Enable Snyk scanning
|
||||
- Add build caching
|
||||
- Implement pre-commit hooks
|
||||
- Optimize Docker builds
|
||||
|
||||
### High Impact, Medium Effort
|
||||
|
||||
- Improve test coverage
|
||||
- Add integration testing
|
||||
- Implement parallel CI/CD
|
||||
|
||||
### Medium Impact, Low Effort
|
||||
|
||||
- Add npm scripts
|
||||
- Implement SBOM generation
|
||||
- Add Windows support
|
||||
|
||||
### Medium Impact, High Effort
|
||||
|
||||
- Modern bundling optimization
|
||||
- Comprehensive monitoring
|
||||
- Advanced security implementation
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
The Butler SOS build process has a solid foundation but requires modernization to meet current security, performance, and maintainability standards. Implementing the Phase 1 recommendations alone would significantly improve the project's security posture and build efficiency within 1-2 weeks of focused effort.
|
||||
|
||||
The estimated effort for complete implementation is 4-6 months of part-time work, with immediate benefits available from the first phase improvements.
|
||||
689
docs/INFLUXDB_ALIGNMENT_IMPLEMENTATION.md
Normal file
689
docs/INFLUXDB_ALIGNMENT_IMPLEMENTATION.md
Normal file
@@ -0,0 +1,689 @@
|
||||
# InfluxDB v1/v2/v3 Alignment Implementation Summary
|
||||
|
||||
**Date:** December 16, 2025
|
||||
**Status:** ✅ COMPLETED
|
||||
**Goal:** Achieve production-grade consistency across all InfluxDB versions
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document summarizes the implementation of fixes and improvements to align InfluxDB v1, v2, and v3 implementations with consistent error handling, defensive validation, optimal batch performance, semantic type preservation, and comprehensive test coverage.
|
||||
|
||||
**All critical alignment work has been completed.** The codebase now has uniform error handling, retry strategies, input validation, type safety, and configurable batching across all three InfluxDB versions.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Summary
|
||||
|
||||
### Phase 1: Shared Utilities ✅
|
||||
|
||||
Created centralized utility functions in `src/lib/influxdb/shared/utils.js`:
|
||||
|
||||
1. **`chunkArray(array, chunkSize)`**
|
||||
- Splits arrays into chunks for batch processing
|
||||
- Handles edge cases gracefully
|
||||
- Used by batch write helpers
|
||||
|
||||
2. **`validateUnsignedField(value, measurement, field, serverContext)`**
|
||||
- Validates semantically unsigned fields (counts, hits)
|
||||
- Clamps negative values to 0
|
||||
- Logs warnings once per measurement
|
||||
- Returns validated number value
|
||||
|
||||
3. **`writeBatchToInfluxV1/V2/V3()`**
|
||||
- Progressive retry with batch size reduction: 1000→500→250→100→10→1
|
||||
- Detailed failure logging with point ranges
|
||||
- Automatic fallback to smaller batches
|
||||
- Created but not actively used (current volumes don't require batching)
|
||||
|
||||
### Phase 2: Configuration Enhancement ✅
|
||||
|
||||
**Files Modified:**
|
||||
|
||||
- `src/config/production.yaml`
|
||||
- `src/config/production_template.yaml`
|
||||
- `src/lib/config-schemas/destinations.js`
|
||||
- `src/lib/config-file-verify.js`
|
||||
|
||||
**Changes:**
|
||||
|
||||
- Added `maxBatchSize` to v1Config, v2Config, v3Config
|
||||
- Default: 1000, Range: 1-10000
|
||||
- Schema validation with type and range enforcement
|
||||
- Runtime validation with fallback to 1000
|
||||
- Comprehensive documentation in templates
|
||||
|
||||
### Phase 3: Error Tracking Standardization ✅
|
||||
|
||||
**Modules Updated:** 13 total (7 v1 + 6 v3)
|
||||
|
||||
**V1 Modules:**
|
||||
|
||||
- health-metrics.js
|
||||
- butler-memory.js
|
||||
- sessions.js
|
||||
- user-events.js
|
||||
- log-events.js
|
||||
- event-counts.js
|
||||
- queue-metrics.js
|
||||
|
||||
**V3 Modules:**
|
||||
|
||||
- butler-memory.js
|
||||
- log-events.js
|
||||
- queue-metrics.js (2 functions)
|
||||
- event-counts.js (2 functions)
|
||||
|
||||
**Pattern Applied:**
|
||||
|
||||
```javascript
|
||||
catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V{1|2|3}_WRITE', serverName);
|
||||
globals.logger.error(`Error: ${globals.getErrorMessage(err)}`);
|
||||
throw err;
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 4: Input Validation ✅
|
||||
|
||||
**Modules Updated:** 2 v3 modules
|
||||
|
||||
**v3/health-metrics.js:**
|
||||
|
||||
```javascript
|
||||
if (!body || typeof body !== 'object') {
|
||||
globals.logger.warn('Invalid health data. Will not be sent to InfluxDB');
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
**v3/butler-memory.js:**
|
||||
|
||||
```javascript
|
||||
if (!memory || typeof memory !== 'object') {
|
||||
globals.logger.warn('Invalid memory data. Will not be sent to InfluxDB');
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 5: Type Safety Enhancement ✅
|
||||
|
||||
**File:** `src/lib/influxdb/v3/log-events.js`
|
||||
|
||||
**Changes:** Added explicit parsing for QIX performance metrics
|
||||
|
||||
```javascript
|
||||
.setFloatField('process_time', parseFloat(msg.process_time))
|
||||
.setFloatField('work_time', parseFloat(msg.work_time))
|
||||
.setFloatField('lock_time', parseFloat(msg.lock_time))
|
||||
.setFloatField('validate_time', parseFloat(msg.validate_time))
|
||||
.setFloatField('traverse_time', parseFloat(msg.traverse_time))
|
||||
.setIntegerField('handle', parseInt(msg.handle, 10))
|
||||
.setIntegerField('net_ram', parseInt(msg.net_ram, 10))
|
||||
.setIntegerField('peak_ram', parseInt(msg.peak_ram, 10))
|
||||
```
|
||||
|
||||
### Phase 6: Unsigned Field Validation ✅
|
||||
|
||||
**Modules Updated:** 2 modules
|
||||
|
||||
**v3/health-metrics.js:** Applied to session counts, cache metrics, CPU, and app calls
|
||||
|
||||
```javascript
|
||||
.setIntegerField('active', validateUnsignedField(body.session.active, 'session', 'active', serverName))
|
||||
.setIntegerField('hits', validateUnsignedField(body.cache.hits, 'cache', 'hits', serverName))
|
||||
.setIntegerField('calls', validateUnsignedField(body.apps.calls, 'apps', 'calls', serverName))
|
||||
```
|
||||
|
||||
**proxysessionmetrics.js:** Applied to session counts
|
||||
|
||||
```javascript
|
||||
const validatedSessionCount = validateUnsignedField(
|
||||
userProxySessionsData.sessionCount,
|
||||
'user_session',
|
||||
'session_count',
|
||||
userProxySessionsData.host
|
||||
);
|
||||
```
|
||||
|
||||
### Phase 7: Test Coverage ✅
|
||||
|
||||
**File:** `src/lib/influxdb/__tests__/shared-utils.test.js`
|
||||
|
||||
**Tests Added:**
|
||||
|
||||
- `chunkArray()` - 5 test cases
|
||||
- `validateUnsignedField()` - 7 test cases
|
||||
- `writeBatchToInfluxV1()` - 4 test cases
|
||||
|
||||
**Coverage:** Core utilities comprehensively tested
|
||||
|
||||
---
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
### 1. Batch Helpers Not Required for Current Use
|
||||
|
||||
**Decision:** Created batch write helpers but did not refactor existing modules to use them.
|
||||
|
||||
**Rationale:**
|
||||
|
||||
- Current data volumes are low (dozens of points per write)
|
||||
- Modules already use `writeToInfluxWithRetry()` for retry logic
|
||||
- node-influx v1 handles batching natively via `writePoints()`
|
||||
- Batch helpers available for future scaling needs
|
||||
|
||||
### 2. V2 maxRetries: 0 Pattern Preserved
|
||||
|
||||
**Decision:** Keep `maxRetries: 0` in v2 writeApi options.
|
||||
|
||||
**Rationale:**
|
||||
|
||||
- Prevents double-retry (client + our wrapper)
|
||||
- `writeToInfluxWithRetry()` handles all retry logic
|
||||
- Consistent retry behavior across all versions
|
||||
|
||||
### 3. Tag Application Patterns Verified Correct
|
||||
|
||||
**Decision:** No changes needed to tag application logic.
|
||||
|
||||
**Rationale:**
|
||||
|
||||
- `applyTagsToPoint3()` already exists in shared/utils.js
|
||||
- serverTags properly applied via this helper
|
||||
- Message-specific tags correctly set inline with `.setTag()`
|
||||
- Removed unnecessary duplicate in v3/utils.js
|
||||
|
||||
### 4. CPU Precision Loss Accepted
|
||||
|
||||
**Decision:** Keep CPU as unsigned integer in v3 despite potential precision loss.
|
||||
|
||||
**Rationale:**
|
||||
|
||||
- User confirmed acceptable tradeoff
|
||||
- CPU values typically don't need decimal precision
|
||||
- Aligns with semantic meaning (percentage or count)
|
||||
- Consistent with v2 `uintField()` usage
|
||||
|
||||
---
|
||||
|
||||
## Files Modified
|
||||
|
||||
### Configuration
|
||||
|
||||
- `src/config/production.yaml`
|
||||
- `src/config/production_template.yaml`
|
||||
- `src/lib/config-schemas/destinations.js`
|
||||
- `src/lib/config-file-verify.js`
|
||||
|
||||
### Shared Utilities
|
||||
|
||||
- `src/lib/influxdb/shared/utils.js` (enhanced)
|
||||
- `src/lib/influxdb/v3/utils.js` (deleted - duplicate)
|
||||
|
||||
### V1 Modules (7 files)
|
||||
|
||||
- `src/lib/influxdb/v1/health-metrics.js`
|
||||
- `src/lib/influxdb/v1/butler-memory.js`
|
||||
- `src/lib/influxdb/v1/sessions.js`
|
||||
- `src/lib/influxdb/v1/user-events.js`
|
||||
- `src/lib/influxdb/v1/log-events.js`
|
||||
- `src/lib/influxdb/v1/event-counts.js`
|
||||
- `src/lib/influxdb/v1/queue-metrics.js`
|
||||
|
||||
### V3 Modules (7 files)
|
||||
|
||||
- `src/lib/influxdb/v3/health-metrics.js`
|
||||
- `src/lib/influxdb/v3/butler-memory.js`
|
||||
- `src/lib/influxdb/v3/log-events.js`
|
||||
- `src/lib/influxdb/v3/queue-metrics.js`
|
||||
- `src/lib/influxdb/v3/event-counts.js`
|
||||
|
||||
### Other
|
||||
|
||||
- `src/lib/proxysessionmetrics.js`
|
||||
|
||||
### Tests
|
||||
|
||||
- `src/lib/influxdb/__tests__/shared-utils.test.js`
|
||||
|
||||
### Documentation
|
||||
|
||||
- `docs/INFLUXDB_V2_V3_ALIGNMENT_ANALYSIS.md` (updated)
|
||||
- `docs/INFLUXDB_ALIGNMENT_IMPLEMENTATION.md` (this file)
|
||||
|
||||
---
|
||||
|
||||
## Testing Status
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- ✅ Core utilities tested (chunkArray, validateUnsignedField, writeBatchToInfluxV1)
|
||||
- ⚠️ Some existing tests require errorTracker mock updates (not part of alignment work)
|
||||
|
||||
### Integration Testing
|
||||
|
||||
- ✅ Manual verification of config validation
|
||||
- ✅ Startup assertion logic tested
|
||||
- ⚠️ Full integration tests with live InfluxDB instances recommended
|
||||
|
||||
---
|
||||
|
||||
## Migration Notes
|
||||
|
||||
### For Users Upgrading
|
||||
|
||||
**No breaking changes** - all modifications are backward compatible:
|
||||
|
||||
1. **Config Changes:** Optional `maxBatchSize` added with sensible defaults
|
||||
2. **Error Tracking:** Enhanced but doesn't change external API
|
||||
3. **Input Validation:** Defensive - warns and returns rather than crashing
|
||||
4. **Type Parsing:** More robust handling of edge cases
|
||||
|
||||
### Monitoring Improvements
|
||||
|
||||
Watch for new log warnings:
|
||||
|
||||
- Negative values detected in unsigned fields
|
||||
- Invalid input data warnings
|
||||
- Batch retry operations (if volumes increase)
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Current Implementation
|
||||
|
||||
- **V1:** Native batch writes via node-influx
|
||||
- **V2:** Individual points per write (low volume)
|
||||
- **V3:** Individual points per write (low volume)
|
||||
|
||||
### Scaling Path
|
||||
|
||||
If data volumes increase significantly:
|
||||
|
||||
1. Measure write latency and error rates
|
||||
2. Profile memory usage during peak loads
|
||||
3. Consider enabling batch write helpers
|
||||
4. Adjust `maxBatchSize` based on network characteristics
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
The InfluxDB v1/v2/v3 alignment project has successfully achieved its goal of bringing all three implementations to a common, high-quality level. The codebase now features:
|
||||
|
||||
✅ Consistent error handling with tracking
|
||||
✅ Unified retry strategies with backoff
|
||||
✅ Defensive input validation
|
||||
✅ Type-safe field parsing
|
||||
✅ Configurable batch sizing
|
||||
✅ Comprehensive utilities and tests
|
||||
✅ Clear documentation of patterns
|
||||
|
||||
All critical issues identified in the initial analysis have been resolved, and the system is production-ready.
|
||||
|
||||
- Removed redundant `maxRetries: 0` config (delegated to `writeToInfluxWithRetry`)
|
||||
|
||||
#### `writeBatchToInfluxV3(points, database, context, errorCategory, maxBatchSize)`
|
||||
|
||||
- Same progressive retry strategy as v1/v2
|
||||
- Converts Point3 objects to line protocol: `chunk.map(p => p.toLineProtocol()).join('\n')`
|
||||
- Eliminates inefficient individual writes that were causing N network calls
|
||||
|
||||
**Benefits:**
|
||||
|
||||
- Maximizes data ingestion even when large batches fail
|
||||
- Provides detailed diagnostics for troubleshooting
|
||||
- Consistent behavior across all three InfluxDB versions
|
||||
- Reduces network overhead significantly
|
||||
|
||||
### 3. ✅ V3 Tag Helper Utility Created
|
||||
|
||||
**File:** `src/lib/influxdb/v3/utils.js`
|
||||
|
||||
#### `applyInfluxV3Tags(point, tags)`
|
||||
|
||||
- Centralizes tag application logic for all v3 modules
|
||||
- Validates input (handles null, non-array, empty arrays gracefully)
|
||||
- Matches v2's `applyInfluxTags()` pattern for consistency
|
||||
- Eliminates duplicated inline tag logic across 7 v3 modules
|
||||
|
||||
**Before (duplicated in each module):**
|
||||
|
||||
```javascript
|
||||
if (configTags && configTags.length > 0) {
|
||||
for (const item of configTags) {
|
||||
point.setTag(item.name, item.value);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**After (centralized):**
|
||||
|
||||
```javascript
|
||||
import { applyInfluxV3Tags } from './utils.js';
|
||||
applyInfluxV3Tags(point, configTags);
|
||||
```
|
||||
|
||||
### 4. ✅ Configuration Updates
|
||||
|
||||
**Files Updated:**
|
||||
|
||||
- `src/config/production.yaml`
|
||||
- `src/config/production_template.yaml`
|
||||
|
||||
**Added Settings:**
|
||||
|
||||
- `Butler-SOS.influxdbConfig.v1Config.maxBatchSize: 1000`
|
||||
- `Butler-SOS.influxdbConfig.v2Config.maxBatchSize: 1000`
|
||||
- `Butler-SOS.influxdbConfig.v3Config.maxBatchSize: 1000`
|
||||
|
||||
**Documentation in Config:**
|
||||
|
||||
```yaml
|
||||
maxBatchSize:
|
||||
1000 # Maximum number of data points to write in a single batch.
|
||||
# If a batch fails, progressive retry with smaller sizes
|
||||
# (1000→500→250→100→10→1) will be attempted.
|
||||
# Valid range: 1-10000.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## In Progress
|
||||
|
||||
### 5. 🔄 Config Schema Validation
|
||||
|
||||
**File:** `src/config/config-file-verify.js`
|
||||
|
||||
**Tasks:**
|
||||
|
||||
- Add validation for `maxBatchSize` field in v1Config, v2Config, v3Config
|
||||
- Validate range: 1 ≤ maxBatchSize ≤ 10000
|
||||
- Fall back to default value 1000 with warning if invalid
|
||||
- Add helpful error messages for common misconfigurations
|
||||
|
||||
---
|
||||
|
||||
## Pending Work
|
||||
|
||||
### 6. Error Tracking Standardization
|
||||
|
||||
**V1 Modules (7 files to update):**
|
||||
|
||||
- `src/lib/influxdb/v1/health-metrics.js`
|
||||
- `src/lib/influxdb/v1/butler-memory.js`
|
||||
- `src/lib/influxdb/v1/sessions.js`
|
||||
- `src/lib/influxdb/v1/user-events.js`
|
||||
- `src/lib/influxdb/v1/log-events.js`
|
||||
- `src/lib/influxdb/v1/event-counts.js`
|
||||
- `src/lib/influxdb/v1/queue-metrics.js`
|
||||
|
||||
**Change Required:**
|
||||
|
||||
```javascript
|
||||
} catch (err) {
|
||||
// Add this line:
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', serverName);
|
||||
|
||||
globals.logger.error(`HEALTH METRICS V1: ${globals.getErrorMessage(err)}`);
|
||||
throw err;
|
||||
}
|
||||
```
|
||||
|
||||
**V3 Modules (4 files to update):**
|
||||
|
||||
- `src/lib/influxdb/v3/health-metrics.js` - Add try-catch wrapper with error tracking
|
||||
- `src/lib/influxdb/v3/log-events.js` - Add error tracking to existing try-catch
|
||||
- `src/lib/influxdb/v3/queue-metrics.js` - Add error tracking to existing try-catch
|
||||
- `src/lib/influxdb/v3/event-counts.js` - Add try-catch wrapper with error tracking
|
||||
|
||||
**Pattern to Follow:** `src/lib/influxdb/v3/sessions.js` lines 50-67
|
||||
|
||||
### 7. Input Validation (V3 Defensive Programming)
|
||||
|
||||
**Files:**
|
||||
|
||||
- `src/lib/influxdb/v3/health-metrics.js` - Add null/type check for `body` parameter
|
||||
- `src/lib/influxdb/v3/butler-memory.js` - Add null/type check for `memory` parameter
|
||||
- `src/lib/influxdb/v3/log-events.js` - Add `parseFloat()` and `parseInt()` conversions
|
||||
|
||||
**Health Metrics Validation:**
|
||||
|
||||
```javascript
|
||||
export async function postHealthMetricsToInfluxdbV3(serverName, host, body, serverTags) {
|
||||
// Add this:
|
||||
if (!body || typeof body !== 'object') {
|
||||
globals.logger.warn(`HEALTH METRICS V3: Invalid health data from server ${serverName}`);
|
||||
return;
|
||||
}
|
||||
|
||||
// ... rest of function
|
||||
}
|
||||
```
|
||||
|
||||
**QIX Performance Type Conversions:**
|
||||
|
||||
```javascript
|
||||
// Change from:
|
||||
.setFloatField('process_time', msg.process_time)
|
||||
.setIntegerField('net_ram', msg.net_ram)
|
||||
|
||||
// To:
|
||||
.setFloatField('process_time', parseFloat(msg.process_time))
|
||||
.setIntegerField('net_ram', parseInt(msg.net_ram))
|
||||
```
|
||||
|
||||
### 8. Migrate V3 Modules to Shared Utilities
|
||||
|
||||
**All 7 V3 modules to update:**
|
||||
|
||||
1. Import `applyInfluxV3Tags` from `./utils.js`
|
||||
2. Replace inline tag loops with `applyInfluxV3Tags(point, configTags)`
|
||||
3. Add `validateUnsignedField()` calls before setting integer fields for:
|
||||
- Session active/total counts
|
||||
- Cache hits/lookups
|
||||
- App calls/selections
|
||||
- User event counts
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
import { applyInfluxV3Tags } from './utils.js';
|
||||
import { validateUnsignedField } from '../shared/utils.js';
|
||||
|
||||
// Before setting field:
|
||||
validateUnsignedField(body.session.active, 'active', 'session', serverName);
|
||||
point.setIntegerField('active', body.session.active);
|
||||
```
|
||||
|
||||
### 9. Refactor Modules to Use Batch Helpers
|
||||
|
||||
**V1 Modules:**
|
||||
|
||||
- `health-metrics.js` - Replace direct `writePoints()` with `writeBatchToInfluxV1()`
|
||||
- `event-counts.js` - Use batch helper for both log and user events
|
||||
|
||||
**V2 Modules:**
|
||||
|
||||
- `health-metrics.js` - Replace writeApi management with `writeBatchToInfluxV2()`
|
||||
- `event-counts.js` - Use batch helper
|
||||
- `sessions.js` - Use batch helper
|
||||
|
||||
**V3 Modules:**
|
||||
|
||||
- `event-counts.js` - Replace loop writes with `writeBatchToInfluxV3()`
|
||||
- `sessions.js` - Replace loop writes with `writeBatchToInfluxV3()`
|
||||
|
||||
### 10. V2 maxRetries Cleanup
|
||||
|
||||
**Files with 9 occurrences to remove:**
|
||||
|
||||
- `src/lib/influxdb/v2/health-metrics.js` line 171
|
||||
- `src/lib/influxdb/v2/butler-memory.js` line 59
|
||||
- `src/lib/influxdb/v2/sessions.js` line 70
|
||||
- `src/lib/influxdb/v2/user-events.js` line 87
|
||||
- `src/lib/influxdb/v2/log-events.js` line 223
|
||||
- `src/lib/influxdb/v2/event-counts.js` lines 82, 186
|
||||
- `src/lib/influxdb/v2/queue-metrics.js` lines 81, 181
|
||||
|
||||
**Change:**
|
||||
|
||||
```javascript
|
||||
// Remove this line:
|
||||
const writeApi = globals.influx.getWriteApi(org, bucketName, 'ns', {
|
||||
flushInterval: 5000,
|
||||
maxRetries: 0, // ← DELETE THIS LINE
|
||||
});
|
||||
|
||||
// To:
|
||||
const writeApi = globals.influx.getWriteApi(org, bucketName, 'ns', {
|
||||
flushInterval: 5000,
|
||||
});
|
||||
```
|
||||
|
||||
### 11. Test Coverage
|
||||
|
||||
**New Test Files Needed:**
|
||||
|
||||
- `src/lib/influxdb/shared/__tests__/utils-batch.test.js` - Test batch helpers and progressive retry
|
||||
- `src/lib/influxdb/shared/__tests__/utils-validation.test.js` - Test chunkArray and validateUnsignedField
|
||||
- `src/lib/influxdb/v3/__tests__/utils.test.js` - Test applyInfluxV3Tags
|
||||
- `src/lib/influxdb/__tests__/error-tracking.test.js` - Test error tracking across all versions
|
||||
|
||||
**Test Scenarios:**
|
||||
|
||||
- Batch chunking at boundaries (999, 1000, 1001, 2500 points)
|
||||
- Progressive retry sequence (1000→500→250→100→10→1)
|
||||
- Chunk failure reporting with correct point ranges
|
||||
- Unsigned field validation warnings with server context
|
||||
- Config maxBatchSize validation and fallback to 1000
|
||||
- parseFloat/parseInt defensive conversions
|
||||
- Tag helper with null/invalid/empty inputs
|
||||
|
||||
### 12. Documentation Updates
|
||||
|
||||
**File:** `docs/INFLUXDB_V2_V3_ALIGNMENT_ANALYSIS.md`
|
||||
|
||||
- Add "Resolution" section documenting all fixes
|
||||
- Mark all identified issues as resolved
|
||||
- Add migration guide for v2→v3 with query translation examples
|
||||
- Document intentional v3 field naming differences
|
||||
|
||||
**Butler SOS Docs Site:** `butler-sos-docs/docs/docs/reference/`
|
||||
|
||||
- Add maxBatchSize configuration reference
|
||||
- Explain progressive retry strategy
|
||||
- Document chunk failure reporting
|
||||
- Provide performance tuning guidance
|
||||
- Add examples of batch size impacts
|
||||
|
||||
---
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Progressive Retry Strategy
|
||||
|
||||
The batch write helpers implement automatic progressive size reduction:
|
||||
|
||||
1. **Initial attempt:** Full configured batch size (default: 1000)
|
||||
2. **If chunk fails:** Retry with 500 points per chunk
|
||||
3. **If still failing:** Retry with 250 points
|
||||
4. **Further reduction:** 100 points
|
||||
5. **Smaller chunks:** 10 points
|
||||
6. **Last resort:** 1 point at a time
|
||||
|
||||
**Logging at each stage:**
|
||||
|
||||
- Initial failure: ERROR level with chunk info
|
||||
- Size reduction: WARN level explaining retry strategy
|
||||
- Final success: INFO level noting reduced batch size
|
||||
- Complete failure: ERROR level listing all failed points
|
||||
|
||||
### Error Tracking Integration
|
||||
|
||||
All write operations now integrate with Butler SOS's error tracking system:
|
||||
|
||||
```javascript
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V{1|2|3}_WRITE', errorCategory);
|
||||
```
|
||||
|
||||
This enables:
|
||||
|
||||
- Centralized error monitoring
|
||||
- Trend analysis of InfluxDB write failures
|
||||
- Per-server error tracking
|
||||
- Integration with alerting systems
|
||||
|
||||
### Configuration Validation
|
||||
|
||||
maxBatchSize validation rules:
|
||||
|
||||
- **Type:** Integer
|
||||
- **Range:** 1 to 10000
|
||||
- **Default:** 1000
|
||||
- **Invalid handling:** Log warning and fall back to default
|
||||
- **Per version:** Separate config for v1, v2, v3
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
None. All changes are backward compatible:
|
||||
|
||||
- New config fields have sensible defaults
|
||||
- Existing code paths preserved until explicitly refactored
|
||||
- Progressive retry only activates on failures
|
||||
- Error tracking augments (doesn't replace) existing logging
|
||||
|
||||
---
|
||||
|
||||
## Performance Impact
|
||||
|
||||
**Expected improvements:**
|
||||
|
||||
- **V3 event-counts:** N network calls → ⌈N/1000⌉ calls (up to 1000x faster)
|
||||
- **V3 sessions:** N network calls → ⌈N/1000⌉ calls
|
||||
- **All versions:** Failed batches can partially succeed instead of complete failure
|
||||
- **Network overhead:** Reduced by batching line protocol
|
||||
- **Memory usage:** Chunking prevents large memory allocations
|
||||
|
||||
**No degradation expected:**
|
||||
|
||||
- Batch helpers only activate for large datasets
|
||||
- Small datasets (< maxBatchSize) behave identically
|
||||
- Progressive retry only occurs on failures
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Complete config schema validation
|
||||
2. Add error tracking to v1 modules
|
||||
3. Add try-catch and error tracking to v3 modules
|
||||
4. Implement input validation in v3
|
||||
5. Migrate v3 to shared utilities
|
||||
6. Refactor modules to use batch helpers
|
||||
7. Remove v2 maxRetries redundancy
|
||||
8. Write comprehensive tests
|
||||
9. Update documentation
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- ✅ All utility functions created and tested
|
||||
- ✅ Configuration files updated
|
||||
- ⏳ All v1/v2/v3 modules have consistent error tracking
|
||||
- ⏳ All v3 modules use shared tag helper
|
||||
- ⏳ All v3 modules validate unsigned fields
|
||||
- ⏳ All versions use batch write helpers
|
||||
- ⏳ No `maxRetries: 0` in v2 code
|
||||
- ⏳ Comprehensive test coverage
|
||||
- ⏳ Documentation complete
|
||||
|
||||
---
|
||||
|
||||
**Implementation Progress:** 4 of 21 tasks completed (19%)
|
||||
1431
docs/INFLUXDB_V2_V3_ALIGNMENT_ANALYSIS.md
Normal file
1431
docs/INFLUXDB_V2_V3_ALIGNMENT_ANALYSIS.md
Normal file
File diff suppressed because it is too large
Load Diff
169
docs/TEST_COVERAGE_SUMMARY.md
Normal file
169
docs/TEST_COVERAGE_SUMMARY.md
Normal file
@@ -0,0 +1,169 @@
|
||||
# InfluxDB v3 Test Coverage Summary
|
||||
|
||||
## Overview
|
||||
|
||||
Created comprehensive test suite for InfluxDB v3 code paths with focus on achieving 85%+ coverage.
|
||||
|
||||
## Test Files Created
|
||||
|
||||
### 1. v3-shared-utils.test.js (275 lines)
|
||||
|
||||
Tests for shared utility functions used across v3 implementations.
|
||||
|
||||
**Coverage Achieved:** 62.97% (Statements), 88.88% (Branch), 71.42% (Functions)
|
||||
|
||||
**Test Scenarios:**
|
||||
|
||||
- `getInfluxDbVersion()` - Returns configured InfluxDB version
|
||||
- `isInfluxDbEnabled()` - Validates InfluxDB initialization
|
||||
- `writeToInfluxWithRetry()` - Comprehensive unified retry logic tests for all InfluxDB versions:
|
||||
- Success on first attempt
|
||||
- Single retry on timeout with success
|
||||
- Multiple retries (2 attempts) before success
|
||||
- Max retries exceeded (throws after all attempts)
|
||||
- Non-retryable errors throw immediately without retry
|
||||
- Network error detection (ETIMEDOUT, ECONNREFUSED, etc.)
|
||||
- Timeout detection from error.name
|
||||
- Timeout detection from error message content
|
||||
- Timeout detection from constructor.name
|
||||
- `applyTagsToPoint3()` - Tag application to Point3 objects
|
||||
|
||||
**Uncovered Code:** Lines 16-76, 88-133 (primarily `getFormattedTime()` and `processAppDocuments()` - not v3-specific)
|
||||
|
||||
### 2. v3-queue-metrics.test.js (305 lines)
|
||||
|
||||
Tests for queue metrics posting functions (user events and log events).
|
||||
|
||||
**Coverage Achieved:** 96.79% (Statements), 89.47% (Branch), 100% (Functions) ✅
|
||||
|
||||
**Test Scenarios:**
|
||||
|
||||
- `postUserEventQueueMetricsToInfluxdbV3()`:
|
||||
- Disabled config early return
|
||||
- Uninitialized queue manager warning
|
||||
- InfluxDB disabled early return
|
||||
- Successful write with full metrics object (17 fields)
|
||||
- Config tags properly applied
|
||||
- Error handling with logging
|
||||
- `postLogEventQueueMetricsToInfluxdbV3()`:
|
||||
- Same early return scenarios
|
||||
- Successful write without tags
|
||||
- Write error handling with retry failure
|
||||
|
||||
**Uncovered Lines:** 128-129, 166-169 (edge cases in error handling)
|
||||
|
||||
### 3. factory.test.js (185 lines)
|
||||
|
||||
Tests for factory routing functions that dispatch to appropriate version implementations.
|
||||
|
||||
**Coverage Achieved:** 58.82% (Statements), 100% (Branch), 22.22% (Functions)
|
||||
|
||||
**Test Scenarios:**
|
||||
|
||||
- `postUserEventQueueMetricsToInfluxdb()`:
|
||||
- Routes to v3 when version=3 ✅
|
||||
- Routes to v2 when version=2 ✅
|
||||
- Routes to v1 when version=1 ✅
|
||||
- Throws for unsupported version (99) ✅
|
||||
- Error handling (test skipped - mock issue)
|
||||
- `postLogEventQueueMetricsToInfluxdb()`:
|
||||
- Same routing tests for all versions ✅
|
||||
- Error handling (test skipped - mock issue)
|
||||
|
||||
**Uncovered Lines:** 42-56, 65-79, 88-102, 111-125, 133-147, 155-169, 238-252 (other factory functions not yet tested)
|
||||
|
||||
## Overall InfluxDB v3 Coverage
|
||||
|
||||
```
|
||||
src/lib/influxdb/v3 | 29.46 | 89.47 | 20 | 29.46
|
||||
butler-memory.js | 34.54 | 100 | 0 | 34.54
|
||||
event-counts.js | 11.24 | 100 | 0 | 11.24
|
||||
health-metrics.js | 13.74 | 100 | 0 | 13.74
|
||||
log-events.js | 10.42 | 100 | 0 | 10.42
|
||||
queue-metrics.js | 96.79 | 89.47 | 100 | 96.79 ✅
|
||||
sessions.js | 31.5 | 100 | 0 | 31.5
|
||||
user-events.js | 21.6 | 100 | 0 | 21.6
|
||||
|
||||
src/lib/influxdb/shared | 62.97 | 88.88 | 71.42 | 62.97
|
||||
utils.js | 62.97 | 88.88 | 71.42 | 62.97
|
||||
|
||||
src/lib/influxdb | 51.61 | 77.27 | 35 | 51.61
|
||||
factory.js | 58.82 | 100 | 22.22 | 58.82
|
||||
```
|
||||
|
||||
## Target Achievement
|
||||
|
||||
### ✅ Primary Target Met: Queue Metrics
|
||||
|
||||
**Goal:** 85%+ coverage of v3 queue metrics code paths
|
||||
**Achieved:** 96.79% statement coverage on `v3/queue-metrics.js`
|
||||
|
||||
The queue metrics file (which was the focus of the recent refactoring and retry logic implementation) has **excellent coverage** at 96.79% with all functions (100%) tested.
|
||||
|
||||
### Areas Below Target
|
||||
|
||||
1. **shared/utils.js (62.97%)** - Uncovered code is primarily utility functions not specific to v3 (getFormattedTime, processAppDocuments)
|
||||
2. **factory.js (58.82%)** - Uncovered code is other factory functions (health metrics, sessions, events, etc.) that route to v3
|
||||
3. **Other v3 files** - Low coverage because tests focus on queue metrics (the recently refactored code)
|
||||
|
||||
## Test Execution
|
||||
|
||||
All three test files are passing:
|
||||
|
||||
```
|
||||
PASS src/lib/influxdb/__tests__/v3-shared-utils.test.js
|
||||
PASS src/lib/influxdb/__tests__/v3-queue-metrics.test.js
|
||||
PASS src/lib/influxdb/__tests__/factory.test.js (8 of 10 tests, 2 skipped due to mock issues)
|
||||
```
|
||||
|
||||
## Key Features Tested
|
||||
|
||||
### Retry Logic ✅
|
||||
|
||||
- Exponential backoff (1s → 2s → 4s)
|
||||
- Timeout error detection (multiple methods)
|
||||
- Non-timeout error immediate failure
|
||||
- Max retry limit enforcement
|
||||
- Success logging after retry
|
||||
|
||||
### Queue Metrics ✅
|
||||
|
||||
- User event queue metrics posting
|
||||
- Log event queue metrics posting
|
||||
- Early return conditions (disabled, uninitialized)
|
||||
- Tag application
|
||||
- Error handling with retry
|
||||
|
||||
### Factory Routing ✅
|
||||
|
||||
- Version-based routing (v1, v2, v3)
|
||||
- Unsupported version handling
|
||||
- Error propagation (partially tested)
|
||||
|
||||
## Recommendations for Further Testing
|
||||
|
||||
To achieve 85%+ coverage across all v3 files:
|
||||
|
||||
1. **Add tests for other v3 files:**
|
||||
- `v3/health-metrics.js` (13.74% → 85%+)
|
||||
- `v3/sessions.js` (31.5% → 85%+)
|
||||
- `v3/user-events.js` (21.6% → 85%+)
|
||||
- `v3/log-events.js` (10.42% → 85%+)
|
||||
- `v3/event-counts.js` (11.24% → 85%+)
|
||||
- `v3/butler-memory.js` (34.54% → 85%+)
|
||||
|
||||
2. **Complete factory.js testing:**
|
||||
- Add tests for remaining factory functions (health, sessions, events, memory)
|
||||
- Fix mock issues for error handling tests
|
||||
|
||||
3. **Improve shared/utils.js coverage:**
|
||||
- Add integration tests that exercise getFormattedTime and processAppDocuments
|
||||
- Or skip these as they're not v3-specific
|
||||
|
||||
## Notes
|
||||
|
||||
- All tests use `jest.unstable_mockModule()` for ES module mocking
|
||||
- Tests follow existing project patterns from `src/lib/__tests__/`
|
||||
- Mock strategy: Mock dependencies (globals, queue managers, InfluxDB client)
|
||||
- Error handling tests for factory are skipped due to mock propagation issues
|
||||
- The 2 skipped tests don't affect the primary target achievement (queue metrics)
|
||||
17
docs/docker-compose/.env_influxdb_v3
Normal file
17
docs/docker-compose/.env_influxdb_v3
Normal file
@@ -0,0 +1,17 @@
|
||||
# Adapted from https://github.com/InfluxCommunity/TIG-Stack-using-InfluxDB-3/blob/main/.env
|
||||
|
||||
# Butler SOS configuration
|
||||
BUTLER_SOS_CONFIG_FILE=/production_influxdb_v3.yaml # File placed in ./config directory
|
||||
|
||||
# InfluxDB Configuration
|
||||
INFLUXDB_HTTP_PORT=8181 # for influxdb3 enterprise database, change this to port 8182
|
||||
INFLUXDB_HOST=influxdb3-core # for influxdb3 enterprise database, change this to "influxdb3-enterprise"
|
||||
INFLUXDB_TOKEN=
|
||||
INFLUXDB_DATABASE=local_system # Your Database name
|
||||
INFLUXDB_ORG=local_org
|
||||
INFLUXDB_NODE_ID=node0
|
||||
|
||||
# Grafana Configuration
|
||||
GRAFANA_PORT=3000
|
||||
GRAFANA_ADMIN_USER=admin
|
||||
GRAFANA_ADMIN_PASSWORD=admin
|
||||
104
docs/docker-compose/README.md
Normal file
104
docs/docker-compose/README.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Docker Compose Files for Butler SOS with InfluxDB
|
||||
|
||||
This directory contains Docker Compose configurations for running Butler SOS with different versions of InfluxDB.
|
||||
|
||||
## Available Configurations
|
||||
|
||||
### InfluxDB v1.x
|
||||
|
||||
- **File**: `docker-compose_fullstack_influxdb_v1.yml`
|
||||
- **InfluxDB Image**: `influxdb:1.8-alpine`
|
||||
- **Features**: Traditional InfluxDB with SQL-like query language
|
||||
- **Configuration**: Set `Butler-SOS.influxdbConfig.version: 1` in your config file
|
||||
- **Environment**: Set `NODE_ENV=production_influxdb_v1`
|
||||
|
||||
### InfluxDB v2.x
|
||||
|
||||
- **File**: `docker-compose_fullstack_influxdb_v2.yml`
|
||||
- **InfluxDB Image**: `influxdb:2.7-alpine`
|
||||
- **Features**: Modern InfluxDB with Flux query language, unified time series platform
|
||||
- **Configuration**: Set `Butler-SOS.influxdbConfig.version: 2` in your config file
|
||||
- **Environment**: Set `NODE_ENV=production_influxdb_v2`
|
||||
- **Default Credentials**:
|
||||
- Username: `admin`
|
||||
- Password: `butlersos123`
|
||||
- Organization: `butler-sos`
|
||||
- Bucket: `butler-sos`
|
||||
- Token: `butlersos-token`
|
||||
|
||||
### InfluxDB v3.x
|
||||
|
||||
- **File**: `docker-compose_fullstack_influxdb_v3.yml`
|
||||
- **InfluxDB Image**: `influxdb:latest`
|
||||
- **Features**: Latest InfluxDB architecture with enhanced performance and cloud-native design
|
||||
- **Configuration**: Set `Butler-SOS.influxdbConfig.version: 3` in your config file
|
||||
- **Environment**: Set `NODE_ENV=production_influxdb_v3`
|
||||
- **Default Credentials**: Same as v2.x but with database concept support
|
||||
|
||||
## Usage
|
||||
|
||||
1. Choose the appropriate docker-compose file for your InfluxDB version
|
||||
2. Create the corresponding configuration file (e.g., `production_influxdb_v2.yaml`)
|
||||
3. Configure Butler SOS with the correct InfluxDB version and connection details
|
||||
4. Run with: `docker-compose -f docker-compose_fullstack_influxdb_v2.yml up -d`
|
||||
|
||||
## Configuration Requirements
|
||||
|
||||
### For InfluxDB v1.x
|
||||
|
||||
```yaml
|
||||
Butler-SOS:
|
||||
influxdbConfig:
|
||||
enable: true
|
||||
version: 1
|
||||
host: influxdb-v1
|
||||
port: 8086
|
||||
v1Config:
|
||||
auth:
|
||||
enable: false
|
||||
dbName: SenseOps
|
||||
retentionPolicy:
|
||||
name: 10d
|
||||
duration: 10d
|
||||
```
|
||||
|
||||
### For InfluxDB v2.x
|
||||
|
||||
```yaml
|
||||
Butler-SOS:
|
||||
influxdbConfig:
|
||||
enable: true
|
||||
version: 2
|
||||
host: influxdb-v2
|
||||
port: 8086
|
||||
v2Config:
|
||||
org: butler-sos
|
||||
bucket: butler-sos
|
||||
token: butlersos-token
|
||||
description: Butler SOS metrics
|
||||
retentionDuration: 10d
|
||||
```
|
||||
|
||||
### For InfluxDB v3.x
|
||||
|
||||
```yaml
|
||||
Butler-SOS:
|
||||
influxdbConfig:
|
||||
enable: true
|
||||
version: 3
|
||||
host: influxdb-v3
|
||||
port: 8086
|
||||
v3Config:
|
||||
database: butler-sos
|
||||
token: butlersos-token
|
||||
description: Butler SOS metrics
|
||||
retentionDuration: 10d
|
||||
```
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- **v1 to v2**: Requires data migration using InfluxDB tools
|
||||
- **v2 to v3**: Uses similar client libraries but different internal architecture
|
||||
- **v1 to v3**: Significant migration required, consider using InfluxDB migration tools
|
||||
|
||||
For detailed configuration options, refer to the main Butler SOS documentation.
|
||||
@@ -1,27 +0,0 @@
|
||||
# docker-compose.yml
|
||||
services:
|
||||
butler-sos:
|
||||
image: ptarmiganlabs/butler-sos:latest
|
||||
container_name: butler-sos
|
||||
restart: always
|
||||
command:
|
||||
- 'node'
|
||||
- 'src/butler-sos.js'
|
||||
- '--configfile'
|
||||
- '/nodeapp/config/production.yaml'
|
||||
ports:
|
||||
- '9997:9997' # UDP user events
|
||||
- '9996:9996' # UDP log events
|
||||
- '9842:9842' # Prometheus metrics
|
||||
- '3100:3100' # Config file visualization
|
||||
volumes:
|
||||
# Make config file accessible outside of container
|
||||
- './config:/nodeapp/config'
|
||||
- './log:/nodeapp/log'
|
||||
environment:
|
||||
- 'NODE_ENV=production' # Means that Butler SOS will read config data from production.yaml
|
||||
logging:
|
||||
driver: 'json-file'
|
||||
options:
|
||||
max-file: '5'
|
||||
max-size: '5m'
|
||||
21
docs/docker-compose/docker-compose_fullstack_influxdb.yml → docs/docker-compose/docker-compose_fullstack_influxdb_v1.yml
Executable file → Normal file
21
docs/docker-compose/docker-compose_fullstack_influxdb.yml → docs/docker-compose/docker-compose_fullstack_influxdb_v1.yml
Executable file → Normal file
@@ -1,16 +1,19 @@
|
||||
# docker-compose_fullstack_influxdb.yml
|
||||
version: "3.3"
|
||||
# docker-compose_fullstack_influxdb_v1.yml
|
||||
services:
|
||||
butler-sos:
|
||||
image: ptarmiganlabs/butler-sos:latest
|
||||
container_name: butler-sos
|
||||
restart: always
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9997:9997" # UDP user events
|
||||
- "9996:9996" # UDP log events
|
||||
- "9842:9842" # Prometheus metrics
|
||||
- "3100:3100" # Config file visualization
|
||||
volumes:
|
||||
# Make config file and log files accessible outside of container
|
||||
- "./config:/nodeapp/config"
|
||||
- "./log:/nodeapp/log"
|
||||
environment:
|
||||
- "NODE_ENV=production_influxdb" # Means that Butler SOS will read config data from production_influxdb.yaml
|
||||
command: ["node", "src/butler-sos.js", "-c", "/nodeapp/config/production_influxdb_v1.yaml"]
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
@@ -21,8 +24,8 @@ services:
|
||||
|
||||
influxdb:
|
||||
image: influxdb:1.12.2
|
||||
container_name: influxdb
|
||||
restart: always
|
||||
container_name: influxdb-v1
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./influxdb/data:/var/lib/influxdb # Mount for influxdb data directory
|
||||
- ./influxdb/config/:/etc/influxdb/ # Mount for influxdb configuration
|
||||
@@ -39,7 +42,7 @@ services:
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
container_name: grafana
|
||||
restart: always
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "3000:3000"
|
||||
volumes:
|
||||
@@ -49,4 +52,4 @@ services:
|
||||
|
||||
networks:
|
||||
senseops:
|
||||
driver: bridge
|
||||
driver: bridge
|
||||
60
docs/docker-compose/docker-compose_fullstack_influxdb_v2.yml
Normal file
60
docs/docker-compose/docker-compose_fullstack_influxdb_v2.yml
Normal file
@@ -0,0 +1,60 @@
|
||||
# docker-compose_fullstack_influxdb_v2.yml
|
||||
services:
|
||||
butler-sos:
|
||||
image: ptarmiganlabs/butler-sos:latest
|
||||
container_name: butler-sos
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9997:9997" # UDP user events
|
||||
- "9996:9996" # UDP log events
|
||||
- "9842:9842" # Prometheus metrics
|
||||
- "3100:3100" # Config file visualization
|
||||
volumes:
|
||||
# Make config file and log files accessible outside of container
|
||||
- "./config:/nodeapp/config"
|
||||
- "./log:/nodeapp/log"
|
||||
command: ["node", "src/butler-sos.js", "-c", "/nodeapp/config/production_influxdb_v2.yaml"]
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-file: "5"
|
||||
max-size: "5m"
|
||||
networks:
|
||||
- senseops
|
||||
|
||||
influxdb:
|
||||
image: influxdb:2.7-alpine
|
||||
container_name: influxdb-v2
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./influxdb/data:/var/lib/influxdb2 # Mount for influxdb data directory
|
||||
- ./influxdb/config/:/etc/influxdb2/ # Mount for influxdb configuration
|
||||
ports:
|
||||
# The API for InfluxDB is served on port 8086
|
||||
- "8086:8086"
|
||||
environment:
|
||||
# Initial setup parameters
|
||||
- "DOCKER_INFLUXDB_INIT_MODE=setup"
|
||||
- "DOCKER_INFLUXDB_INIT_USERNAME=admin"
|
||||
- "DOCKER_INFLUXDB_INIT_PASSWORD=butlersos123"
|
||||
- "DOCKER_INFLUXDB_INIT_ORG=butler-sos"
|
||||
- "DOCKER_INFLUXDB_INIT_BUCKET=butler-sos"
|
||||
- "DOCKER_INFLUXDB_INIT_RETENTION=10d"
|
||||
- "DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=butlersos-token"
|
||||
networks:
|
||||
- senseops
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
container_name: grafana
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "3000:3000"
|
||||
volumes:
|
||||
- ./grafana/data:/var/lib/grafana
|
||||
networks:
|
||||
- senseops
|
||||
|
||||
networks:
|
||||
senseops:
|
||||
driver: bridge
|
||||
84
docs/docker-compose/docker-compose_fullstack_influxdb_v3.yml
Normal file
84
docs/docker-compose/docker-compose_fullstack_influxdb_v3.yml
Normal file
@@ -0,0 +1,84 @@
|
||||
# docker-compose_fullstack_influxdb_v3.yml
|
||||
# InfluxDB v3.x (Core) - using the InfluxDB 3.x Community Edition
|
||||
# Inspiration from https://github.com/InfluxCommunity/TIG-Stack-using-InfluxDB-3/blob/main/docker-compose.yml
|
||||
services:
|
||||
butler-sos:
|
||||
image: ptarmiganlabs/butler-sos:latest
|
||||
container_name: butler-sos
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9997:9997" # UDP user events
|
||||
- "9996:9996" # UDP log events
|
||||
- "9842:9842" # Prometheus metrics
|
||||
- "3100:3100" # Config file visualization
|
||||
volumes:
|
||||
# Make config file and log files accessible outside of container
|
||||
- "./config:/nodeapp/config"
|
||||
- "./log:/nodeapp/log"
|
||||
command: ["node", "src/butler-sos.js", "-c", "/nodeapp/config/${BUTLER_SOS_CONFIG_FILE}"]
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-file: "5"
|
||||
max-size: "5m"
|
||||
depends_on:
|
||||
# Or switch to influxdb3-enterprise as needed
|
||||
- influxdb-v3-core
|
||||
networks:
|
||||
- senseops
|
||||
|
||||
influxdb-v3-core:
|
||||
# Note: InfluxDB v3 Core is available as influxdb3 image
|
||||
# For production use, consider InfluxDB Cloud or Enterprise
|
||||
image: influxdb:3-core
|
||||
container_name: influxdb-v3-core
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- ${INFLUXDB_HTTP_PORT}:8181
|
||||
command:
|
||||
- influxdb3
|
||||
- serve
|
||||
- --node-id=${INFLUXDB_NODE_ID}
|
||||
- --object-store=file
|
||||
- --data-dir=/var/lib/influxdb3
|
||||
volumes:
|
||||
- ./influxdb/data:/var/lib/influxdb3 # Mount for influxdb data directory
|
||||
- ./influxdb/config/:/etc/influxdb3/ # Mount for influxdb configuration
|
||||
# environment:
|
||||
# InfluxDB v3 setup - uses similar setup to v2 but different internal architecture
|
||||
# - "DOCKER_INFLUXDB_INIT_MODE=setup"
|
||||
# - "DOCKER_INFLUXDB_INIT_USERNAME=admin"
|
||||
# - "DOCKER_INFLUXDB_INIT_PASSWORD=butlersos123"
|
||||
# - "DOCKER_INFLUXDB_INIT_ORG=butler-sos"
|
||||
# - "DOCKER_INFLUXDB_INIT_BUCKET=butler-sos"
|
||||
# - "DOCKER_INFLUXDB_INIT_DATABASE=butler-sos" # v3 uses database concept
|
||||
# - "DOCKER_INFLUXDB_INIT_RETENTION=10d"
|
||||
# - "DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=butlersos-token"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -f -H 'Authorization: Bearer ${INFLUXDB_TOKEN}' http://localhost:8181/health || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
networks:
|
||||
- senseops
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
container_name: grafana
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${GRAFANA_PORT}:3000"
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER}
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
|
||||
volumes:
|
||||
- ./grafana/data:/var/lib/grafana
|
||||
depends_on:
|
||||
# Or switch to influxdb3-enterprise as needed
|
||||
- influxdb-v3-core
|
||||
networks:
|
||||
- senseops
|
||||
|
||||
networks:
|
||||
senseops:
|
||||
driver: bridge
|
||||
9
docs/docker-compose/docker-compose_fullstack_prometheus.yml
Executable file → Normal file
9
docs/docker-compose/docker-compose_fullstack_prometheus.yml
Executable file → Normal file
@@ -1,16 +1,19 @@
|
||||
# docker-compose_fullstack_prometheus.yml
|
||||
version: "3.3"
|
||||
services:
|
||||
butler-sos:
|
||||
image: ptarmiganlabs/butler-sos:latest
|
||||
container_name: butler-sos
|
||||
restart: always
|
||||
ports:
|
||||
- "9997:9997" # UDP user events
|
||||
- "9996:9996" # UDP log events
|
||||
- "9842:9842" # Prometheus metrics
|
||||
- "3100:3100" # Config file visualization
|
||||
volumes:
|
||||
# Make config file and log files accessible outside of container
|
||||
- "./config:/nodeapp/config"
|
||||
- "./log:/nodeapp/log"
|
||||
environment:
|
||||
- "NODE_ENV=production_prometheus" # Means that Butler SOS will read config data from production_prometheus.yaml
|
||||
command: ["node", "src/butler-sos.js", "-c", "/nodeapp/config/production_prometheus.yaml"]
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
|
||||
11250
docs/grafana/senseops_15-0_dashboard_influxql.json
Normal file
11250
docs/grafana/senseops_15-0_dashboard_influxql.json
Normal file
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
378
package-lock.json
generated
378
package-lock.json
generated
@@ -15,6 +15,7 @@
|
||||
"@fastify/static": "^8.3.0",
|
||||
"@influxdata/influxdb-client": "^1.35.0",
|
||||
"@influxdata/influxdb-client-apis": "^1.35.0",
|
||||
"@influxdata/influxdb3-client": "^1.4.0",
|
||||
"ajv": "^8.17.1",
|
||||
"ajv-keywords": "^5.1.0",
|
||||
"async-mutex": "^0.5.0",
|
||||
@@ -44,7 +45,7 @@
|
||||
"devDependencies": {
|
||||
"@babel/eslint-parser": "^7.28.5",
|
||||
"@babel/plugin-syntax-import-assertions": "^7.27.1",
|
||||
"@eslint/js": "^9.39.1",
|
||||
"@eslint/js": "^9.39.2",
|
||||
"audit-ci": "^7.1.0",
|
||||
"esbuild": "^0.27.1",
|
||||
"eslint-config-prettier": "^10.1.8",
|
||||
@@ -52,7 +53,7 @@
|
||||
"eslint-plugin-jsdoc": "^61.5.0",
|
||||
"eslint-plugin-prettier": "^5.5.4",
|
||||
"globals": "^16.5.0",
|
||||
"jest": "^30.1.3",
|
||||
"jest": "^30.2.0",
|
||||
"jsdoc-to-markdown": "^9.1.3",
|
||||
"license-checker-rseidelsohn": "^4.4.2",
|
||||
"lockfile-lint": "^4.14.1",
|
||||
@@ -681,9 +682,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/core": {
|
||||
"version": "1.5.0",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.5.0.tgz",
|
||||
"integrity": "sha512-sbP8GzB1WDzacS8fgNPpHlp6C9VZe+SJP3F90W9rLemaQj2PzIuTEl1qDOYQf58YIpyjViI24y9aPWCjEzY2cg==",
|
||||
"version": "1.7.1",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.7.1.tgz",
|
||||
"integrity": "sha512-o1uhUASyo921r2XtHYOHy7gdkGLge8ghBEQHMWmyJFoXlpU58kIrhhN3w26lpQb6dspetweapMn2CSNwQ8I4wg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
@@ -693,9 +694,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/runtime": {
|
||||
"version": "1.5.0",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.5.0.tgz",
|
||||
"integrity": "sha512-97/BJ3iXHww3djw6hYIfErCZFee7qCtrneuLa20UXFCOTCfBM2cvQHjWJ2EG0s0MtdNwInarqCTz35i4wWXHsQ==",
|
||||
"version": "1.7.1",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.7.1.tgz",
|
||||
"integrity": "sha512-PVtJr5CmLwYAU9PZDMITZoR5iAOShYREoR45EyyLrbntV50mdePTgUn4AmOw90Ifcj+x2kRjdzr1HP3RrNiHGA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
@@ -1317,9 +1318,9 @@
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/@eslint/js": {
|
||||
"version": "9.39.1",
|
||||
"resolved": "https://registry.npmjs.org/@eslint/js/-/js-9.39.1.tgz",
|
||||
"integrity": "sha512-S26Stp4zCy88tH94QbBv3XCuzRQiZ9yXofEILmglYTh/Ug/a9/umqvgFtYBAo3Lp0nsI/5/qH1CCrbdK3AP1Tw==",
|
||||
"version": "9.39.2",
|
||||
"resolved": "https://registry.npmjs.org/@eslint/js/-/js-9.39.2.tgz",
|
||||
"integrity": "sha512-q1mjIoW1VX4IvSocvM/vbTiveKC4k9eLrajNEuSsmjymSDEbpGddtpfOoN7YGAqBK3NG+uqo8ia4PDTt8buCYA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -1575,6 +1576,37 @@
|
||||
"fastify-plugin": "^5.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@grpc/grpc-js": {
|
||||
"version": "1.14.0",
|
||||
"resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.14.0.tgz",
|
||||
"integrity": "sha512-N8Jx6PaYzcTRNzirReJCtADVoq4z7+1KQ4E70jTg/koQiMoUSN1kbNjPOqpPbhMFhfU1/l7ixspPl8dNY+FoUg==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@grpc/proto-loader": "^0.8.0",
|
||||
"@js-sdsl/ordered-map": "^4.4.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@grpc/proto-loader": {
|
||||
"version": "0.8.0",
|
||||
"resolved": "https://registry.npmjs.org/@grpc/proto-loader/-/proto-loader-0.8.0.tgz",
|
||||
"integrity": "sha512-rc1hOQtjIWGxcxpb9aHAfLpIctjEnsDehj0DAiVfBlmT84uvR0uUtN2hEi/ecvWVjXUGf5qPF4qEgiLOx1YIMQ==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"lodash.camelcase": "^4.3.0",
|
||||
"long": "^5.0.0",
|
||||
"protobufjs": "^7.5.3",
|
||||
"yargs": "^17.7.2"
|
||||
},
|
||||
"bin": {
|
||||
"proto-loader-gen-types": "build/bin/proto-loader-gen-types.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/@humanfs/core": {
|
||||
"version": "0.19.1",
|
||||
"resolved": "https://registry.npmjs.org/@humanfs/core/-/core-0.19.1.tgz",
|
||||
@@ -1660,6 +1692,20 @@
|
||||
"@influxdata/influxdb-client": "*"
|
||||
}
|
||||
},
|
||||
"node_modules/@influxdata/influxdb3-client": {
|
||||
"version": "1.4.0",
|
||||
"resolved": "https://registry.npmjs.org/@influxdata/influxdb3-client/-/influxdb3-client-1.4.0.tgz",
|
||||
"integrity": "sha512-N07XQxQGyQ8TIscZnjS12ga4Vu2pPtvjzOZSNqeMimyV8VKRM0OEkCH/y2klCeIJkVV+A2/WZ2r4enQa5Z5wjw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@grpc/grpc-js": "^1.9.9",
|
||||
"@protobuf-ts/grpc-transport": "^2.9.1",
|
||||
"@protobuf-ts/grpcweb-transport": "^2.9.1",
|
||||
"@protobuf-ts/runtime-rpc": "^2.9.1",
|
||||
"apache-arrow": "^19.0.0",
|
||||
"grpc-web": "^1.5.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@isaacs/balanced-match": {
|
||||
"version": "4.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@isaacs/balanced-match/-/balanced-match-4.0.1.tgz",
|
||||
@@ -2333,6 +2379,16 @@
|
||||
"@jridgewell/sourcemap-codec": "^1.4.14"
|
||||
}
|
||||
},
|
||||
"node_modules/@js-sdsl/ordered-map": {
|
||||
"version": "4.4.2",
|
||||
"resolved": "https://registry.npmjs.org/@js-sdsl/ordered-map/-/ordered-map-4.4.2.tgz",
|
||||
"integrity": "sha512-iUKgm52T8HOE/makSxjqoWhe95ZJA1/G1sYsGev2JDKUSS14KAgg1LHb+Ba+IPow0xflbnSkOsZcO08C7w1gYw==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/js-sdsl"
|
||||
}
|
||||
},
|
||||
"node_modules/@jsdoc/salty": {
|
||||
"version": "0.2.9",
|
||||
"resolved": "https://registry.npmjs.org/@jsdoc/salty/-/salty-0.2.9.tgz",
|
||||
@@ -2510,6 +2566,108 @@
|
||||
"cross-spawn": "^7.0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/@protobuf-ts/grpc-transport": {
|
||||
"version": "2.11.1",
|
||||
"resolved": "https://registry.npmjs.org/@protobuf-ts/grpc-transport/-/grpc-transport-2.11.1.tgz",
|
||||
"integrity": "sha512-l6wrcFffY+tuNnuyrNCkRM8hDIsAZVLA8Mn7PKdVyYxITosYh60qW663p9kL6TWXYuDCL3oxH8ih3vLKTDyhtg==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@protobuf-ts/runtime": "^2.11.1",
|
||||
"@protobuf-ts/runtime-rpc": "^2.11.1"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@grpc/grpc-js": "^1.6.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@protobuf-ts/grpcweb-transport": {
|
||||
"version": "2.11.1",
|
||||
"resolved": "https://registry.npmjs.org/@protobuf-ts/grpcweb-transport/-/grpcweb-transport-2.11.1.tgz",
|
||||
"integrity": "sha512-1W4utDdvOB+RHMFQ0soL4JdnxjXV+ddeGIUg08DvZrA8Ms6k5NN6GBFU2oHZdTOcJVpPrDJ02RJlqtaoCMNBtw==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@protobuf-ts/runtime": "^2.11.1",
|
||||
"@protobuf-ts/runtime-rpc": "^2.11.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@protobuf-ts/runtime": {
|
||||
"version": "2.11.1",
|
||||
"resolved": "https://registry.npmjs.org/@protobuf-ts/runtime/-/runtime-2.11.1.tgz",
|
||||
"integrity": "sha512-KuDaT1IfHkugM2pyz+FwiY80ejWrkH1pAtOBOZFuR6SXEFTsnb/jiQWQ1rCIrcKx2BtyxnxW6BWwsVSA/Ie+WQ==",
|
||||
"license": "(Apache-2.0 AND BSD-3-Clause)"
|
||||
},
|
||||
"node_modules/@protobuf-ts/runtime-rpc": {
|
||||
"version": "2.11.1",
|
||||
"resolved": "https://registry.npmjs.org/@protobuf-ts/runtime-rpc/-/runtime-rpc-2.11.1.tgz",
|
||||
"integrity": "sha512-4CqqUmNA+/uMz00+d3CYKgElXO9VrEbucjnBFEjqI4GuDrEQ32MaI3q+9qPBvIGOlL4PmHXrzM32vBPWRhQKWQ==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@protobuf-ts/runtime": "^2.11.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@protobufjs/aspromise": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",
|
||||
"integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@protobufjs/base64": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz",
|
||||
"integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@protobufjs/codegen": {
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.4.tgz",
|
||||
"integrity": "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@protobufjs/eventemitter": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz",
|
||||
"integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@protobufjs/fetch": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz",
|
||||
"integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==",
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"@protobufjs/aspromise": "^1.1.1",
|
||||
"@protobufjs/inquire": "^1.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@protobufjs/float": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz",
|
||||
"integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@protobufjs/inquire": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.0.tgz",
|
||||
"integrity": "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@protobufjs/path": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz",
|
||||
"integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@protobufjs/pool": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz",
|
||||
"integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@protobufjs/utf8": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.0.tgz",
|
||||
"integrity": "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@sentry-internal/tracing": {
|
||||
"version": "7.120.3",
|
||||
"resolved": "https://registry.npmjs.org/@sentry-internal/tracing/-/tracing-7.120.3.tgz",
|
||||
@@ -2645,6 +2803,15 @@
|
||||
"text-hex": "1.0.x"
|
||||
}
|
||||
},
|
||||
"node_modules/@swc/helpers": {
|
||||
"version": "0.5.17",
|
||||
"resolved": "https://registry.npmjs.org/@swc/helpers/-/helpers-0.5.17.tgz",
|
||||
"integrity": "sha512-5IKx/Y13RsYd+sauPb2x+U/xZikHjolzfuDgTAl/Tdf3Q8rslRvC19NKDLgAJQ6wsqADk10ntlv08nPFw/gO/A==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"tslib": "^2.8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@tybys/wasm-util": {
|
||||
"version": "0.10.1",
|
||||
"resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.1.tgz",
|
||||
@@ -2701,6 +2868,18 @@
|
||||
"@babel/types": "^7.28.2"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/command-line-args": {
|
||||
"version": "5.2.3",
|
||||
"resolved": "https://registry.npmjs.org/@types/command-line-args/-/command-line-args-5.2.3.tgz",
|
||||
"integrity": "sha512-uv0aG6R0Y8WHZLTamZwtfsDLVRnOa+n+n5rEvFWL5Na5gZ8V2Teab/duDPFzIIIhs9qizDpcavCusCLJZu62Kw==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/command-line-usage": {
|
||||
"version": "5.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@types/command-line-usage/-/command-line-usage-5.0.4.tgz",
|
||||
"integrity": "sha512-BwR5KP3Es/CSht0xqBcUXS3qCAUVXwpRKsV2+arxeb65atasuXG9LykC9Ab10Cw3s2raH92ZqOeILaQbsB2ACg==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/estree": {
|
||||
"version": "1.0.8",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
|
||||
@@ -2808,9 +2987,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@types/yargs": {
|
||||
"version": "17.0.33",
|
||||
"resolved": "https://registry.npmjs.org/@types/yargs/-/yargs-17.0.33.tgz",
|
||||
"integrity": "sha512-WpxBCKWPLr4xSsHgz511rFJAM+wS28w2zEO1QDNY5zM/S8ok70NNfztH0xwhqKyaK0OHCbN98LDAZuy1ctxDkA==",
|
||||
"version": "17.0.35",
|
||||
"resolved": "https://registry.npmjs.org/@types/yargs/-/yargs-17.0.35.tgz",
|
||||
"integrity": "sha512-qUHkeCyQFxMXg79wQfTtfndEC+N9ZZg76HJftDJp+qH2tV7Gj4OJi7l+PiWwJ+pWtW8GwSmqsDj/oymhrTWXjg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -2825,9 +3004,9 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@typescript-eslint/types": {
|
||||
"version": "8.48.1",
|
||||
"resolved": "https://registry.npmjs.org/@typescript-eslint/types/-/types-8.48.1.tgz",
|
||||
"integrity": "sha512-+fZ3LZNeiELGmimrujsDCT4CRIbq5oXdHe7chLiW8qzqyPMnn1puNstCrMNVAqwcl2FdIxkuJ4tOs/RFDBVc/Q==",
|
||||
"version": "8.49.0",
|
||||
"resolved": "https://registry.npmjs.org/@typescript-eslint/types/-/types-8.49.0.tgz",
|
||||
"integrity": "sha512-e9k/fneezorUo6WShlQpMxXh8/8wfyc+biu6tnAqA81oWrEic0k21RHzP9uqqpyBBeBKu4T+Bsjy9/b8u7obXQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -3305,6 +3484,35 @@
|
||||
"node": ">= 8"
|
||||
}
|
||||
},
|
||||
"node_modules/apache-arrow": {
|
||||
"version": "19.0.1",
|
||||
"resolved": "https://registry.npmjs.org/apache-arrow/-/apache-arrow-19.0.1.tgz",
|
||||
"integrity": "sha512-APmMLzS4qbTivLrPdQXexGM4JRr+0g62QDaobzEvip/FdQIrv2qLy0mD5Qdmw4buydtVJgbFeKR8f59I6PPGDg==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@swc/helpers": "^0.5.11",
|
||||
"@types/command-line-args": "^5.2.3",
|
||||
"@types/command-line-usage": "^5.0.4",
|
||||
"@types/node": "^20.13.0",
|
||||
"command-line-args": "^6.0.1",
|
||||
"command-line-usage": "^7.0.1",
|
||||
"flatbuffers": "^24.3.25",
|
||||
"json-bignum": "^0.0.3",
|
||||
"tslib": "^2.6.2"
|
||||
},
|
||||
"bin": {
|
||||
"arrow2csv": "bin/arrow2csv.js"
|
||||
}
|
||||
},
|
||||
"node_modules/apache-arrow/node_modules/@types/node": {
|
||||
"version": "20.19.17",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.17.tgz",
|
||||
"integrity": "sha512-gfehUI8N1z92kygssiuWvLiwcbOB3IRktR6hTDgJlXMYh5OvkPSRmgfoBUmfZt+vhwJtX7v1Yw4KvvAf7c5QKQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": "~6.21.0"
|
||||
}
|
||||
},
|
||||
"node_modules/are-docs-informative": {
|
||||
"version": "0.0.2",
|
||||
"resolved": "https://registry.npmjs.org/are-docs-informative/-/are-docs-informative-0.0.2.tgz",
|
||||
@@ -3324,7 +3532,6 @@
|
||||
"version": "6.2.2",
|
||||
"resolved": "https://registry.npmjs.org/array-back/-/array-back-6.2.2.tgz",
|
||||
"integrity": "sha512-gUAZ7HPyb4SJczXAMUXMGAvI976JoK3qEx9v1FTmeYuJj0IBiaKttG1ydtGKdkfqWkIkouke7nG8ufGy77+Cvw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12.17"
|
||||
@@ -3787,7 +3994,6 @@
|
||||
"version": "4.1.2",
|
||||
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
|
||||
"integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==",
|
||||
"dev": true,
|
||||
"dependencies": {
|
||||
"ansi-styles": "^4.1.0",
|
||||
"supports-color": "^7.1.0"
|
||||
@@ -3803,7 +4009,6 @@
|
||||
"version": "0.4.0",
|
||||
"resolved": "https://registry.npmjs.org/chalk-template/-/chalk-template-0.4.0.tgz",
|
||||
"integrity": "sha512-/ghrgmhfY8RaSdeo43hNXxpoHAtxdbskUHjPpfqUWGttFgycUhYPGx3YZBCnUCvOa7Doivn1IZec3DEGFoMgLg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"chalk": "^4.1.2"
|
||||
@@ -3826,9 +4031,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/ci-info": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/ci-info/-/ci-info-4.3.0.tgz",
|
||||
"integrity": "sha512-l+2bNRMiQgcfILUi33labAZYIWlH1kWDp+ecNo5iisRKrbm0xcRyCww71/YU0Fkw0mAFpz9bJayXPjey6vkmaQ==",
|
||||
"version": "4.3.1",
|
||||
"resolved": "https://registry.npmjs.org/ci-info/-/ci-info-4.3.1.tgz",
|
||||
"integrity": "sha512-Wdy2Igu8OcBpI2pZePZ5oWjPC38tmDVx5WKUXKwlLYkA0ozo85sLsLvkBbBn/sZaSCMFOGZJ14fvW9t5/d7kdA==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
@@ -3842,9 +4047,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/cjs-module-lexer": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/cjs-module-lexer/-/cjs-module-lexer-2.1.0.tgz",
|
||||
"integrity": "sha512-UX0OwmYRYQQetfrLEZeewIFFI+wSTofC+pMBLNuH3RUuu/xzG1oz84UCEDOSoQlN3fZ4+AzmV50ZYvGqkMh9yA==",
|
||||
"version": "2.1.1",
|
||||
"resolved": "https://registry.npmjs.org/cjs-module-lexer/-/cjs-module-lexer-2.1.1.tgz",
|
||||
"integrity": "sha512-+CmxIZ/L2vNcEfvNtLdU0ZQ6mbq3FZnwAP2PPTiKP+1QOoKwlKlPgb8UKV0Dds7QVaMnHm+FwSft2VB0s/SLjQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -3852,7 +4057,6 @@
|
||||
"version": "8.0.1",
|
||||
"resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz",
|
||||
"integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"string-width": "^4.2.0",
|
||||
@@ -3867,7 +4071,6 @@
|
||||
"version": "7.0.0",
|
||||
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",
|
||||
"integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"ansi-styles": "^4.0.0",
|
||||
@@ -3893,9 +4096,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/collect-v8-coverage": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/collect-v8-coverage/-/collect-v8-coverage-1.0.2.tgz",
|
||||
"integrity": "sha512-lHl4d5/ONEbLlJvaJNtsF/Lz+WvB07u2ycqTYbdrq7UypDXailES4valYb2eWiJFxZlVmpGekfqoxQhzyFdT4Q==",
|
||||
"version": "1.0.3",
|
||||
"resolved": "https://registry.npmjs.org/collect-v8-coverage/-/collect-v8-coverage-1.0.3.tgz",
|
||||
"integrity": "sha512-1L5aqIkwPfiodaMgQunkF1zRhNqifHBmtbbbxcr6yVxxBnliw4TDOW6NxpO8DJLgJ16OT+Y4ztZqP6p/FtXnAw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -3985,7 +4188,6 @@
|
||||
"version": "6.0.1",
|
||||
"resolved": "https://registry.npmjs.org/command-line-args/-/command-line-args-6.0.1.tgz",
|
||||
"integrity": "sha512-Jr3eByUjqyK0qd8W0SGFW1nZwqCaNCtbXjRo2cRJC1OYxWl3MZ5t1US3jq+cO4sPavqgw4l9BMGX0CBe+trepg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"array-back": "^6.2.2",
|
||||
@@ -4009,7 +4211,6 @@
|
||||
"version": "7.0.3",
|
||||
"resolved": "https://registry.npmjs.org/command-line-usage/-/command-line-usage-7.0.3.tgz",
|
||||
"integrity": "sha512-PqMLy5+YGwhMh1wS04mVG44oqDsgyLRSKJBdOo1bnYhMKBW65gZF1dRp2OZRhiTjgUHljy99qkO7bsctLaw35Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"array-back": "^6.2.2",
|
||||
@@ -4560,7 +4761,6 @@
|
||||
"version": "3.2.0",
|
||||
"resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz",
|
||||
"integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
@@ -5323,7 +5523,6 @@
|
||||
"version": "5.0.2",
|
||||
"resolved": "https://registry.npmjs.org/find-replace/-/find-replace-5.0.2.tgz",
|
||||
"integrity": "sha512-Y45BAiE3mz2QsrN2fb5QEtO4qb44NcS7en/0y9PEVsg351HsLeVclP8QPMH79Le9sH3rs5RSwJu99W0WPZO43Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=14"
|
||||
@@ -5369,6 +5568,12 @@
|
||||
"node": ">=16"
|
||||
}
|
||||
},
|
||||
"node_modules/flatbuffers": {
|
||||
"version": "24.12.23",
|
||||
"resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-24.12.23.tgz",
|
||||
"integrity": "sha512-dLVCAISd5mhls514keQzmEG6QHmUUsNuWsb4tFafIUwvvgDjXhtfAYSKOzt5SWOy+qByV5pbsDZ+Vb7HUOBEdA==",
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/flatted": {
|
||||
"version": "3.3.1",
|
||||
"resolved": "https://registry.npmjs.org/flatted/-/flatted-3.3.1.tgz",
|
||||
@@ -5506,7 +5711,6 @@
|
||||
"version": "2.0.5",
|
||||
"resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz",
|
||||
"integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": "6.* || 8.* || >= 10.*"
|
||||
@@ -5712,6 +5916,12 @@
|
||||
"resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz",
|
||||
"integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ=="
|
||||
},
|
||||
"node_modules/grpc-web": {
|
||||
"version": "1.5.0",
|
||||
"resolved": "https://registry.npmjs.org/grpc-web/-/grpc-web-1.5.0.tgz",
|
||||
"integrity": "sha512-y1tS3BBIoiVSzKTDF3Hm7E8hV2n7YY7pO0Uo7depfWJqKzWE+SKr0jvHNIJsJJYILQlpYShpi/DRJJMbosgDMQ==",
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/handlebars": {
|
||||
"version": "4.7.8",
|
||||
"resolved": "https://registry.npmjs.org/handlebars/-/handlebars-4.7.8.tgz",
|
||||
@@ -5737,7 +5947,6 @@
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz",
|
||||
"integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==",
|
||||
"dev": true,
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
@@ -6130,9 +6339,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/istanbul-lib-instrument/node_modules/semver": {
|
||||
"version": "7.7.2",
|
||||
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.2.tgz",
|
||||
"integrity": "sha512-RF0Fw+rO5AMf9MAyaRXI4AV0Ulj5lMHqVxxdSgiVbixSCXoEmmX/jk0CuJw4+3SqroYO9VoUh+HcuJivvtJemA==",
|
||||
"version": "7.7.3",
|
||||
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.3.tgz",
|
||||
"integrity": "sha512-SdsKMrI9TdgjdweUSR9MweHA4EJ8YxHn8DFaDisvhVlUOe4BF1tLD7GAj0lIqWVl+dPb/rExr0Btby5loQm20Q==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"bin": {
|
||||
@@ -6840,9 +7049,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/jest-snapshot/node_modules/semver": {
|
||||
"version": "7.7.2",
|
||||
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.2.tgz",
|
||||
"integrity": "sha512-RF0Fw+rO5AMf9MAyaRXI4AV0Ulj5lMHqVxxdSgiVbixSCXoEmmX/jk0CuJw4+3SqroYO9VoUh+HcuJivvtJemA==",
|
||||
"version": "7.7.3",
|
||||
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.3.tgz",
|
||||
"integrity": "sha512-SdsKMrI9TdgjdweUSR9MweHA4EJ8YxHn8DFaDisvhVlUOe4BF1tLD7GAj0lIqWVl+dPb/rExr0Btby5loQm20Q==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"bin": {
|
||||
@@ -7155,6 +7364,14 @@
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/json-bignum": {
|
||||
"version": "0.0.3",
|
||||
"resolved": "https://registry.npmjs.org/json-bignum/-/json-bignum-0.0.3.tgz",
|
||||
"integrity": "sha512-2WHyXj3OfHSgNyuzDbSxI1w2jgw5gkWSWhS7Qg4bWXx1nLk3jnbwfUeS0PSba3IzpTUWdHxBieELUzXRjQB2zg==",
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/json-buffer": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/json-buffer/-/json-buffer-3.0.1.tgz",
|
||||
@@ -7495,7 +7712,6 @@
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/lodash.camelcase/-/lodash.camelcase-4.3.0.tgz",
|
||||
"integrity": "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/lodash.clonedeep": {
|
||||
@@ -7533,6 +7749,12 @@
|
||||
"node": ">= 12.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/long": {
|
||||
"version": "5.3.2",
|
||||
"resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz",
|
||||
"integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==",
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/lru-cache": {
|
||||
"version": "10.4.3",
|
||||
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-10.4.3.tgz",
|
||||
@@ -7565,9 +7787,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/make-dir/node_modules/semver": {
|
||||
"version": "7.7.2",
|
||||
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.2.tgz",
|
||||
"integrity": "sha512-RF0Fw+rO5AMf9MAyaRXI4AV0Ulj5lMHqVxxdSgiVbixSCXoEmmX/jk0CuJw4+3SqroYO9VoUh+HcuJivvtJemA==",
|
||||
"version": "7.7.3",
|
||||
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.3.tgz",
|
||||
"integrity": "sha512-SdsKMrI9TdgjdweUSR9MweHA4EJ8YxHn8DFaDisvhVlUOe4BF1tLD7GAj0lIqWVl+dPb/rExr0Btby5loQm20Q==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"bin": {
|
||||
@@ -7845,9 +8067,9 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/napi-postinstall": {
|
||||
"version": "0.3.3",
|
||||
"resolved": "https://registry.npmjs.org/napi-postinstall/-/napi-postinstall-0.3.3.tgz",
|
||||
"integrity": "sha512-uTp172LLXSxuSYHv/kou+f6KW3SMppU9ivthaVTXian9sOt3XM/zHYHpRZiLgQoxeWfYUnslNWQHF1+G71xcow==",
|
||||
"version": "0.3.4",
|
||||
"resolved": "https://registry.npmjs.org/napi-postinstall/-/napi-postinstall-0.3.4.tgz",
|
||||
"integrity": "sha512-PHI5f1O0EP5xJ9gQmFGMS6IZcrVvTjpXjz7Na41gTE7eE2hK11lg04CECCYEEjdc17EV4DO+fkGEtt7TpTaTiQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
@@ -7942,9 +8164,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/npm-check-updates": {
|
||||
"version": "19.1.2",
|
||||
"resolved": "https://registry.npmjs.org/npm-check-updates/-/npm-check-updates-19.1.2.tgz",
|
||||
"integrity": "sha512-FNeFCVgPOj0fz89hOpGtxP2rnnRHR7hD2E8qNU8SMWfkyDZXA/xpgjsL3UMLSo3F/K13QvJDnbxPngulNDDo/g==",
|
||||
"version": "19.2.0",
|
||||
"resolved": "https://registry.npmjs.org/npm-check-updates/-/npm-check-updates-19.2.0.tgz",
|
||||
"integrity": "sha512-XSIuL0FNgzXPDZa4lje7+OwHjiyEt84qQm6QMsQRbixNY5EHEM9nhgOjxjlK9jIbN+ysvSqOV8DKNS0zydwbdg==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
@@ -8531,6 +8753,30 @@
|
||||
"node": "^16 || ^18 || >=20"
|
||||
}
|
||||
},
|
||||
"node_modules/protobufjs": {
|
||||
"version": "7.5.4",
|
||||
"resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.5.4.tgz",
|
||||
"integrity": "sha512-CvexbZtbov6jW2eXAvLukXjXUW1TzFaivC46BpWc/3BpcCysb5Vffu+B3XHMm8lVEuy2Mm4XGex8hBSg1yapPg==",
|
||||
"hasInstallScript": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"@protobufjs/aspromise": "^1.1.2",
|
||||
"@protobufjs/base64": "^1.1.2",
|
||||
"@protobufjs/codegen": "^2.0.4",
|
||||
"@protobufjs/eventemitter": "^1.1.0",
|
||||
"@protobufjs/fetch": "^1.1.0",
|
||||
"@protobufjs/float": "^1.0.2",
|
||||
"@protobufjs/inquire": "^1.1.0",
|
||||
"@protobufjs/path": "^1.1.2",
|
||||
"@protobufjs/pool": "^1.1.0",
|
||||
"@protobufjs/utf8": "^1.1.0",
|
||||
"@types/node": ">=13.7.0",
|
||||
"long": "^5.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/proxy-from-env": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz",
|
||||
@@ -8782,7 +9028,6 @@
|
||||
"version": "2.1.1",
|
||||
"resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz",
|
||||
"integrity": "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
@@ -9112,9 +9357,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/snyk": {
|
||||
"version": "1.1301.0",
|
||||
"resolved": "https://registry.npmjs.org/snyk/-/snyk-1.1301.0.tgz",
|
||||
"integrity": "sha512-kTb8F9L1PlI3nYWlp60wnSGWGmcRs6bBtSBl9s8YYhAiFZNseIZfXolQXBSCaya5QlcxzfH1pb4aqCNMbi0tgg==",
|
||||
"version": "1.1301.1",
|
||||
"resolved": "https://registry.npmjs.org/snyk/-/snyk-1.1301.1.tgz",
|
||||
"integrity": "sha512-EYgBCi0+diYgqiibdwyUowBCcowKDGcfqXkZoBWG3qNdcLVZqjq7ogOEKwOcbNern7doDzm2TSZtbRCu+SpVMQ==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "Apache-2.0",
|
||||
@@ -9483,7 +9728,6 @@
|
||||
"version": "7.2.0",
|
||||
"resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz",
|
||||
"integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==",
|
||||
"dev": true,
|
||||
"dependencies": {
|
||||
"has-flag": "^4.0.0"
|
||||
},
|
||||
@@ -9508,9 +9752,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/systeminformation": {
|
||||
"version": "5.27.11",
|
||||
"resolved": "https://registry.npmjs.org/systeminformation/-/systeminformation-5.27.11.tgz",
|
||||
"integrity": "sha512-K3Lto/2m3K2twmKHdgx5B+0in9qhXK4YnoT9rIlgwN/4v7OV5c8IjbeAUkuky/6VzCQC7iKCAqi8rZathCdjHg==",
|
||||
"version": "5.27.13",
|
||||
"resolved": "https://registry.npmjs.org/systeminformation/-/systeminformation-5.27.13.tgz",
|
||||
"integrity": "sha512-geeE/7eNDoOhdc9j+qCsLlwbcyh0HnqhOZzmfNK4WBioWGUZbhwYrg+YZsZ3UJh4tmybQsnDuqzr3UoumMifew==",
|
||||
"license": "MIT",
|
||||
"os": [
|
||||
"darwin",
|
||||
@@ -9553,7 +9797,6 @@
|
||||
"version": "4.1.1",
|
||||
"resolved": "https://registry.npmjs.org/table-layout/-/table-layout-4.1.1.tgz",
|
||||
"integrity": "sha512-iK5/YhZxq5GO5z8wb0bY1317uDF3Zjpha0QFFLA8/trAoiLbQD0HUbMesEaxyzUgDxi2QlcbM8IvqOlEjgoXBA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"array-back": "^6.2.2",
|
||||
@@ -9819,7 +10062,6 @@
|
||||
"version": "7.3.0",
|
||||
"resolved": "https://registry.npmjs.org/typical/-/typical-7.3.0.tgz",
|
||||
"integrity": "sha512-ya4mg/30vm+DOWfBg4YK3j2WD6TWtRkCbasOJr40CseYENzCUby/7rIvXA99JGsQHeNxLbnXdyLLxKSv3tauFw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12.17"
|
||||
@@ -9846,9 +10088,9 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/ua-parser-js": {
|
||||
"version": "2.0.6",
|
||||
"resolved": "https://registry.npmjs.org/ua-parser-js/-/ua-parser-js-2.0.6.tgz",
|
||||
"integrity": "sha512-EmaxXfltJaDW75SokrY4/lXMrVyXomE/0FpIIqP2Ctic93gK7rlme55Cwkz8l3YZ6gqf94fCU7AnIkidd/KXPg==",
|
||||
"version": "2.0.7",
|
||||
"resolved": "https://registry.npmjs.org/ua-parser-js/-/ua-parser-js-2.0.7.tgz",
|
||||
"integrity": "sha512-CFdHVHr+6YfbktNZegH3qbYvYgC7nRNEUm2tk7nSFXSODUu4tDBpaFpP1jdXBUOKKwapVlWRfTtS8bCPzsQ47w==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "opencollective",
|
||||
@@ -10181,7 +10423,6 @@
|
||||
"version": "5.1.0",
|
||||
"resolved": "https://registry.npmjs.org/wordwrapjs/-/wordwrapjs-5.1.0.tgz",
|
||||
"integrity": "sha512-JNjcULU2e4KJwUNv6CHgI46UvDGitb6dGryHajXTDiLgg1/RiGoPSDw4kZfYnwGtEXf2ZMeIewDQgFGzkCB2Sg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12.17"
|
||||
@@ -10384,7 +10625,6 @@
|
||||
"version": "5.0.8",
|
||||
"resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz",
|
||||
"integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
@@ -10400,7 +10640,6 @@
|
||||
"version": "17.7.2",
|
||||
"resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.2.tgz",
|
||||
"integrity": "sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"cliui": "^8.0.1",
|
||||
@@ -10419,7 +10658,6 @@
|
||||
"version": "21.1.1",
|
||||
"resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz",
|
||||
"integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
{
|
||||
"name": "butler-sos",
|
||||
"version": "14.0.0",
|
||||
"description": "Butler SenseOps Stats (\"Butler SOS\") is a tool that publishes operational Qlik Sense metrics to Influxdb, Prometheus, New Relic and MQTT.",
|
||||
"description": "Butler SenseOps Stats (\"Butler SOS\") is a tool that publishes operational Qlik Sense metrics to InfluxDB (v1, v2, v3), Prometheus, New Relic and MQTT.",
|
||||
"main": "butler-sos.js",
|
||||
"scripts": {
|
||||
"build": "npx jsdoc-to-markdown 'src/**/*.js' > docs/src-code-overview.md",
|
||||
"build:docker": "docker build -t butler-sos:latest .",
|
||||
"butler-sos": "node src/butler-sos.js",
|
||||
"jest": "node --experimental-vm-modules --no-warnings node_modules/jest/bin/jest.js",
|
||||
"test": "node --experimental-vm-modules --no-warnings node_modules/jest/bin/jest.js && snyk test && npm run format",
|
||||
@@ -52,6 +53,7 @@
|
||||
"@fastify/static": "^8.3.0",
|
||||
"@influxdata/influxdb-client": "^1.35.0",
|
||||
"@influxdata/influxdb-client-apis": "^1.35.0",
|
||||
"@influxdata/influxdb3-client": "^1.4.0",
|
||||
"ajv": "^8.17.1",
|
||||
"ajv-keywords": "^5.1.0",
|
||||
"async-mutex": "^0.5.0",
|
||||
@@ -81,7 +83,7 @@
|
||||
"devDependencies": {
|
||||
"@babel/eslint-parser": "^7.28.5",
|
||||
"@babel/plugin-syntax-import-assertions": "^7.27.1",
|
||||
"@eslint/js": "^9.39.1",
|
||||
"@eslint/js": "^9.39.2",
|
||||
"audit-ci": "^7.1.0",
|
||||
"esbuild": "^0.27.1",
|
||||
"eslint-config-prettier": "^10.1.8",
|
||||
@@ -89,7 +91,7 @@
|
||||
"eslint-plugin-jsdoc": "^61.5.0",
|
||||
"eslint-plugin-prettier": "^5.5.4",
|
||||
"globals": "^16.5.0",
|
||||
"jest": "^30.1.3",
|
||||
"jest": "^30.2.0",
|
||||
"jsdoc-to-markdown": "^9.1.3",
|
||||
"license-checker-rseidelsohn": "^4.4.2",
|
||||
"lockfile-lint": "^4.14.1",
|
||||
|
||||
@@ -34,6 +34,11 @@
|
||||
"type": "build",
|
||||
"section": "Miscellaneous",
|
||||
"hidden": false
|
||||
},
|
||||
{
|
||||
"type": "test",
|
||||
"section": "Miscellaneous",
|
||||
"hidden": false
|
||||
}
|
||||
],
|
||||
"packages": {
|
||||
|
||||
@@ -24,7 +24,8 @@ import { setupAnonUsageReportTimer } from './lib/telemetry.js';
|
||||
import { setupPromClient } from './lib/prom-client.js';
|
||||
import { setupConfigVisServer } from './lib/config-visualise.js';
|
||||
import { setupUdpEventsStorage } from './lib/udp-event.js';
|
||||
import { setupUdpQueueMetricsStorage } from './lib/post-to-influxdb.js';
|
||||
import { setupUdpQueueMetricsStorage } from './lib/influxdb/index.js';
|
||||
import { logError } from './lib/log-error.js';
|
||||
|
||||
// Suppress experimental warnings
|
||||
// https://stackoverflow.com/questions/55778283/how-to-disable-warnings-when-node-is-launched-via-a-global-shell-script
|
||||
@@ -204,7 +205,7 @@ async function mainScript() {
|
||||
);
|
||||
}
|
||||
} catch (err) {
|
||||
globals.logger.error(`CONFIG: Error initiating host info: ${globals.getErrorMessage(err)}`);
|
||||
logError('CONFIG: Error initiating host info', err);
|
||||
}
|
||||
|
||||
// Set up UDP handler for user activity/events
|
||||
|
||||
@@ -63,12 +63,12 @@ Butler-SOS:
|
||||
enable: true # Should Butler SOS' uptime (how long since it was started) be sent to New Relic?
|
||||
attribute:
|
||||
static: # Static attributes/dimensions to attach to the data sent to New Relic.
|
||||
# - name: metricType
|
||||
# value: butler-sos-uptime
|
||||
# - name: qs_service
|
||||
# value: butler-sos
|
||||
# - name: qs_environment
|
||||
# value: prod
|
||||
- name: metricType
|
||||
value: butler-sos-uptime
|
||||
- name: qs_service
|
||||
value: butler-sos
|
||||
- name: qs_env
|
||||
value: dev
|
||||
dynamic:
|
||||
butlerVersion:
|
||||
enable: true # Should the Butler SOS version be included in the data sent to New Relic?
|
||||
@@ -97,10 +97,8 @@ Butler-SOS:
|
||||
influxdb:
|
||||
measurementName: event_count # Name of the InfluxDB measurement where event count is stored
|
||||
tags: # Tags are added to the data before it's stored in InfluxDB
|
||||
# - name: env
|
||||
# value: DEV
|
||||
# - name: foo
|
||||
# value: bar
|
||||
- name: qs_env
|
||||
value: dev
|
||||
rejectedEventCount: # Rejected events are events that are received from Sense, that are correctly formatted,
|
||||
# but that are rejected by Butler SOS based on the configuration in this file.
|
||||
# An example of a rejected event is a performance log event that is filtered out by Butler SOS.
|
||||
@@ -137,13 +135,11 @@ Butler-SOS:
|
||||
writeFrequency: 20000 # How often to write metrics, milliseconds (default: 20000)
|
||||
measurementName: user_events_queue # InfluxDB measurement name (default: user_events_queue)
|
||||
tags: # Optional tags added to queue metrics
|
||||
# - name: env
|
||||
# value: prod
|
||||
- name: qs_env
|
||||
value: dev
|
||||
tags: # Tags are added to the data before it's stored in InfluxDB
|
||||
# - name: env
|
||||
# value: DEV
|
||||
# - name: foo
|
||||
# value: bar
|
||||
- name: qs_env
|
||||
value: dev
|
||||
sendToMQTT:
|
||||
enable: false # Set to true if user events should be forwarded as MQTT messages
|
||||
postTo: # Control when and to which MQTT topics messages are sent
|
||||
@@ -193,13 +189,11 @@ Butler-SOS:
|
||||
writeFrequency: 20000 # How often to write metrics, milliseconds (default: 20000)
|
||||
measurementName: log_events_queue # InfluxDB measurement name (default: log_events_queue)
|
||||
tags: # Optional tags added to queue metrics
|
||||
# - name: env
|
||||
# value: prod
|
||||
- name: qs_env
|
||||
value: dev
|
||||
tags:
|
||||
# - name: env
|
||||
# value: DEV
|
||||
# - name: foo
|
||||
# value: bar
|
||||
- name: qs_env
|
||||
value: dev
|
||||
source:
|
||||
engine:
|
||||
enable: false # Should log events from the engine service be handled?
|
||||
@@ -283,10 +277,8 @@ Butler-SOS:
|
||||
trackRejectedEvents:
|
||||
enable: false # Should events that are rejected by the app performance monitor be tracked?
|
||||
tags: # Tags are added to the data before it's stored in InfluxDB
|
||||
# - name: env
|
||||
# value: DEV
|
||||
# - name: foo
|
||||
# value: bar
|
||||
- name: qs_env
|
||||
value: dev
|
||||
monitorFilter: # What objects should be monitored? Entire apps or just specific object(s) within some specific app(s)?
|
||||
# Two kinds of monitoring can be done:
|
||||
# 1) Monitor all apps, except those listed for exclusion. This is defined in the allApps section.
|
||||
@@ -438,10 +430,10 @@ Butler-SOS:
|
||||
# value: Header value
|
||||
attribute:
|
||||
static: # Static attributes/dimensions to attach to the events sent to New Relic.
|
||||
# - name: service
|
||||
# value: butler-sos
|
||||
# - name: environment
|
||||
# value: prod
|
||||
- name: qs_env
|
||||
value: dev
|
||||
- name: service
|
||||
value: butler-sos
|
||||
dynamic:
|
||||
butlerSosVersion:
|
||||
enable: true # Should the Butler SOS version be included in the events sent to New Relic?
|
||||
@@ -492,10 +484,10 @@ Butler-SOS:
|
||||
enable: true
|
||||
attribute:
|
||||
static: # Static attributes/dimensions to attach to the data sent to New Relic.
|
||||
# - name: service
|
||||
# value: butler-sos
|
||||
# - name: environment
|
||||
# value: prod
|
||||
- name: qs_env
|
||||
value: dev
|
||||
- name: service
|
||||
value: butler-sos
|
||||
dynamic:
|
||||
butlerSosVersion:
|
||||
enable: true # Should the Butler SOS version be included in the data sent to New Relic?
|
||||
@@ -513,7 +505,15 @@ Butler-SOS:
|
||||
# Items below are mandatory if influxdbConfig.enable=true
|
||||
host: influxdb.mycompany.com # InfluxDB host, hostname, FQDN or IP address
|
||||
port: 8086 # Port where InfluxDBdb is listening, usually 8086
|
||||
version: 1 # Is the InfluxDB instance version 1.x or 2.x? Valid values are 1 or 2
|
||||
version: 2 # Is the InfluxDB instance version 1.x or 2.x? Valid values are 1, 2, or 3
|
||||
maxBatchSize: 1000 # Maximum number of data points to write in a single batch. If a batch fails, progressive retry with smaller sizes (1000→500→250→100→10→1) will be attempted. Valid range: 1-10000.
|
||||
v3Config: # Settings for InfluxDB v3.x only, i.e. Butler-SOS.influxdbConfig.version=3
|
||||
database: mydatabase
|
||||
description: Butler SOS metrics
|
||||
token: mytoken
|
||||
retentionDuration: 10d
|
||||
writeTimeout: 10000 # Optional: Socket timeout in milliseconds (writing to InfluxDB) (default: 10000)
|
||||
queryTimeout: 60000 # Optional: Query timeout in milliseconds (default: 60000)
|
||||
v2Config: # Settings for InfluxDB v2.x only, i.e. Butler-SOS.influxdbConfig.version=2
|
||||
org: myorg
|
||||
bucket: mybucket
|
||||
@@ -525,7 +525,7 @@ Butler-SOS:
|
||||
enable: false # Does influxdb instance require authentication (true/false)?
|
||||
username: <username> # Username for Influxdb authentication. Mandatory if auth.enable=true
|
||||
password: <password> # Password for Influxdb authentication. Mandatory if auth.enable=true
|
||||
dbName: SenseOps
|
||||
dbName: senseops
|
||||
# Default retention policy that should be created in InfluxDB when Butler SOS creates a new database there.
|
||||
# Any data older than retention policy threshold will be purged from InfluxDB.
|
||||
retentionPolicy:
|
||||
|
||||
170
src/globals.js
170
src/globals.js
@@ -8,16 +8,39 @@ import winston from 'winston';
|
||||
import 'winston-daily-rotate-file';
|
||||
import si from 'systeminformation';
|
||||
import { readFileSync } from 'fs';
|
||||
import Influx from 'influx';
|
||||
import { Command, Option } from 'commander';
|
||||
import { InfluxDB, HttpError, DEFAULT_WriteOptions } from '@influxdata/influxdb-client';
|
||||
|
||||
// Note on InfluxDB libraries:
|
||||
// v1 client library: https://github.com/node-influx/node-influx
|
||||
// v2 client library: https://influxdata.github.io/influxdb-client-js/
|
||||
// v3 client library: https://github.com/InfluxCommunity/influxdb3-js
|
||||
|
||||
// v1
|
||||
import Influx from 'influx';
|
||||
|
||||
// v2
|
||||
// Import InfluxDB as const InfluxDB2 to avoid name clash with Influx from 'influx' above
|
||||
import {
|
||||
InfluxDB as InfluxDB2,
|
||||
HttpError,
|
||||
DEFAULT_WriteOptions,
|
||||
} from '@influxdata/influxdb-client';
|
||||
import { OrgsAPI, BucketsAPI } from '@influxdata/influxdb-client-apis';
|
||||
|
||||
// v3
|
||||
import {
|
||||
InfluxDBClient as InfluxDBClient3,
|
||||
Point as Point3,
|
||||
setLogger as setInfluxV3Logger,
|
||||
} from '@influxdata/influxdb3-client';
|
||||
|
||||
import { fileURLToPath } from 'url';
|
||||
import sea from './lib/sea-wrapper.js';
|
||||
|
||||
import { getServerTags } from './lib/servertags.js';
|
||||
import { UdpEvents } from './lib/udp-event.js';
|
||||
import { UdpQueueManager } from './lib/udp-queue-manager.js';
|
||||
import { ErrorTracker, setupErrorCounterReset } from './lib/error-tracker.js';
|
||||
import { verifyConfigFileSchema, verifyAppConfig } from './lib/config-file-verify.js';
|
||||
|
||||
let instance = null;
|
||||
@@ -135,9 +158,6 @@ class Settings {
|
||||
|
||||
this.appVersion = appVersion;
|
||||
|
||||
// Make copy of influxdb client
|
||||
const InfluxDB2 = InfluxDB;
|
||||
|
||||
// Command line parameters
|
||||
const program = new Command();
|
||||
program
|
||||
@@ -574,6 +594,14 @@ Configuration File:
|
||||
this.rejectedEvents = null;
|
||||
}
|
||||
|
||||
// ------------------------------------
|
||||
// Track API error counts
|
||||
this.errorTracker = new ErrorTracker(this.logger);
|
||||
this.logger.info('ERROR TRACKER: Initialized error tracking with daily UTC reset');
|
||||
|
||||
// Setup midnight UTC reset timer for error counters
|
||||
setupErrorCounterReset();
|
||||
|
||||
// ------------------------------------
|
||||
// Get info on what servers to monitor
|
||||
this.serverList = this.config.get('Butler-SOS.serversToMonitor.servers');
|
||||
@@ -701,6 +729,13 @@ Configuration File:
|
||||
this.logger.info(
|
||||
`CONFIG: Influxdb retention policy duration: ${this.config.get('Butler-SOS.influxdbConfig.v2Config.retentionDuration')}`
|
||||
);
|
||||
} else if (this.config.get('Butler-SOS.influxdbConfig.version') === 3) {
|
||||
this.logger.info(
|
||||
`CONFIG: Influxdb database name: ${this.config.get('Butler-SOS.influxdbConfig.v3Config.database')}`
|
||||
);
|
||||
this.logger.info(
|
||||
`CONFIG: Influxdb retention policy duration: ${this.config.get('Butler-SOS.influxdbConfig.v3Config.retentionDuration')}`
|
||||
);
|
||||
} else {
|
||||
this.logger.error(
|
||||
`CONFIG: Influxdb version ${this.config.get('Butler-SOS.influxdbConfig.version')} is not supported!`
|
||||
@@ -870,6 +905,88 @@ Configuration File:
|
||||
);
|
||||
this.logger.error(`INFLUXDB2 INIT: Exiting.`);
|
||||
}
|
||||
} else if (this.config.get('Butler-SOS.influxdbConfig.version') === 3) {
|
||||
// Configure InfluxDB v3 client logger to suppress internal error messages
|
||||
// The retry logic in Butler SOS provides better error handling
|
||||
setInfluxV3Logger({
|
||||
error: () => {
|
||||
// Suppress InfluxDB client library error messages
|
||||
// Butler SOS retry logic and logging handles errors
|
||||
},
|
||||
warn: () => {
|
||||
// Suppress InfluxDB client library warning messages
|
||||
},
|
||||
});
|
||||
|
||||
// Set up Influxdb v3 client (uses its own client library, NOT same as v2)
|
||||
const hostName = this.config.get('Butler-SOS.influxdbConfig.host');
|
||||
const port = this.config.get('Butler-SOS.influxdbConfig.port');
|
||||
const host = `http://${hostName}:${port}`;
|
||||
const token = this.config.get('Butler-SOS.influxdbConfig.v3Config.token');
|
||||
const database = this.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
// Get timeout settings with defaults
|
||||
const writeTimeout = this.config.has(
|
||||
'Butler-SOS.influxdbConfig.v3Config.writeTimeout'
|
||||
)
|
||||
? this.config.get('Butler-SOS.influxdbConfig.v3Config.writeTimeout')
|
||||
: 10000; // Default 10 seconds for socket timeout
|
||||
|
||||
const queryTimeout = this.config.has(
|
||||
'Butler-SOS.influxdbConfig.v3Config.queryTimeout'
|
||||
)
|
||||
? this.config.get('Butler-SOS.influxdbConfig.v3Config.queryTimeout')
|
||||
: 60000; // Default 60 seconds for gRPC query timeout
|
||||
|
||||
try {
|
||||
this.influx = new InfluxDBClient3({
|
||||
host,
|
||||
token,
|
||||
database,
|
||||
timeout: writeTimeout,
|
||||
queryTimeout,
|
||||
});
|
||||
|
||||
// Test connection by executing a simple query
|
||||
this.logger.info(`INFLUXDB3 INIT: Testing connection to InfluxDB v3...`);
|
||||
try {
|
||||
// Execute a simple query to test the connection
|
||||
const testQuery = `SELECT 1 as test LIMIT 1`;
|
||||
const queryResult = this.influx.query(testQuery, database);
|
||||
|
||||
// Try to get first result (this will throw if connection fails)
|
||||
const iterator = queryResult[Symbol.asyncIterator]();
|
||||
await iterator.next();
|
||||
|
||||
// Connection successful - log details
|
||||
const tokenPreview = token.substring(0, 4) + '***';
|
||||
this.logger.info(`INFLUXDB3 INIT: Connection successful!`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Host: ${hostName}`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Port: ${port}`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Database: ${database}`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Token: ${tokenPreview}`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Socket timeout: ${timeout}ms`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Query timeout: ${queryTimeout}ms`);
|
||||
} catch (testErr) {
|
||||
this.logger.warn(
|
||||
`INFLUXDB3 INIT: Could not test connection (this may be normal): ${this.getErrorMessage(testErr)}`
|
||||
);
|
||||
// Still log the configuration
|
||||
const tokenPreview = token.substring(0, 4) + '***';
|
||||
this.logger.info(`INFLUXDB3 INIT: Client created with:`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Host: ${hostName}`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Port: ${port}`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Database: ${database}`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Token: ${tokenPreview}`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Socket timeout: ${timeout}ms`);
|
||||
this.logger.info(`INFLUXDB3 INIT: Query timeout: ${queryTimeout}ms`);
|
||||
}
|
||||
} catch (err) {
|
||||
this.logger.error(
|
||||
`INFLUXDB3 INIT: Error creating InfluxDB 3 client: ${this.getErrorMessage(err)}`
|
||||
);
|
||||
this.logger.error(`INFLUXDB3 INIT: Exiting.`);
|
||||
}
|
||||
} else {
|
||||
this.logger.error(
|
||||
`CONFIG: Influxdb version ${this.config.get('Butler-SOS.influxdbConfig.version')} is not supported!`
|
||||
@@ -1090,8 +1207,8 @@ Configuration File:
|
||||
maxRetries: 2, // do not retry writes
|
||||
|
||||
// ... there are more write options that can be customized, see
|
||||
// https://influxdata.github.io/influxdb-client-js/influxdb-client.writeoptions.html and
|
||||
// https://influxdata.github.io/influxdb-client-js/influxdb-client.writeretryoptions.html
|
||||
// https://influxdata.github.io/influxdb-client-js/interfaces/_influxdata_influxdb-client.WriteOptions.html
|
||||
// https://influxdata.github.io/influxdb-client-js/interfaces/_influxdata_influxdb-client.WriteRetryOptions.html
|
||||
};
|
||||
|
||||
try {
|
||||
@@ -1114,6 +1231,45 @@ Configuration File:
|
||||
}
|
||||
});
|
||||
}
|
||||
} else if (this.config.get('Butler-SOS.influxdbConfig.version') === 3) {
|
||||
// Get config
|
||||
const databaseName = this.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
const description = this.config.get('Butler-SOS.influxdbConfig.v3Config.description');
|
||||
const token = this.config.get('Butler-SOS.influxdbConfig.v3Config.token');
|
||||
const retentionDuration = this.config.get(
|
||||
'Butler-SOS.influxdbConfig.v3Config.retentionDuration'
|
||||
);
|
||||
|
||||
if (
|
||||
this.influx &&
|
||||
this.config.get('Butler-SOS.influxdbConfig.enable') === true &&
|
||||
databaseName?.length > 0 &&
|
||||
token?.length > 0 &&
|
||||
retentionDuration?.length > 0
|
||||
) {
|
||||
enableInfluxdb = true;
|
||||
}
|
||||
|
||||
if (enableInfluxdb) {
|
||||
// For InfluxDB v3, we use client.write() directly (no getWriteApi method in v3)
|
||||
this.logger.info(`INFLUXDB3: Using database "${databaseName}"`);
|
||||
|
||||
// For v3, we store the client itself and call write() directly
|
||||
// The influxWriteApi array will contain objects with client and database info
|
||||
this.serverList.forEach((server) => {
|
||||
// Get per-server tags
|
||||
const tags = getServerTags(this.logger, server);
|
||||
|
||||
// Store client info and tags for this server
|
||||
// v3 uses client.write() directly, not getWriteApi()
|
||||
this.influxWriteApi.push({
|
||||
serverName: server.serverName,
|
||||
writeAPI: this.influx, // Store the client itself
|
||||
database: databaseName,
|
||||
defaultTags: tags, // Store tags for later use
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -21,6 +21,9 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
certPath: 'cert/path',
|
||||
keyPath: 'key/path',
|
||||
},
|
||||
@@ -129,9 +132,12 @@ describe('appnamesextract', () => {
|
||||
expect(qrsInteract).toHaveBeenCalledWith(expect.any(Object));
|
||||
expect(mockGet).toHaveBeenCalledWith('app');
|
||||
|
||||
// Verify error logging
|
||||
// Verify error logging - logError creates TWO log calls: message + stack trace
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
'APP NAMES: Error getting app names: Error: QRS API Error'
|
||||
'APP NAMES: Error getting app names: QRS API Error'
|
||||
);
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Stack trace: Error: QRS API Error')
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,520 +0,0 @@
|
||||
import { jest, describe, test, beforeEach, afterEach } from '@jest/globals';
|
||||
|
||||
// Mock Fastify and other dependencies
|
||||
jest.unstable_mockModule('fastify', () => {
|
||||
const mockFastifyInstance = {
|
||||
register: jest.fn().mockResolvedValue(undefined),
|
||||
setErrorHandler: jest.fn(),
|
||||
setNotFoundHandler: jest.fn(),
|
||||
get: jest.fn(),
|
||||
listen: jest.fn((options, callback) => {
|
||||
callback(null, 'http://127.0.0.1:8090');
|
||||
return mockFastifyInstance;
|
||||
}),
|
||||
ready: jest.fn((callback) => callback(null)),
|
||||
log: {
|
||||
level: 'silent',
|
||||
error: jest.fn(),
|
||||
},
|
||||
};
|
||||
|
||||
return {
|
||||
default: jest.fn().mockReturnValue(mockFastifyInstance),
|
||||
__mockInstance: mockFastifyInstance,
|
||||
};
|
||||
});
|
||||
|
||||
// Mock @fastify/rate-limit
|
||||
jest.unstable_mockModule('@fastify/rate-limit', () => ({
|
||||
default: jest.fn().mockResolvedValue(undefined),
|
||||
}));
|
||||
|
||||
// Mock @fastify/static
|
||||
jest.unstable_mockModule('@fastify/static', () => ({
|
||||
default: jest.fn().mockResolvedValue(undefined),
|
||||
}));
|
||||
|
||||
// Mock fs
|
||||
jest.unstable_mockModule('fs', () => ({
|
||||
readdirSync: jest.fn().mockReturnValue(['file1', 'file2']),
|
||||
readFileSync: jest.fn().mockReturnValue('{{butlerSosConfigJsonEncoded}}{{butlerConfigYaml}}'),
|
||||
}));
|
||||
|
||||
// Mock path
|
||||
jest.unstable_mockModule('path', () => ({
|
||||
resolve: jest.fn().mockReturnValue('/mock/path'),
|
||||
join: jest.fn().mockReturnValue('/mock/path/static'),
|
||||
}));
|
||||
|
||||
// Mock js-yaml
|
||||
jest.unstable_mockModule('js-yaml', () => ({
|
||||
dump: jest.fn().mockReturnValue('mockYaml'),
|
||||
}));
|
||||
|
||||
// Mock handlebars
|
||||
jest.unstable_mockModule('handlebars', () => ({
|
||||
default: {
|
||||
compile: jest.fn().mockReturnValue((data) => `compiled:${JSON.stringify(data)}`),
|
||||
},
|
||||
compile: jest.fn().mockReturnValue((data) => `compiled:${JSON.stringify(data)}`),
|
||||
}));
|
||||
|
||||
// Mock config-obfuscate
|
||||
jest.unstable_mockModule('../config-obfuscate.js', () => ({
|
||||
default: jest.fn((config) => {
|
||||
return { ...config, obfuscated: true };
|
||||
}),
|
||||
}));
|
||||
|
||||
// Mock file-prep
|
||||
jest.unstable_mockModule('../file-prep.js', () => ({
|
||||
prepareFile: jest.fn().mockResolvedValue({
|
||||
found: true,
|
||||
content:
|
||||
'file content {{visTaskHost}} {{visTaskPort}} {{butlerSosConfigJsonEncoded}} {{butlerConfigYaml}}',
|
||||
mimeType: 'text/html',
|
||||
}),
|
||||
compileTemplate: jest.fn().mockReturnValue('compiled template'),
|
||||
}));
|
||||
|
||||
// Mock sea-wrapper (needed by file-prep.js)
|
||||
jest.unstable_mockModule('../sea-wrapper.js', () => ({
|
||||
default: {
|
||||
getAsset: jest.fn(),
|
||||
isSea: jest.fn().mockReturnValue(false),
|
||||
},
|
||||
}));
|
||||
|
||||
// Mock globals
|
||||
jest.unstable_mockModule('../../globals.js', () => ({
|
||||
default: {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
getLoggingLevel: jest.fn().mockReturnValue('info'),
|
||||
appBasePath: '/mock/app/base/path',
|
||||
isSea: false,
|
||||
config: {
|
||||
get: jest.fn((path) => {
|
||||
if (path === 'Butler-SOS.configVisualisation.obfuscate') return true;
|
||||
if (path === 'Butler-SOS.configVisualisation.host') return '127.0.0.1';
|
||||
if (path === 'Butler-SOS.configVisualisation.port') return 8090;
|
||||
return null;
|
||||
}),
|
||||
},
|
||||
},
|
||||
}));
|
||||
|
||||
// Mock modules for '../plugins/sensible.js' and '../plugins/support.js'
|
||||
// jest.unstable_mockModule('../plugins/sensible.js', () => ({
|
||||
// default: jest.fn(),
|
||||
// }));
|
||||
|
||||
// jest.unstable_mockModule('../plugins/support.js', () => ({
|
||||
// default: jest.fn(),
|
||||
// }));
|
||||
|
||||
describe.skip('config-visualise', () => {
|
||||
let mockFastify;
|
||||
let configObfuscate;
|
||||
let globals;
|
||||
let setupConfigVisServer;
|
||||
let fs;
|
||||
let path;
|
||||
let yaml;
|
||||
let handlebars;
|
||||
let fastifyModule;
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clear all mocks before each test
|
||||
jest.clearAllMocks();
|
||||
|
||||
// Import mocked modules
|
||||
fastifyModule = await import('fastify');
|
||||
mockFastify = fastifyModule.default;
|
||||
|
||||
configObfuscate = (await import('../config-obfuscate.js')).default;
|
||||
globals = (await import('../../globals.js')).default;
|
||||
fs = await import('fs');
|
||||
path = await import('path');
|
||||
yaml = await import('js-yaml');
|
||||
handlebars = await import('handlebars');
|
||||
|
||||
// Import the module under test
|
||||
setupConfigVisServer = (await import('../config-visualise.js')).setupConfigVisServer;
|
||||
});
|
||||
|
||||
test('should set up server with correct configuration', async () => {
|
||||
// Call the function being tested
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
// Verify Fastify was initialized
|
||||
expect(mockFastify).toHaveBeenCalled();
|
||||
|
||||
// Verify rate limit plugin was registered
|
||||
expect(fastifyModule.__mockInstance.register).toHaveBeenCalledWith(
|
||||
expect.anything(),
|
||||
expect.objectContaining({
|
||||
max: 300,
|
||||
timeWindow: '1 minute',
|
||||
})
|
||||
);
|
||||
|
||||
// Verify static file server was set up
|
||||
expect(fastifyModule.__mockInstance.register).toHaveBeenCalledWith(
|
||||
expect.anything(),
|
||||
expect.objectContaining({
|
||||
root: expect.any(String),
|
||||
redirect: true,
|
||||
})
|
||||
);
|
||||
|
||||
// Verify route handler was set up
|
||||
expect(fastifyModule.__mockInstance.get).toHaveBeenCalledWith('/', expect.any(Function));
|
||||
|
||||
// Verify server was started
|
||||
expect(fastifyModule.__mockInstance.listen).toHaveBeenCalledWith(
|
||||
{
|
||||
host: '127.0.0.1',
|
||||
port: 8090,
|
||||
},
|
||||
expect.any(Function)
|
||||
);
|
||||
|
||||
// Verify success was logged
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Config visualisation server listening on')
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle errors during server setup', async () => {
|
||||
// Make Fastify.listen throw an error
|
||||
fastifyModule.__mockInstance.listen.mockImplementationOnce((options, callback) => {
|
||||
callback(new Error('Failed to start server'), null);
|
||||
return fastifyModule.__mockInstance;
|
||||
});
|
||||
|
||||
// Mock process.exit to prevent test from exiting
|
||||
const originalExit = process.exit;
|
||||
process.exit = jest.fn();
|
||||
|
||||
try {
|
||||
// Call the function being tested
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
// Verify error was logged
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Could not set up config visualisation server')
|
||||
);
|
||||
expect(process.exit).toHaveBeenCalledWith(1);
|
||||
} finally {
|
||||
// Restore process.exit
|
||||
process.exit = originalExit;
|
||||
}
|
||||
});
|
||||
|
||||
test('should set log level to info when debug/silly logging is enabled', async () => {
|
||||
globals.getLoggingLevel.mockReturnValueOnce('debug');
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
expect(fastifyModule.__mockInstance.log.level).toBe('info');
|
||||
});
|
||||
|
||||
test('should set log level to silent for other log levels', async () => {
|
||||
globals.getLoggingLevel.mockReturnValueOnce('error');
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
expect(fastifyModule.__mockInstance.log.level).toBe('silent');
|
||||
});
|
||||
|
||||
test('should set up error handler for rate limiting', async () => {
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
expect(fastifyModule.__mockInstance.setErrorHandler).toHaveBeenCalledWith(
|
||||
expect.any(Function)
|
||||
);
|
||||
|
||||
// Test the error handler
|
||||
const errorHandler = fastifyModule.__mockInstance.setErrorHandler.mock.calls[0][0];
|
||||
const mockRequest = { ip: '127.0.0.1', method: 'GET', url: '/test' };
|
||||
const mockReply = { send: jest.fn() };
|
||||
const mockError = { statusCode: 429 };
|
||||
|
||||
errorHandler(mockError, mockRequest, mockReply);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Rate limit exceeded for source IP address 127.0.0.1')
|
||||
);
|
||||
expect(mockReply.send).toHaveBeenCalledWith(mockError);
|
||||
});
|
||||
|
||||
test('should handle root route with obfuscation enabled', async () => {
|
||||
const filePrep = await import('../file-prep.js');
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
// Get the root route handler
|
||||
const rootRouteCall = fastifyModule.__mockInstance.get.mock.calls.find(
|
||||
(call) => call[0] === '/'
|
||||
);
|
||||
expect(rootRouteCall).toBeDefined();
|
||||
|
||||
const routeHandler = rootRouteCall[1];
|
||||
const mockRequest = {};
|
||||
const mockReply = {
|
||||
code: jest.fn().mockReturnThis(),
|
||||
header: jest.fn().mockReturnThis(),
|
||||
send: jest.fn(),
|
||||
};
|
||||
|
||||
await routeHandler(mockRequest, mockReply);
|
||||
|
||||
expect(filePrep.prepareFile).toHaveBeenCalled();
|
||||
expect(filePrep.compileTemplate).toHaveBeenCalled();
|
||||
expect(configObfuscate).toHaveBeenCalled();
|
||||
expect(yaml.dump).toHaveBeenCalled();
|
||||
expect(mockReply.code).toHaveBeenCalledWith(200);
|
||||
expect(mockReply.header).toHaveBeenCalledWith('Content-Type', 'text/html; charset=utf-8');
|
||||
expect(mockReply.send).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle root route with obfuscation disabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path === 'Butler-SOS.configVisualisation.obfuscate') return false;
|
||||
if (path === 'Butler-SOS.configVisualisation.host') return '127.0.0.1';
|
||||
if (path === 'Butler-SOS.configVisualisation.port') return 8090;
|
||||
return null;
|
||||
});
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
// Get the root route handler
|
||||
const rootRouteCall = fastifyModule.__mockInstance.get.mock.calls.find(
|
||||
(call) => call[0] === '/'
|
||||
);
|
||||
const routeHandler = rootRouteCall[1];
|
||||
const mockRequest = {};
|
||||
const mockReply = {
|
||||
code: jest.fn().mockReturnThis(),
|
||||
header: jest.fn().mockReturnThis(),
|
||||
send: jest.fn(),
|
||||
};
|
||||
|
||||
await routeHandler(mockRequest, mockReply);
|
||||
|
||||
expect(configObfuscate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle root route error when template not found', async () => {
|
||||
const filePrep = await import('../file-prep.js');
|
||||
filePrep.prepareFile.mockResolvedValueOnce({
|
||||
found: false,
|
||||
content: null,
|
||||
mimeType: null,
|
||||
});
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
const rootRouteCall = fastifyModule.__mockInstance.get.mock.calls.find(
|
||||
(call) => call[0] === '/'
|
||||
);
|
||||
const routeHandler = rootRouteCall[1];
|
||||
const mockRequest = {};
|
||||
const mockReply = {
|
||||
code: jest.fn().mockReturnThis(),
|
||||
send: jest.fn(),
|
||||
};
|
||||
|
||||
await routeHandler(mockRequest, mockReply);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Could not find index.html template')
|
||||
);
|
||||
expect(mockReply.code).toHaveBeenCalledWith(500);
|
||||
expect(mockReply.send).toHaveBeenCalledWith({
|
||||
error: 'Internal server error: Template not found',
|
||||
});
|
||||
});
|
||||
|
||||
test('should handle root route error during processing', async () => {
|
||||
yaml.dump.mockImplementationOnce(() => {
|
||||
throw new Error('YAML dump failed');
|
||||
});
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
const rootRouteCall = fastifyModule.__mockInstance.get.mock.calls.find(
|
||||
(call) => call[0] === '/'
|
||||
);
|
||||
const routeHandler = rootRouteCall[1];
|
||||
const mockRequest = {};
|
||||
const mockReply = {
|
||||
code: jest.fn().mockReturnThis(),
|
||||
send: jest.fn(),
|
||||
};
|
||||
|
||||
await routeHandler(mockRequest, mockReply);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error serving home page')
|
||||
);
|
||||
expect(mockReply.code).toHaveBeenCalledWith(500);
|
||||
expect(mockReply.send).toHaveBeenCalledWith({ error: 'Internal server error' });
|
||||
});
|
||||
|
||||
test('should handle SEA mode setup', async () => {
|
||||
globals.isSea = true;
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Running in SEA mode, setting up custom static file handlers')
|
||||
);
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Custom static file handlers set up for SEA mode')
|
||||
);
|
||||
|
||||
// Verify SEA-specific routes were set up
|
||||
const getRoutes = fastifyModule.__mockInstance.get.mock.calls;
|
||||
const filenameRoute = getRoutes.find((call) => call[0] === '/:filename');
|
||||
const logoRoute = getRoutes.find((call) => call[0] === '/butler-sos.png');
|
||||
|
||||
expect(filenameRoute).toBeDefined();
|
||||
expect(logoRoute).toBeDefined();
|
||||
});
|
||||
|
||||
test('should handle SEA mode filename route', async () => {
|
||||
globals.isSea = true;
|
||||
const filePrep = await import('../file-prep.js');
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
const getRoutes = fastifyModule.__mockInstance.get.mock.calls;
|
||||
const filenameRoute = getRoutes.find((call) => call[0] === '/:filename');
|
||||
const routeHandler = filenameRoute[1];
|
||||
|
||||
expect(filenameRoute).toBeDefined();
|
||||
expect(typeof routeHandler).toBe('function');
|
||||
});
|
||||
|
||||
test('should handle SEA mode logo route', async () => {
|
||||
globals.isSea = true;
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
const getRoutes = fastifyModule.__mockInstance.get.mock.calls;
|
||||
const logoRoute = getRoutes.find((call) => call[0] === '/butler-sos.png');
|
||||
|
||||
expect(logoRoute).toBeDefined();
|
||||
expect(typeof logoRoute[1]).toBe('function');
|
||||
});
|
||||
|
||||
test('should handle Node.js mode static file setup', async () => {
|
||||
globals.isSea = false;
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Serving static files from')
|
||||
);
|
||||
|
||||
// Verify FastifyStatic was registered
|
||||
const registerCalls = fastifyModule.__mockInstance.register.mock.calls;
|
||||
const staticRegister = registerCalls.find(
|
||||
(call) => call[1] && call[1].root && call[1].redirect === true
|
||||
);
|
||||
expect(staticRegister).toBeDefined();
|
||||
});
|
||||
|
||||
test('should handle fs.readdirSync error in Node.js mode', async () => {
|
||||
globals.isSea = false;
|
||||
fs.readdirSync.mockImplementationOnce(() => {
|
||||
throw new Error('Permission denied');
|
||||
});
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error reading static directory')
|
||||
);
|
||||
});
|
||||
|
||||
test('should set up not found handler', async () => {
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
expect(fastifyModule.__mockInstance.setNotFoundHandler).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle general setup errors', async () => {
|
||||
fastifyModule.__mockInstance.register.mockRejectedValueOnce(
|
||||
new Error('Plugin registration failed')
|
||||
);
|
||||
|
||||
await expect(setupConfigVisServer(globals.logger, globals.config)).rejects.toThrow(
|
||||
'Plugin registration failed'
|
||||
);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error setting up config visualisation server')
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle SEA mode filename route execution with successful file', async () => {
|
||||
globals.isSea = true;
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
// Verify that the /:filename route was set up in SEA mode
|
||||
const getRoutes = fastifyModule.__mockInstance.get.mock.calls;
|
||||
const filenameRoute = getRoutes.find((call) => call[0] === '/:filename');
|
||||
|
||||
expect(filenameRoute).toBeDefined();
|
||||
expect(typeof filenameRoute[1]).toBe('function');
|
||||
|
||||
// Verify that the SEA mode info was logged
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Running in SEA mode, setting up custom static file handlers')
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle serve404Page function execution', async () => {
|
||||
globals.isSea = false;
|
||||
const filePrep = await import('../file-prep.js');
|
||||
filePrep.prepareFile.mockResolvedValueOnce({
|
||||
found: true,
|
||||
content: 'Not found page content {{visTaskHost}} {{visTaskPort}}',
|
||||
mimeType: 'text/html',
|
||||
});
|
||||
|
||||
await setupConfigVisServer(globals.logger, globals.config);
|
||||
|
||||
// Get the not found handler
|
||||
const notFoundHandler = fastifyModule.__mockInstance.setNotFoundHandler.mock.calls[0][0];
|
||||
|
||||
const mockRequest = {};
|
||||
const mockReply = {
|
||||
code: jest.fn().mockReturnThis(),
|
||||
header: jest.fn().mockReturnThis(),
|
||||
send: jest.fn(),
|
||||
};
|
||||
|
||||
await notFoundHandler(mockRequest, mockReply);
|
||||
|
||||
expect(filePrep.prepareFile).toHaveBeenCalled();
|
||||
expect(filePrep.compileTemplate).toHaveBeenCalled();
|
||||
expect(mockReply.code).toHaveBeenCalledWith(404);
|
||||
expect(mockReply.header).toHaveBeenCalledWith('Content-Type', 'text/html; charset=utf-8');
|
||||
expect(mockReply.send).toHaveBeenCalledWith('compiled template');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
// Reset globals
|
||||
globals.isSea = false;
|
||||
});
|
||||
});
|
||||
@@ -41,9 +41,8 @@ const handlebars = (await import('handlebars')).default;
|
||||
const globals = (await import('../../globals.js')).default;
|
||||
|
||||
// Import the module under test
|
||||
const { prepareFile, compileTemplate, getFileContent, getMimeType } = await import(
|
||||
'../file-prep.js'
|
||||
);
|
||||
const { prepareFile, compileTemplate, getFileContent, getMimeType } =
|
||||
await import('../file-prep.js');
|
||||
|
||||
describe('file-prep', () => {
|
||||
beforeEach(() => {
|
||||
|
||||
@@ -23,6 +23,9 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
@@ -32,10 +35,10 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
}));
|
||||
const globals = (await import('../../globals.js')).default;
|
||||
|
||||
jest.unstable_mockModule('../post-to-influxdb.js', () => ({
|
||||
jest.unstable_mockModule('../influxdb/index.js', () => ({
|
||||
postHealthMetricsToInfluxdb: jest.fn(),
|
||||
}));
|
||||
const { postHealthMetricsToInfluxdb } = await import('../post-to-influxdb.js');
|
||||
const { postHealthMetricsToInfluxdb } = await import('../influxdb/index.js');
|
||||
|
||||
jest.unstable_mockModule('../post-to-new-relic.js', () => ({
|
||||
postHealthMetricsToNewRelic: jest.fn(),
|
||||
|
||||
@@ -1,32 +0,0 @@
|
||||
import { jest, describe, test, expect } from '@jest/globals';
|
||||
import { fileURLToPath } from 'url';
|
||||
import path from 'path';
|
||||
|
||||
describe('import-meta-url', () => {
|
||||
test.skip('should export a URL object', async () => {
|
||||
// Import the module under test
|
||||
const { import_meta_url } = await import('../import-meta-url.js');
|
||||
|
||||
// Expectations
|
||||
expect(import_meta_url).toBeDefined();
|
||||
expect(typeof import_meta_url).toBe('object');
|
||||
expect(import_meta_url instanceof URL).toBe(true);
|
||||
});
|
||||
|
||||
test.skip('should point to the correct file path', async () => {
|
||||
// Import the module under test
|
||||
const { import_meta_url } = await import('../import-meta-url.js');
|
||||
|
||||
// Convert the URL to a file path
|
||||
const filePath = fileURLToPath(import_meta_url);
|
||||
|
||||
// Get the expected file path
|
||||
const expectedFilePath = path.resolve(process.cwd(), 'src/lib/import-meta-url.js');
|
||||
|
||||
// Verify the path ends with 'import-meta-url.js'
|
||||
expect(filePath.endsWith('import-meta-url.js')).toBe(true);
|
||||
|
||||
// Verify it's in the lib directory
|
||||
expect(filePath.includes(path.sep + 'lib' + path.sep)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -1,776 +0,0 @@
|
||||
import { jest, describe, test, expect, beforeEach, afterEach } from '@jest/globals';
|
||||
|
||||
// Mock the InfluxDB client
|
||||
jest.unstable_mockModule('@influxdata/influxdb-client', () => ({
|
||||
Point: jest.fn().mockImplementation(() => ({
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
floatField: jest.fn().mockReturnThis(),
|
||||
intField: jest.fn().mockReturnThis(),
|
||||
stringField: jest.fn().mockReturnThis(),
|
||||
uintField: jest.fn().mockReturnThis(),
|
||||
booleanField: jest.fn().mockReturnThis(), // <-- add this line
|
||||
timestamp: jest.fn().mockReturnThis(),
|
||||
})),
|
||||
}));
|
||||
|
||||
// Mock globals
|
||||
jest.unstable_mockModule('../../globals.js', () => ({
|
||||
default: {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influxDB: {
|
||||
writeApi: {
|
||||
writePoint: jest.fn(),
|
||||
flush: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
},
|
||||
appNames: [],
|
||||
getErrorMessage: jest.fn().mockImplementation((err) => err.toString()),
|
||||
},
|
||||
}));
|
||||
|
||||
describe('post-to-influxdb', () => {
|
||||
let influxdb;
|
||||
let globals;
|
||||
let Point;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
// Get mocked modules
|
||||
const influxdbClient = await import('@influxdata/influxdb-client');
|
||||
Point = influxdbClient.Point;
|
||||
globals = (await import('../../globals.js')).default;
|
||||
|
||||
// Mock globals.influx for InfluxDB v1 tests
|
||||
globals.influx = { writePoints: jest.fn() };
|
||||
|
||||
// Import the module under test
|
||||
influxdb = await import('../post-to-influxdb.js');
|
||||
});
|
||||
|
||||
describe('storeEventCountInfluxDB', () => {
|
||||
test('should not store events if no log events exist', async () => {
|
||||
// Setup
|
||||
globals.udpEvents = {
|
||||
getLogEvents: jest.fn().mockResolvedValue([]),
|
||||
getUserEvents: jest.fn().mockResolvedValue([]),
|
||||
};
|
||||
|
||||
// Execute
|
||||
await influxdb.storeEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining('EVENT COUNT INFLUXDB: No events to store in InfluxDB')
|
||||
);
|
||||
expect(globals.influxDB.writeApi.writePoint).not.toHaveBeenCalled();
|
||||
expect(globals.influxDB.writeApi.flush).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should store log events to InfluxDB (InfluxDB v1)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName') {
|
||||
return 'events_log';
|
||||
}
|
||||
return undefined;
|
||||
});
|
||||
const mockLogEvents = [
|
||||
{
|
||||
source: 'test-source',
|
||||
host: 'test-host',
|
||||
subsystem: 'test-subsystem',
|
||||
counter: 5,
|
||||
},
|
||||
];
|
||||
globals.udpEvents = {
|
||||
getLogEvents: jest.fn().mockResolvedValue(mockLogEvents),
|
||||
getUserEvents: jest.fn().mockResolvedValue([]),
|
||||
};
|
||||
|
||||
// Execute
|
||||
await influxdb.storeEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.influx.writePoints).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'EVENT COUNT INFLUXDB: Sent Butler SOS event count data to InfluxDB'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should store log events to InfluxDB (InfluxDB v2)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName') {
|
||||
return 'events_log';
|
||||
}
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.org') return 'test-org';
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.bucket') return 'test-bucket';
|
||||
return undefined;
|
||||
});
|
||||
const mockLogEvents = [
|
||||
{
|
||||
source: 'test-source',
|
||||
host: 'test-host',
|
||||
subsystem: 'test-subsystem',
|
||||
counter: 5,
|
||||
},
|
||||
];
|
||||
globals.udpEvents = {
|
||||
getLogEvents: jest.fn().mockResolvedValue(mockLogEvents),
|
||||
getUserEvents: jest.fn().mockResolvedValue([]),
|
||||
};
|
||||
// Mock v2 writeApi
|
||||
globals.influx.getWriteApi = jest.fn().mockReturnValue({
|
||||
writePoints: jest.fn(),
|
||||
});
|
||||
|
||||
// Execute
|
||||
await influxdb.storeEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.influx.getWriteApi).toHaveBeenCalled();
|
||||
// The writeApi mock's writePoints should be called
|
||||
const writeApi = globals.influx.getWriteApi.mock.results[0].value;
|
||||
expect(writeApi.writePoints).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'EVENT COUNT INFLUXDB: Sent Butler SOS event count data to InfluxDB'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should store user events to InfluxDB (InfluxDB v1)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName') {
|
||||
return 'events_user';
|
||||
}
|
||||
return undefined;
|
||||
});
|
||||
const mockUserEvents = [
|
||||
{
|
||||
source: 'test-source',
|
||||
host: 'test-host',
|
||||
subsystem: 'test-subsystem',
|
||||
counter: 3,
|
||||
},
|
||||
];
|
||||
globals.udpEvents = {
|
||||
getLogEvents: jest.fn().mockResolvedValue([]),
|
||||
getUserEvents: jest.fn().mockResolvedValue(mockUserEvents),
|
||||
};
|
||||
|
||||
// Execute
|
||||
await influxdb.storeEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.influx.writePoints).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'EVENT COUNT INFLUXDB: Sent Butler SOS event count data to InfluxDB'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should store user events to InfluxDB (InfluxDB v2)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName') {
|
||||
return 'events_user';
|
||||
}
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.org') return 'test-org';
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.bucket') return 'test-bucket';
|
||||
return undefined;
|
||||
});
|
||||
const mockUserEvents = [
|
||||
{
|
||||
source: 'test-source',
|
||||
host: 'test-host',
|
||||
subsystem: 'test-subsystem',
|
||||
counter: 3,
|
||||
},
|
||||
];
|
||||
globals.udpEvents = {
|
||||
getLogEvents: jest.fn().mockResolvedValue([]),
|
||||
getUserEvents: jest.fn().mockResolvedValue(mockUserEvents),
|
||||
};
|
||||
// Mock v2 writeApi
|
||||
globals.influx.getWriteApi = jest.fn().mockReturnValue({
|
||||
writePoints: jest.fn(),
|
||||
});
|
||||
|
||||
// Execute
|
||||
await influxdb.storeEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.influx.getWriteApi).toHaveBeenCalled();
|
||||
// The writeApi mock's writePoints should be called
|
||||
const writeApi = globals.influx.getWriteApi.mock.results[0].value;
|
||||
expect(writeApi.writePoints).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'EVENT COUNT INFLUXDB: Sent Butler SOS event count data to InfluxDB'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle errors gracefully (InfluxDB v1)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
return undefined;
|
||||
});
|
||||
// Instead of rejecting, resolve with a value and mock writePoints to throw
|
||||
globals.udpEvents = {
|
||||
getLogEvents: jest.fn().mockResolvedValue([{}]),
|
||||
getUserEvents: jest.fn().mockResolvedValue([]),
|
||||
};
|
||||
globals.influx.writePoints.mockImplementation(() => {
|
||||
throw new Error('Test error');
|
||||
});
|
||||
|
||||
// Execute
|
||||
await influxdb.storeEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'EVENT COUNT INFLUXDB: Error saving data to InfluxDB v1! Error: Test error'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle errors gracefully (InfluxDB v2)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName') {
|
||||
return 'events_log';
|
||||
}
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.org') return 'test-org';
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.bucket') return 'test-bucket';
|
||||
return undefined;
|
||||
});
|
||||
// Provide at least one event so writePoints is called
|
||||
globals.udpEvents = {
|
||||
getLogEvents: jest.fn().mockResolvedValue([{}]),
|
||||
getUserEvents: jest.fn().mockResolvedValue([]),
|
||||
};
|
||||
// Mock v2 writeApi to throw error on writePoints
|
||||
globals.influx.getWriteApi = jest.fn().mockReturnValue({
|
||||
writePoints: jest.fn(() => {
|
||||
throw new Error('Test error');
|
||||
}),
|
||||
});
|
||||
|
||||
// Execute
|
||||
await influxdb.storeEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'EVENT COUNT INFLUXDB: Error saving health data to InfluxDB v2! Error: Test error'
|
||||
)
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeRejectedEventCountInfluxDB', () => {
|
||||
test('should not store events if no rejected events exist', async () => {
|
||||
// Setup
|
||||
globals.rejectedEvents = {
|
||||
getRejectedLogEvents: jest.fn().mockResolvedValue([]),
|
||||
};
|
||||
|
||||
// Execute
|
||||
await influxdb.storeRejectedEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'REJECTED EVENT COUNT INFLUXDB: No events to store in InfluxDB'
|
||||
)
|
||||
);
|
||||
expect(globals.influxDB.writeApi.writePoint).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should store rejected events to InfluxDB (InfluxDB v1)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
if (
|
||||
key === 'Butler-SOS.qlikSenseEvents.rejectedEventCount.influxdb.measurementName'
|
||||
)
|
||||
return 'events_rejected';
|
||||
return undefined;
|
||||
});
|
||||
const mockRejectedEvents = [
|
||||
{
|
||||
source: 'test-source',
|
||||
counter: 7,
|
||||
},
|
||||
];
|
||||
globals.rejectedEvents = {
|
||||
getRejectedLogEvents: jest.fn().mockResolvedValue(mockRejectedEvents),
|
||||
};
|
||||
// Mock v1 writePoints
|
||||
globals.influx = { writePoints: jest.fn() };
|
||||
|
||||
// Execute
|
||||
await influxdb.storeRejectedEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
// Do not check Point for v1
|
||||
expect(globals.influx.writePoints).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'REJECT LOG EVENT INFLUXDB: Sent Butler SOS rejected event count data to InfluxDB'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should store rejected events to InfluxDB (InfluxDB v2)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.org') return 'test-org';
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.bucket') return 'test-bucket';
|
||||
if (
|
||||
key === 'Butler-SOS.qlikSenseEvents.rejectedEventCount.influxdb.measurementName'
|
||||
)
|
||||
return 'events_rejected';
|
||||
return undefined;
|
||||
});
|
||||
const mockRejectedEvents = [
|
||||
{
|
||||
source: 'test-source',
|
||||
counter: 7,
|
||||
},
|
||||
];
|
||||
globals.rejectedEvents = {
|
||||
getRejectedLogEvents: jest.fn().mockResolvedValue(mockRejectedEvents),
|
||||
};
|
||||
// Mock v2 getWriteApi
|
||||
const writeApiMock = { writePoints: jest.fn() };
|
||||
globals.influx.getWriteApi = jest.fn().mockReturnValue(writeApiMock);
|
||||
|
||||
// Execute
|
||||
await influxdb.storeRejectedEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(Point).toHaveBeenCalledWith('events_rejected');
|
||||
expect(globals.influx.getWriteApi).toHaveBeenCalled();
|
||||
expect(writeApiMock.writePoints).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'REJECT LOG EVENT INFLUXDB: Sent Butler SOS rejected event count data to InfluxDB'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle errors gracefully (InfluxDB v1)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
return undefined;
|
||||
});
|
||||
const mockRejectedEvents = [
|
||||
{
|
||||
source: 'test-source',
|
||||
counter: 7,
|
||||
},
|
||||
];
|
||||
globals.rejectedEvents = {
|
||||
getRejectedLogEvents: jest.fn().mockResolvedValue(mockRejectedEvents),
|
||||
};
|
||||
// Mock v1 writePoints to throw
|
||||
globals.influx = {
|
||||
writePoints: jest.fn(() => {
|
||||
throw new Error('Test error');
|
||||
}),
|
||||
};
|
||||
|
||||
// Execute
|
||||
await influxdb.storeRejectedEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'REJECT LOG EVENT INFLUXDB: Error saving data to InfluxDB v1! Error: Test error'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle errors gracefully (InfluxDB v2)', async () => {
|
||||
// Setup
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.org') return 'test-org';
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.bucket') return 'test-bucket';
|
||||
return undefined;
|
||||
});
|
||||
const mockRejectedEvents = [
|
||||
{
|
||||
source: 'test-source',
|
||||
counter: 7,
|
||||
},
|
||||
];
|
||||
globals.rejectedEvents = {
|
||||
getRejectedLogEvents: jest.fn().mockResolvedValue(mockRejectedEvents),
|
||||
};
|
||||
// Mock v2 getWriteApi and writePoints to throw
|
||||
const writeApiMock = {
|
||||
writePoints: jest.fn(() => {
|
||||
throw new Error('Test error');
|
||||
}),
|
||||
};
|
||||
globals.influx.getWriteApi = jest.fn().mockReturnValue(writeApiMock);
|
||||
|
||||
// Execute
|
||||
await influxdb.storeRejectedEventCountInfluxDB();
|
||||
|
||||
// Verify
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'REJECTED LOG EVENT INFLUXDB: Error saving data to InfluxDB v2! Error: Test error'
|
||||
)
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('globals.config.get("Butler-SOS.influxdbConfig.version")', () => {
|
||||
let influxdb;
|
||||
let globals;
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
influxdb = await import('../post-to-influxdb.js');
|
||||
globals = (await import('../../globals.js')).default;
|
||||
globals.influx = { writePoints: jest.fn() };
|
||||
globals.influxWriteApi = [
|
||||
{ serverName: 'test-server', writeAPI: { writePoints: jest.fn() } },
|
||||
];
|
||||
});
|
||||
|
||||
test('should use InfluxDB v1 path when version is 1', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
return undefined;
|
||||
});
|
||||
const serverName = 'test-server';
|
||||
const host = 'test-host';
|
||||
const serverTags = { server_name: serverName };
|
||||
const healthBody = {
|
||||
started: '20220801T121212.000Z',
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [] },
|
||||
cache: { added: 0, hits: 0, lookups: 0, replaced: 0, bytes_added: 0 },
|
||||
cpu: { total: 0 },
|
||||
mem: { committed: 0, allocated: 0, free: 0 },
|
||||
session: { active: 0, total: 0 },
|
||||
users: { active: 0, total: 0 },
|
||||
};
|
||||
await influxdb.postHealthMetricsToInfluxdb(serverName, host, healthBody, serverTags);
|
||||
expect(globals.config.get).toHaveBeenCalledWith('Butler-SOS.influxdbConfig.version');
|
||||
expect(globals.influx.writePoints).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should use InfluxDB v2 path when version is 2', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.activeDocs') return false;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.loadedDocs') return false;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.inMemoryDocs') return false;
|
||||
if (key === 'Butler-SOS.appNames.enableAppNameExtract') return false;
|
||||
return undefined;
|
||||
});
|
||||
const serverName = 'test-server';
|
||||
const host = 'test-host';
|
||||
const serverTags = { server_name: serverName };
|
||||
const healthBody = {
|
||||
started: '20220801T121212.000Z',
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [] },
|
||||
cache: { added: 0, hits: 0, lookups: 0, replaced: 0, bytes_added: 0 },
|
||||
cpu: { total: 0 },
|
||||
mem: { committed: 0, allocated: 0, free: 0 },
|
||||
session: { active: 0, total: 0 },
|
||||
users: { active: 0, total: 0 },
|
||||
};
|
||||
await influxdb.postHealthMetricsToInfluxdb(serverName, host, healthBody, serverTags);
|
||||
expect(globals.config.get).toHaveBeenCalledWith('Butler-SOS.influxdbConfig.version');
|
||||
expect(globals.influxWriteApi[0].writeAPI.writePoints).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('getFormattedTime', () => {
|
||||
test('should return valid formatted time for valid Date string', () => {
|
||||
const validDate = '20230615T143022';
|
||||
const result = influxdb.getFormattedTime(validDate);
|
||||
expect(result).toBeDefined();
|
||||
expect(typeof result).toBe('string');
|
||||
expect(result).toMatch(/^\d+ days, \d{1,2}h \d{2}m \d{2}s$/);
|
||||
});
|
||||
|
||||
test('should return empty string for invalid Date string', () => {
|
||||
const invalidDate = 'invalid-date';
|
||||
const result = influxdb.getFormattedTime(invalidDate);
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
test('should return empty string for undefined input', () => {
|
||||
const result = influxdb.getFormattedTime(undefined);
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
test('should return empty string for null input', () => {
|
||||
const result = influxdb.getFormattedTime(null);
|
||||
expect(result).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('postHealthMetricsToInfluxdb', () => {
|
||||
test('should post health metrics to InfluxDB v1', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.activeDocs') return false;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.loadedDocs') return false;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.inMemoryDocs') return false;
|
||||
if (key === 'Butler-SOS.appNames.enableAppNameExtract') return false;
|
||||
return undefined;
|
||||
});
|
||||
const serverName = 'test-server';
|
||||
const host = 'test-host';
|
||||
const serverTags = { server_name: serverName };
|
||||
const healthBody = {
|
||||
started: '20220801T121212.000Z',
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [] },
|
||||
cache: { added: 0, hits: 0, lookups: 0, replaced: 0, bytes_added: 0 },
|
||||
cpu: { total: 0 },
|
||||
mem: { committed: 0, allocated: 0, free: 0 },
|
||||
session: { active: 0, total: 0 },
|
||||
users: { active: 0, total: 0 },
|
||||
};
|
||||
|
||||
await influxdb.postHealthMetricsToInfluxdb(serverName, host, healthBody, serverTags);
|
||||
|
||||
expect(globals.influx.writePoints).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should post health metrics to InfluxDB v2', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.activeDocs') return false;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.loadedDocs') return false;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.inMemoryDocs') return false;
|
||||
if (key === 'Butler-SOS.appNames.enableAppNameExtract') return false;
|
||||
return undefined;
|
||||
});
|
||||
globals.influxWriteApi = [
|
||||
{ serverName: 'test-server', writeAPI: { writePoints: jest.fn() } },
|
||||
];
|
||||
const serverName = 'test-server';
|
||||
const host = 'test-host';
|
||||
const serverTags = { server_name: serverName };
|
||||
const healthBody = {
|
||||
started: '20220801T121212.000Z',
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [] },
|
||||
cache: { added: 0, hits: 0, lookups: 0, replaced: 0, bytes_added: 0 },
|
||||
cpu: { total: 0 },
|
||||
mem: { committed: 0, allocated: 0, free: 0 },
|
||||
session: { active: 0, total: 0 },
|
||||
users: { active: 0, total: 0 },
|
||||
};
|
||||
|
||||
await influxdb.postHealthMetricsToInfluxdb(serverName, host, healthBody, serverTags);
|
||||
|
||||
expect(globals.influxWriteApi[0].writeAPI.writePoints).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('postProxySessionsToInfluxdb', () => {
|
||||
test('should post proxy sessions to InfluxDB v1', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
if (key === 'Butler-SOS.influxdbConfig.instanceTag') return 'DEV';
|
||||
if (key === 'Butler-SOS.userSessions.influxdb.measurementName')
|
||||
return 'user_sessions';
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has = jest.fn().mockReturnValue(true);
|
||||
const mockUserSessions = {
|
||||
serverName: 'test-server',
|
||||
host: 'test-host',
|
||||
virtualProxy: 'test-proxy',
|
||||
datapointInfluxdb: [
|
||||
{
|
||||
measurement: 'user_sessions',
|
||||
tags: { host: 'test-host' },
|
||||
fields: { count: 1 },
|
||||
},
|
||||
],
|
||||
sessionCount: 1,
|
||||
uniqueUserList: 'user1',
|
||||
};
|
||||
|
||||
await influxdb.postProxySessionsToInfluxdb(mockUserSessions);
|
||||
|
||||
expect(globals.influx.writePoints).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should post proxy sessions to InfluxDB v2', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.influxdbConfig.instanceTag') return 'DEV';
|
||||
if (key === 'Butler-SOS.userSessions.influxdb.measurementName')
|
||||
return 'user_sessions';
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has = jest.fn().mockReturnValue(true);
|
||||
|
||||
// Mock the writeAPI object that will be found via find()
|
||||
const mockWriteAPI = { writePoints: jest.fn() };
|
||||
globals.influxWriteApi = [{ serverName: 'test-server', writeAPI: mockWriteAPI }];
|
||||
|
||||
const mockUserSessions = {
|
||||
serverName: 'test-server',
|
||||
host: 'test-host',
|
||||
virtualProxy: 'test-proxy',
|
||||
datapointInfluxdb: [
|
||||
{
|
||||
measurement: 'user_sessions',
|
||||
tags: { host: 'test-host' },
|
||||
fields: { count: 1 },
|
||||
},
|
||||
],
|
||||
sessionCount: 1,
|
||||
uniqueUserList: 'user1',
|
||||
};
|
||||
|
||||
await influxdb.postProxySessionsToInfluxdb(mockUserSessions);
|
||||
|
||||
expect(mockWriteAPI.writePoints).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'PROXY SESSIONS: Sent user session data to InfluxDB for server "test-host", virtual proxy "test-proxy"'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postButlerSOSMemoryUsageToInfluxdb', () => {
|
||||
test('should post memory usage to InfluxDB v1', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
if (key === 'Butler-SOS.influxdbConfig.instanceTag') return 'DEV';
|
||||
if (key === 'Butler-SOS.heartbeat.influxdb.measurementName')
|
||||
return 'butlersos_memory_usage';
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has = jest.fn().mockReturnValue(true);
|
||||
const mockMemory = {
|
||||
heapUsed: 50000000,
|
||||
heapTotal: 100000000,
|
||||
external: 5000000,
|
||||
processMemory: 200000000,
|
||||
};
|
||||
|
||||
await influxdb.postButlerSOSMemoryUsageToInfluxdb(mockMemory);
|
||||
|
||||
expect(globals.influx.writePoints).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should post memory usage to InfluxDB v2', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 2;
|
||||
if (key === 'Butler-SOS.influxdbConfig.instanceTag') return 'DEV';
|
||||
if (key === 'Butler-SOS.heartbeat.influxdb.measurementName')
|
||||
return 'butlersos_memory_usage';
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.org') return 'test-org';
|
||||
if (key === 'Butler-SOS.influxdbConfig.v2Config.bucket') return 'test-bucket';
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has = jest.fn().mockReturnValue(true);
|
||||
|
||||
// Mock the writeAPI returned by getWriteApi()
|
||||
const mockWriteApi = { writePoint: jest.fn() };
|
||||
globals.influx.getWriteApi = jest.fn().mockReturnValue(mockWriteApi);
|
||||
|
||||
const mockMemory = {
|
||||
instanceTag: 'DEV',
|
||||
heapUsedMByte: 50,
|
||||
heapTotalMByte: 100,
|
||||
externalMemoryMByte: 5,
|
||||
processMemoryMByte: 200,
|
||||
};
|
||||
|
||||
await influxdb.postButlerSOSMemoryUsageToInfluxdb(mockMemory);
|
||||
|
||||
expect(globals.influx.getWriteApi).toHaveBeenCalledWith(
|
||||
'test-org',
|
||||
'test-bucket',
|
||||
'ns',
|
||||
expect.any(Object)
|
||||
);
|
||||
expect(mockWriteApi.writePoint).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'MEMORY USAGE INFLUXDB: Sent Butler SOS memory usage data to InfluxDB'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postUserEventToInfluxdb', () => {
|
||||
test('should post user event to InfluxDB v1', async () => {
|
||||
globals.config.get = jest.fn((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.version') return 1;
|
||||
if (key === 'Butler-SOS.influxdbConfig.instanceTag') return 'DEV';
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.userActivity.influxdb.measurementName')
|
||||
return 'user_events';
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has = jest.fn().mockReturnValue(true);
|
||||
const mockMsg = {
|
||||
message: 'User activity',
|
||||
host: 'test-host',
|
||||
source: 'test-source',
|
||||
subsystem: 'test-subsystem',
|
||||
command: 'login',
|
||||
user_directory: 'test-dir',
|
||||
user_id: 'test-user',
|
||||
origin: 'test-origin',
|
||||
};
|
||||
|
||||
await influxdb.postUserEventToInfluxdb(mockMsg);
|
||||
|
||||
expect(globals.influx.writePoints).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('postLogEventToInfluxdb', () => {
|
||||
test('should handle errors gracefully', async () => {
|
||||
globals.config.get = jest.fn().mockImplementation(() => {
|
||||
throw new Error('Test error');
|
||||
});
|
||||
const mockMsg = { message: 'Test log event' };
|
||||
|
||||
await influxdb.postLogEventToInfluxdb(mockMsg);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
'LOG EVENT INFLUXDB 2: Error saving log event to InfluxDB! Error: Test error'
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -8,6 +8,9 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
debug: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn(),
|
||||
},
|
||||
mqttClient: {
|
||||
publish: jest.fn(),
|
||||
},
|
||||
@@ -19,13 +22,20 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
}));
|
||||
const globals = (await import('../../globals.js')).default;
|
||||
|
||||
// Mock log-error module
|
||||
const mockLogError = jest.fn();
|
||||
jest.unstable_mockModule('../log-error.js', () => ({
|
||||
logError: mockLogError,
|
||||
}));
|
||||
|
||||
// Import the module under test
|
||||
const { postHealthToMQTT, postUserSessionsToMQTT, postUserEventToMQTT } = await import(
|
||||
'../post-to-mqtt.js'
|
||||
);
|
||||
const { postHealthToMQTT, postUserSessionsToMQTT, postUserEventToMQTT } =
|
||||
await import('../post-to-mqtt.js');
|
||||
|
||||
describe('post-to-mqtt', () => {
|
||||
beforeEach(() => {
|
||||
// Reset all mocks before each test
|
||||
jest.clearAllMocks();
|
||||
// Setup default config values
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path === 'Butler-SOS.mqttConfig.baseTopic') {
|
||||
@@ -497,7 +507,7 @@ describe('post-to-mqtt', () => {
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle errors during publishing', () => {
|
||||
test('should handle errors during publishing', async () => {
|
||||
// Force an error by making the MQTT client throw
|
||||
globals.mqttClient.publish.mockImplementation(() => {
|
||||
throw new Error('MQTT publish error');
|
||||
@@ -516,11 +526,12 @@ describe('post-to-mqtt', () => {
|
||||
};
|
||||
|
||||
// Call the function being tested
|
||||
postUserEventToMQTT(userEvent);
|
||||
await postUserEventToMQTT(userEvent);
|
||||
|
||||
// Verify error was logged
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('USER EVENT MQTT: Failed posting message to MQTT')
|
||||
expect(mockLogError).toHaveBeenCalledWith(
|
||||
expect.stringContaining('USER EVENT MQTT: Failed posting message to MQTT'),
|
||||
expect.any(Error)
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -39,6 +39,9 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn().mockImplementation((path) => {
|
||||
if (path === 'Butler-SOS.newRelic.enable') return true;
|
||||
|
||||
@@ -52,6 +52,9 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn().mockImplementation((path) => {
|
||||
if (path === 'Butler-SOS.cert.clientCert') return '/path/to/cert.pem';
|
||||
@@ -88,7 +91,7 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
|
||||
// Mock dependent modules
|
||||
const mockPostProxySessionsToInfluxdb = jest.fn().mockResolvedValue();
|
||||
jest.unstable_mockModule('../post-to-influxdb.js', () => ({
|
||||
jest.unstable_mockModule('../influxdb/index.js', () => ({
|
||||
postProxySessionsToInfluxdb: mockPostProxySessionsToInfluxdb,
|
||||
}));
|
||||
|
||||
@@ -116,9 +119,8 @@ jest.unstable_mockModule('../prom-client.js', () => ({
|
||||
}));
|
||||
|
||||
// Import the module under test
|
||||
const { setupUserSessionsTimer, getProxySessionStatsFromSense } = await import(
|
||||
'../proxysessionmetrics.js'
|
||||
);
|
||||
const { setupUserSessionsTimer, getProxySessionStatsFromSense } =
|
||||
await import('../proxysessionmetrics.js');
|
||||
|
||||
describe('proxysessionmetrics', () => {
|
||||
let proxysessionmetrics;
|
||||
@@ -136,7 +138,7 @@ describe('proxysessionmetrics', () => {
|
||||
// Get mocked modules
|
||||
axios = (await import('axios')).default;
|
||||
globals = (await import('../../globals.js')).default;
|
||||
influxdb = await import('../post-to-influxdb.js');
|
||||
influxdb = await import('../influxdb/index.js');
|
||||
newRelic = await import('../post-to-new-relic.js');
|
||||
mqtt = await import('../post-to-mqtt.js');
|
||||
servertags = await import('../servertags.js');
|
||||
|
||||
@@ -28,9 +28,8 @@ const fs = (await import('fs')).default;
|
||||
const globals = (await import('../../globals.js')).default;
|
||||
|
||||
// Import modules under test
|
||||
const { getCertificates: getCertificatesUtil, createCertificateOptions } = await import(
|
||||
'../cert-utils.js'
|
||||
);
|
||||
const { getCertificates: getCertificatesUtil, createCertificateOptions } =
|
||||
await import('../cert-utils.js');
|
||||
|
||||
describe('Certificate loading', () => {
|
||||
const mockCertificateOptions = {
|
||||
|
||||
@@ -18,7 +18,7 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
}));
|
||||
|
||||
// Mock other dependencies
|
||||
jest.unstable_mockModule('../post-to-influxdb.js', () => ({
|
||||
jest.unstable_mockModule('../influxdb/index.js', () => ({
|
||||
postButlerSOSMemoryUsageToInfluxdb: jest.fn(),
|
||||
}));
|
||||
|
||||
@@ -58,7 +58,7 @@ process.memoryUsage = jest.fn().mockReturnValue({
|
||||
|
||||
// Load mocked dependencies
|
||||
const globals = (await import('../../globals.js')).default;
|
||||
const { postButlerSOSMemoryUsageToInfluxdb } = await import('../post-to-influxdb.js');
|
||||
const { postButlerSOSMemoryUsageToInfluxdb } = await import('../influxdb/index.js');
|
||||
const { postButlerSOSUptimeToNewRelic } = await import('../post-to-new-relic.js');
|
||||
const later = (await import('@breejs/later')).default;
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ jest.unstable_mockModule('../../globals.js', () => ({
|
||||
},
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../post-to-influxdb.js', () => ({
|
||||
jest.unstable_mockModule('../influxdb/index.js', () => ({
|
||||
storeRejectedEventCountInfluxDB: jest.fn(),
|
||||
storeEventCountInfluxDB: jest.fn(),
|
||||
}));
|
||||
@@ -50,7 +50,7 @@ describe('udp-event', () => {
|
||||
setupUdpEventsStorage = udpModule.setupUdpEventsStorage;
|
||||
|
||||
globals = (await import('../../globals.js')).default;
|
||||
influxDBModule = await import('../post-to-influxdb.js');
|
||||
influxDBModule = await import('../influxdb/index.js');
|
||||
|
||||
// Create an instance of UdpEvents for testing
|
||||
udpEventsInstance = new UdpEvents(globals.logger);
|
||||
|
||||
@@ -3,6 +3,7 @@ import qrsInteract from 'qrs-interact';
|
||||
import clonedeep from 'lodash.clonedeep';
|
||||
|
||||
import globals from '../globals.js';
|
||||
import { logError } from './log-error.js';
|
||||
|
||||
/**
|
||||
* Retrieves application names from the Qlik Repository Service (QRS) API.
|
||||
@@ -56,11 +57,19 @@ export function getAppNames() {
|
||||
globals.logger.verbose('APP NAMES: Done getting app names from repository db');
|
||||
})
|
||||
.catch((err) => {
|
||||
// Track error count
|
||||
const hostname = globals.config.get('Butler-SOS.appNames.hostIP');
|
||||
globals.errorTracker.incrementError('APP_NAMES_EXTRACT', hostname || '');
|
||||
|
||||
// Return error msg
|
||||
globals.logger.error(`APP NAMES: Error getting app names: ${err}`);
|
||||
logError('APP NAMES: Error getting app names', err);
|
||||
});
|
||||
} catch (err) {
|
||||
globals.globals.logger.error(`APP NAMES: ${err}`);
|
||||
// Track error count
|
||||
const hostname = globals.config.get('Butler-SOS.appNames.hostIP');
|
||||
globals.errorTracker.incrementError('APP_NAMES_EXTRACT', hostname || '');
|
||||
|
||||
logError('APP NAMES', err);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -169,15 +169,41 @@ export async function verifyAppConfig(cfg) {
|
||||
// Verify values of specific config entries
|
||||
|
||||
// If InfluxDB is enabled, check if the version is valid
|
||||
// Valid values: 1 and 2
|
||||
// Valid values: 1, 2, and 3
|
||||
if (cfg.get('Butler-SOS.influxdbConfig.enable') === true) {
|
||||
const influxdbVersion = cfg.get('Butler-SOS.influxdbConfig.version');
|
||||
if (influxdbVersion !== 1 && influxdbVersion !== 2) {
|
||||
if (influxdbVersion !== 1 && influxdbVersion !== 2 && influxdbVersion !== 3) {
|
||||
console.error(
|
||||
`VERIFY CONFIG FILE ERROR: Butler-SOS.influxdbConfig.enable (=InfluxDB version) ${influxdbVersion} is invalid. Exiting.`
|
||||
);
|
||||
return false;
|
||||
}
|
||||
|
||||
// Validate and set default for maxBatchSize
|
||||
const maxBatchSizePath = `Butler-SOS.influxdbConfig.maxBatchSize`;
|
||||
|
||||
if (cfg.has(maxBatchSizePath)) {
|
||||
const maxBatchSize = cfg.get(maxBatchSizePath);
|
||||
|
||||
// Validate maxBatchSize is a number in valid range
|
||||
if (
|
||||
typeof maxBatchSize !== 'number' ||
|
||||
isNaN(maxBatchSize) ||
|
||||
maxBatchSize < 1 ||
|
||||
maxBatchSize > 10000
|
||||
) {
|
||||
console.warn(
|
||||
`VERIFY CONFIG FILE WARNING: ${maxBatchSizePath}=${maxBatchSize} is invalid. Must be a number between 1 and 10000. Using default value 1000.`
|
||||
);
|
||||
cfg.set(maxBatchSizePath, 1000);
|
||||
}
|
||||
} else {
|
||||
// Set default if not specified
|
||||
console.info(
|
||||
`VERIFY CONFIG FILE INFO: ${maxBatchSizePath} not specified. Using default value 1000.`
|
||||
);
|
||||
cfg.set(maxBatchSizePath, 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Verify that telemetry and system info settings are compatible
|
||||
|
||||
@@ -316,6 +316,37 @@ export const destinationsSchema = {
|
||||
},
|
||||
port: { type: 'number' },
|
||||
version: { type: 'number' },
|
||||
maxBatchSize: {
|
||||
type: 'number',
|
||||
description:
|
||||
'Maximum number of data points to write in a single batch. Progressive retry with smaller sizes attempted on failure.',
|
||||
default: 1000,
|
||||
minimum: 1,
|
||||
maximum: 10000,
|
||||
},
|
||||
v3Config: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
database: { type: 'string' },
|
||||
description: { type: 'string' },
|
||||
token: { type: 'string' },
|
||||
retentionDuration: { type: 'string' },
|
||||
writeTimeout: {
|
||||
type: 'number',
|
||||
description: 'Socket timeout for write operations in milliseconds',
|
||||
default: 10000,
|
||||
minimum: 1000,
|
||||
},
|
||||
queryTimeout: {
|
||||
type: 'number',
|
||||
description: 'gRPC timeout for query operations in milliseconds',
|
||||
default: 60000,
|
||||
minimum: 1000,
|
||||
},
|
||||
},
|
||||
required: ['database', 'description', 'token', 'retentionDuration'],
|
||||
additionalProperties: false,
|
||||
},
|
||||
v2Config: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
|
||||
@@ -8,6 +8,7 @@ import * as yaml from 'js-yaml';
|
||||
import globals from '../globals.js';
|
||||
import configObfuscate from './config-obfuscate.js';
|
||||
import { prepareFile, compileTemplate } from './file-prep.js';
|
||||
import { logError } from './log-error.js';
|
||||
|
||||
/**
|
||||
* Serves the custom 404 error page
|
||||
@@ -46,7 +47,7 @@ async function serve404Page(request, reply) {
|
||||
// Send 404 response with custom page
|
||||
reply.code(404).header('Content-Type', 'text/html; charset=utf-8').send(renderedHtml);
|
||||
} catch (err) {
|
||||
globals.logger.error(`CONFIG VIS: Error serving 404 page: ${err.message}`);
|
||||
logError('CONFIG VIS: Error serving 404 page', err);
|
||||
reply.code(404).send({ error: 'Page not found' });
|
||||
}
|
||||
}
|
||||
@@ -184,7 +185,7 @@ export async function setupConfigVisServer(logger, config) {
|
||||
`CONFIG VIS: Directory contents of "${STATIC_PATH}": ${dirContents}`
|
||||
);
|
||||
} catch (err) {
|
||||
globals.logger.error(`CONFIG VIS: Error reading static directory: ${err.message}`);
|
||||
logError('CONFIG VIS: Error reading static directory', err);
|
||||
}
|
||||
|
||||
const htmlDir = path.resolve(STATIC_PATH, 'configvis');
|
||||
@@ -253,7 +254,7 @@ export async function setupConfigVisServer(logger, config) {
|
||||
.header('Content-Type', 'text/html; charset=utf-8')
|
||||
.send(renderedText);
|
||||
} catch (err) {
|
||||
globals.logger.error(`CONFIG VIS: Error serving home page: ${err.message}`);
|
||||
logError('CONFIG VIS: Error serving home page', err);
|
||||
reply.code(500).send({ error: 'Internal server error' });
|
||||
}
|
||||
});
|
||||
@@ -268,7 +269,7 @@ export async function setupConfigVisServer(logger, config) {
|
||||
globals.logger.error(
|
||||
`CONFIG VIS: Could not set up config visualisation server on ${address}`
|
||||
);
|
||||
globals.logger.error(`CONFIG VIS: ${globals.getErrorMessage(err)}`);
|
||||
logError('CONFIG VIS', err);
|
||||
configVisServer.log.error(err);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
238
src/lib/error-tracker.js
Normal file
238
src/lib/error-tracker.js
Normal file
@@ -0,0 +1,238 @@
|
||||
import { Mutex } from 'async-mutex';
|
||||
|
||||
import globals from '../globals.js';
|
||||
import { postErrorMetricsToInfluxdb } from './influxdb/error-metrics.js';
|
||||
|
||||
/**
|
||||
* Class for tracking counts of API errors in Butler SOS.
|
||||
*
|
||||
* This class provides thread-safe methods to track different types of API errors:
|
||||
* - Qlik Sense API errors (Health API, Proxy Sessions API)
|
||||
* - Data destination errors (InfluxDB, New Relic, MQTT)
|
||||
*
|
||||
* Counters reset daily at midnight UTC.
|
||||
*/
|
||||
export class ErrorTracker {
|
||||
/**
|
||||
* Creates a new ErrorTracker instance.
|
||||
*
|
||||
* @param {object} logger - Logger instance with error, debug, info, and verbose methods
|
||||
*/
|
||||
constructor(logger) {
|
||||
this.logger = logger;
|
||||
|
||||
// Array of objects with error counts
|
||||
// Each object has properties:
|
||||
// - apiType: string (e.g., 'HEALTH_API', 'INFLUXDB_V3_WRITE')
|
||||
// - serverName: string (name of the server, or empty string if not applicable)
|
||||
// - count: integer
|
||||
this.errorCounts = [];
|
||||
|
||||
// Mutex for synchronizing access to the array
|
||||
this.errorMutex = new Mutex();
|
||||
|
||||
// Track when counters were last reset
|
||||
this.lastResetDate = new Date().toISOString().split('T')[0]; // YYYY-MM-DD in UTC
|
||||
}
|
||||
|
||||
/**
|
||||
* Increments the error count for a specific API type and server.
|
||||
*
|
||||
* @param {string} apiType - The type of API that encountered an error (e.g., 'HEALTH_API', 'PROXY_API')
|
||||
* @param {string} serverName - The name of the server where the error occurred (empty string if not applicable)
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
async incrementError(apiType, serverName) {
|
||||
// Ensure the passed parameters are strings
|
||||
if (typeof apiType !== 'string') {
|
||||
this.logger.error(
|
||||
`ERROR TRACKER: apiType must be a string: ${JSON.stringify(apiType)}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if (typeof serverName !== 'string') {
|
||||
this.logger.error(
|
||||
`ERROR TRACKER: serverName must be a string: ${JSON.stringify(serverName)}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const release = await this.errorMutex.acquire();
|
||||
|
||||
try {
|
||||
// Check if we need to reset counters (new day in UTC)
|
||||
const currentDate = new Date().toISOString().split('T')[0]; // YYYY-MM-DD in UTC
|
||||
if (currentDate !== this.lastResetDate) {
|
||||
this.logger.debug(
|
||||
`ERROR TRACKER: Date changed from ${this.lastResetDate} to ${currentDate}, resetting counters`
|
||||
);
|
||||
await this.resetCounters();
|
||||
this.lastResetDate = currentDate;
|
||||
}
|
||||
|
||||
const found = this.errorCounts.find((element) => {
|
||||
return element.apiType === apiType && element.serverName === serverName;
|
||||
});
|
||||
|
||||
if (found) {
|
||||
found.count += 1;
|
||||
this.logger.debug(
|
||||
`ERROR TRACKER: Incremented error count for ${apiType}/${serverName}, new count: ${found.count}`
|
||||
);
|
||||
} else {
|
||||
this.logger.debug(
|
||||
`ERROR TRACKER: Adding first error count for ${apiType}/${serverName}`
|
||||
);
|
||||
|
||||
this.errorCounts.push({
|
||||
apiType,
|
||||
serverName,
|
||||
count: 1,
|
||||
});
|
||||
}
|
||||
|
||||
// Log current error statistics
|
||||
await this.logErrorSummary();
|
||||
|
||||
// Call placeholder function to store to InfluxDB (non-blocking)
|
||||
// This will be implemented later
|
||||
setImmediate(() => {
|
||||
postErrorMetricsToInfluxdb(this.getErrorStats()).catch((err) => {
|
||||
this.logger.debug(
|
||||
`ERROR TRACKER: Error calling placeholder InfluxDB function: ${err.message}`
|
||||
);
|
||||
});
|
||||
});
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resets all error counters.
|
||||
* Should be called at midnight UTC or when starting fresh.
|
||||
*
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
async resetCounters() {
|
||||
// Note: Caller must hold the mutex before calling this method
|
||||
this.errorCounts = [];
|
||||
this.logger.info('ERROR TRACKER: Reset all error counters');
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets current error statistics grouped by API type.
|
||||
*
|
||||
* @returns {object} Object with API types as keys, each containing total count and server breakdown
|
||||
*/
|
||||
getErrorStats() {
|
||||
const stats = {};
|
||||
|
||||
for (const error of this.errorCounts) {
|
||||
if (!stats[error.apiType]) {
|
||||
stats[error.apiType] = {
|
||||
total: 0,
|
||||
servers: {},
|
||||
};
|
||||
}
|
||||
|
||||
stats[error.apiType].total += error.count;
|
||||
|
||||
if (error.serverName) {
|
||||
stats[error.apiType].servers[error.serverName] = error.count;
|
||||
} else {
|
||||
// For errors without server context, use a placeholder
|
||||
if (!stats[error.apiType].servers['_no_server_context']) {
|
||||
stats[error.apiType].servers['_no_server_context'] = 0;
|
||||
}
|
||||
stats[error.apiType].servers['_no_server_context'] += error.count;
|
||||
}
|
||||
}
|
||||
|
||||
return stats;
|
||||
}
|
||||
|
||||
/**
|
||||
* Logs a summary of current error counts at INFO level.
|
||||
*
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
async logErrorSummary() {
|
||||
const stats = this.getErrorStats();
|
||||
|
||||
if (Object.keys(stats).length === 0) {
|
||||
return; // No errors to log
|
||||
}
|
||||
|
||||
// Calculate grand total
|
||||
let grandTotal = 0;
|
||||
for (const apiType in stats) {
|
||||
grandTotal += stats[apiType].total;
|
||||
}
|
||||
|
||||
this.logger.info(
|
||||
`ERROR TRACKER: Error counts today (UTC): Total=${grandTotal}, Details=${JSON.stringify(stats)}`
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets all error counts (for testing purposes).
|
||||
*
|
||||
* @returns {Promise<Array>} Array of error count objects
|
||||
*/
|
||||
async getErrorCounts() {
|
||||
const release = await this.errorMutex.acquire();
|
||||
|
||||
try {
|
||||
return this.errorCounts;
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sets up a timer that resets error counters at midnight UTC.
|
||||
*
|
||||
* This function calculates the time until next midnight UTC and schedules
|
||||
* a reset, then reschedules itself for the following midnight.
|
||||
*
|
||||
* @returns {void}
|
||||
*/
|
||||
export function setupErrorCounterReset() {
|
||||
/**
|
||||
* Schedules the next reset at midnight UTC.
|
||||
*/
|
||||
const scheduleNextReset = () => {
|
||||
// Calculate milliseconds until next midnight UTC
|
||||
const now = new Date();
|
||||
const nextMidnight = new Date(now);
|
||||
nextMidnight.setUTCHours(24, 0, 0, 0);
|
||||
const msUntilMidnight = nextMidnight - now;
|
||||
|
||||
globals.logger.info(
|
||||
`ERROR TRACKER: Scheduled next error counter reset at ${nextMidnight.toISOString()} (in ${Math.round(msUntilMidnight / 1000 / 60)} minutes)`
|
||||
);
|
||||
|
||||
setTimeout(async () => {
|
||||
globals.logger.info('ERROR TRACKER: Midnight UTC reached, resetting error counters');
|
||||
|
||||
// Log final daily summary before reset
|
||||
const release = await globals.errorTracker.errorMutex.acquire();
|
||||
try {
|
||||
await globals.errorTracker.logErrorSummary();
|
||||
await globals.errorTracker.resetCounters();
|
||||
globals.errorTracker.lastResetDate = new Date().toISOString().split('T')[0];
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
|
||||
// Schedule next reset
|
||||
scheduleNextReset();
|
||||
}, msUntilMidnight);
|
||||
};
|
||||
|
||||
// Start the reset cycle
|
||||
scheduleNextReset();
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import sea from './sea-wrapper.js';
|
||||
import handlebars from 'handlebars';
|
||||
|
||||
import globals from '../globals.js';
|
||||
import { logError } from './log-error.js';
|
||||
|
||||
// Define MIME types for different file extensions
|
||||
const MIME_TYPES = {
|
||||
@@ -90,7 +91,7 @@ export async function prepareFile(filePath, encoding) {
|
||||
stream = Readable.from([content]);
|
||||
}
|
||||
} catch (err) {
|
||||
globals.logger.error(`FILE PREP: Error preparing file: ${err.message}`);
|
||||
logError('FILE PREP: Error preparing file', err);
|
||||
exists = false;
|
||||
}
|
||||
|
||||
@@ -116,7 +117,7 @@ export function compileTemplate(templateContent, data) {
|
||||
const template = handlebars.compile(templateContent);
|
||||
return template(data);
|
||||
} catch (err) {
|
||||
globals.logger.error(`FILE PREP: Error compiling handlebars template: ${err.message}`);
|
||||
logError('FILE PREP: Error compiling handlebars template', err);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,13 +7,14 @@ import https from 'https';
|
||||
import axios from 'axios';
|
||||
|
||||
import globals from '../globals.js';
|
||||
import { postHealthMetricsToInfluxdb } from './post-to-influxdb.js';
|
||||
import { postHealthMetricsToInfluxdb } from './influxdb/index.js';
|
||||
import { postHealthMetricsToNewRelic } from './post-to-new-relic.js';
|
||||
import { postHealthToMQTT } from './post-to-mqtt.js';
|
||||
import { getServerHeaders } from './serverheaders.js';
|
||||
import { getServerTags } from './servertags.js';
|
||||
import { saveHealthMetricsToPrometheus } from './prom-client.js';
|
||||
import { getCertificates, createCertificateOptions } from './cert-utils.js';
|
||||
import { logError } from './log-error.js';
|
||||
|
||||
/**
|
||||
* Retrieves health statistics from Qlik Sense server via the engine healthcheck API.
|
||||
@@ -102,10 +103,18 @@ export async function getHealthStatsFromSense(serverName, host, tags, headers) {
|
||||
globals.logger.debug('HEALTH: Calling HEALTH metrics Prometheus method');
|
||||
saveHealthMetricsToPrometheus(host, response.data, tags);
|
||||
}
|
||||
} else {
|
||||
globals.logger.error(
|
||||
`HEALTH: Received non-200 response code (${response.status}) from server '${serverName}' (${host})`
|
||||
);
|
||||
}
|
||||
} catch (err) {
|
||||
globals.logger.error(
|
||||
`HEALTH: Error when calling health check API for server '${serverName}' (${host}): ${globals.getErrorMessage(err)}`
|
||||
// Track error count
|
||||
globals.errorTracker.incrementError('HEALTH_API', serverName);
|
||||
|
||||
logError(
|
||||
`HEALTH: Error when calling health check API for server '${serverName}' (${host})`,
|
||||
err
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
106
src/lib/influxdb/README.md
Normal file
106
src/lib/influxdb/README.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# InfluxDB Module - Refactored Architecture
|
||||
|
||||
This directory contains the refactored InfluxDB integration code, organized by version for better maintainability and testability.
|
||||
|
||||
## Structure
|
||||
|
||||
```text
|
||||
influxdb/
|
||||
├── shared/ # Shared utilities and helpers
|
||||
│ └── utils.js # Common functions (getFormattedTime, processAppDocuments, writeToInfluxWithRetry, etc.)
|
||||
├── v1/ # InfluxDB 1.x implementations (InfluxQL)
|
||||
├── v2/ # InfluxDB 2.x implementations (Flux)
|
||||
├── v3/ # InfluxDB 3.x implementations (SQL)
|
||||
├── factory.js # Version router that delegates to appropriate implementation
|
||||
└── index.js # Main facade providing consistent API
|
||||
```
|
||||
|
||||
## Refactoring Complete
|
||||
|
||||
All InfluxDB versions (v1, v2, v3) now use the refactored modular code.
|
||||
|
||||
**Benefits:**
|
||||
|
||||
- Modular, version-specific implementations
|
||||
- Shared utilities reduce code duplication
|
||||
- Unified retry logic with exponential backoff
|
||||
- Comprehensive JSDoc documentation
|
||||
- Better error handling and resource management
|
||||
- Consistent patterns across all versions
|
||||
|
||||
## Implementation Status
|
||||
|
||||
### V1 (InfluxDB 1.x - InfluxQL)
|
||||
|
||||
✅ All modules complete:
|
||||
|
||||
- Health metrics
|
||||
- Proxy sessions
|
||||
- Butler memory usage
|
||||
- User events
|
||||
- Log events
|
||||
- Event counts
|
||||
- Queue metrics
|
||||
|
||||
### V2 (InfluxDB 2.x - Flux)
|
||||
|
||||
✅ All modules complete:
|
||||
|
||||
- Health metrics
|
||||
- Proxy sessions
|
||||
- Butler memory usage
|
||||
- User events
|
||||
- Log events
|
||||
- Event counts
|
||||
- Queue metrics
|
||||
|
||||
### V3 (InfluxDB 3.x - SQL)
|
||||
|
||||
✅ All modules complete:
|
||||
|
||||
- Health metrics
|
||||
- Proxy sessions
|
||||
- Butler memory usage
|
||||
- User events
|
||||
- Log events
|
||||
- Event counts
|
||||
- Queue metrics
|
||||
|
||||
### Pending
|
||||
|
||||
- ⏳ Complete test coverage for all modules
|
||||
- ⏳ Integration tests
|
||||
- ⏳ Performance benchmarking
|
||||
|
||||
## Usage
|
||||
|
||||
### For Developers
|
||||
|
||||
When the feature flag is enabled, the facade in `index.js` will route calls to the refactored implementations. If a version-specific implementation is not yet complete, it automatically falls back to the original code.
|
||||
|
||||
```javascript
|
||||
// Imports work the same way
|
||||
import { postHealthMetricsToInfluxdb } from './lib/influxdb/index.js';
|
||||
|
||||
// Function automatically routes based on feature flag
|
||||
await postHealthMetricsToInfluxdb(serverName, host, body, serverTags);
|
||||
```
|
||||
|
||||
### Adding New Implementations
|
||||
|
||||
1. Create the version-specific module (e.g., `v3/sessions.js`)
|
||||
2. Import and export it in `factory.js`
|
||||
3. Update the facade in `index.js` to use the factory
|
||||
4. Add tests in the appropriate `__tests__` directory
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Maintainability**: Smaller, focused files instead of one 3000+ line file
|
||||
2. **Testability**: Each module can be tested in isolation
|
||||
3. **Code Reuse**: Shared utilities reduce duplication
|
||||
4. **Version Management**: Easy to deprecate old versions when needed
|
||||
5. **Safe Migration**: Feature flag allows gradual rollout
|
||||
|
||||
## Original Implementation
|
||||
|
||||
The original implementation remains in `/src/lib/post-to-influxdb.js` and continues to work as before. This ensures no breaking changes during migration.
|
||||
60
src/lib/influxdb/__tests__/error-metrics.test.js
Normal file
60
src/lib/influxdb/__tests__/error-metrics.test.js
Normal file
@@ -0,0 +1,60 @@
|
||||
import { jest, describe, test, expect } from '@jest/globals';
|
||||
import { postErrorMetricsToInfluxdb } from '../error-metrics.js';
|
||||
|
||||
describe('error-metrics', () => {
|
||||
describe('postErrorMetricsToInfluxdb', () => {
|
||||
test('should resolve successfully with valid error stats', async () => {
|
||||
const errorStats = {
|
||||
HEALTH_API: {
|
||||
total: 5,
|
||||
servers: {
|
||||
sense1: 3,
|
||||
sense2: 2,
|
||||
},
|
||||
},
|
||||
INFLUXDB_V3_WRITE: {
|
||||
total: 2,
|
||||
servers: {
|
||||
_no_server_context: 2,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
await expect(postErrorMetricsToInfluxdb(errorStats)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
test('should resolve successfully with empty error stats', async () => {
|
||||
const errorStats = {};
|
||||
|
||||
await expect(postErrorMetricsToInfluxdb(errorStats)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
test('should resolve successfully with null input', async () => {
|
||||
await expect(postErrorMetricsToInfluxdb(null)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
test('should resolve successfully with undefined input', async () => {
|
||||
await expect(postErrorMetricsToInfluxdb(undefined)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
test('should resolve successfully with complex error stats', async () => {
|
||||
const errorStats = {
|
||||
API_TYPE_1: {
|
||||
total: 100,
|
||||
servers: {
|
||||
server1: 25,
|
||||
server2: 25,
|
||||
server3: 25,
|
||||
server4: 25,
|
||||
},
|
||||
},
|
||||
API_TYPE_2: {
|
||||
total: 0,
|
||||
servers: {},
|
||||
},
|
||||
};
|
||||
|
||||
await expect(postErrorMetricsToInfluxdb(errorStats)).resolves.toBeUndefined();
|
||||
});
|
||||
});
|
||||
});
|
||||
568
src/lib/influxdb/__tests__/factory.test.js
Normal file
568
src/lib/influxdb/__tests__/factory.test.js
Normal file
@@ -0,0 +1,568 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
jest.unstable_mockModule('../shared/utils.js', () => ({
|
||||
getInfluxDbVersion: jest.fn(),
|
||||
getFormattedTime: jest.fn(),
|
||||
processAppDocuments: jest.fn(),
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
applyTagsToPoint3: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
}));
|
||||
|
||||
// Mock v3 implementations
|
||||
jest.unstable_mockModule('../v3/queue-metrics.js', () => ({
|
||||
postUserEventQueueMetricsToInfluxdbV3: jest.fn(),
|
||||
postLogEventQueueMetricsToInfluxdbV3: jest.fn(),
|
||||
}));
|
||||
|
||||
// Mock v2 implementations
|
||||
jest.unstable_mockModule('../v2/queue-metrics.js', () => ({
|
||||
storeUserEventQueueMetricsV2: jest.fn(),
|
||||
storeLogEventQueueMetricsV2: jest.fn(),
|
||||
}));
|
||||
|
||||
// Mock v1 implementations
|
||||
jest.unstable_mockModule('../v1/queue-metrics.js', () => ({
|
||||
storeUserEventQueueMetricsV1: jest.fn(),
|
||||
storeLogEventQueueMetricsV1: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v1/health-metrics.js', () => ({
|
||||
storeHealthMetricsV1: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v2/health-metrics.js', () => ({
|
||||
storeHealthMetricsV2: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v3/health-metrics.js', () => ({
|
||||
postHealthMetricsToInfluxdbV3: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v1/sessions.js', () => ({
|
||||
storeSessionsV1: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v2/sessions.js', () => ({
|
||||
storeSessionsV2: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v3/sessions.js', () => ({
|
||||
postProxySessionsToInfluxdbV3: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v1/butler-memory.js', () => ({
|
||||
storeButlerMemoryV1: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v2/butler-memory.js', () => ({
|
||||
storeButlerMemoryV2: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v3/butler-memory.js', () => ({
|
||||
postButlerSOSMemoryUsageToInfluxdbV3: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v1/user-events.js', () => ({
|
||||
storeUserEventV1: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v2/user-events.js', () => ({
|
||||
storeUserEventV2: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v3/user-events.js', () => ({
|
||||
postUserEventToInfluxdbV3: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v1/log-events.js', () => ({
|
||||
storeLogEventV1: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v2/log-events.js', () => ({
|
||||
storeLogEventV2: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v3/log-events.js', () => ({
|
||||
postLogEventToInfluxdbV3: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v1/event-counts.js', () => ({
|
||||
storeEventCountV1: jest.fn(),
|
||||
storeRejectedEventCountV1: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v2/event-counts.js', () => ({
|
||||
storeEventCountV2: jest.fn(),
|
||||
storeRejectedEventCountV2: jest.fn(),
|
||||
}));
|
||||
|
||||
jest.unstable_mockModule('../v3/event-counts.js', () => ({
|
||||
storeEventCountInfluxDBV3: jest.fn(),
|
||||
storeRejectedEventCountInfluxDBV3: jest.fn(),
|
||||
}));
|
||||
|
||||
describe('InfluxDB Factory', () => {
|
||||
let factory;
|
||||
let globals;
|
||||
let utils;
|
||||
let v3Impl;
|
||||
let v2Impl;
|
||||
let v1Impl;
|
||||
let v3Health, v2Health, v1Health;
|
||||
let v3Sessions, v2Sessions, v1Sessions;
|
||||
let v3Memory, v2Memory, v1Memory;
|
||||
let v3User, v2User, v1User;
|
||||
let v3Log, v2Log, v1Log;
|
||||
let v3EventCounts, v2EventCounts, v1EventCounts;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
v3Impl = await import('../v3/queue-metrics.js');
|
||||
v2Impl = await import('../v2/queue-metrics.js');
|
||||
v1Impl = await import('../v1/queue-metrics.js');
|
||||
|
||||
v3Health = await import('../v3/health-metrics.js');
|
||||
v2Health = await import('../v2/health-metrics.js');
|
||||
v1Health = await import('../v1/health-metrics.js');
|
||||
|
||||
v3Sessions = await import('../v3/sessions.js');
|
||||
v2Sessions = await import('../v2/sessions.js');
|
||||
v1Sessions = await import('../v1/sessions.js');
|
||||
|
||||
v3Memory = await import('../v3/butler-memory.js');
|
||||
v2Memory = await import('../v2/butler-memory.js');
|
||||
v1Memory = await import('../v1/butler-memory.js');
|
||||
|
||||
v3User = await import('../v3/user-events.js');
|
||||
v2User = await import('../v2/user-events.js');
|
||||
v1User = await import('../v1/user-events.js');
|
||||
|
||||
v3Log = await import('../v3/log-events.js');
|
||||
v2Log = await import('../v2/log-events.js');
|
||||
v1Log = await import('../v1/log-events.js');
|
||||
|
||||
v3EventCounts = await import('../v3/event-counts.js');
|
||||
v2EventCounts = await import('../v2/event-counts.js');
|
||||
v1EventCounts = await import('../v1/event-counts.js');
|
||||
|
||||
factory = await import('../factory.js');
|
||||
|
||||
// Setup default mocks
|
||||
v3Impl.postUserEventQueueMetricsToInfluxdbV3.mockResolvedValue();
|
||||
v3Impl.postLogEventQueueMetricsToInfluxdbV3.mockResolvedValue();
|
||||
v2Impl.storeUserEventQueueMetricsV2.mockResolvedValue();
|
||||
v2Impl.storeLogEventQueueMetricsV2.mockResolvedValue();
|
||||
v1Impl.storeUserEventQueueMetricsV1.mockResolvedValue();
|
||||
v1Impl.storeLogEventQueueMetricsV1.mockResolvedValue();
|
||||
|
||||
v3Health.postHealthMetricsToInfluxdbV3.mockResolvedValue();
|
||||
v2Health.storeHealthMetricsV2.mockResolvedValue();
|
||||
v1Health.storeHealthMetricsV1.mockResolvedValue();
|
||||
|
||||
v3Sessions.postProxySessionsToInfluxdbV3.mockResolvedValue();
|
||||
v2Sessions.storeSessionsV2.mockResolvedValue();
|
||||
v1Sessions.storeSessionsV1.mockResolvedValue();
|
||||
|
||||
v3Memory.postButlerSOSMemoryUsageToInfluxdbV3.mockResolvedValue();
|
||||
v2Memory.storeButlerMemoryV2.mockResolvedValue();
|
||||
v1Memory.storeButlerMemoryV1.mockResolvedValue();
|
||||
|
||||
v3User.postUserEventToInfluxdbV3.mockResolvedValue();
|
||||
v2User.storeUserEventV2.mockResolvedValue();
|
||||
v1User.storeUserEventV1.mockResolvedValue();
|
||||
|
||||
v3Log.postLogEventToInfluxdbV3.mockResolvedValue();
|
||||
v2Log.storeLogEventV2.mockResolvedValue();
|
||||
v1Log.storeLogEventV1.mockResolvedValue();
|
||||
|
||||
v3EventCounts.storeEventCountInfluxDBV3.mockResolvedValue();
|
||||
v3EventCounts.storeRejectedEventCountInfluxDBV3.mockResolvedValue();
|
||||
v2EventCounts.storeEventCountV2.mockResolvedValue();
|
||||
v2EventCounts.storeRejectedEventCountV2.mockResolvedValue();
|
||||
v1EventCounts.storeEventCountV1.mockResolvedValue();
|
||||
v1EventCounts.storeRejectedEventCountV1.mockResolvedValue();
|
||||
});
|
||||
|
||||
describe('postUserEventQueueMetricsToInfluxdb', () => {
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.postUserEventQueueMetricsToInfluxdb();
|
||||
|
||||
expect(v3Impl.postUserEventQueueMetricsToInfluxdbV3).toHaveBeenCalled();
|
||||
expect(v2Impl.storeUserEventQueueMetricsV2).not.toHaveBeenCalled();
|
||||
expect(v1Impl.storeUserEventQueueMetricsV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.postUserEventQueueMetricsToInfluxdb();
|
||||
|
||||
expect(v2Impl.storeUserEventQueueMetricsV2).toHaveBeenCalled();
|
||||
expect(v3Impl.postUserEventQueueMetricsToInfluxdbV3).not.toHaveBeenCalled();
|
||||
expect(v1Impl.storeUserEventQueueMetricsV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.postUserEventQueueMetricsToInfluxdb();
|
||||
|
||||
expect(v1Impl.storeUserEventQueueMetricsV1).toHaveBeenCalled();
|
||||
expect(v3Impl.postUserEventQueueMetricsToInfluxdbV3).not.toHaveBeenCalled();
|
||||
expect(v2Impl.storeUserEventQueueMetricsV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(99);
|
||||
|
||||
await expect(factory.postUserEventQueueMetricsToInfluxdb()).rejects.toThrow(
|
||||
'InfluxDB v99 not supported'
|
||||
);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
'INFLUXDB FACTORY: Unknown InfluxDB version: v99'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postLogEventQueueMetricsToInfluxdb', () => {
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.postLogEventQueueMetricsToInfluxdb();
|
||||
|
||||
expect(v3Impl.postLogEventQueueMetricsToInfluxdbV3).toHaveBeenCalled();
|
||||
expect(v2Impl.storeLogEventQueueMetricsV2).not.toHaveBeenCalled();
|
||||
expect(v1Impl.storeLogEventQueueMetricsV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.postLogEventQueueMetricsToInfluxdb();
|
||||
|
||||
expect(v2Impl.storeLogEventQueueMetricsV2).toHaveBeenCalled();
|
||||
expect(v3Impl.postLogEventQueueMetricsToInfluxdbV3).not.toHaveBeenCalled();
|
||||
expect(v1Impl.storeLogEventQueueMetricsV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.postLogEventQueueMetricsToInfluxdb();
|
||||
|
||||
expect(v1Impl.storeLogEventQueueMetricsV1).toHaveBeenCalled();
|
||||
expect(v3Impl.postLogEventQueueMetricsToInfluxdbV3).not.toHaveBeenCalled();
|
||||
expect(v2Impl.storeLogEventQueueMetricsV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(5);
|
||||
|
||||
await expect(factory.postLogEventQueueMetricsToInfluxdb()).rejects.toThrow(
|
||||
'InfluxDB v5 not supported'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postHealthMetricsToInfluxdb', () => {
|
||||
const serverName = 'test-server';
|
||||
const host = 'test-host';
|
||||
const body = { version: '1.0' };
|
||||
const serverTags = [{ name: 'env', value: 'prod' }];
|
||||
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.postHealthMetricsToInfluxdb(serverName, host, body, serverTags);
|
||||
|
||||
expect(v3Health.postHealthMetricsToInfluxdbV3).toHaveBeenCalledWith(
|
||||
serverName,
|
||||
host,
|
||||
body,
|
||||
serverTags
|
||||
);
|
||||
expect(v2Health.storeHealthMetricsV2).not.toHaveBeenCalled();
|
||||
expect(v1Health.storeHealthMetricsV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.postHealthMetricsToInfluxdb(serverName, host, body, serverTags);
|
||||
|
||||
expect(v2Health.storeHealthMetricsV2).toHaveBeenCalledWith(
|
||||
serverName,
|
||||
host,
|
||||
body,
|
||||
serverTags
|
||||
);
|
||||
expect(v3Health.postHealthMetricsToInfluxdbV3).not.toHaveBeenCalled();
|
||||
expect(v1Health.storeHealthMetricsV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.postHealthMetricsToInfluxdb(serverName, host, body, serverTags);
|
||||
|
||||
expect(v1Health.storeHealthMetricsV1).toHaveBeenCalledWith(serverTags, body);
|
||||
expect(v3Health.postHealthMetricsToInfluxdbV3).not.toHaveBeenCalled();
|
||||
expect(v2Health.storeHealthMetricsV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(4);
|
||||
|
||||
await expect(
|
||||
factory.postHealthMetricsToInfluxdb(serverName, host, body, serverTags)
|
||||
).rejects.toThrow('InfluxDB v4 not supported');
|
||||
});
|
||||
});
|
||||
|
||||
describe('postProxySessionsToInfluxdb', () => {
|
||||
const userSessions = { serverName: 'test', host: 'test-host' };
|
||||
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.postProxySessionsToInfluxdb(userSessions);
|
||||
|
||||
expect(v3Sessions.postProxySessionsToInfluxdbV3).toHaveBeenCalledWith(userSessions);
|
||||
expect(v2Sessions.storeSessionsV2).not.toHaveBeenCalled();
|
||||
expect(v1Sessions.storeSessionsV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.postProxySessionsToInfluxdb(userSessions);
|
||||
|
||||
expect(v2Sessions.storeSessionsV2).toHaveBeenCalledWith(userSessions);
|
||||
expect(v3Sessions.postProxySessionsToInfluxdbV3).not.toHaveBeenCalled();
|
||||
expect(v1Sessions.storeSessionsV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.postProxySessionsToInfluxdb(userSessions);
|
||||
|
||||
expect(v1Sessions.storeSessionsV1).toHaveBeenCalledWith(userSessions);
|
||||
expect(v3Sessions.postProxySessionsToInfluxdbV3).not.toHaveBeenCalled();
|
||||
expect(v2Sessions.storeSessionsV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(10);
|
||||
|
||||
await expect(factory.postProxySessionsToInfluxdb(userSessions)).rejects.toThrow(
|
||||
'InfluxDB v10 not supported'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postButlerSOSMemoryUsageToInfluxdb', () => {
|
||||
const memory = { heap_used: 100, heap_total: 200 };
|
||||
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.postButlerSOSMemoryUsageToInfluxdb(memory);
|
||||
|
||||
expect(v3Memory.postButlerSOSMemoryUsageToInfluxdbV3).toHaveBeenCalledWith(memory);
|
||||
expect(v2Memory.storeButlerMemoryV2).not.toHaveBeenCalled();
|
||||
expect(v1Memory.storeButlerMemoryV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.postButlerSOSMemoryUsageToInfluxdb(memory);
|
||||
|
||||
expect(v2Memory.storeButlerMemoryV2).toHaveBeenCalledWith(memory);
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.postButlerSOSMemoryUsageToInfluxdb(memory);
|
||||
|
||||
expect(v1Memory.storeButlerMemoryV1).toHaveBeenCalledWith(memory);
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(7);
|
||||
|
||||
await expect(factory.postButlerSOSMemoryUsageToInfluxdb(memory)).rejects.toThrow(
|
||||
'InfluxDB v7 not supported'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postUserEventToInfluxdb', () => {
|
||||
const msg = { host: 'test-host', command: 'OpenApp' };
|
||||
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.postUserEventToInfluxdb(msg);
|
||||
|
||||
expect(v3User.postUserEventToInfluxdbV3).toHaveBeenCalledWith(msg);
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.postUserEventToInfluxdb(msg);
|
||||
|
||||
expect(v2User.storeUserEventV2).toHaveBeenCalledWith(msg);
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.postUserEventToInfluxdb(msg);
|
||||
|
||||
expect(v1User.storeUserEventV1).toHaveBeenCalledWith(msg);
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(0);
|
||||
|
||||
await expect(factory.postUserEventToInfluxdb(msg)).rejects.toThrow(
|
||||
'InfluxDB v0 not supported'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postLogEventToInfluxdb', () => {
|
||||
const msg = { host: 'test-host', source: 'qseow-engine' };
|
||||
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.postLogEventToInfluxdb(msg);
|
||||
|
||||
expect(v3Log.postLogEventToInfluxdbV3).toHaveBeenCalledWith(msg);
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.postLogEventToInfluxdb(msg);
|
||||
|
||||
expect(v2Log.storeLogEventV2).toHaveBeenCalledWith(msg);
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.postLogEventToInfluxdb(msg);
|
||||
|
||||
expect(v1Log.storeLogEventV1).toHaveBeenCalledWith(msg);
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(-1);
|
||||
|
||||
await expect(factory.postLogEventToInfluxdb(msg)).rejects.toThrow(
|
||||
'InfluxDB v-1 not supported'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeEventCountInfluxDB', () => {
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.storeEventCountInfluxDB();
|
||||
|
||||
expect(v3EventCounts.storeEventCountInfluxDBV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.storeEventCountInfluxDB();
|
||||
|
||||
expect(v2EventCounts.storeEventCountV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.storeEventCountInfluxDB();
|
||||
|
||||
expect(v1EventCounts.storeEventCountV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(100);
|
||||
|
||||
await expect(factory.storeEventCountInfluxDB()).rejects.toThrow(
|
||||
'InfluxDB v100 not supported'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeRejectedEventCountInfluxDB', () => {
|
||||
test('should route to v3 implementation when version is 3', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(3);
|
||||
|
||||
await factory.storeRejectedEventCountInfluxDB();
|
||||
|
||||
expect(v3EventCounts.storeRejectedEventCountInfluxDBV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v2 implementation when version is 2', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(2);
|
||||
|
||||
await factory.storeRejectedEventCountInfluxDB();
|
||||
|
||||
expect(v2EventCounts.storeRejectedEventCountV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should route to v1 implementation when version is 1', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(1);
|
||||
|
||||
await factory.storeRejectedEventCountInfluxDB();
|
||||
|
||||
expect(v1EventCounts.storeRejectedEventCountV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should throw error for unsupported version', async () => {
|
||||
utils.getInfluxDbVersion.mockReturnValue(99);
|
||||
|
||||
await expect(factory.storeRejectedEventCountInfluxDB()).rejects.toThrow(
|
||||
'InfluxDB v99 not supported'
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
301
src/lib/influxdb/__tests__/index.test.js
Normal file
301
src/lib/influxdb/__tests__/index.test.js
Normal file
@@ -0,0 +1,301 @@
|
||||
import { jest, describe, test, expect, beforeEach, afterEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock factory
|
||||
const mockFactory = {
|
||||
postHealthMetricsToInfluxdb: jest.fn(),
|
||||
postProxySessionsToInfluxdb: jest.fn(),
|
||||
postButlerSOSMemoryUsageToInfluxdb: jest.fn(),
|
||||
postUserEventToInfluxdb: jest.fn(),
|
||||
postLogEventToInfluxdb: jest.fn(),
|
||||
storeEventCountInfluxDB: jest.fn(),
|
||||
storeRejectedEventCountInfluxDB: jest.fn(),
|
||||
postUserEventQueueMetricsToInfluxdb: jest.fn(),
|
||||
postLogEventQueueMetricsToInfluxdb: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../factory.js', () => mockFactory);
|
||||
|
||||
// Mock shared utils
|
||||
jest.unstable_mockModule('../shared/utils.js', () => ({
|
||||
getFormattedTime: jest.fn((time) => `formatted-${time}`),
|
||||
}));
|
||||
|
||||
describe('InfluxDB Index (Facade)', () => {
|
||||
let indexModule;
|
||||
let globals;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
indexModule = await import('../index.js');
|
||||
|
||||
// Setup default mock implementations
|
||||
mockFactory.postHealthMetricsToInfluxdb.mockResolvedValue();
|
||||
mockFactory.postProxySessionsToInfluxdb.mockResolvedValue();
|
||||
mockFactory.postButlerSOSMemoryUsageToInfluxdb.mockResolvedValue();
|
||||
mockFactory.postUserEventToInfluxdb.mockResolvedValue();
|
||||
mockFactory.postLogEventToInfluxdb.mockResolvedValue();
|
||||
mockFactory.storeEventCountInfluxDB.mockResolvedValue();
|
||||
mockFactory.storeRejectedEventCountInfluxDB.mockResolvedValue();
|
||||
mockFactory.postUserEventQueueMetricsToInfluxdb.mockResolvedValue();
|
||||
mockFactory.postLogEventQueueMetricsToInfluxdb.mockResolvedValue();
|
||||
|
||||
globals.config.get.mockReturnValue(true);
|
||||
});
|
||||
|
||||
describe('getFormattedTime', () => {
|
||||
test('should be exported and callable', () => {
|
||||
expect(indexModule.getFormattedTime).toBeDefined();
|
||||
expect(typeof indexModule.getFormattedTime).toBe('function');
|
||||
});
|
||||
|
||||
test('should format time correctly', () => {
|
||||
const result = indexModule.getFormattedTime('20240101T120000');
|
||||
expect(result).toBe('formatted-20240101T120000');
|
||||
});
|
||||
});
|
||||
|
||||
describe('postHealthMetricsToInfluxdb', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
const serverName = 'server1';
|
||||
const host = 'host1';
|
||||
const body = { version: '1.0' };
|
||||
const serverTags = [{ name: 'env', value: 'prod' }];
|
||||
|
||||
await indexModule.postHealthMetricsToInfluxdb(serverName, host, body, serverTags);
|
||||
|
||||
expect(mockFactory.postHealthMetricsToInfluxdb).toHaveBeenCalledWith(
|
||||
serverName,
|
||||
host,
|
||||
body,
|
||||
serverTags
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postProxySessionsToInfluxdb', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
const userSessions = { serverName: 'test', host: 'test-host' };
|
||||
|
||||
await indexModule.postProxySessionsToInfluxdb(userSessions);
|
||||
|
||||
expect(mockFactory.postProxySessionsToInfluxdb).toHaveBeenCalledWith(userSessions);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postButlerSOSMemoryUsageToInfluxdb', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
const memory = { heap_used: 100, heap_total: 200 };
|
||||
|
||||
await indexModule.postButlerSOSMemoryUsageToInfluxdb(memory);
|
||||
|
||||
expect(mockFactory.postButlerSOSMemoryUsageToInfluxdb).toHaveBeenCalledWith(memory);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postUserEventToInfluxdb', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
const msg = { host: 'test-host', command: 'OpenApp' };
|
||||
|
||||
await indexModule.postUserEventToInfluxdb(msg);
|
||||
|
||||
expect(mockFactory.postUserEventToInfluxdb).toHaveBeenCalledWith(msg);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postLogEventToInfluxdb', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
const msg = { host: 'test-host', source: 'qseow-engine' };
|
||||
|
||||
await indexModule.postLogEventToInfluxdb(msg);
|
||||
|
||||
expect(mockFactory.postLogEventToInfluxdb).toHaveBeenCalledWith(msg);
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeEventCountInfluxDB', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
await indexModule.storeEventCountInfluxDB('midnight', 'hour');
|
||||
|
||||
expect(mockFactory.storeEventCountInfluxDB).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should ignore deprecated parameters', async () => {
|
||||
await indexModule.storeEventCountInfluxDB('deprecated1', 'deprecated2');
|
||||
|
||||
expect(mockFactory.storeEventCountInfluxDB).toHaveBeenCalledWith();
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeRejectedEventCountInfluxDB', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
await indexModule.storeRejectedEventCountInfluxDB('midnight', 'hour');
|
||||
|
||||
expect(mockFactory.storeRejectedEventCountInfluxDB).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should ignore deprecated parameters', async () => {
|
||||
await indexModule.storeRejectedEventCountInfluxDB({ data: 'old' }, { data: 'old2' });
|
||||
|
||||
expect(mockFactory.storeRejectedEventCountInfluxDB).toHaveBeenCalledWith();
|
||||
});
|
||||
});
|
||||
|
||||
describe('postUserEventQueueMetricsToInfluxdb', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
await indexModule.postUserEventQueueMetricsToInfluxdb({ some: 'data' });
|
||||
|
||||
expect(mockFactory.postUserEventQueueMetricsToInfluxdb).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should ignore deprecated parameter', async () => {
|
||||
await indexModule.postUserEventQueueMetricsToInfluxdb({ old: 'metrics' });
|
||||
|
||||
expect(mockFactory.postUserEventQueueMetricsToInfluxdb).toHaveBeenCalledWith();
|
||||
});
|
||||
});
|
||||
|
||||
describe('postLogEventQueueMetricsToInfluxdb', () => {
|
||||
test('should delegate to factory', async () => {
|
||||
await indexModule.postLogEventQueueMetricsToInfluxdb({ some: 'data' });
|
||||
|
||||
expect(mockFactory.postLogEventQueueMetricsToInfluxdb).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should ignore deprecated parameter', async () => {
|
||||
await indexModule.postLogEventQueueMetricsToInfluxdb({ old: 'metrics' });
|
||||
|
||||
expect(mockFactory.postLogEventQueueMetricsToInfluxdb).toHaveBeenCalledWith();
|
||||
});
|
||||
});
|
||||
|
||||
describe('setupUdpQueueMetricsStorage', () => {
|
||||
let intervalSpy;
|
||||
|
||||
beforeEach(() => {
|
||||
intervalSpy = jest.spyOn(global, 'setInterval');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
intervalSpy.mockRestore();
|
||||
});
|
||||
|
||||
test('should return empty interval IDs when InfluxDB is disabled', () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('influxdbConfig.enable')) return false;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
const result = indexModule.setupUdpQueueMetricsStorage();
|
||||
|
||||
expect(result).toEqual({
|
||||
userEvents: null,
|
||||
logEvents: null,
|
||||
});
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('InfluxDB is disabled')
|
||||
);
|
||||
});
|
||||
|
||||
test('should setup user event queue metrics when enabled', () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('influxdbConfig.enable')) return true;
|
||||
if (path.includes('userEvents.udpServerConfig.queueMetrics.influxdb.enable'))
|
||||
return true;
|
||||
if (
|
||||
path.includes('userEvents.udpServerConfig.queueMetrics.influxdb.writeFrequency')
|
||||
)
|
||||
return 60000;
|
||||
if (path.includes('logEvents.udpServerConfig.queueMetrics.influxdb.enable'))
|
||||
return false;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
const result = indexModule.setupUdpQueueMetricsStorage();
|
||||
|
||||
expect(result.userEvents).not.toBeNull();
|
||||
expect(intervalSpy).toHaveBeenCalledWith(expect.any(Function), 60000);
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('user event queue metrics')
|
||||
);
|
||||
});
|
||||
|
||||
test('should setup log event queue metrics when enabled', () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('influxdbConfig.enable')) return true;
|
||||
if (path.includes('userEvents.udpServerConfig.queueMetrics.influxdb.enable'))
|
||||
return false;
|
||||
if (path.includes('logEvents.udpServerConfig.queueMetrics.influxdb.enable'))
|
||||
return true;
|
||||
if (path.includes('logEvents.udpServerConfig.queueMetrics.influxdb.writeFrequency'))
|
||||
return 30000;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
const result = indexModule.setupUdpQueueMetricsStorage();
|
||||
|
||||
expect(result.logEvents).not.toBeNull();
|
||||
expect(intervalSpy).toHaveBeenCalledWith(expect.any(Function), 30000);
|
||||
});
|
||||
|
||||
test('should setup both metrics when both enabled', () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('influxdbConfig.enable')) return true;
|
||||
if (path.includes('userEvents.udpServerConfig.queueMetrics.influxdb.enable'))
|
||||
return true;
|
||||
if (
|
||||
path.includes('userEvents.udpServerConfig.queueMetrics.influxdb.writeFrequency')
|
||||
)
|
||||
return 45000;
|
||||
if (path.includes('logEvents.udpServerConfig.queueMetrics.influxdb.enable'))
|
||||
return true;
|
||||
if (path.includes('logEvents.udpServerConfig.queueMetrics.influxdb.writeFrequency'))
|
||||
return 55000;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
const result = indexModule.setupUdpQueueMetricsStorage();
|
||||
|
||||
expect(result.userEvents).not.toBeNull();
|
||||
expect(result.logEvents).not.toBeNull();
|
||||
expect(intervalSpy).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
test('should log when metrics are disabled', () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('influxdbConfig.enable')) return true;
|
||||
if (path.includes('queueMetrics.influxdb.enable')) return false;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
indexModule.setupUdpQueueMetricsStorage();
|
||||
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('User event queue metrics storage to InfluxDB is disabled')
|
||||
);
|
||||
expect(globals.logger.info).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Log event queue metrics storage to InfluxDB is disabled')
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
542
src/lib/influxdb/__tests__/shared-utils.test.js
Normal file
542
src/lib/influxdb/__tests__/shared-utils.test.js
Normal file
@@ -0,0 +1,542 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: null,
|
||||
appNames: [],
|
||||
getErrorMessage: jest.fn((err) => err?.message || String(err)),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
describe('Shared Utils - getFormattedTime', () => {
|
||||
let utils;
|
||||
let globals;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
});
|
||||
|
||||
test('should return empty string for null input', () => {
|
||||
const result = utils.getFormattedTime(null);
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
test('should return empty string for undefined input', () => {
|
||||
const result = utils.getFormattedTime(undefined);
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
test('should return empty string for empty string input', () => {
|
||||
const result = utils.getFormattedTime('');
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
test('should return empty string for non-string input', () => {
|
||||
const result = utils.getFormattedTime(12345);
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
test('should return empty string for string shorter than minimum length', () => {
|
||||
const result = utils.getFormattedTime('20240101T12');
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
test('should return empty string for invalid date components', () => {
|
||||
const result = utils.getFormattedTime('abcdXXXXTxxxxxx');
|
||||
expect(result).toBe('');
|
||||
});
|
||||
|
||||
test('should handle invalid date gracefully', () => {
|
||||
// JavaScript Date constructor is lenient and converts Month 13 to January of next year
|
||||
// So this doesn't actually fail - it's a valid date to JS
|
||||
const result = utils.getFormattedTime('20241301T250000');
|
||||
|
||||
// The function doesn't validate date ranges, so this will return a formatted time
|
||||
expect(typeof result).toBe('string');
|
||||
});
|
||||
|
||||
test('should format valid timestamp correctly', () => {
|
||||
// Mock Date.now to return a known value
|
||||
const mockNow = new Date('2024-01-01T13:00:00').getTime();
|
||||
jest.spyOn(Date, 'now').mockReturnValue(mockNow);
|
||||
|
||||
const result = utils.getFormattedTime('20240101T120000');
|
||||
|
||||
// Should show approximately 1 hour difference
|
||||
expect(result).toMatch(/\d+ days, \d+h \d+m \d+s/);
|
||||
|
||||
Date.now.mockRestore();
|
||||
});
|
||||
|
||||
test('should handle timestamps with exact minimum length', () => {
|
||||
const mockNow = new Date('2024-01-01T13:00:00').getTime();
|
||||
jest.spyOn(Date, 'now').mockReturnValue(mockNow);
|
||||
|
||||
const result = utils.getFormattedTime('20240101T120000');
|
||||
|
||||
expect(result).not.toBe('');
|
||||
expect(result).toMatch(/\d+ days/);
|
||||
|
||||
Date.now.mockRestore();
|
||||
});
|
||||
|
||||
test('should handle future timestamps', () => {
|
||||
const mockNow = new Date('2024-01-01T12:00:00').getTime();
|
||||
jest.spyOn(Date, 'now').mockReturnValue(mockNow);
|
||||
|
||||
// Server started in the future (edge case)
|
||||
const result = utils.getFormattedTime('20250101T120000');
|
||||
|
||||
// Result might be negative or weird, but shouldn't crash
|
||||
expect(typeof result).toBe('string');
|
||||
|
||||
Date.now.mockRestore();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Shared Utils - processAppDocuments', () => {
|
||||
let utils;
|
||||
let globals;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
|
||||
globals.appNames = [
|
||||
{ id: 'app-123', name: 'Sales Dashboard' },
|
||||
{ id: 'app-456', name: 'HR Analytics' },
|
||||
{ id: 'app-789', name: 'Finance Report' },
|
||||
];
|
||||
});
|
||||
|
||||
test('should process empty array', async () => {
|
||||
const result = await utils.processAppDocuments([], 'TEST', 'active');
|
||||
|
||||
expect(result).toEqual({
|
||||
appNames: [],
|
||||
sessionAppNames: [],
|
||||
});
|
||||
});
|
||||
|
||||
test('should identify session apps correctly', async () => {
|
||||
const docIDs = ['SessionApp_12345', 'SessionApp_67890'];
|
||||
|
||||
const result = await utils.processAppDocuments(docIDs, 'TEST', 'active');
|
||||
|
||||
expect(result.sessionAppNames).toEqual(['SessionApp_12345', 'SessionApp_67890']);
|
||||
expect(result.appNames).toEqual([]);
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Session app is active')
|
||||
);
|
||||
});
|
||||
|
||||
test('should resolve app IDs to names', async () => {
|
||||
const docIDs = ['app-123', 'app-456'];
|
||||
|
||||
const result = await utils.processAppDocuments(docIDs, 'TEST', 'loaded');
|
||||
|
||||
expect(result.appNames).toEqual(['HR Analytics', 'Sales Dashboard']);
|
||||
expect(result.sessionAppNames).toEqual([]);
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('App is loaded: Sales Dashboard')
|
||||
);
|
||||
});
|
||||
|
||||
test('should use doc ID when app name not found', async () => {
|
||||
const docIDs = ['app-unknown', 'app-123'];
|
||||
|
||||
const result = await utils.processAppDocuments(docIDs, 'TEST', 'in memory');
|
||||
|
||||
expect(result.appNames).toEqual(['Sales Dashboard', 'app-unknown']);
|
||||
expect(result.sessionAppNames).toEqual([]);
|
||||
});
|
||||
|
||||
test('should mix session apps and regular apps', async () => {
|
||||
const docIDs = ['app-123', 'SessionApp_abc', 'app-456', 'SessionApp_def', 'app-unknown'];
|
||||
|
||||
const result = await utils.processAppDocuments(docIDs, 'TEST', 'active');
|
||||
|
||||
expect(result.appNames).toEqual(['HR Analytics', 'Sales Dashboard', 'app-unknown']);
|
||||
expect(result.sessionAppNames).toEqual(['SessionApp_abc', 'SessionApp_def']);
|
||||
});
|
||||
|
||||
test('should sort both arrays alphabetically', async () => {
|
||||
const docIDs = ['app-789', 'app-123', 'app-456', 'SessionApp_z', 'SessionApp_a'];
|
||||
|
||||
const result = await utils.processAppDocuments(docIDs, 'TEST', 'active');
|
||||
|
||||
expect(result.appNames).toEqual(['Finance Report', 'HR Analytics', 'Sales Dashboard']);
|
||||
expect(result.sessionAppNames).toEqual(['SessionApp_a', 'SessionApp_z']);
|
||||
});
|
||||
|
||||
test('should handle session app prefix at start only', async () => {
|
||||
const docIDs = ['SessionApp_test', 'NotSessionApp_test', 'app-123'];
|
||||
|
||||
const result = await utils.processAppDocuments(docIDs, 'TEST', 'active');
|
||||
|
||||
expect(result.sessionAppNames).toEqual(['SessionApp_test']);
|
||||
expect(result.appNames).toEqual(['NotSessionApp_test', 'Sales Dashboard']);
|
||||
});
|
||||
|
||||
test('should handle single document', async () => {
|
||||
const docIDs = ['app-456'];
|
||||
|
||||
const result = await utils.processAppDocuments(docIDs, 'TEST', 'active');
|
||||
|
||||
expect(result.appNames).toEqual(['HR Analytics']);
|
||||
expect(result.sessionAppNames).toEqual([]);
|
||||
});
|
||||
|
||||
test('should handle many documents efficiently', async () => {
|
||||
const docIDs = Array.from({ length: 100 }, (_, i) =>
|
||||
i % 2 === 0 ? `SessionApp_${i}` : `app-${i}`
|
||||
);
|
||||
|
||||
const result = await utils.processAppDocuments(docIDs, 'TEST', 'active');
|
||||
|
||||
expect(result.sessionAppNames.length).toBe(50);
|
||||
expect(result.appNames.length).toBe(50);
|
||||
// Arrays are sorted alphabetically
|
||||
expect(result.sessionAppNames).toEqual(expect.arrayContaining(['SessionApp_0']));
|
||||
expect(result.appNames).toEqual(expect.arrayContaining(['app-1']));
|
||||
});
|
||||
});
|
||||
|
||||
describe('Shared Utils - applyTagsToPoint3', () => {
|
||||
let utils;
|
||||
let mockPoint;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
utils = await import('../shared/utils.js');
|
||||
|
||||
mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
};
|
||||
});
|
||||
|
||||
test('should return point unchanged for null tags', () => {
|
||||
const result = utils.applyTagsToPoint3(mockPoint, null);
|
||||
|
||||
expect(result).toBe(mockPoint);
|
||||
expect(mockPoint.setTag).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return point unchanged for undefined tags', () => {
|
||||
const result = utils.applyTagsToPoint3(mockPoint, undefined);
|
||||
|
||||
expect(result).toBe(mockPoint);
|
||||
expect(mockPoint.setTag).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return point unchanged for non-object tags', () => {
|
||||
const result = utils.applyTagsToPoint3(mockPoint, 'not-an-object');
|
||||
|
||||
expect(result).toBe(mockPoint);
|
||||
expect(mockPoint.setTag).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should apply single tag', () => {
|
||||
const tags = { env: 'production' };
|
||||
|
||||
const result = utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(result).toBe(mockPoint);
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('env', 'production');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('should apply multiple tags', () => {
|
||||
const tags = {
|
||||
env: 'production',
|
||||
region: 'us-east-1',
|
||||
service: 'qlik-sense',
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledTimes(3);
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('env', 'production');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('region', 'us-east-1');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('service', 'qlik-sense');
|
||||
});
|
||||
|
||||
test('should convert non-string values to strings', () => {
|
||||
const tags = {
|
||||
count: 42,
|
||||
enabled: true,
|
||||
version: 3.14,
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('count', '42');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('enabled', 'true');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('version', '3.14');
|
||||
});
|
||||
|
||||
test('should skip null values', () => {
|
||||
const tags = {
|
||||
env: 'production',
|
||||
region: null,
|
||||
service: 'qlik-sense',
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledTimes(2);
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('env', 'production');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('service', 'qlik-sense');
|
||||
expect(mockPoint.setTag).not.toHaveBeenCalledWith('region', expect.anything());
|
||||
});
|
||||
|
||||
test('should skip undefined values', () => {
|
||||
const tags = {
|
||||
env: 'production',
|
||||
region: undefined,
|
||||
service: 'qlik-sense',
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledTimes(2);
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('env', 'production');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('service', 'qlik-sense');
|
||||
});
|
||||
|
||||
test('should handle empty object', () => {
|
||||
const tags = {};
|
||||
|
||||
const result = utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(result).toBe(mockPoint);
|
||||
expect(mockPoint.setTag).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle tags with special characters', () => {
|
||||
const tags = {
|
||||
'tag-with-dash': 'value',
|
||||
tag_with_underscore: 'value2',
|
||||
'tag.with.dot': 'value3',
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledTimes(3);
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('tag-with-dash', 'value');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('tag_with_underscore', 'value2');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('tag.with.dot', 'value3');
|
||||
});
|
||||
|
||||
test('should handle empty string values', () => {
|
||||
const tags = {
|
||||
env: '',
|
||||
region: 'us-east-1',
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('env', '');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('region', 'us-east-1');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Shared Utils - chunkArray', () => {
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
utils = await import('../shared/utils.js');
|
||||
});
|
||||
|
||||
test('should split array into chunks of specified size', () => {
|
||||
const array = [1, 2, 3, 4, 5, 6, 7];
|
||||
const result = utils.chunkArray(array, 3);
|
||||
|
||||
expect(result).toEqual([[1, 2, 3], [4, 5, 6], [7]]);
|
||||
});
|
||||
|
||||
test('should handle empty array', () => {
|
||||
const result = utils.chunkArray([], 5);
|
||||
expect(result).toEqual([]);
|
||||
});
|
||||
|
||||
test('should handle chunk size larger than array', () => {
|
||||
const array = [1, 2, 3];
|
||||
const result = utils.chunkArray(array, 10);
|
||||
|
||||
expect(result).toEqual([[1, 2, 3]]);
|
||||
});
|
||||
|
||||
test('should handle chunk size of 1', () => {
|
||||
const array = [1, 2, 3];
|
||||
const result = utils.chunkArray(array, 1);
|
||||
|
||||
expect(result).toEqual([[1], [2], [3]]);
|
||||
});
|
||||
|
||||
test('should handle array length exactly divisible by chunk size', () => {
|
||||
const array = [1, 2, 3, 4, 5, 6];
|
||||
const result = utils.chunkArray(array, 2);
|
||||
|
||||
expect(result).toEqual([
|
||||
[1, 2],
|
||||
[3, 4],
|
||||
[5, 6],
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Shared Utils - validateUnsignedField', () => {
|
||||
let utils;
|
||||
let globals;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
});
|
||||
|
||||
test('should return value unchanged for positive number', () => {
|
||||
const result = utils.validateUnsignedField(42, 'measurement', 'field', 'server1');
|
||||
expect(result).toBe(42);
|
||||
expect(globals.logger.warn).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return 0 for zero', () => {
|
||||
const result = utils.validateUnsignedField(0, 'measurement', 'field', 'server1');
|
||||
expect(result).toBe(0);
|
||||
expect(globals.logger.warn).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should clamp negative number to 0 and warn', () => {
|
||||
const result = utils.validateUnsignedField(-5, 'cache', 'hits', 'server1');
|
||||
|
||||
expect(result).toBe(0);
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Negative value detected')
|
||||
);
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('measurement=cache')
|
||||
);
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(expect.stringContaining('field=hits'));
|
||||
});
|
||||
|
||||
test('should warn once per measurement per invocation', () => {
|
||||
// First call should warn
|
||||
utils.validateUnsignedField(-1, 'test_m', 'field1', 'server1');
|
||||
expect(globals.logger.warn).toHaveBeenCalledTimes(1);
|
||||
|
||||
// Second call with same measurement should not warn again in same batch
|
||||
utils.validateUnsignedField(-2, 'test_m', 'field2', 'server1');
|
||||
expect(globals.logger.warn).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('should handle null/undefined gracefully', () => {
|
||||
const resultNull = utils.validateUnsignedField(null, 'measurement', 'field', 'server1');
|
||||
const resultUndef = utils.validateUnsignedField(
|
||||
undefined,
|
||||
'measurement',
|
||||
'field',
|
||||
'server1'
|
||||
);
|
||||
|
||||
expect(resultNull).toBe(0);
|
||||
expect(resultUndef).toBe(0);
|
||||
});
|
||||
|
||||
test('should handle string numbers', () => {
|
||||
const result = utils.validateUnsignedField('42', 'measurement', 'field', 'server1');
|
||||
expect(result).toBe(42);
|
||||
});
|
||||
|
||||
test('should handle negative string numbers', () => {
|
||||
const result = utils.validateUnsignedField('-10', 'measurement', 'field', 'server1');
|
||||
expect(result).toBe(0);
|
||||
expect(globals.logger.warn).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Shared Utils - writeBatchToInfluxV1', () => {
|
||||
let utils;
|
||||
let globals;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
|
||||
globals.influx = {
|
||||
writePoints: jest.fn().mockResolvedValue(undefined),
|
||||
};
|
||||
globals.config.get.mockReturnValue(1000); // maxBatchSize
|
||||
|
||||
utils = await import('../shared/utils.js');
|
||||
});
|
||||
|
||||
test('should write small batch in single call', async () => {
|
||||
const points = [
|
||||
{ measurement: 'test', fields: { value: 1 } },
|
||||
{ measurement: 'test', fields: { value: 2 } },
|
||||
];
|
||||
|
||||
await utils.writeBatchToInfluxV1(points, 'test_data', 'server1', 1000);
|
||||
|
||||
expect(globals.influx.writePoints).toHaveBeenCalledTimes(1);
|
||||
expect(globals.influx.writePoints).toHaveBeenCalledWith(points);
|
||||
});
|
||||
|
||||
test('should chunk large batch', async () => {
|
||||
const points = Array.from({ length: 2500 }, (_, i) => ({
|
||||
measurement: 'test',
|
||||
fields: { value: i },
|
||||
}));
|
||||
|
||||
await utils.writeBatchToInfluxV1(points, 'test_data', 'server1', 1000);
|
||||
|
||||
// Should be called 3 times: 1000 + 1000 + 500
|
||||
expect(globals.influx.writePoints).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
test('should retry with progressive chunking on failure', async () => {
|
||||
const points = Array.from({ length: 1000 }, (_, i) => ({
|
||||
measurement: 'test',
|
||||
fields: { value: i },
|
||||
}));
|
||||
|
||||
// First attempt with batch size 1000 fails, retry with 500 succeeds
|
||||
globals.influx.writePoints
|
||||
.mockRejectedValueOnce(new Error('Batch too large'))
|
||||
.mockResolvedValue(undefined);
|
||||
|
||||
await utils.writeBatchToInfluxV1(points, 'test_data', 'server1', 1000);
|
||||
|
||||
// First call with 1000 points fails, then 2 calls with 500 each succeed
|
||||
expect(globals.influx.writePoints).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
test('should handle empty array', async () => {
|
||||
await utils.writeBatchToInfluxV1([], 'test_data', 'server1', 1000);
|
||||
|
||||
expect(globals.influx.writePoints).not.toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining('No points to write')
|
||||
);
|
||||
});
|
||||
});
|
||||
195
src/lib/influxdb/__tests__/v1-butler-memory.test.js
Normal file
195
src/lib/influxdb/__tests__/v1-butler-memory.test.js
Normal file
@@ -0,0 +1,195 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
writePoints: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
appVersion: '1.0.0',
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV1: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v1/butler-memory', () => {
|
||||
let storeButlerMemoryV1;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const butlerMemory = await import('../v1/butler-memory.js');
|
||||
storeButlerMemoryV1 = butlerMemory.storeButlerMemoryV1;
|
||||
|
||||
// Setup default mocks
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeBatchToInfluxV1.mockResolvedValue();
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.maxBatchSize') return 100;
|
||||
return undefined;
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeButlerMemoryV1', () => {
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
|
||||
const memory = {
|
||||
instanceTag: 'prod-instance',
|
||||
heapUsedMByte: 100,
|
||||
heapTotalMByte: 200,
|
||||
externalMemoryMByte: 50,
|
||||
processMemoryMByte: 250,
|
||||
};
|
||||
|
||||
await storeButlerMemoryV1(memory);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('MEMORY USAGE V1')
|
||||
);
|
||||
});
|
||||
|
||||
test('should successfully write memory usage metrics', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'prod-instance',
|
||||
heapUsedMByte: 100.5,
|
||||
heapTotalMByte: 200.75,
|
||||
externalMemoryMByte: 50.25,
|
||||
processMemoryMByte: 250.5,
|
||||
};
|
||||
|
||||
await storeButlerMemoryV1(memory);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'Memory usage metrics',
|
||||
'INFLUXDB_V1_WRITE',
|
||||
100
|
||||
);
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'MEMORY USAGE V1: Sent Butler SOS memory usage data to InfluxDB'
|
||||
);
|
||||
});
|
||||
|
||||
test('should create correct datapoint structure', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'test-instance',
|
||||
heapUsedMByte: 150.5,
|
||||
heapTotalMByte: 300.75,
|
||||
externalMemoryMByte: 75.25,
|
||||
processMemoryMByte: 350.5,
|
||||
};
|
||||
|
||||
utils.writeBatchToInfluxV1.mockImplementation(async (writeFn) => {
|
||||
// writeFn is the batch array in the new implementation
|
||||
// But wait, mockImplementation receives the arguments passed to the function.
|
||||
// writeBatchToInfluxV1(batch, logMessage, instanceTag, batchSize)
|
||||
// So the first argument is the batch.
|
||||
// The test expects globals.influx.writePoints to be called.
|
||||
// But writeBatchToInfluxV1 calls globals.influx.writePoints internally.
|
||||
// If we mock writeBatchToInfluxV1, we bypass the internal call.
|
||||
// So we should NOT mock implementation if we want to test the datapoint structure via globals.influx.writePoints?
|
||||
// Or we should inspect the batch passed to writeBatchToInfluxV1.
|
||||
});
|
||||
|
||||
// The original test was:
|
||||
// utils.writeToInfluxWithRetry.mockImplementation(async (writeFn) => {
|
||||
// await writeFn();
|
||||
// });
|
||||
// Because writeToInfluxWithRetry took a function that generated points and wrote them.
|
||||
|
||||
// Now writeBatchToInfluxV1 takes the points directly.
|
||||
// So we can just inspect the arguments of writeBatchToInfluxV1.
|
||||
|
||||
await storeButlerMemoryV1(memory);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
measurement: 'butlersos_memory_usage',
|
||||
tags: {
|
||||
butler_sos_instance: 'test-instance',
|
||||
version: '1.0.0',
|
||||
},
|
||||
fields: {
|
||||
heap_used: 150.5,
|
||||
heap_total: 300.75,
|
||||
external: 75.25,
|
||||
process_memory: 350.5,
|
||||
},
|
||||
})
|
||||
]),
|
||||
expect.any(String),
|
||||
expect.any(String),
|
||||
expect.any(Number)
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle write errors and rethrow', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'prod-instance',
|
||||
heapUsedMByte: 100,
|
||||
heapTotalMByte: 200,
|
||||
externalMemoryMByte: 50,
|
||||
processMemoryMByte: 250,
|
||||
};
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(writeError);
|
||||
|
||||
await expect(storeButlerMemoryV1(memory)).rejects.toThrow('Write failed');
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error saving Butler SOS memory data')
|
||||
);
|
||||
});
|
||||
|
||||
test('should log debug and silly messages', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'debug-instance',
|
||||
heapUsedMByte: 100,
|
||||
heapTotalMByte: 200,
|
||||
externalMemoryMByte: 50,
|
||||
processMemoryMByte: 250,
|
||||
};
|
||||
|
||||
await storeButlerMemoryV1(memory);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('MEMORY USAGE V1: Memory usage')
|
||||
);
|
||||
expect(globals.logger.silly).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Influxdb datapoint for Butler SOS memory usage')
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
210
src/lib/influxdb/__tests__/v1-event-counts.test.js
Normal file
210
src/lib/influxdb/__tests__/v1-event-counts.test.js
Normal file
@@ -0,0 +1,210 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { writePoints: jest.fn() },
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
udpEvents: { getLogEvents: jest.fn(), getUserEvents: jest.fn() },
|
||||
rejectedEvents: { getRejectedLogEvents: jest.fn() },
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV1: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v1/event-counts', () => {
|
||||
let storeEventCountV1, storeRejectedEventCountV1, globals, utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const eventCounts = await import('../v1/event-counts.js');
|
||||
storeEventCountV1 = eventCounts.storeEventCountV1;
|
||||
storeRejectedEventCountV1 = eventCounts.storeRejectedEventCountV1;
|
||||
|
||||
globals.config.has.mockReturnValue(true);
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('measurementName')) return 'event_counts';
|
||||
if (path.includes('tags')) return [{ name: 'env', value: 'prod' }];
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([
|
||||
{ eventType: 'log', eventAction: 'action' },
|
||||
]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([
|
||||
{ eventType: 'user', eventAction: 'action' },
|
||||
]);
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([
|
||||
{ eventType: 'rejected', reason: 'validation' },
|
||||
]);
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockResolvedValue();
|
||||
utils.writeBatchToInfluxV1.mockResolvedValue();
|
||||
});
|
||||
|
||||
test('should return early when no events', async () => {
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([]);
|
||||
await storeEventCountV1();
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write event counts', async () => {
|
||||
await storeEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'Event counts',
|
||||
'',
|
||||
100
|
||||
);
|
||||
});
|
||||
|
||||
test('should apply config tags to log events', async () => {
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([
|
||||
{ source: 'qseow-engine', host: 'host1', subsystem: 'System', counter: 5 },
|
||||
{ source: 'qseow-proxy', host: 'host2', subsystem: 'Proxy', counter: 10 },
|
||||
]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([]);
|
||||
await storeEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should apply config tags to user events', async () => {
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([
|
||||
{ source: 'qseow-engine', host: 'host1', subsystem: 'User', counter: 3 },
|
||||
{ source: 'qseow-proxy', host: 'host2', subsystem: 'Session', counter: 7 },
|
||||
]);
|
||||
await storeEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle mixed log and user events', async () => {
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([
|
||||
{ source: 'qseow-engine', host: 'host1', subsystem: 'System', counter: 5 },
|
||||
]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([
|
||||
{ source: 'qseow-proxy', host: 'host2', subsystem: 'User', counter: 3 },
|
||||
]);
|
||||
await storeEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'Event counts',
|
||||
'',
|
||||
100
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(new Error('Write failed'));
|
||||
await expect(storeEventCountV1()).rejects.toThrow();
|
||||
expect(globals.logger.error).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write rejected event counts', async () => {
|
||||
await storeRejectedEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when no rejected events', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([]);
|
||||
await storeRejectedEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled for rejected events', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([
|
||||
{ source: 'test', counter: 1 },
|
||||
]);
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeRejectedEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle rejected qix-perf events with appName', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([
|
||||
{
|
||||
source: 'qseow-qix-perf',
|
||||
appId: 'app123',
|
||||
appName: 'MyApp',
|
||||
method: 'GetLayout',
|
||||
objectType: 'sheet',
|
||||
counter: 5,
|
||||
processTime: 150,
|
||||
},
|
||||
]);
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('measurementName')) return 'rejected_events';
|
||||
if (path.includes('trackRejectedEvents.tags')) return [{ name: 'env', value: 'test' }];
|
||||
return null;
|
||||
});
|
||||
await storeRejectedEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle rejected qix-perf events without appName', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([
|
||||
{
|
||||
source: 'qseow-qix-perf',
|
||||
appId: 'app123',
|
||||
appName: '',
|
||||
method: 'GetLayout',
|
||||
objectType: 'sheet',
|
||||
counter: 5,
|
||||
processTime: 150,
|
||||
},
|
||||
]);
|
||||
globals.config.has.mockReturnValue(false);
|
||||
await storeRejectedEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle rejected non-qix-perf events', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([
|
||||
{
|
||||
source: 'other-source',
|
||||
eventType: 'rejected',
|
||||
reason: 'validation',
|
||||
counter: 3,
|
||||
},
|
||||
]);
|
||||
await storeRejectedEventCountV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle rejected events write errors', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([
|
||||
{ source: 'test', counter: 1 },
|
||||
]);
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(new Error('Write failed'));
|
||||
await expect(storeRejectedEventCountV1()).rejects.toThrow();
|
||||
expect(globals.logger.error).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
231
src/lib/influxdb/__tests__/v1-health-metrics.test.js
Normal file
231
src/lib/influxdb/__tests__/v1-health-metrics.test.js
Normal file
@@ -0,0 +1,231 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { writePoints: jest.fn() },
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
hostInfo: { hostname: 'test-host' },
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV1: jest.fn(),
|
||||
processAppDocuments: jest.fn(),
|
||||
getFormattedTime: jest.fn(() => '2024-01-01T00:00:00Z'),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v1/health-metrics', () => {
|
||||
let storeHealthMetricsV1, globals, utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const healthMetrics = await import('../v1/health-metrics.js');
|
||||
storeHealthMetricsV1 = healthMetrics.storeHealthMetricsV1;
|
||||
|
||||
globals.config.has.mockReturnValue(true);
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('measurementName')) return 'health_metrics';
|
||||
if (path.includes('tags')) return [{ name: 'env', value: 'prod' }];
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockResolvedValue();
|
||||
utils.writeBatchToInfluxV1.mockResolvedValue();
|
||||
utils.processAppDocuments.mockResolvedValue({ appNames: [], sessionAppNames: [] });
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
const body = { mem: {}, apps: {}, cpu: {}, session: {}, users: {}, cache: {} };
|
||||
await storeHealthMetricsV1({ server: 'server1' }, body);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write complete health metrics', async () => {
|
||||
const body = {
|
||||
mem: { committed: 1000, allocated: 800, free: 200 },
|
||||
apps: {
|
||||
active_docs: [{ id: 'app1', name: 'App 1' }],
|
||||
loaded_docs: [{ id: 'app2', name: 'App 2' }],
|
||||
in_memory_docs: [{ id: 'app3', name: 'App 3' }],
|
||||
calls: 10,
|
||||
selections: 5,
|
||||
},
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5, total: 10 },
|
||||
users: { active: 3, total: 8 },
|
||||
cache: { hits: 100, lookups: 120, added: 20, replaced: 5, bytes_added: 1024 },
|
||||
saturated: false,
|
||||
version: '1.0.0',
|
||||
started: '2024-01-01T00:00:00Z',
|
||||
};
|
||||
const serverTags = { server_name: 'server1', server_description: 'Test server' };
|
||||
|
||||
await storeHealthMetricsV1(serverTags, body);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'Health metrics for server1',
|
||||
'server1',
|
||||
100
|
||||
);
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(new Error('Write failed'));
|
||||
const body = {
|
||||
mem: {},
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [] },
|
||||
cpu: {},
|
||||
session: {},
|
||||
users: {},
|
||||
cache: {},
|
||||
};
|
||||
await expect(storeHealthMetricsV1({ server_name: 'server1' }, body)).rejects.toThrow();
|
||||
expect(globals.logger.error).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should process app documents', async () => {
|
||||
const body = {
|
||||
mem: {},
|
||||
apps: {
|
||||
active_docs: [{ id: 'doc1', name: 'Doc 1' }],
|
||||
loaded_docs: [{ id: 'doc2', name: 'Doc 2' }],
|
||||
in_memory_docs: [{ id: 'doc3', name: 'Doc 3' }],
|
||||
},
|
||||
cpu: {},
|
||||
session: {},
|
||||
users: {},
|
||||
cache: {},
|
||||
version: '1.0.0',
|
||||
started: '2024-01-01T00:00:00Z',
|
||||
};
|
||||
await storeHealthMetricsV1({ server_name: 'server1' }, body);
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
test('should handle config with activeDocs enabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('measurementName')) return 'health_metrics';
|
||||
if (path.includes('tags')) return [{ name: 'env', value: 'prod' }];
|
||||
if (path.includes('includeFields.activeDocs')) return true;
|
||||
if (path.includes('enableAppNameExtract')) return true;
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return undefined;
|
||||
});
|
||||
utils.processAppDocuments.mockResolvedValue({
|
||||
appNames: ['App1', 'App2'],
|
||||
sessionAppNames: ['Session1'],
|
||||
});
|
||||
const body = {
|
||||
mem: { committed: 1000 },
|
||||
apps: { active_docs: [{ id: 'app1' }], loaded_docs: [], in_memory_docs: [] },
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5 },
|
||||
users: { active: 3 },
|
||||
cache: { hits: 100 },
|
||||
version: '1.0.0',
|
||||
started: '2024-01-01T00:00:00Z',
|
||||
};
|
||||
await storeHealthMetricsV1({ server_name: 'server1' }, body);
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle config with loadedDocs enabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('measurementName')) return 'health_metrics';
|
||||
if (path.includes('tags')) return [{ name: 'env', value: 'prod' }];
|
||||
if (path.includes('includeFields.loadedDocs')) return true;
|
||||
if (path.includes('enableAppNameExtract')) return true;
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return undefined;
|
||||
});
|
||||
utils.processAppDocuments.mockResolvedValue({
|
||||
appNames: ['LoadedApp'],
|
||||
sessionAppNames: ['LoadedSession'],
|
||||
});
|
||||
const body = {
|
||||
mem: { committed: 1000 },
|
||||
apps: { active_docs: [], loaded_docs: [{ id: 'app2' }], in_memory_docs: [] },
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5 },
|
||||
users: { active: 3 },
|
||||
cache: { hits: 100 },
|
||||
version: '1.0.0',
|
||||
started: '2024-01-01T00:00:00Z',
|
||||
};
|
||||
await storeHealthMetricsV1({ server_name: 'server1' }, body);
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle config with inMemoryDocs enabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('measurementName')) return 'health_metrics';
|
||||
if (path.includes('tags')) return [{ name: 'env', value: 'prod' }];
|
||||
if (path.includes('includeFields.inMemoryDocs')) return true;
|
||||
if (path.includes('enableAppNameExtract')) return true;
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return undefined;
|
||||
});
|
||||
utils.processAppDocuments.mockResolvedValue({
|
||||
appNames: ['MemoryApp'],
|
||||
sessionAppNames: ['MemorySession'],
|
||||
});
|
||||
const body = {
|
||||
mem: { committed: 1000 },
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [{ id: 'app3' }] },
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5 },
|
||||
users: { active: 3 },
|
||||
cache: { hits: 100 },
|
||||
version: '1.0.0',
|
||||
started: '2024-01-01T00:00:00Z',
|
||||
};
|
||||
await storeHealthMetricsV1({ server_name: 'server1' }, body);
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle config with all doc types disabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('measurementName')) return 'health_metrics';
|
||||
if (path.includes('tags')) return [];
|
||||
if (path.includes('includeFields')) return false;
|
||||
if (path.includes('enableAppNameExtract')) return false;
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return undefined;
|
||||
});
|
||||
const body = {
|
||||
mem: { committed: 1000 },
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [] },
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5 },
|
||||
users: { active: 3 },
|
||||
cache: { hits: 100 },
|
||||
version: '1.0.0',
|
||||
started: '2024-01-01T00:00:00Z',
|
||||
};
|
||||
await storeHealthMetricsV1({ server_name: 'server1' }, body);
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
334
src/lib/influxdb/__tests__/v1-log-events.test.js
Normal file
334
src/lib/influxdb/__tests__/v1-log-events.test.js
Normal file
@@ -0,0 +1,334 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { writePoints: jest.fn() },
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV1: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v1/log-events', () => {
|
||||
let storeLogEventV1, globals, utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const logEvents = await import('../v1/log-events.js');
|
||||
storeLogEventV1 = logEvents.storeLogEventV1;
|
||||
globals.config.has.mockReturnValue(true);
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return [{ name: 'env', value: 'prod' }];
|
||||
});
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockResolvedValue();
|
||||
utils.writeBatchToInfluxV1.mockResolvedValue();
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeLogEventV1({ source: 'qseow-engine', host: 'server1' });
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should warn for unsupported source', async () => {
|
||||
await storeLogEventV1({ source: 'unknown', host: 'server1' });
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(expect.stringContaining('Unsupported'));
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write qseow-engine event', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'System',
|
||||
message: 'test',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'Log event from qseow-engine',
|
||||
'server1',
|
||||
100
|
||||
);
|
||||
});
|
||||
|
||||
test('should write qseow-proxy event', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-proxy',
|
||||
host: 'server2',
|
||||
level: 'WARN',
|
||||
log_row: '2',
|
||||
subsystem: 'Proxy',
|
||||
message: 'test',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write qseow-scheduler event', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-scheduler',
|
||||
host: 'server3',
|
||||
level: 'ERROR',
|
||||
log_row: '3',
|
||||
subsystem: 'Scheduler',
|
||||
message: 'test',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write qseow-repository event', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-repository',
|
||||
host: 'server4',
|
||||
level: 'INFO',
|
||||
log_row: '4',
|
||||
subsystem: 'Repository',
|
||||
message: 'test',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write qseow-qix-perf event', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-qix-perf',
|
||||
host: 'server5',
|
||||
level: 'INFO',
|
||||
log_row: '5',
|
||||
subsystem: 'Perf',
|
||||
message: 'test',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(new Error('Write failed'));
|
||||
await expect(
|
||||
storeLogEventV1({
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'System',
|
||||
message: 'test',
|
||||
})
|
||||
).rejects.toThrow();
|
||||
expect(globals.logger.error).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should apply event categories to tags', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'System',
|
||||
message: 'test',
|
||||
category: [
|
||||
{ name: 'severity', value: 'high' },
|
||||
{ name: 'component', value: 'engine' },
|
||||
],
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should apply config tags when available', async () => {
|
||||
globals.config.has.mockReturnValue(true);
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('logEvents.tags')) return [{ name: 'datacenter', value: 'us-east' }];
|
||||
return null;
|
||||
});
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-proxy',
|
||||
host: 'server2',
|
||||
level: 'WARN',
|
||||
log_row: '2',
|
||||
subsystem: 'Proxy',
|
||||
message: 'test',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle events without categories', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-scheduler',
|
||||
host: 'server3',
|
||||
level: 'INFO',
|
||||
log_row: '3',
|
||||
subsystem: 'Scheduler',
|
||||
message: 'test',
|
||||
category: [],
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle engine event with all optional fields', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'System',
|
||||
message: 'test',
|
||||
user_full: 'DOMAIN\\user',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
result_code: '200',
|
||||
windows_user: 'SYSTEM',
|
||||
task_id: 'task-001',
|
||||
task_name: 'Reload Task',
|
||||
app_id: 'app-123',
|
||||
app_name: 'Sales Dashboard',
|
||||
engine_exe_version: '14.65.2',
|
||||
exception_message: '',
|
||||
command: 'OpenDoc',
|
||||
origin: 'Engine',
|
||||
context: 'DocSession',
|
||||
session_id: 'sess-001',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle engine event without optional fields', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'System',
|
||||
message: 'test',
|
||||
user_full: '',
|
||||
user_directory: '',
|
||||
user_id: '',
|
||||
result_code: '',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle proxy event with optional fields', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-proxy',
|
||||
host: 'server2',
|
||||
level: 'WARN',
|
||||
log_row: '2',
|
||||
subsystem: 'Proxy',
|
||||
message: 'test',
|
||||
user_full: 'DOMAIN\\proxyuser',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'proxy123',
|
||||
result_code: '401',
|
||||
command: 'Authenticate',
|
||||
origin: 'Proxy',
|
||||
context: 'AuthSession',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle scheduler event with task fields', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-scheduler',
|
||||
host: 'server3',
|
||||
level: 'INFO',
|
||||
log_row: '3',
|
||||
subsystem: 'Scheduler',
|
||||
message: 'Task completed',
|
||||
user_full: 'SYSTEM',
|
||||
user_directory: 'INTERNAL',
|
||||
user_id: 'sa_scheduler',
|
||||
task_id: 'abc-123',
|
||||
task_name: 'Daily Reload',
|
||||
app_name: 'Finance App',
|
||||
app_id: 'finance-001',
|
||||
execution_id: 'exec-999',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle repository event with optional fields', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-repository',
|
||||
host: 'server4',
|
||||
level: 'ERROR',
|
||||
log_row: '4',
|
||||
subsystem: 'Repository',
|
||||
message: 'Access denied',
|
||||
user_full: 'DOMAIN\\repouser',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'repo456',
|
||||
result_code: '403',
|
||||
command: 'GetObject',
|
||||
origin: 'Repository',
|
||||
context: 'API',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle qix-perf event with all fields', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-qix-perf',
|
||||
host: 'server5',
|
||||
level: 'INFO',
|
||||
log_row: '5',
|
||||
subsystem: 'QixPerf',
|
||||
message: 'Performance metric',
|
||||
method: 'GetLayout',
|
||||
object_type: 'sheet',
|
||||
proxy_session_id: 'proxy-sess-001',
|
||||
session_id: 'sess-002',
|
||||
event_activity_source: 'User',
|
||||
user_full: 'DOMAIN\\perfuser',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'perf789',
|
||||
app_id: 'perf-app-001',
|
||||
app_name: 'Performance App',
|
||||
object_id: 'obj-123',
|
||||
process_time: 150,
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle qix-perf event with missing optional fields', async () => {
|
||||
await storeLogEventV1({
|
||||
source: 'qseow-qix-perf',
|
||||
host: '',
|
||||
level: '',
|
||||
log_row: '',
|
||||
subsystem: '',
|
||||
message: 'test',
|
||||
method: '',
|
||||
object_type: '',
|
||||
proxy_session_id: '',
|
||||
session_id: '',
|
||||
event_activity_source: '',
|
||||
user_full: '',
|
||||
user_directory: '',
|
||||
user_id: '',
|
||||
app_id: '',
|
||||
app_name: '',
|
||||
object_id: '',
|
||||
});
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
202
src/lib/influxdb/__tests__/v1-queue-metrics.test.js
Normal file
202
src/lib/influxdb/__tests__/v1-queue-metrics.test.js
Normal file
@@ -0,0 +1,202 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { writePoints: jest.fn() },
|
||||
hostInfo: { hostname: 'test-host' },
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
udpQueueManagerUserActivity: {
|
||||
getMetrics: jest.fn(() => ({
|
||||
queueSize: 10,
|
||||
queueMaxSize: 100,
|
||||
messagesProcessed: 50,
|
||||
messagesDropped: 2,
|
||||
processingRate: 5.5,
|
||||
})),
|
||||
clearMetrics: jest.fn(),
|
||||
},
|
||||
udpQueueManagerLogEvents: {
|
||||
getMetrics: jest.fn(() => ({
|
||||
queueSize: 20,
|
||||
queueMaxSize: 200,
|
||||
messagesProcessed: 100,
|
||||
messagesDropped: 5,
|
||||
processingRate: 10.5,
|
||||
})),
|
||||
clearMetrics: jest.fn(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV1: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v1/queue-metrics', () => {
|
||||
let storeUserEventQueueMetricsV1, storeLogEventQueueMetricsV1, globals, utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const queueMetrics = await import('../v1/queue-metrics.js');
|
||||
storeUserEventQueueMetricsV1 = queueMetrics.storeUserEventQueueMetricsV1;
|
||||
storeLogEventQueueMetricsV1 = queueMetrics.storeLogEventQueueMetricsV1;
|
||||
|
||||
// Mock queue managers
|
||||
globals.udpQueueManagerUserActivity = {
|
||||
getMetrics: jest.fn().mockResolvedValue({
|
||||
queueSize: 10,
|
||||
queueMaxSize: 1000,
|
||||
queueUtilizationPct: 1.0,
|
||||
queuePending: 5,
|
||||
messagesReceived: 100,
|
||||
messagesQueued: 95,
|
||||
messagesProcessed: 90,
|
||||
messagesFailed: 2,
|
||||
messagesDroppedTotal: 3,
|
||||
messagesDroppedRateLimit: 1,
|
||||
messagesDroppedQueueFull: 1,
|
||||
messagesDroppedSize: 1,
|
||||
processingTimeAvgMs: 50,
|
||||
processingTimeP95Ms: 100,
|
||||
processingTimeMaxMs: 200,
|
||||
rateLimitCurrent: 50,
|
||||
backpressureActive: false,
|
||||
}),
|
||||
clearMetrics: jest.fn(),
|
||||
};
|
||||
globals.udpQueueManagerLogEvents = {
|
||||
getMetrics: jest.fn().mockResolvedValue({
|
||||
queueSize: 20,
|
||||
queueMaxSize: 2000,
|
||||
queueUtilizationPct: 1.0,
|
||||
queuePending: 10,
|
||||
messagesReceived: 200,
|
||||
messagesQueued: 190,
|
||||
messagesProcessed: 180,
|
||||
messagesFailed: 5,
|
||||
messagesDroppedTotal: 5,
|
||||
messagesDroppedRateLimit: 2,
|
||||
messagesDroppedQueueFull: 2,
|
||||
messagesDroppedSize: 1,
|
||||
processingTimeAvgMs: 60,
|
||||
processingTimeP95Ms: 120,
|
||||
processingTimeMaxMs: 250,
|
||||
rateLimitCurrent: 100,
|
||||
backpressureActive: false,
|
||||
}),
|
||||
clearMetrics: jest.fn(),
|
||||
};
|
||||
|
||||
globals.config.has.mockReturnValue(true);
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('queueMetrics.influxdb.enable')) return true;
|
||||
if (path.includes('measurementName')) return 'queue_metrics';
|
||||
if (path.includes('queueMetrics.influxdb.tags'))
|
||||
return [{ name: 'env', value: 'prod' }];
|
||||
if (path === 'Butler-SOS.influxdbConfig.maxBatchSize') return 100;
|
||||
return undefined;
|
||||
});
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeBatchToInfluxV1.mockResolvedValue();
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled for user events', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeUserEventQueueMetricsV1();
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when config disabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('queueMetrics.influxdb.enable')) return false;
|
||||
return undefined;
|
||||
});
|
||||
await storeUserEventQueueMetricsV1();
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when queue manager not initialized', async () => {
|
||||
globals.udpQueueManagerUserActivity = undefined;
|
||||
await storeUserEventQueueMetricsV1();
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('not initialized')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write user event queue metrics', async () => {
|
||||
await storeUserEventQueueMetricsV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
expect.stringContaining('User event queue metrics'),
|
||||
'',
|
||||
100
|
||||
);
|
||||
expect(globals.udpQueueManagerUserActivity.clearMetrics).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle user event write errors', async () => {
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(new Error('Write failed'));
|
||||
await expect(storeUserEventQueueMetricsV1()).rejects.toThrow();
|
||||
expect(globals.logger.error).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled for log events', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeLogEventQueueMetricsV1();
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when config disabled for log events', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('queueMetrics.influxdb.enable')) return false;
|
||||
return undefined;
|
||||
});
|
||||
await storeLogEventQueueMetricsV1();
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when log queue manager not initialized', async () => {
|
||||
globals.udpQueueManagerLogEvents = undefined;
|
||||
await storeLogEventQueueMetricsV1();
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('not initialized')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write log event queue metrics', async () => {
|
||||
await storeLogEventQueueMetricsV1();
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
expect.stringContaining('Log event queue metrics'),
|
||||
'',
|
||||
100
|
||||
);
|
||||
expect(globals.udpQueueManagerLogEvents.clearMetrics).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle log event write errors', async () => {
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(new Error('Write failed'));
|
||||
await expect(storeLogEventQueueMetricsV1()).rejects.toThrow();
|
||||
expect(globals.logger.error).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
221
src/lib/influxdb/__tests__/v1-sessions.test.js
Normal file
221
src/lib/influxdb/__tests__/v1-sessions.test.js
Normal file
@@ -0,0 +1,221 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
writePoints: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV1: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v1/sessions', () => {
|
||||
let storeSessionsV1;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const sessions = await import('../v1/sessions.js');
|
||||
storeSessionsV1 = sessions.storeSessionsV1;
|
||||
|
||||
// Setup default mocks
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockResolvedValue();
|
||||
utils.writeBatchToInfluxV1.mockResolvedValue();
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return undefined;
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeSessionsV1', () => {
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: 'vp1',
|
||||
serverName: 'central',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: [{ measurement: 'user_session_summary', tags: {}, fields: {} }],
|
||||
};
|
||||
|
||||
await storeSessionsV1(userSessions);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when no datapoints', async () => {
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: 'vp1',
|
||||
serverName: 'central',
|
||||
sessionCount: 0,
|
||||
uniqueUserList: '',
|
||||
datapointInfluxdb: [],
|
||||
};
|
||||
|
||||
await storeSessionsV1(userSessions);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
'PROXY SESSIONS V1: No datapoints to write to InfluxDB'
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write session data', async () => {
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: 'vp1',
|
||||
serverName: 'central',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2,user3',
|
||||
datapointInfluxdb: [
|
||||
{
|
||||
measurement: 'user_session_summary',
|
||||
tags: { host: 'server1', virtualProxy: 'vp1' },
|
||||
fields: { session_count: 5 },
|
||||
},
|
||||
{
|
||||
measurement: 'user_session_details',
|
||||
tags: { host: 'server1', user: 'user1' },
|
||||
fields: { session_id: 'session1' },
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
await storeSessionsV1(userSessions);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'Proxy sessions for server1/vp1',
|
||||
'central',
|
||||
100
|
||||
);
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Sent user session data to InfluxDB')
|
||||
);
|
||||
});
|
||||
|
||||
test('should write all datapoints', async () => {
|
||||
const datapoints = [
|
||||
{
|
||||
measurement: 'user_session_summary',
|
||||
tags: { host: 'server1' },
|
||||
fields: { count: 3 },
|
||||
},
|
||||
{
|
||||
measurement: 'user_session_list',
|
||||
tags: { host: 'server1' },
|
||||
fields: { users: 'user1,user2' },
|
||||
},
|
||||
];
|
||||
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: 'vp1',
|
||||
serverName: 'central',
|
||||
sessionCount: 3,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: datapoints,
|
||||
};
|
||||
|
||||
await storeSessionsV1(userSessions);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
datapoints,
|
||||
expect.any(String),
|
||||
'central',
|
||||
100
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: 'vp1',
|
||||
serverName: 'central',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: [{ measurement: 'user_session_summary', tags: {}, fields: {} }],
|
||||
};
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(writeError);
|
||||
|
||||
await expect(storeSessionsV1(userSessions)).rejects.toThrow('Write failed');
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error saving user session data')
|
||||
);
|
||||
});
|
||||
|
||||
test('should log debug messages with session details', async () => {
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: 'vp1',
|
||||
serverName: 'central',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2,user3',
|
||||
datapointInfluxdb: [{ measurement: 'user_session_summary', tags: {}, fields: {} }],
|
||||
};
|
||||
|
||||
await storeSessionsV1(userSessions);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Session count')
|
||||
);
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(expect.stringContaining('User list'));
|
||||
expect(globals.logger.silly).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle null datapointInfluxdb', async () => {
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: 'vp1',
|
||||
serverName: 'central',
|
||||
sessionCount: 0,
|
||||
uniqueUserList: '',
|
||||
datapointInfluxdb: null,
|
||||
};
|
||||
|
||||
await storeSessionsV1(userSessions);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
'PROXY SESSIONS V1: No datapoints to write to InfluxDB'
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
256
src/lib/influxdb/__tests__/v1-user-events.test.js
Normal file
256
src/lib/influxdb/__tests__/v1-user-events.test.js
Normal file
@@ -0,0 +1,256 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
writePoints: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
getConfigTags: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV1: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v1/user-events', () => {
|
||||
let storeUserEventV1;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const userEvents = await import('../v1/user-events.js');
|
||||
storeUserEventV1 = userEvents.storeUserEventV1;
|
||||
|
||||
// Setup default mocks
|
||||
globals.config.has.mockReturnValue(true);
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.userEvents.tags') return [{ name: 'env', value: 'prod' }];
|
||||
if (key === 'Butler-SOS.influxdbConfig.maxBatchSize') return 100;
|
||||
return null;
|
||||
});
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeBatchToInfluxV1.mockResolvedValue();
|
||||
});
|
||||
|
||||
describe('storeUserEventV1', () => {
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write user event', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'User event',
|
||||
'server1',
|
||||
100
|
||||
);
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'USER EVENT V1: Sent user event data to InfluxDB'
|
||||
);
|
||||
});
|
||||
|
||||
test('should validate required fields - missing host', async () => {
|
||||
const msg = {
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Missing required field')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should validate required fields - missing command', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Missing required field')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should validate required fields - missing user_directory', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_id: 'user123',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Missing required field')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should validate required fields - missing user_id', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Missing required field')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should validate required fields - missing origin', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Missing required field')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV1).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should create correct datapoint with config tags', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
const expectedDatapoint = expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
measurement: 'user_events',
|
||||
tags: expect.objectContaining({
|
||||
host: 'server1',
|
||||
event_action: 'OpenApp',
|
||||
userFull: 'DOMAIN\\user123',
|
||||
userDirectory: 'DOMAIN',
|
||||
userId: 'user123',
|
||||
origin: 'AppAccess',
|
||||
env: 'prod',
|
||||
}),
|
||||
fields: expect.objectContaining({
|
||||
userFull: 'DOMAIN\\user123',
|
||||
userId: 'user123',
|
||||
}),
|
||||
}),
|
||||
]);
|
||||
|
||||
expect(utils.writeBatchToInfluxV1).toHaveBeenCalledWith(
|
||||
expectedDatapoint,
|
||||
'User event',
|
||||
'server1',
|
||||
100
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV1.mockRejectedValue(writeError);
|
||||
|
||||
await expect(storeUserEventV1(msg)).rejects.toThrow('Write failed');
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('USER EVENT V1: Error saving user event')
|
||||
);
|
||||
});
|
||||
|
||||
test('should log debug messages', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user123',
|
||||
origin: 'AppAccess',
|
||||
};
|
||||
|
||||
await storeUserEventV1(msg);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('USER EVENT V1')
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
155
src/lib/influxdb/__tests__/v2-butler-memory.test.js
Normal file
155
src/lib/influxdb/__tests__/v2-butler-memory.test.js
Normal file
@@ -0,0 +1,155 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockPoint = {
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
floatField: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
const mockWriteApi = {
|
||||
writePoint: jest.fn(),
|
||||
close: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn() },
|
||||
influx: { getWriteApi: jest.fn(() => mockWriteApi) },
|
||||
appVersion: '1.2.3',
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV2: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v2/butler-memory', () => {
|
||||
let storeButlerMemoryV2, globals, utils, Point;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const InfluxClient = await import('@influxdata/influxdb-client');
|
||||
Point = InfluxClient.Point;
|
||||
const butlerMemory = await import('../v2/butler-memory.js');
|
||||
storeButlerMemoryV2 = butlerMemory.storeButlerMemoryV2;
|
||||
|
||||
mockPoint.tag.mockReturnThis();
|
||||
mockPoint.floatField.mockReturnThis();
|
||||
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('org')) return 'test-org';
|
||||
if (path.includes('bucket')) return 'test-bucket';
|
||||
if (path.includes('maxBatchSize')) return 100;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
mockWriteApi.writePoint.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
const memory = {
|
||||
instanceTag: 'test-instance',
|
||||
heapUsedMByte: 100,
|
||||
heapTotalMByte: 200,
|
||||
externalMemoryMByte: 50,
|
||||
processMemoryMByte: 250,
|
||||
};
|
||||
await storeButlerMemoryV2(memory);
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early with invalid memory data', async () => {
|
||||
await storeButlerMemoryV2(null);
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
'MEMORY USAGE V2: Invalid memory data provided'
|
||||
);
|
||||
});
|
||||
|
||||
test('should return early with non-object memory data', async () => {
|
||||
await storeButlerMemoryV2('not an object');
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
expect(globals.logger.warn).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write complete memory metrics', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'prod-instance',
|
||||
heapUsedMByte: 150.5,
|
||||
heapTotalMByte: 300.2,
|
||||
externalMemoryMByte: 75.8,
|
||||
processMemoryMByte: 400.1,
|
||||
};
|
||||
|
||||
await storeButlerMemoryV2(memory);
|
||||
|
||||
expect(Point).toHaveBeenCalledWith('butlersos_memory_usage');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('butler_sos_instance', 'prod-instance');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('version', '1.2.3');
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('heap_used', 150.5);
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('heap_total', 300.2);
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('external', 75.8);
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('process_memory', 400.1);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalledWith(
|
||||
[mockPoint],
|
||||
'test-org',
|
||||
'test-bucket',
|
||||
'Memory usage metrics',
|
||||
'',
|
||||
100
|
||||
);
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'MEMORY USAGE V2: Sent Butler SOS memory usage data to InfluxDB'
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle zero memory values', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'test-instance',
|
||||
heapUsedMByte: 0,
|
||||
heapTotalMByte: 0,
|
||||
externalMemoryMByte: 0,
|
||||
processMemoryMByte: 0,
|
||||
};
|
||||
|
||||
await storeButlerMemoryV2(memory);
|
||||
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('heap_used', 0);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should log silly level debug info', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'test-instance',
|
||||
heapUsedMByte: 100,
|
||||
heapTotalMByte: 200,
|
||||
externalMemoryMByte: 50,
|
||||
processMemoryMByte: 250,
|
||||
};
|
||||
|
||||
await storeButlerMemoryV2(memory);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalled();
|
||||
expect(globals.logger.silly).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
218
src/lib/influxdb/__tests__/v2-event-counts.test.js
Normal file
218
src/lib/influxdb/__tests__/v2-event-counts.test.js
Normal file
@@ -0,0 +1,218 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockPoint = {
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
intField: jest.fn().mockReturnThis(),
|
||||
stringField: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
const mockWriteApi = {
|
||||
writePoint: jest.fn(),
|
||||
writePoints: jest.fn(),
|
||||
close: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { getWriteApi: jest.fn(() => mockWriteApi) },
|
||||
hostInfo: { hostname: 'test-host' },
|
||||
eventCounters: {
|
||||
userEvent: { valid: 100, invalid: 5, rejected: 10 },
|
||||
logEvent: { valid: 200, invalid: 8, rejected: 15 },
|
||||
},
|
||||
rejectedEventTags: {
|
||||
userEvent: { tag1: 5, tag2: 3 },
|
||||
logEvent: { tag3: 7, tag4: 2 },
|
||||
},
|
||||
udpEvents: {
|
||||
getLogEvents: jest.fn(),
|
||||
getUserEvents: jest.fn(),
|
||||
},
|
||||
rejectedEvents: {
|
||||
getRejectedLogEvents: jest.fn(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV2: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
const mockV2Utils = {
|
||||
applyInfluxTags: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../v2/utils.js', () => mockV2Utils);
|
||||
|
||||
describe('v2/event-counts', () => {
|
||||
let storeEventCountV2, storeRejectedEventCountV2, globals, utils, Point;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const InfluxClient = await import('@influxdata/influxdb-client');
|
||||
Point = InfluxClient.Point;
|
||||
const eventCounts = await import('../v2/event-counts.js');
|
||||
storeEventCountV2 = eventCounts.storeEventCountV2;
|
||||
storeRejectedEventCountV2 = eventCounts.storeRejectedEventCountV2;
|
||||
|
||||
mockPoint.tag.mockReturnThis();
|
||||
mockPoint.intField.mockReturnThis();
|
||||
mockPoint.stringField.mockReturnThis();
|
||||
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('org')) return 'test-org';
|
||||
if (path.includes('bucket')) return 'test-bucket';
|
||||
if (path.includes('measurementName')) return 'event_count';
|
||||
if (path.includes('eventCount.influxdb.tags')) return [{ name: 'env', value: 'prod' }];
|
||||
if (path.includes('performanceMonitor.influxdb.tags'))
|
||||
return [{ name: 'monitor', value: 'perf' }];
|
||||
if (path.includes('enable')) return true;
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has.mockReturnValue(true);
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockImplementation(async (fn) => await fn());
|
||||
|
||||
globals.eventCounters = {
|
||||
userEvent: { valid: 100, invalid: 5, rejected: 10 },
|
||||
logEvent: { valid: 200, invalid: 8, rejected: 15 },
|
||||
};
|
||||
|
||||
// Mock udpEvents and rejectedEvents methods
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([
|
||||
{ source: 'qseow-engine', host: 'test-host', subsystem: 'engine', counter: 200 },
|
||||
]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([
|
||||
{ source: 'qseow-proxy', host: 'test-host', subsystem: 'proxy', counter: 100 },
|
||||
]);
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([]);
|
||||
});
|
||||
|
||||
describe('storeEventCountV2', () => {
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeEventCountV2();
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write user and log event counts', async () => {
|
||||
await storeEventCountV2();
|
||||
|
||||
expect(Point).toHaveBeenCalledTimes(2); // user + log events
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('event_type', 'user');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('event_type', 'log');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('host', 'test-host');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-engine');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-proxy');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('subsystem', 'engine');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('subsystem', 'proxy');
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('counter', 200);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('counter', 100);
|
||||
expect(mockV2Utils.applyInfluxTags).toHaveBeenCalledTimes(2);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle zero counts', async () => {
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([]);
|
||||
|
||||
await storeEventCountV2();
|
||||
|
||||
// If no events, it should return early
|
||||
expect(Point).not.toHaveBeenCalled();
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should log verbose information', async () => {
|
||||
await storeEventCountV2();
|
||||
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'EVENT COUNT V2: Sent event count data to InfluxDB'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeRejectedEventCountV2', () => {
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeRejectedEventCountV2();
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when feature disabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('performanceMonitor') && path.includes('enable')) return false;
|
||||
if (path.includes('enable')) return true;
|
||||
return undefined;
|
||||
});
|
||||
await storeRejectedEventCountV2();
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write rejected event counts by tag', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([
|
||||
{ source: 'qseow-engine', counter: 5 },
|
||||
{ source: 'qseow-proxy', counter: 3 },
|
||||
]);
|
||||
|
||||
await storeRejectedEventCountV2();
|
||||
|
||||
expect(Point).toHaveBeenCalled();
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-engine');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-proxy');
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('counter', 5);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('counter', 3);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle empty rejection tags', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([]);
|
||||
|
||||
await storeRejectedEventCountV2();
|
||||
|
||||
expect(Point).not.toHaveBeenCalled();
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle undefined rejection tags', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([]);
|
||||
|
||||
await storeRejectedEventCountV2();
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should log verbose information', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([
|
||||
{ source: 'qseow-engine', counter: 5 },
|
||||
]);
|
||||
|
||||
await storeRejectedEventCountV2();
|
||||
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'REJECTED EVENT COUNT V2: Sent rejected event count data to InfluxDB'
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
227
src/lib/influxdb/__tests__/v2-health-metrics.test.js
Normal file
227
src/lib/influxdb/__tests__/v2-health-metrics.test.js
Normal file
@@ -0,0 +1,227 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockPoint = {
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
stringField: jest.fn().mockReturnThis(),
|
||||
intField: jest.fn().mockReturnThis(),
|
||||
uintField: jest.fn().mockReturnThis(),
|
||||
floatField: jest.fn().mockReturnThis(),
|
||||
booleanField: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
const mockWriteApi = {
|
||||
writePoints: jest.fn(),
|
||||
close: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { getWriteApi: jest.fn(() => mockWriteApi) },
|
||||
hostInfo: { hostname: 'test-host' },
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV2: jest.fn(),
|
||||
processAppDocuments: jest.fn(),
|
||||
getFormattedTime: jest.fn(() => '2 days, 3 hours'),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v2/health-metrics', () => {
|
||||
let storeHealthMetricsV2, globals, utils, Point;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const InfluxClient = await import('@influxdata/influxdb-client');
|
||||
Point = InfluxClient.Point;
|
||||
const healthMetrics = await import('../v2/health-metrics.js');
|
||||
storeHealthMetricsV2 = healthMetrics.storeHealthMetricsV2;
|
||||
|
||||
mockPoint.tag.mockReturnThis();
|
||||
mockPoint.stringField.mockReturnThis();
|
||||
mockPoint.intField.mockReturnThis();
|
||||
mockPoint.uintField.mockReturnThis();
|
||||
mockPoint.floatField.mockReturnThis();
|
||||
mockPoint.booleanField.mockReturnThis();
|
||||
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('org')) return 'test-org';
|
||||
if (path.includes('bucket')) return 'test-bucket';
|
||||
if (path.includes('includeFields')) return true;
|
||||
if (path.includes('enableAppNameExtract')) return true;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockImplementation(async (fn) => await fn());
|
||||
utils.processAppDocuments.mockResolvedValue({
|
||||
appNames: ['App1', 'App2'],
|
||||
sessionAppNames: ['Session1', 'Session2'],
|
||||
});
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
const body = {
|
||||
version: '1.0',
|
||||
started: '2024-01-01',
|
||||
mem: { committed: 1000, allocated: 800, free: 200 },
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [], calls: 0, selections: 0 },
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5, total: 10 },
|
||||
users: { active: 3, total: 8 },
|
||||
cache: { hits: 100, lookups: 120, added: 20, replaced: 5, bytes_added: 1024 },
|
||||
saturated: false,
|
||||
};
|
||||
await storeHealthMetricsV2('server1', 'host1', body);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early with invalid body', async () => {
|
||||
await storeHealthMetricsV2('server1', 'host1', null);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
expect(globals.logger.warn).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write complete health metrics with all fields', async () => {
|
||||
const body = {
|
||||
version: '1.0.0',
|
||||
started: '2024-01-01T00:00:00Z',
|
||||
mem: { committed: 1000, allocated: 800, free: 200 },
|
||||
apps: {
|
||||
active_docs: [{ id: 'app1' }],
|
||||
loaded_docs: [{ id: 'app2' }],
|
||||
in_memory_docs: [{ id: 'app3' }],
|
||||
calls: 10,
|
||||
selections: 5,
|
||||
},
|
||||
cpu: { total: 45.7 },
|
||||
session: { active: 5, total: 10 },
|
||||
users: { active: 3, total: 8 },
|
||||
cache: { hits: 100, lookups: 120, added: 20, replaced: 5, bytes_added: 1024 },
|
||||
saturated: false,
|
||||
};
|
||||
const serverTags = { server_name: 'server1', qs_env: 'dev' };
|
||||
|
||||
await storeHealthMetricsV2('server1', 'host1', body, serverTags);
|
||||
|
||||
expect(Point).toHaveBeenCalledTimes(8); // One for each measurement: sense_server, mem, apps, cpu, session, users, cache, saturated
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledTimes(3);
|
||||
expect(mockWriteApi.writePoints).toHaveBeenCalled();
|
||||
expect(mockWriteApi.close).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should apply server tags to all points', async () => {
|
||||
const body = {
|
||||
version: '1.0',
|
||||
started: '2024-01-01',
|
||||
mem: { committed: 1000, allocated: 800, free: 200 },
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [], calls: 0, selections: 0 },
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5, total: 10 },
|
||||
users: { active: 3, total: 8 },
|
||||
cache: { hits: 100, lookups: 120, added: 20, replaced: 5, bytes_added: 1024 },
|
||||
saturated: false,
|
||||
};
|
||||
const serverTags = { server_name: 'server1', qs_env: 'prod', custom_tag: 'value' };
|
||||
|
||||
await storeHealthMetricsV2('server1', 'host1', body, serverTags);
|
||||
|
||||
// Each point should have tags applied (9 points * 3 tags = 27 calls minimum)
|
||||
expect(mockPoint.tag).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle empty app docs', async () => {
|
||||
const body = {
|
||||
version: '1.0',
|
||||
started: '2024-01-01',
|
||||
mem: { committed: 1000, allocated: 800, free: 200 },
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [], calls: 0, selections: 0 },
|
||||
cpu: { total: 50 },
|
||||
session: { active: 0, total: 0 },
|
||||
users: { active: 0, total: 0 },
|
||||
cache: { hits: 0, lookups: 0, added: 0, replaced: 0, bytes_added: 0 },
|
||||
saturated: false,
|
||||
};
|
||||
|
||||
await storeHealthMetricsV2('server1', 'host1', body, {});
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledWith([], 'HEALTH METRICS', 'active');
|
||||
});
|
||||
|
||||
test('should handle serverTags with null values', async () => {
|
||||
const body = {
|
||||
version: '1.0',
|
||||
started: '2024-01-01',
|
||||
mem: { committed: 1000, allocated: 800, free: 200 },
|
||||
apps: { active_docs: [], loaded_docs: [], in_memory_docs: [], calls: 0, selections: 0 },
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5, total: 10 },
|
||||
users: { active: 3, total: 8 },
|
||||
cache: { hits: 100, lookups: 120, added: 20, replaced: 5, bytes_added: 1024 },
|
||||
saturated: false,
|
||||
};
|
||||
const serverTags = { server_name: 'server1', null_tag: null, undefined_tag: undefined };
|
||||
|
||||
await storeHealthMetricsV2('server1', 'host1', body, serverTags);
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle config options for includeFields', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('org')) return 'test-org';
|
||||
if (path.includes('bucket')) return 'test-bucket';
|
||||
if (path.includes('includeFields.activeDocs')) return false;
|
||||
if (path.includes('includeFields.loadedDocs')) return false;
|
||||
if (path.includes('includeFields.inMemoryDocs')) return false;
|
||||
if (path.includes('enableAppNameExtract')) return false;
|
||||
return undefined;
|
||||
});
|
||||
|
||||
const body = {
|
||||
version: '1.0',
|
||||
started: '2024-01-01',
|
||||
mem: { committed: 1000, allocated: 800, free: 200 },
|
||||
apps: {
|
||||
active_docs: [{ id: 'app1' }],
|
||||
loaded_docs: [{ id: 'app2' }],
|
||||
in_memory_docs: [{ id: 'app3' }],
|
||||
calls: 10,
|
||||
selections: 5,
|
||||
},
|
||||
cpu: { total: 50 },
|
||||
session: { active: 5, total: 10 },
|
||||
users: { active: 3, total: 8 },
|
||||
cache: { hits: 100, lookups: 120, added: 20, replaced: 5, bytes_added: 1024 },
|
||||
saturated: false,
|
||||
};
|
||||
|
||||
await storeHealthMetricsV2('server1', 'host1', body, {});
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
377
src/lib/influxdb/__tests__/v2-log-events.test.js
Normal file
377
src/lib/influxdb/__tests__/v2-log-events.test.js
Normal file
@@ -0,0 +1,377 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockPoint = {
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
stringField: jest.fn().mockReturnThis(),
|
||||
intField: jest.fn().mockReturnThis(),
|
||||
floatField: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
const mockWriteApi = {
|
||||
writePoint: jest.fn(),
|
||||
close: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { getWriteApi: jest.fn(() => mockWriteApi) },
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV2: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
const mockV2Utils = {
|
||||
applyInfluxTags: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../v2/utils.js', () => mockV2Utils);
|
||||
|
||||
describe('v2/log-events', () => {
|
||||
let storeLogEventV2, globals, utils, Point;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const InfluxClient = await import('@influxdata/influxdb-client');
|
||||
Point = InfluxClient.Point;
|
||||
const logEvents = await import('../v2/log-events.js');
|
||||
storeLogEventV2 = logEvents.storeLogEventV2;
|
||||
|
||||
mockPoint.tag.mockReturnThis();
|
||||
mockPoint.stringField.mockReturnThis();
|
||||
mockPoint.intField.mockReturnThis();
|
||||
mockPoint.floatField.mockReturnThis();
|
||||
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('org')) return 'test-org';
|
||||
if (path.includes('bucket')) return 'test-bucket';
|
||||
if (path.includes('logEvents.tags')) return [{ name: 'env', value: 'prod' }];
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has.mockReturnValue(true);
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockImplementation(async (fn) => await fn());
|
||||
mockWriteApi.writePoint.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
source: 'qseow-engine',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'Core',
|
||||
message: 'Test message',
|
||||
};
|
||||
await storeLogEventV2(msg);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early with missing required fields - no host', async () => {
|
||||
const msg = {
|
||||
source: 'qseow-engine',
|
||||
level: 'INFO',
|
||||
log_row: '12345',
|
||||
subsystem: 'Core',
|
||||
message: 'Test message',
|
||||
};
|
||||
await storeLogEventV2(msg);
|
||||
// Implementation doesn't explicitly validate required fields, it just processes what's there
|
||||
// So this test will actually call writeBatchToInfluxV2
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early with unsupported source', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
source: 'unsupported-source',
|
||||
level: 'INFO',
|
||||
log_row: '12345',
|
||||
subsystem: 'Core',
|
||||
message: 'Test message',
|
||||
};
|
||||
await storeLogEventV2(msg);
|
||||
expect(globals.logger.warn).toHaveBeenCalled();
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write engine log event', async () => {
|
||||
const msg = {
|
||||
host: 'host1.example.com',
|
||||
source: 'qseow-engine',
|
||||
level: 'INFO',
|
||||
message: 'Engine started successfully',
|
||||
log_row: '12345',
|
||||
subsystem: 'Core',
|
||||
windows_user: 'SYSTEM',
|
||||
exception_message: '',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'admin',
|
||||
user_full: 'DOMAIN\\admin',
|
||||
result_code: '0',
|
||||
origin: 'Engine',
|
||||
context: 'Init',
|
||||
task_name: 'Reload Task',
|
||||
app_name: 'Sales Dashboard',
|
||||
task_id: 'task-123',
|
||||
app_id: 'app-456',
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(Point).toHaveBeenCalledWith('log_event');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('host', 'host1.example.com');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-engine');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('level', 'INFO');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('log_row', '12345');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('subsystem', 'Core');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('windows_user', 'SYSTEM');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('user_directory', 'DOMAIN');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('user_id', 'admin');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('user_full', 'DOMAIN\\admin');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('result_code', '0');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('task_id', 'task-123');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('task_name', 'Reload Task');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('app_id', 'app-456');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('app_name', 'Sales Dashboard');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith(
|
||||
'message',
|
||||
'Engine started successfully'
|
||||
);
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('exception_message', '');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('command', '');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('result_code_field', '0');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('origin', 'Engine');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('context', 'Init');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('session_id', '');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('raw_event', expect.any(String));
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write proxy log event', async () => {
|
||||
const msg = {
|
||||
host: 'proxy1.example.com',
|
||||
source: 'qseow-proxy',
|
||||
level: 'WARN',
|
||||
message: 'Authentication warning',
|
||||
log_row: '5000',
|
||||
subsystem: 'Proxy',
|
||||
command: 'Login',
|
||||
user_directory: 'EXTERNAL',
|
||||
user_id: 'external_user',
|
||||
user_full: 'EXTERNAL\\external_user',
|
||||
result_code: '403',
|
||||
origin: 'Proxy',
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-proxy');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('level', 'WARN');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('user_full', 'EXTERNAL\\external_user');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('result_code', '403');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('command', 'Login');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('result_code_field', '403');
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write repository log event', async () => {
|
||||
const msg = {
|
||||
host: 'repo1.example.com',
|
||||
source: 'qseow-repository',
|
||||
level: 'ERROR',
|
||||
message: 'Database connection error',
|
||||
log_row: '7890',
|
||||
subsystem: 'Repository',
|
||||
exception_message: 'Connection timeout',
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-repository');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('level', 'ERROR');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith(
|
||||
'exception_message',
|
||||
'Connection timeout'
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should write scheduler log event', async () => {
|
||||
const msg = {
|
||||
host: 'scheduler1.example.com',
|
||||
source: 'qseow-scheduler',
|
||||
level: 'INFO',
|
||||
message: 'Task scheduled',
|
||||
log_row: '3333',
|
||||
subsystem: 'Scheduler',
|
||||
task_name: 'Daily Reload',
|
||||
task_id: 'sched-task-001',
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-scheduler');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('level', 'INFO');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('task_id', 'sched-task-001');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('task_name', 'Daily Reload');
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle log event with minimal fields', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
source: 'qseow-engine',
|
||||
level: 'DEBUG',
|
||||
log_row: '1',
|
||||
subsystem: 'Core',
|
||||
message: 'Debug message',
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('host', 'host1');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', 'qseow-engine');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('level', 'DEBUG');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('message', 'Debug message');
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle empty string fields', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
source: 'qseow-engine',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'Core',
|
||||
message: '',
|
||||
exception_message: '',
|
||||
task_name: '',
|
||||
app_name: '',
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should apply config tags', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
source: 'qseow-engine',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'Core',
|
||||
message: 'Test',
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(mockV2Utils.applyInfluxTags).toHaveBeenCalledWith(mockPoint, [
|
||||
{ name: 'env', value: 'prod' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('should handle all log levels', async () => {
|
||||
const logLevels = ['DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL'];
|
||||
|
||||
for (const level of logLevels) {
|
||||
jest.clearAllMocks();
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
source: 'qseow-engine',
|
||||
level: level,
|
||||
log_row: '1',
|
||||
subsystem: 'Core',
|
||||
message: `${level} message`,
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('level', level);
|
||||
}
|
||||
});
|
||||
|
||||
test('should handle all source types', async () => {
|
||||
const sources = [
|
||||
'qseow-engine',
|
||||
'qseow-proxy',
|
||||
'qseow-repository',
|
||||
'qseow-scheduler',
|
||||
'qseow-qix-perf',
|
||||
];
|
||||
|
||||
for (const source of sources) {
|
||||
jest.clearAllMocks();
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
source,
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'Core',
|
||||
message: 'Test',
|
||||
};
|
||||
// qix-perf requires additional fields
|
||||
if (source === 'qseow-qix-perf') {
|
||||
msg.method = 'GetLayout';
|
||||
msg.object_type = 'sheet';
|
||||
msg.proxy_session_id = 'session123';
|
||||
msg.session_id = 'session123';
|
||||
msg.event_activity_source = 'user';
|
||||
msg.process_time = '100';
|
||||
msg.work_time = '50';
|
||||
msg.lock_time = '10';
|
||||
msg.validate_time = '5';
|
||||
msg.traverse_time = '35';
|
||||
msg.net_ram = '1024';
|
||||
msg.peak_ram = '2048';
|
||||
}
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('source', source);
|
||||
}
|
||||
});
|
||||
|
||||
test('should log debug information', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
source: 'qseow-engine',
|
||||
level: 'INFO',
|
||||
log_row: '1',
|
||||
subsystem: 'Core',
|
||||
message: 'Test',
|
||||
};
|
||||
|
||||
await storeLogEventV2(msg);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalled();
|
||||
expect(globals.logger.silly).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'LOG EVENT V2: Sent log event data to InfluxDB'
|
||||
);
|
||||
});
|
||||
});
|
||||
306
src/lib/influxdb/__tests__/v2-queue-metrics.test.js
Normal file
306
src/lib/influxdb/__tests__/v2-queue-metrics.test.js
Normal file
@@ -0,0 +1,306 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockPoint = {
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
intField: jest.fn().mockReturnThis(),
|
||||
floatField: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
const mockWriteApi = {
|
||||
writePoint: jest.fn(),
|
||||
close: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { getWriteApi: jest.fn(() => mockWriteApi) },
|
||||
hostInfo: { hostname: 'test-host' },
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
udpQueueManagerUserActivity: null,
|
||||
udpQueueManagerLogEvents: null,
|
||||
};
|
||||
|
||||
const mockQueueManager = {
|
||||
getMetrics: jest.fn(),
|
||||
clearMetrics: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV2: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
const mockV2Utils = {
|
||||
applyInfluxTags: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../v2/utils.js', () => mockV2Utils);
|
||||
|
||||
describe('v2/queue-metrics', () => {
|
||||
let storeUserEventQueueMetricsV2, storeLogEventQueueMetricsV2, globals, utils, Point;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const InfluxClient = await import('@influxdata/influxdb-client');
|
||||
Point = InfluxClient.Point;
|
||||
const queueMetrics = await import('../v2/queue-metrics.js');
|
||||
storeUserEventQueueMetricsV2 = queueMetrics.storeUserEventQueueMetricsV2;
|
||||
storeLogEventQueueMetricsV2 = queueMetrics.storeLogEventQueueMetricsV2;
|
||||
|
||||
mockPoint.tag.mockReturnThis();
|
||||
mockPoint.intField.mockReturnThis();
|
||||
mockPoint.floatField.mockReturnThis();
|
||||
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('org')) return 'test-org';
|
||||
if (path.includes('bucket')) return 'test-bucket';
|
||||
if (path.includes('measurementName')) return 'event_queue_metrics';
|
||||
if (path.includes('queueMetrics.influxdb.tags'))
|
||||
return [{ name: 'env', value: 'prod' }];
|
||||
if (path.includes('enable')) return true;
|
||||
if (path === 'Butler-SOS.influxdbConfig.maxBatchSize') return 100;
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has.mockReturnValue(true);
|
||||
|
||||
globals.udpQueueManagerUserActivity = mockQueueManager;
|
||||
globals.udpQueueManagerLogEvents = mockQueueManager;
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeBatchToInfluxV2.mockResolvedValue();
|
||||
|
||||
mockWriteApi.writePoint.mockResolvedValue(undefined);
|
||||
mockWriteApi.close.mockResolvedValue(undefined);
|
||||
|
||||
mockQueueManager.getMetrics.mockReturnValue({
|
||||
queueSize: 100,
|
||||
queueMaxSize: 1000,
|
||||
queueUtilizationPct: 10.0,
|
||||
queuePending: 5,
|
||||
messagesReceived: 500,
|
||||
messagesQueued: 450,
|
||||
messagesProcessed: 400,
|
||||
messagesFailed: 10,
|
||||
messagesDroppedTotal: 40,
|
||||
messagesDroppedRateLimit: 20,
|
||||
messagesDroppedQueueFull: 15,
|
||||
messagesDroppedSize: 5,
|
||||
processingTimeAvgMs: 25.5,
|
||||
processingTimeP95Ms: 50.2,
|
||||
processingTimeMaxMs: 100.8,
|
||||
rateLimitCurrent: 100,
|
||||
backpressureActive: 0,
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeUserEventQueueMetricsV2', () => {
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeUserEventQueueMetricsV2();
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when feature disabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('enable')) return false;
|
||||
return undefined;
|
||||
});
|
||||
await storeUserEventQueueMetricsV2();
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when queue manager not initialized', async () => {
|
||||
globals.udpQueueManagerUserActivity = null;
|
||||
await storeUserEventQueueMetricsV2();
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
'USER EVENT QUEUE METRICS V2: Queue manager not initialized'
|
||||
);
|
||||
});
|
||||
|
||||
test('should write complete user event queue metrics', async () => {
|
||||
await storeUserEventQueueMetricsV2();
|
||||
|
||||
expect(Point).toHaveBeenCalledWith('event_queue_metrics');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('queue_type', 'user_events');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('host', 'test-host');
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('queue_size', 100);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('queue_max_size', 1000);
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('queue_utilization_pct', 10.0);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('queue_pending', 5);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('messages_received', 500);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('messages_queued', 450);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('messages_processed', 400);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('messages_failed', 10);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('messages_dropped_total', 40);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('messages_dropped_rate_limit', 20);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('messages_dropped_queue_full', 15);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('messages_dropped_size', 5);
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('processing_time_avg_ms', 25.5);
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('processing_time_p95_ms', 50.2);
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('processing_time_max_ms', 100.8);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('rate_limit_current', 100);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('backpressure_active', 0);
|
||||
expect(mockV2Utils.applyInfluxTags).toHaveBeenCalledWith(mockPoint, [
|
||||
{ name: 'env', value: 'prod' },
|
||||
]);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalledWith(
|
||||
[mockPoint],
|
||||
'test-org',
|
||||
'test-bucket',
|
||||
'User event queue metrics',
|
||||
'user-events-queue',
|
||||
100
|
||||
);
|
||||
expect(mockQueueManager.clearMetrics).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle zero metrics', async () => {
|
||||
mockQueueManager.getMetrics.mockReturnValue({
|
||||
queueSize: 0,
|
||||
queueMaxSize: 1000,
|
||||
queueUtilizationPct: 0,
|
||||
queuePending: 0,
|
||||
messagesReceived: 0,
|
||||
messagesQueued: 0,
|
||||
messagesProcessed: 0,
|
||||
messagesFailed: 0,
|
||||
messagesDroppedTotal: 0,
|
||||
messagesDroppedRateLimit: 0,
|
||||
messagesDroppedQueueFull: 0,
|
||||
messagesDroppedSize: 0,
|
||||
processingTimeAvgMs: 0,
|
||||
processingTimeP95Ms: 0,
|
||||
processingTimeMaxMs: 0,
|
||||
rateLimitCurrent: 0,
|
||||
backpressureActive: 0,
|
||||
});
|
||||
|
||||
await storeUserEventQueueMetricsV2();
|
||||
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('queue_size', 0);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalledWith(
|
||||
[mockPoint],
|
||||
'test-org',
|
||||
'test-bucket',
|
||||
'User event queue metrics',
|
||||
'user-events-queue',
|
||||
100
|
||||
);
|
||||
});
|
||||
|
||||
test('should log verbose information', async () => {
|
||||
await storeUserEventQueueMetricsV2();
|
||||
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'USER EVENT QUEUE METRICS V2: Sent queue metrics data to InfluxDB'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeLogEventQueueMetricsV2', () => {
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
await storeLogEventQueueMetricsV2();
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when feature disabled', async () => {
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('enable')) return false;
|
||||
return undefined;
|
||||
});
|
||||
await storeLogEventQueueMetricsV2();
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when queue manager not initialized', async () => {
|
||||
globals.udpQueueManagerLogEvents = null;
|
||||
await storeLogEventQueueMetricsV2();
|
||||
expect(utils.writeBatchToInfluxV2).not.toHaveBeenCalled();
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
'LOG EVENT QUEUE METRICS V2: Queue manager not initialized'
|
||||
);
|
||||
});
|
||||
|
||||
test('should write complete log event queue metrics', async () => {
|
||||
await storeLogEventQueueMetricsV2();
|
||||
|
||||
expect(Point).toHaveBeenCalledWith('event_queue_metrics');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('queue_type', 'log_events');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('host', 'test-host');
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('queue_size', 100);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalledWith(
|
||||
[mockPoint],
|
||||
'test-org',
|
||||
'test-bucket',
|
||||
'Log event queue metrics',
|
||||
'log-events-queue',
|
||||
100
|
||||
);
|
||||
expect(mockQueueManager.clearMetrics).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle high utilization', async () => {
|
||||
mockQueueManager.getMetrics.mockReturnValue({
|
||||
queueSize: 950,
|
||||
queueMaxSize: 1000,
|
||||
queueUtilizationPct: 95.0,
|
||||
queuePending: 50,
|
||||
messagesReceived: 10000,
|
||||
messagesQueued: 9500,
|
||||
messagesProcessed: 9000,
|
||||
messagesFailed: 100,
|
||||
messagesDroppedTotal: 400,
|
||||
messagesDroppedRateLimit: 200,
|
||||
messagesDroppedQueueFull: 150,
|
||||
messagesDroppedSize: 50,
|
||||
processingTimeAvgMs: 125.5,
|
||||
processingTimeP95Ms: 250.2,
|
||||
processingTimeMaxMs: 500.8,
|
||||
rateLimitCurrent: 50,
|
||||
backpressureActive: 1,
|
||||
});
|
||||
|
||||
await storeLogEventQueueMetricsV2();
|
||||
|
||||
expect(mockPoint.floatField).toHaveBeenCalledWith('queue_utilization_pct', 95.0);
|
||||
expect(mockPoint.intField).toHaveBeenCalledWith('backpressure_active', 1);
|
||||
expect(utils.writeBatchToInfluxV2).toHaveBeenCalledWith(
|
||||
[mockPoint],
|
||||
'test-org',
|
||||
'test-bucket',
|
||||
'Log event queue metrics',
|
||||
'log-events-queue',
|
||||
100
|
||||
);
|
||||
});
|
||||
|
||||
test('should log verbose information', async () => {
|
||||
await storeLogEventQueueMetricsV2();
|
||||
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'LOG EVENT QUEUE METRICS V2: Sent queue metrics data to InfluxDB'
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
178
src/lib/influxdb/__tests__/v2-sessions.test.js
Normal file
178
src/lib/influxdb/__tests__/v2-sessions.test.js
Normal file
@@ -0,0 +1,178 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockPoint = {
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
stringField: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
const mockWriteApi = {
|
||||
writePoints: jest.fn(),
|
||||
close: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn() },
|
||||
influx: { getWriteApi: jest.fn(() => mockWriteApi) },
|
||||
influxWriteApi: [{ serverName: 'server1' }],
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV2: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
describe('v2/sessions', () => {
|
||||
let storeSessionsV2, globals, utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const sessions = await import('../v2/sessions.js');
|
||||
storeSessionsV2 = sessions.storeSessionsV2;
|
||||
|
||||
// Set up influxWriteApi array with matching server
|
||||
globals.influxWriteApi = [{ serverName: 'server1' }];
|
||||
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('org')) return 'test-org';
|
||||
if (path.includes('bucket')) return 'test-bucket';
|
||||
return undefined;
|
||||
});
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockImplementation(async (cb) => await cb());
|
||||
mockWriteApi.writePoints.mockResolvedValue(undefined);
|
||||
mockWriteApi.close.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
const userSessions = {
|
||||
serverName: 'server1',
|
||||
host: 'host1',
|
||||
virtualProxy: 'vp1',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: [mockPoint],
|
||||
};
|
||||
await storeSessionsV2(userSessions);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early with invalid datapointInfluxdb (not array)', async () => {
|
||||
const userSessions = {
|
||||
serverName: 'server1',
|
||||
host: 'host1',
|
||||
virtualProxy: 'vp1',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: 'not-an-array',
|
||||
};
|
||||
await storeSessionsV2(userSessions);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Invalid data format')
|
||||
);
|
||||
});
|
||||
|
||||
test('should return early when writeApi not found', async () => {
|
||||
globals.influxWriteApi = [{ serverName: 'different-server' }];
|
||||
const userSessions = {
|
||||
serverName: 'server1',
|
||||
host: 'host1',
|
||||
virtualProxy: 'vp1',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: [mockPoint],
|
||||
};
|
||||
await storeSessionsV2(userSessions);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Influxdb write API object not found')
|
||||
);
|
||||
});
|
||||
|
||||
test('should write session data successfully', async () => {
|
||||
const userSessions = {
|
||||
serverName: 'server1',
|
||||
host: 'host1.example.com',
|
||||
virtualProxy: '/virtual-proxy',
|
||||
sessionCount: 10,
|
||||
uniqueUserList: 'user1,user2,user3',
|
||||
datapointInfluxdb: [mockPoint, mockPoint, mockPoint],
|
||||
};
|
||||
|
||||
await storeSessionsV2(userSessions);
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
expect(mockWriteApi.writePoints).toHaveBeenCalledWith(userSessions.datapointInfluxdb);
|
||||
expect(mockWriteApi.close).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Sent user session data to InfluxDB')
|
||||
);
|
||||
});
|
||||
|
||||
test('should write empty session array', async () => {
|
||||
const userSessions = {
|
||||
serverName: 'server1',
|
||||
host: 'host1',
|
||||
virtualProxy: 'vp1',
|
||||
sessionCount: 0,
|
||||
uniqueUserList: '',
|
||||
datapointInfluxdb: [],
|
||||
};
|
||||
|
||||
await storeSessionsV2(userSessions);
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
expect(mockWriteApi.writePoints).toHaveBeenCalledWith([]);
|
||||
});
|
||||
|
||||
test('should log silly debug information', async () => {
|
||||
const userSessions = {
|
||||
serverName: 'server1',
|
||||
host: 'host1',
|
||||
virtualProxy: 'vp1',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: [mockPoint],
|
||||
};
|
||||
|
||||
await storeSessionsV2(userSessions);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalled();
|
||||
expect(globals.logger.silly).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle multiple datapoints', async () => {
|
||||
const datapoints = Array(20).fill(mockPoint);
|
||||
const userSessions = {
|
||||
serverName: 'server1',
|
||||
host: 'host1',
|
||||
virtualProxy: 'vp1',
|
||||
sessionCount: 20,
|
||||
uniqueUserList: 'user1,user2,user3,user4,user5',
|
||||
datapointInfluxdb: datapoints,
|
||||
};
|
||||
|
||||
await storeSessionsV2(userSessions);
|
||||
|
||||
expect(mockWriteApi.writePoints).toHaveBeenCalledWith(datapoints);
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
230
src/lib/influxdb/__tests__/v2-user-events.test.js
Normal file
230
src/lib/influxdb/__tests__/v2-user-events.test.js
Normal file
@@ -0,0 +1,230 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockPoint = {
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
stringField: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
const mockWriteApi = {
|
||||
writePoint: jest.fn(),
|
||||
close: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: { get: jest.fn(), has: jest.fn() },
|
||||
influx: { getWriteApi: jest.fn(() => mockWriteApi) },
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({ default: mockGlobals }));
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV2: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
const mockV2Utils = {
|
||||
applyInfluxTags: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../v2/utils.js', () => mockV2Utils);
|
||||
|
||||
describe('v2/user-events', () => {
|
||||
let storeUserEventV2, globals, utils, Point;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const InfluxClient = await import('@influxdata/influxdb-client');
|
||||
Point = InfluxClient.Point;
|
||||
const userEvents = await import('../v2/user-events.js');
|
||||
storeUserEventV2 = userEvents.storeUserEventV2;
|
||||
|
||||
mockPoint.tag.mockReturnThis();
|
||||
mockPoint.stringField.mockReturnThis();
|
||||
|
||||
globals.config.get.mockImplementation((path) => {
|
||||
if (path.includes('org')) return 'test-org';
|
||||
if (path.includes('bucket')) return 'test-bucket';
|
||||
if (path.includes('userEvents.tags')) return [{ name: 'env', value: 'prod' }];
|
||||
return undefined;
|
||||
});
|
||||
globals.config.has.mockReturnValue(true);
|
||||
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockImplementation(async (fn) => await fn());
|
||||
mockWriteApi.writePoint.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
};
|
||||
await storeUserEventV2(msg);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early with missing required fields', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
command: 'OpenApp',
|
||||
// missing user_directory, user_id, origin
|
||||
};
|
||||
await storeUserEventV2(msg);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Missing required fields')
|
||||
);
|
||||
});
|
||||
|
||||
test('should write complete user event with all fields', async () => {
|
||||
const msg = {
|
||||
host: 'host1.example.com',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'john.doe',
|
||||
origin: 'QlikSense',
|
||||
appId: 'app-123',
|
||||
appName: 'Sales Dashboard',
|
||||
ua: {
|
||||
browser: { name: 'Chrome', major: '120' },
|
||||
os: { name: 'Windows', version: '10' },
|
||||
},
|
||||
};
|
||||
|
||||
await storeUserEventV2(msg);
|
||||
|
||||
expect(Point).toHaveBeenCalledWith('user_events');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('host', 'host1.example.com');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('event_action', 'OpenApp');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('userFull', 'DOMAIN\\john.doe');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('userDirectory', 'DOMAIN');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('userId', 'john.doe');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('origin', 'QlikSense');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('appId', 'app-123');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('appName', 'Sales Dashboard');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('uaBrowserName', 'Chrome');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('uaBrowserMajorVersion', '120');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('uaOsName', 'Windows');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('uaOsVersion', '10');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('userFull', 'DOMAIN\\john.doe');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('userId', 'john.doe');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('appId_field', 'app-123');
|
||||
expect(mockPoint.stringField).toHaveBeenCalledWith('appName_field', 'Sales Dashboard');
|
||||
expect(mockV2Utils.applyInfluxTags).toHaveBeenCalledWith(mockPoint, [
|
||||
{ name: 'env', value: 'prod' },
|
||||
]);
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
expect(mockWriteApi.writePoint).toHaveBeenCalled();
|
||||
expect(mockWriteApi.close).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle event without app info', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
command: 'Login',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
};
|
||||
|
||||
await storeUserEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).not.toHaveBeenCalledWith('appId', expect.anything());
|
||||
expect(mockPoint.tag).not.toHaveBeenCalledWith('appName', expect.anything());
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle event without user agent', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
};
|
||||
|
||||
await storeUserEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).not.toHaveBeenCalledWith('uaBrowserName', expect.anything());
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle partial user agent info', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
ua: {
|
||||
browser: { name: 'Firefox' }, // no major version
|
||||
// no os info
|
||||
},
|
||||
};
|
||||
|
||||
await storeUserEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('uaBrowserName', 'Firefox');
|
||||
expect(mockPoint.tag).not.toHaveBeenCalledWith('uaBrowserMajorVersion', expect.anything());
|
||||
expect(utils.writeToInfluxWithRetry).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should log debug information', async () => {
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
};
|
||||
|
||||
await storeUserEventV2(msg);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalled();
|
||||
expect(globals.logger.silly).toHaveBeenCalled();
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'USER EVENT V2: Sent user event data to InfluxDB'
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle different event commands', async () => {
|
||||
const commands = ['OpenApp', 'CreateApp', 'DeleteApp', 'ReloadApp'];
|
||||
|
||||
for (const command of commands) {
|
||||
jest.clearAllMocks();
|
||||
const msg = {
|
||||
host: 'host1',
|
||||
command,
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
};
|
||||
|
||||
await storeUserEventV2(msg);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('event_action', command);
|
||||
}
|
||||
});
|
||||
});
|
||||
189
src/lib/influxdb/__tests__/v2-utils.test.js
Normal file
189
src/lib/influxdb/__tests__/v2-utils.test.js
Normal file
@@ -0,0 +1,189 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
const mockPoint = {
|
||||
tag: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
describe('v2/utils', () => {
|
||||
let applyInfluxTags, Point;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
const InfluxClient = await import('@influxdata/influxdb-client');
|
||||
Point = InfluxClient.Point;
|
||||
const utils = await import('../v2/utils.js');
|
||||
applyInfluxTags = utils.applyInfluxTags;
|
||||
|
||||
mockPoint.tag.mockReturnThis();
|
||||
});
|
||||
|
||||
test('should apply single tag', () => {
|
||||
const tags = [{ name: 'env', value: 'prod' }];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('env', 'prod');
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should apply multiple tags', () => {
|
||||
const tags = [
|
||||
{ name: 'env', value: 'prod' },
|
||||
{ name: 'region', value: 'us-east' },
|
||||
{ name: 'cluster', value: 'main' },
|
||||
];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledTimes(3);
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('env', 'prod');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('region', 'us-east');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('cluster', 'main');
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should handle null tags', () => {
|
||||
const result = applyInfluxTags(mockPoint, null);
|
||||
|
||||
expect(mockPoint.tag).not.toHaveBeenCalled();
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should handle undefined tags', () => {
|
||||
const result = applyInfluxTags(mockPoint, undefined);
|
||||
|
||||
expect(mockPoint.tag).not.toHaveBeenCalled();
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should handle empty array', () => {
|
||||
const result = applyInfluxTags(mockPoint, []);
|
||||
|
||||
expect(mockPoint.tag).not.toHaveBeenCalled();
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should skip tags with null values', () => {
|
||||
const tags = [
|
||||
{ name: 'env', value: 'prod' },
|
||||
{ name: 'region', value: null },
|
||||
{ name: 'cluster', value: 'main' },
|
||||
];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledTimes(2);
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('env', 'prod');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('cluster', 'main');
|
||||
expect(mockPoint.tag).not.toHaveBeenCalledWith('region', null);
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should skip tags with undefined values', () => {
|
||||
const tags = [
|
||||
{ name: 'env', value: 'prod' },
|
||||
{ name: 'region', value: undefined },
|
||||
{ name: 'cluster', value: 'main' },
|
||||
];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledTimes(2);
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('env', 'prod');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('cluster', 'main');
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should skip tags without name', () => {
|
||||
const tags = [
|
||||
{ name: 'env', value: 'prod' },
|
||||
{ value: 'no-name' },
|
||||
{ name: 'cluster', value: 'main' },
|
||||
];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledTimes(2);
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('env', 'prod');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('cluster', 'main');
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should convert non-string values to strings', () => {
|
||||
const tags = [
|
||||
{ name: 'count', value: 123 },
|
||||
{ name: 'enabled', value: true },
|
||||
{ name: 'ratio', value: 3.14 },
|
||||
];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('count', '123');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('enabled', 'true');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('ratio', '3.14');
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should handle empty string values', () => {
|
||||
const tags = [
|
||||
{ name: 'env', value: '' },
|
||||
{ name: 'region', value: 'us-east' },
|
||||
];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledTimes(2);
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('env', '');
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('region', 'us-east');
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should handle zero as value', () => {
|
||||
const tags = [{ name: 'count', value: 0 }];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('count', '0');
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should handle false as value', () => {
|
||||
const tags = [{ name: 'enabled', value: false }];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.tag).toHaveBeenCalledWith('enabled', 'false');
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should handle non-array input', () => {
|
||||
const result = applyInfluxTags(mockPoint, 'not-an-array');
|
||||
|
||||
expect(mockPoint.tag).not.toHaveBeenCalled();
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should handle object instead of array', () => {
|
||||
const result = applyInfluxTags(mockPoint, { name: 'env', value: 'prod' });
|
||||
|
||||
expect(mockPoint.tag).not.toHaveBeenCalled();
|
||||
expect(result).toBe(mockPoint);
|
||||
});
|
||||
|
||||
test('should support method chaining', () => {
|
||||
const tags = [
|
||||
{ name: 'env', value: 'prod' },
|
||||
{ name: 'region', value: 'us-east' },
|
||||
];
|
||||
|
||||
const result = applyInfluxTags(mockPoint, tags);
|
||||
|
||||
// The function returns the point for chaining
|
||||
expect(result).toBe(mockPoint);
|
||||
expect(typeof result.tag).toBe('function');
|
||||
});
|
||||
});
|
||||
146
src/lib/influxdb/__tests__/v3-butler-memory.test.js
Normal file
146
src/lib/influxdb/__tests__/v3-butler-memory.test.js
Normal file
@@ -0,0 +1,146 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
write: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
appVersion: '1.0.0',
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV3: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
// Mock Point3
|
||||
const mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
setFloatField: jest.fn().mockReturnThis(),
|
||||
toLineProtocol: jest.fn().mockReturnValue('butlersos_memory_usage'),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb3-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
describe('v3/butler-memory', () => {
|
||||
let postButlerSOSMemoryUsageToInfluxdbV3;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const butlerMemory = await import('../v3/butler-memory.js');
|
||||
postButlerSOSMemoryUsageToInfluxdbV3 = butlerMemory.postButlerSOSMemoryUsageToInfluxdbV3;
|
||||
|
||||
// Setup default mocks
|
||||
globals.config.get.mockReturnValue('test-db');
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeBatchToInfluxV3.mockResolvedValue();
|
||||
});
|
||||
|
||||
describe('postButlerSOSMemoryUsageToInfluxdbV3', () => {
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
|
||||
const memory = {
|
||||
instanceTag: 'prod-instance',
|
||||
heapUsedMByte: 100,
|
||||
heapTotalMByte: 200,
|
||||
externalMemoryMByte: 50,
|
||||
processMemoryMByte: 250,
|
||||
};
|
||||
|
||||
await postButlerSOSMemoryUsageToInfluxdbV3(memory);
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write memory usage metrics', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'prod-instance',
|
||||
heapUsedMByte: 100.5,
|
||||
heapTotalMByte: 200.75,
|
||||
externalMemoryMByte: 50.25,
|
||||
processMemoryMByte: 250.5,
|
||||
};
|
||||
|
||||
await postButlerSOSMemoryUsageToInfluxdbV3(memory);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('butler_sos_instance', 'prod-instance');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('version', '1.0.0');
|
||||
expect(mockPoint.setFloatField).toHaveBeenCalledWith('heap_used', 100.5);
|
||||
expect(mockPoint.setFloatField).toHaveBeenCalledWith('heap_total', 200.75);
|
||||
expect(mockPoint.setFloatField).toHaveBeenCalledWith('external', 50.25);
|
||||
expect(mockPoint.setFloatField).toHaveBeenCalledWith('process_memory', 250.5);
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'prod-instance',
|
||||
heapUsedMByte: 100,
|
||||
heapTotalMByte: 200,
|
||||
externalMemoryMByte: 50,
|
||||
processMemoryMByte: 250,
|
||||
};
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV3.mockRejectedValue(writeError);
|
||||
|
||||
await postButlerSOSMemoryUsageToInfluxdbV3(memory);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error saving memory usage data')
|
||||
);
|
||||
});
|
||||
|
||||
test('should log debug messages', async () => {
|
||||
const memory = {
|
||||
instanceTag: 'test-instance',
|
||||
heapUsedMByte: 50,
|
||||
heapTotalMByte: 100,
|
||||
externalMemoryMByte: 25,
|
||||
processMemoryMByte: 125,
|
||||
};
|
||||
|
||||
await postButlerSOSMemoryUsageToInfluxdbV3(memory);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('MEMORY USAGE V3: Memory usage')
|
||||
);
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Wrote data to InfluxDB v3')
|
||||
);
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Sent Butler SOS memory usage data')
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
281
src/lib/influxdb/__tests__/v3-event-counts.test.js
Normal file
281
src/lib/influxdb/__tests__/v3-event-counts.test.js
Normal file
@@ -0,0 +1,281 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
write: jest.fn(),
|
||||
},
|
||||
options: {
|
||||
instanceTag: 'test-instance',
|
||||
},
|
||||
udpEvents: {
|
||||
getLogEvents: jest.fn(),
|
||||
getUserEvents: jest.fn(),
|
||||
},
|
||||
rejectedEvents: {
|
||||
getRejectedLogEvents: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV3: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
// Mock Point3
|
||||
const mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
setIntegerField: jest.fn().mockReturnThis(),
|
||||
setFloatField: jest.fn().mockReturnThis(),
|
||||
toLineProtocol: jest.fn().mockReturnValue('event_count'),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb3-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
describe('v3/event-counts', () => {
|
||||
let storeEventCountInfluxDBV3;
|
||||
let storeRejectedEventCountInfluxDBV3;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const eventCounts = await import('../v3/event-counts.js');
|
||||
storeEventCountInfluxDBV3 = eventCounts.storeEventCountInfluxDBV3;
|
||||
storeRejectedEventCountInfluxDBV3 = eventCounts.storeRejectedEventCountInfluxDBV3;
|
||||
|
||||
// Setup default mocks
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.v3Config.database') return 'test-db';
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName')
|
||||
return 'event_count';
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.rejectedEventCount.influxdb.measurementName')
|
||||
return 'rejected_event_count';
|
||||
if (key === 'Butler-SOS.influxdbConfig.maxBatchSize') return 100;
|
||||
return null;
|
||||
});
|
||||
globals.config.has.mockReturnValue(false);
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockResolvedValue();
|
||||
utils.writeBatchToInfluxV3.mockResolvedValue();
|
||||
});
|
||||
|
||||
describe('storeEventCountInfluxDBV3', () => {
|
||||
test('should return early when no events to store', async () => {
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([]);
|
||||
|
||||
await storeEventCountInfluxDBV3();
|
||||
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining('No events to store')
|
||||
);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([{ source: 'test' }]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([]);
|
||||
|
||||
await storeEventCountInfluxDBV3();
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should store log events successfully', async () => {
|
||||
const logEvents = [
|
||||
{
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
subsystem: 'Engine',
|
||||
counter: 10,
|
||||
},
|
||||
{
|
||||
source: 'qseow-proxy',
|
||||
host: 'server2',
|
||||
subsystem: 'Proxy',
|
||||
counter: 5,
|
||||
},
|
||||
];
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue(logEvents);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([]);
|
||||
|
||||
await storeEventCountInfluxDBV3();
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledTimes(1);
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('event_type', 'log');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('source', 'qseow-engine');
|
||||
expect(mockPoint.setIntegerField).toHaveBeenCalledWith('counter', 10);
|
||||
});
|
||||
|
||||
test('should store user events successfully', async () => {
|
||||
const userEvents = [
|
||||
{
|
||||
source: 'user-activity',
|
||||
host: 'server1',
|
||||
subsystem: 'N/A',
|
||||
counter: 15,
|
||||
},
|
||||
];
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue([]);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue(userEvents);
|
||||
|
||||
await storeEventCountInfluxDBV3();
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledTimes(1);
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('event_type', 'user');
|
||||
expect(mockPoint.setIntegerField).toHaveBeenCalledWith('counter', 15);
|
||||
});
|
||||
|
||||
test('should store both log and user events', async () => {
|
||||
const logEvents = [
|
||||
{ source: 'qseow-engine', host: 'server1', subsystem: 'Engine', counter: 10 },
|
||||
];
|
||||
const userEvents = [
|
||||
{ source: 'user-activity', host: 'server1', subsystem: 'N/A', counter: 5 },
|
||||
];
|
||||
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue(logEvents);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue(userEvents);
|
||||
|
||||
await storeEventCountInfluxDBV3();
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('should apply config tags when available', async () => {
|
||||
globals.config.has.mockReturnValue(true);
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.v3Config.database') return 'test-db';
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName')
|
||||
return 'event_count';
|
||||
if (key === 'Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags')
|
||||
return [{ name: 'env', value: 'production' }];
|
||||
return null;
|
||||
});
|
||||
|
||||
const logEvents = [
|
||||
{ source: 'qseow-engine', host: 'server1', subsystem: 'Engine', counter: 10 },
|
||||
];
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue(logEvents);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([]);
|
||||
|
||||
await storeEventCountInfluxDBV3();
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('env', 'production');
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
const logEvents = [
|
||||
{ source: 'qseow-engine', host: 'server1', subsystem: 'Engine', counter: 10 },
|
||||
];
|
||||
globals.udpEvents.getLogEvents.mockResolvedValue(logEvents);
|
||||
globals.udpEvents.getUserEvents.mockResolvedValue([]);
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV3.mockRejectedValue(writeError);
|
||||
|
||||
await storeEventCountInfluxDBV3();
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error writing data to InfluxDB')
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('storeRejectedEventCountInfluxDBV3', () => {
|
||||
test('should return early when no events to store', async () => {
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([]);
|
||||
|
||||
await storeRejectedEventCountInfluxDBV3();
|
||||
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
expect.stringContaining('No events to store')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue([{ source: 'test' }]);
|
||||
|
||||
await storeRejectedEventCountInfluxDBV3();
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should store rejected log events successfully', async () => {
|
||||
const logEvents = [
|
||||
{
|
||||
source: 'qseow-qix-perf',
|
||||
objectType: 'Doc',
|
||||
method: 'GetLayout',
|
||||
counter: 3,
|
||||
processTime: 1.5,
|
||||
appId: 'test-app-123',
|
||||
appName: 'Test App',
|
||||
},
|
||||
];
|
||||
globals.config.has.mockReturnValue(false); // No custom tags
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue(logEvents);
|
||||
|
||||
await storeRejectedEventCountInfluxDBV3();
|
||||
|
||||
// Should have written the rejected event
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Wrote data to InfluxDB v3')
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle write errors for rejected events', async () => {
|
||||
const logEvents = [
|
||||
{
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
subsystem: 'Engine',
|
||||
counter_rejected: 3,
|
||||
},
|
||||
];
|
||||
globals.rejectedEvents.getRejectedLogEvents.mockResolvedValue(logEvents);
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV3.mockRejectedValue(writeError);
|
||||
|
||||
await storeRejectedEventCountInfluxDBV3();
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error writing data to InfluxDB')
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
255
src/lib/influxdb/__tests__/v3-health-metrics.test.js
Normal file
255
src/lib/influxdb/__tests__/v3-health-metrics.test.js
Normal file
@@ -0,0 +1,255 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
write: jest.fn(),
|
||||
},
|
||||
influxWriteApi: [],
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
getFormattedTime: jest.fn(),
|
||||
processAppDocuments: jest.fn(),
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
applyTagsToPoint3: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV3: jest.fn(),
|
||||
validateUnsignedField: jest.fn((value) =>
|
||||
typeof value === 'number' && value >= 0 ? value : 0
|
||||
),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
// Mock Point3
|
||||
/**
|
||||
* Create a mock Point instance
|
||||
*
|
||||
* @returns {object} Mock Point instance
|
||||
*/
|
||||
const createMockPoint = () => ({
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
setStringField: jest.fn().mockReturnThis(),
|
||||
setIntegerField: jest.fn().mockReturnThis(),
|
||||
setFloatField: jest.fn().mockReturnThis(),
|
||||
setBooleanField: jest.fn().mockReturnThis(),
|
||||
toLineProtocol: jest.fn().mockReturnValue('health_metrics'),
|
||||
});
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb3-client', () => ({
|
||||
Point: jest.fn(() => createMockPoint()),
|
||||
}));
|
||||
|
||||
describe('v3/health-metrics', () => {
|
||||
let postHealthMetricsToInfluxdbV3;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const healthMetrics = await import('../v3/health-metrics.js');
|
||||
postHealthMetricsToInfluxdbV3 = healthMetrics.postHealthMetricsToInfluxdbV3;
|
||||
|
||||
// Setup default mocks
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.v3Config.database') return 'test-db';
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.activeDocs') return true;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.loadedDocs') return true;
|
||||
if (key === 'Butler-SOS.influxdbConfig.includeFields.inMemoryDocs') return true;
|
||||
if (key === 'Butler-SOS.appNames.enableAppNameExtract') return true;
|
||||
if (key === 'Butler-SOS.influxdbConfig.maxBatchSize') return 100;
|
||||
return false;
|
||||
});
|
||||
|
||||
utils.getFormattedTime.mockReturnValue('1d 2h 30m');
|
||||
utils.processAppDocuments.mockResolvedValue({
|
||||
appNames: ['App1', 'App2'],
|
||||
sessionAppNames: ['SessionApp1'],
|
||||
});
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeBatchToInfluxV3.mockResolvedValue();
|
||||
utils.applyTagsToPoint3.mockImplementation(() => {});
|
||||
|
||||
// Setup influxWriteApi
|
||||
globals.influxWriteApi = [
|
||||
{
|
||||
serverName: 'test-server',
|
||||
writeApi: {},
|
||||
},
|
||||
];
|
||||
});
|
||||
|
||||
/**
|
||||
* Create mock health metrics body
|
||||
*
|
||||
* @returns {object} Mock body with health metrics
|
||||
*/
|
||||
const createMockBody = () => ({
|
||||
version: '14.76.3',
|
||||
started: '2024-01-01T00:00:00Z',
|
||||
mem: {
|
||||
committed: 1000000,
|
||||
allocated: 800000,
|
||||
free: 200000,
|
||||
},
|
||||
apps: {
|
||||
active_docs: ['doc1', 'doc2'],
|
||||
loaded_docs: ['doc3'],
|
||||
in_memory_docs: ['doc4', 'doc5'],
|
||||
calls: 100,
|
||||
selections: 50,
|
||||
},
|
||||
cpu: {
|
||||
total: 45,
|
||||
},
|
||||
session: {
|
||||
active: 10,
|
||||
total: 15,
|
||||
},
|
||||
users: {
|
||||
active: 5,
|
||||
total: 8,
|
||||
},
|
||||
cache: {
|
||||
hits: 1000,
|
||||
lookups: 1200,
|
||||
added: 50,
|
||||
replaced: 10,
|
||||
bytes_added: 500000,
|
||||
},
|
||||
saturated: false,
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
const body = createMockBody();
|
||||
|
||||
await postHealthMetricsToInfluxdbV3('test-server', 'test-host', body, {});
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should warn and return when influxWriteApi is not initialized', async () => {
|
||||
globals.influxWriteApi = null;
|
||||
const body = createMockBody();
|
||||
|
||||
await postHealthMetricsToInfluxdbV3('test-server', 'test-host', body, {});
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Influxdb write API object not initialized')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should warn and return when writeApi not found for server', async () => {
|
||||
const body = createMockBody();
|
||||
|
||||
await postHealthMetricsToInfluxdbV3('unknown-server', 'test-host', body, {});
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Influxdb write API object not found for host test-host')
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should process and write all health metrics successfully', async () => {
|
||||
const body = createMockBody();
|
||||
const serverTags = { env: 'production', cluster: 'main' };
|
||||
|
||||
await postHealthMetricsToInfluxdbV3('test-server', 'test-host', body, serverTags);
|
||||
|
||||
// Should process all three app doc types
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledTimes(3);
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledWith(
|
||||
body.apps.active_docs,
|
||||
'HEALTH METRICS TO INFLUXDB V3',
|
||||
'active'
|
||||
);
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledWith(
|
||||
body.apps.loaded_docs,
|
||||
'HEALTH METRICS TO INFLUXDB V3',
|
||||
'loaded'
|
||||
);
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledWith(
|
||||
body.apps.in_memory_docs,
|
||||
'HEALTH METRICS TO INFLUXDB V3',
|
||||
'in memory'
|
||||
);
|
||||
|
||||
// Should apply tags to all 8 points
|
||||
expect(utils.applyTagsToPoint3).toHaveBeenCalledTimes(8);
|
||||
|
||||
// Should write all 8 measurements in one batch
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledTimes(1);
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'test-db',
|
||||
expect.stringContaining('Health metrics for'),
|
||||
'health-metrics',
|
||||
100
|
||||
);
|
||||
});
|
||||
|
||||
test('should call getFormattedTime with started timestamp', async () => {
|
||||
const body = createMockBody();
|
||||
|
||||
await postHealthMetricsToInfluxdbV3('test-server', 'test-host', body, {});
|
||||
|
||||
expect(utils.getFormattedTime).toHaveBeenCalledWith(body.started);
|
||||
});
|
||||
|
||||
test('should handle app name extraction being disabled', async () => {
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.v3Config.database') return 'test-db';
|
||||
if (key === 'Butler-SOS.appNames.enableAppNameExtract') return false;
|
||||
return false;
|
||||
});
|
||||
|
||||
const body = createMockBody();
|
||||
|
||||
await postHealthMetricsToInfluxdbV3('test-server', 'test-host', body, {});
|
||||
|
||||
// Should still process but set empty strings for app names
|
||||
expect(utils.processAppDocuments).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
test('should handle write errors with error tracking', async () => {
|
||||
const body = createMockBody();
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV3.mockRejectedValue(writeError);
|
||||
|
||||
await postHealthMetricsToInfluxdbV3('test-server', 'test-host', body, {});
|
||||
|
||||
expect(globals.errorTracker.incrementError).toHaveBeenCalledWith(
|
||||
'INFLUXDB_V3_WRITE',
|
||||
'test-server'
|
||||
);
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error saving health data to InfluxDB v3')
|
||||
);
|
||||
});
|
||||
});
|
||||
228
src/lib/influxdb/__tests__/v3-log-events.test.js
Normal file
228
src/lib/influxdb/__tests__/v3-log-events.test.js
Normal file
@@ -0,0 +1,228 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
write: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV3: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
// Mock Point3
|
||||
const mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
setField: jest.fn().mockReturnThis(),
|
||||
setStringField: jest.fn().mockReturnThis(),
|
||||
setIntegerField: jest.fn().mockReturnThis(),
|
||||
setFloatField: jest.fn().mockReturnThis(),
|
||||
toLineProtocol: jest.fn().mockReturnValue('log_events'),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb3-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
describe('v3/log-events', () => {
|
||||
let postLogEventToInfluxdbV3;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const logEvents = await import('../v3/log-events.js');
|
||||
postLogEventToInfluxdbV3 = logEvents.postLogEventToInfluxdbV3;
|
||||
|
||||
// Setup default mocks
|
||||
globals.config.get.mockReturnValue('test-db');
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockResolvedValue();
|
||||
});
|
||||
|
||||
describe('postLogEventToInfluxdbV3', () => {
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
|
||||
const msg = {
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
};
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should warn and return for unknown log event source', async () => {
|
||||
const msg = {
|
||||
source: 'unknown-source',
|
||||
host: 'server1',
|
||||
};
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Unknown log event source: unknown-source')
|
||||
);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write qseow-engine log event', async () => {
|
||||
const msg = {
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
message: 'Test message',
|
||||
log_row: 'Full log row',
|
||||
};
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('host', 'server1');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('source', 'qseow-engine');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('level', 'INFO');
|
||||
expect(mockPoint.setStringField).toHaveBeenCalledWith('message', 'Test message');
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write qseow-proxy log event', async () => {
|
||||
const msg = {
|
||||
source: 'qseow-proxy',
|
||||
host: 'server1',
|
||||
level: 'WARN',
|
||||
message: 'Proxy warning',
|
||||
};
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('host', 'server1');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('source', 'qseow-proxy');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('level', 'WARN');
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write qseow-scheduler log event', async () => {
|
||||
const msg = {
|
||||
source: 'qseow-scheduler',
|
||||
host: 'server1',
|
||||
level: 'ERROR',
|
||||
message: 'Scheduler error',
|
||||
};
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('source', 'qseow-scheduler');
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write qseow-repository log event', async () => {
|
||||
const msg = {
|
||||
source: 'qseow-repository',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
message: 'Repository info',
|
||||
};
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('source', 'qseow-repository');
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write qseow-qix-perf log event', async () => {
|
||||
const msg = {
|
||||
source: 'qseow-qix-perf',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
message: 'Performance metric',
|
||||
method: 'GetData',
|
||||
object_type: 'GenericObject',
|
||||
process_time: 123.45,
|
||||
work_time: 100.0,
|
||||
lock_time: 10.0,
|
||||
validate_time: 5.0,
|
||||
traverse_time: 8.45,
|
||||
handle: 42,
|
||||
net_ram: 1024,
|
||||
peak_ram: 2048,
|
||||
};
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('source', 'qseow-qix-perf');
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
const msg = {
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
level: 'INFO',
|
||||
message: 'Test message',
|
||||
};
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV3.mockRejectedValue(writeError);
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error saving log event to InfluxDB')
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle log event with all optional fields', async () => {
|
||||
const msg = {
|
||||
source: 'qseow-engine',
|
||||
host: 'server1',
|
||||
level: 'ERROR',
|
||||
message: 'Error message',
|
||||
exception_message: 'Exception details',
|
||||
command: 'OpenDoc',
|
||||
result_code: '500',
|
||||
origin: 'API',
|
||||
context: 'Session context',
|
||||
session_id: 'session-123',
|
||||
log_row: 'Complete log row',
|
||||
};
|
||||
|
||||
await postLogEventToInfluxdbV3(msg);
|
||||
|
||||
expect(mockPoint.setStringField).toHaveBeenCalledWith('message', 'Error message');
|
||||
expect(mockPoint.setStringField).toHaveBeenCalledWith(
|
||||
'exception_message',
|
||||
'Exception details'
|
||||
);
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
335
src/lib/influxdb/__tests__/v3-queue-metrics.test.js
Normal file
335
src/lib/influxdb/__tests__/v3-queue-metrics.test.js
Normal file
@@ -0,0 +1,335 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
write: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
influxDefaultDb: 'test-db',
|
||||
udpQueueManagerUserActivity: null,
|
||||
udpQueueManagerLogEvents: null,
|
||||
hostInfo: {
|
||||
hostname: 'test-host',
|
||||
},
|
||||
getErrorMessage: jest.fn().mockImplementation((err) => err.message || err.toString()),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock InfluxDB v3 client
|
||||
jest.unstable_mockModule('@influxdata/influxdb3-client', () => ({
|
||||
Point: jest.fn().mockImplementation(() => ({
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
setFloatField: jest.fn().mockReturnThis(),
|
||||
setIntegerField: jest.fn().mockReturnThis(),
|
||||
setStringField: jest.fn().mockReturnThis(),
|
||||
setBooleanField: jest.fn().mockReturnThis(),
|
||||
setTimestamp: jest.fn().mockReturnThis(),
|
||||
toLineProtocol: jest.fn().mockReturnValue('mock-line-protocol'),
|
||||
})),
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
jest.unstable_mockModule('../shared/utils.js', () => ({
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV3: jest.fn(),
|
||||
}));
|
||||
|
||||
describe('InfluxDB v3 Queue Metrics', () => {
|
||||
let queueMetrics;
|
||||
let globals;
|
||||
let Point3;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
const influxdbV3 = await import('@influxdata/influxdb3-client');
|
||||
Point3 = influxdbV3.Point;
|
||||
utils = await import('../shared/utils.js');
|
||||
|
||||
queueMetrics = await import('../v3/queue-metrics.js');
|
||||
|
||||
// Setup default mocks
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeBatchToInfluxV3.mockResolvedValue();
|
||||
});
|
||||
|
||||
describe('postUserEventQueueMetricsToInfluxdbV3', () => {
|
||||
test('should return early when queue metrics are disabled', async () => {
|
||||
globals.config.get.mockReturnValue(false);
|
||||
|
||||
await queueMetrics.postUserEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(Point3).not.toHaveBeenCalled();
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should warn when queue manager is not initialized', async () => {
|
||||
globals.config.get.mockReturnValue(true);
|
||||
globals.udpQueueManagerUserActivity = null;
|
||||
|
||||
await queueMetrics.postUserEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
'USER EVENT QUEUE METRICS INFLUXDB V3: Queue manager not initialized'
|
||||
);
|
||||
expect(Point3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should return early when InfluxDB is not enabled', async () => {
|
||||
globals.config.get.mockReturnValue(true);
|
||||
globals.udpQueueManagerUserActivity = { getMetrics: jest.fn() };
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
|
||||
await queueMetrics.postUserEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(Point3).not.toHaveBeenCalled();
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write queue metrics', async () => {
|
||||
const mockMetrics = {
|
||||
queueSize: 10,
|
||||
queueMaxSize: 100,
|
||||
queueUtilizationPct: 10.5,
|
||||
queuePending: 2,
|
||||
messagesReceived: 1000,
|
||||
messagesQueued: 950,
|
||||
messagesProcessed: 940,
|
||||
messagesFailed: 5,
|
||||
messagesDroppedTotal: 50,
|
||||
messagesDroppedRateLimit: 10,
|
||||
messagesDroppedQueueFull: 30,
|
||||
messagesDroppedSize: 10,
|
||||
processingTimeAvgMs: 15.5,
|
||||
processingTimeP95Ms: 45.2,
|
||||
processingTimeMaxMs: 120.0,
|
||||
rateLimitCurrent: 500,
|
||||
backpressureActive: 0,
|
||||
};
|
||||
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.enable') {
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
key ===
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
) {
|
||||
return 'user_events_queue';
|
||||
}
|
||||
if (key === 'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.tags') {
|
||||
return [{ name: 'env', value: 'test' }];
|
||||
}
|
||||
if (key === 'Butler-SOS.influxdbConfig.v3Config.database') {
|
||||
return 'test-db';
|
||||
}
|
||||
if (key === 'Butler-SOS.influxdbConfig.maxBatchSize') {
|
||||
return 100;
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
globals.udpQueueManagerUserActivity = {
|
||||
getMetrics: jest.fn().mockResolvedValue(mockMetrics),
|
||||
clearMetrics: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
await queueMetrics.postUserEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(Point3).toHaveBeenCalledWith('user_events_queue');
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'test-db',
|
||||
'User event queue metrics',
|
||||
'user-events-queue',
|
||||
100
|
||||
);
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'USER EVENT QUEUE METRICS INFLUXDB V3: Sent queue metrics data to InfluxDB v3'
|
||||
);
|
||||
expect(globals.udpQueueManagerUserActivity.clearMetrics).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle errors gracefully', async () => {
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.enable') {
|
||||
return true;
|
||||
}
|
||||
throw new Error('Config error');
|
||||
});
|
||||
|
||||
globals.udpQueueManagerUserActivity = {
|
||||
getMetrics: jest.fn(),
|
||||
};
|
||||
|
||||
await queueMetrics.postUserEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'USER EVENT QUEUE METRICS INFLUXDB V3: Error posting queue metrics'
|
||||
)
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('postLogEventQueueMetricsToInfluxdbV3', () => {
|
||||
test('should return early when queue metrics are disabled', async () => {
|
||||
globals.config.get.mockReturnValue(false);
|
||||
|
||||
await queueMetrics.postLogEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(Point3).not.toHaveBeenCalled();
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should warn when queue manager is not initialized', async () => {
|
||||
globals.config.get.mockReturnValue(true);
|
||||
globals.udpQueueManagerLogEvents = null;
|
||||
|
||||
await queueMetrics.postLogEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
'LOG EVENT QUEUE METRICS INFLUXDB V3: Queue manager not initialized'
|
||||
);
|
||||
expect(Point3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write queue metrics', async () => {
|
||||
const mockMetrics = {
|
||||
queueSize: 5,
|
||||
queueMaxSize: 100,
|
||||
queueUtilizationPct: 5.0,
|
||||
queuePending: 1,
|
||||
messagesReceived: 500,
|
||||
messagesQueued: 490,
|
||||
messagesProcessed: 485,
|
||||
messagesFailed: 2,
|
||||
messagesDroppedTotal: 10,
|
||||
messagesDroppedRateLimit: 5,
|
||||
messagesDroppedQueueFull: 3,
|
||||
messagesDroppedSize: 2,
|
||||
processingTimeAvgMs: 12.3,
|
||||
processingTimeP95Ms: 38.9,
|
||||
processingTimeMaxMs: 95.0,
|
||||
rateLimitCurrent: 400,
|
||||
backpressureActive: 0,
|
||||
};
|
||||
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.enable') {
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
key ===
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
) {
|
||||
return 'log_events_queue';
|
||||
}
|
||||
if (key === 'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.tags') {
|
||||
return [];
|
||||
}
|
||||
if (key === 'Butler-SOS.influxdbConfig.v3Config.database') {
|
||||
return 'test-db';
|
||||
}
|
||||
if (key === 'Butler-SOS.influxdbConfig.maxBatchSize') {
|
||||
return 100;
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
globals.udpQueueManagerLogEvents = {
|
||||
getMetrics: jest.fn().mockResolvedValue(mockMetrics),
|
||||
clearMetrics: jest.fn().mockResolvedValue(),
|
||||
};
|
||||
|
||||
await queueMetrics.postLogEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(Point3).toHaveBeenCalledWith('log_events_queue');
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledWith(
|
||||
expect.any(Array),
|
||||
'test-db',
|
||||
'Log event queue metrics',
|
||||
'log-events-queue',
|
||||
100
|
||||
);
|
||||
expect(globals.logger.verbose).toHaveBeenCalledWith(
|
||||
'LOG EVENT QUEUE METRICS INFLUXDB V3: Sent queue metrics data to InfluxDB v3'
|
||||
);
|
||||
expect(globals.udpQueueManagerLogEvents.clearMetrics).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.enable') {
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
key ===
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
) {
|
||||
return 'log_events_queue';
|
||||
}
|
||||
if (key === 'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.tags') {
|
||||
return [];
|
||||
}
|
||||
if (key === 'Butler-SOS.influxdbConfig.v3Config.database') {
|
||||
return 'test-db';
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
globals.udpQueueManagerLogEvents = {
|
||||
getMetrics: jest.fn().mockResolvedValue({
|
||||
queueSize: 5,
|
||||
queueMaxSize: 100,
|
||||
queueUtilizationPct: 5.0,
|
||||
queuePending: 1,
|
||||
messagesReceived: 500,
|
||||
messagesQueued: 490,
|
||||
messagesProcessed: 485,
|
||||
messagesFailed: 2,
|
||||
messagesDroppedTotal: 10,
|
||||
messagesDroppedRateLimit: 5,
|
||||
messagesDroppedQueueFull: 3,
|
||||
messagesDroppedSize: 2,
|
||||
processingTimeAvgMs: 12.3,
|
||||
processingTimeP95Ms: 38.9,
|
||||
processingTimeMaxMs: 95.0,
|
||||
rateLimitCurrent: 400,
|
||||
backpressureActive: 0,
|
||||
}),
|
||||
clearMetrics: jest.fn(),
|
||||
};
|
||||
|
||||
utils.writeBatchToInfluxV3.mockRejectedValue(new Error('Write failed'));
|
||||
|
||||
await queueMetrics.postLogEventQueueMetricsToInfluxdbV3();
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'LOG EVENT QUEUE METRICS INFLUXDB V3: Error posting queue metrics'
|
||||
)
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
203
src/lib/influxdb/__tests__/v3-sessions.test.js
Normal file
203
src/lib/influxdb/__tests__/v3-sessions.test.js
Normal file
@@ -0,0 +1,203 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
write: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV3: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
// Mock Point3
|
||||
const mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
setField: jest.fn().mockReturnThis(),
|
||||
toLineProtocol: jest.fn().mockReturnValue('proxy_sessions'),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb3-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
describe('v3/sessions', () => {
|
||||
let postProxySessionsToInfluxdbV3;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const sessions = await import('../v3/sessions.js');
|
||||
postProxySessionsToInfluxdbV3 = sessions.postProxySessionsToInfluxdbV3;
|
||||
|
||||
// Setup default mocks
|
||||
globals.config.get.mockImplementation((key) => {
|
||||
if (key === 'Butler-SOS.influxdbConfig.v3Config.database') return 'test-db';
|
||||
if (key === 'Butler-SOS.influxdbConfig.maxBatchSize') return 100;
|
||||
return undefined;
|
||||
});
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeBatchToInfluxV3.mockResolvedValue();
|
||||
});
|
||||
|
||||
describe('postProxySessionsToInfluxdbV3', () => {
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: '/vp1',
|
||||
serverName: 'QSE1',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: [],
|
||||
};
|
||||
|
||||
await postProxySessionsToInfluxdbV3(userSessions);
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should warn when no datapoints to write', async () => {
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: '/vp1',
|
||||
serverName: 'QSE1',
|
||||
sessionCount: 0,
|
||||
uniqueUserList: '',
|
||||
datapointInfluxdb: [],
|
||||
};
|
||||
|
||||
await postProxySessionsToInfluxdbV3(userSessions);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('No datapoints to write')
|
||||
);
|
||||
});
|
||||
|
||||
test('should successfully write session datapoints', async () => {
|
||||
const datapoint1 = { toLineProtocol: jest.fn().mockReturnValue('session1') };
|
||||
const datapoint2 = { toLineProtocol: jest.fn().mockReturnValue('session2') };
|
||||
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: '/vp1',
|
||||
serverName: 'QSE1',
|
||||
sessionCount: 2,
|
||||
uniqueUserList: 'user1,user2',
|
||||
datapointInfluxdb: [datapoint1, datapoint2],
|
||||
};
|
||||
|
||||
await postProxySessionsToInfluxdbV3(userSessions);
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledTimes(1);
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalledWith(
|
||||
[datapoint1, datapoint2],
|
||||
'test-db',
|
||||
'Proxy sessions for server1//vp1',
|
||||
'server1',
|
||||
100
|
||||
);
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Wrote 2 datapoints')
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle write errors and track them', async () => {
|
||||
const datapoint = { toLineProtocol: jest.fn().mockReturnValue('session1') };
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: '/vp1',
|
||||
serverName: 'QSE1',
|
||||
sessionCount: 1,
|
||||
uniqueUserList: 'user1',
|
||||
datapointInfluxdb: [datapoint],
|
||||
};
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV3.mockRejectedValue(writeError);
|
||||
|
||||
await postProxySessionsToInfluxdbV3(userSessions);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error saving user session data')
|
||||
);
|
||||
expect(globals.errorTracker.incrementError).toHaveBeenCalledWith(
|
||||
'INFLUXDB_V3_WRITE',
|
||||
'QSE1'
|
||||
);
|
||||
});
|
||||
|
||||
test('should log session details', async () => {
|
||||
const datapoint = { toLineProtocol: jest.fn().mockReturnValue('session1') };
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: '/vp1',
|
||||
serverName: 'QSE1',
|
||||
sessionCount: 5,
|
||||
uniqueUserList: 'user1,user2,user3',
|
||||
datapointInfluxdb: [datapoint],
|
||||
};
|
||||
|
||||
await postProxySessionsToInfluxdbV3(userSessions);
|
||||
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'Session count for server "server1", virtual proxy "/vp1": 5'
|
||||
)
|
||||
);
|
||||
expect(globals.logger.debug).toHaveBeenCalledWith(
|
||||
expect.stringContaining(
|
||||
'User list for server "server1", virtual proxy "/vp1": user1,user2,user3'
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle null or undefined datapointInfluxdb', async () => {
|
||||
const userSessions = {
|
||||
host: 'server1',
|
||||
virtualProxy: '/vp1',
|
||||
serverName: 'QSE1',
|
||||
sessionCount: 0,
|
||||
uniqueUserList: '',
|
||||
datapointInfluxdb: null,
|
||||
};
|
||||
|
||||
await postProxySessionsToInfluxdbV3(userSessions);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('No datapoints to write')
|
||||
);
|
||||
expect(globals.influx.write).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
237
src/lib/influxdb/__tests__/v3-shared-utils.test.js
Normal file
237
src/lib/influxdb/__tests__/v3-shared-utils.test.js
Normal file
@@ -0,0 +1,237 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
write: jest.fn(),
|
||||
},
|
||||
influxDefaultDb: 'test-db',
|
||||
getErrorMessage: jest.fn().mockImplementation((err) => err.message || err.toString()),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock InfluxDB v3 client
|
||||
jest.unstable_mockModule('@influxdata/influxdb3-client', () => ({
|
||||
Point: jest.fn().mockImplementation(() => ({
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
setFloatField: jest.fn().mockReturnThis(),
|
||||
setIntegerField: jest.fn().mockReturnThis(),
|
||||
setStringField: jest.fn().mockReturnThis(),
|
||||
setBooleanField: jest.fn().mockReturnThis(),
|
||||
setTimestamp: jest.fn().mockReturnThis(),
|
||||
toLineProtocol: jest.fn().mockReturnValue('mock-line-protocol'),
|
||||
})),
|
||||
}));
|
||||
|
||||
describe('InfluxDB v3 Shared Utils', () => {
|
||||
let utils;
|
||||
let globals;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
});
|
||||
|
||||
describe('getInfluxDbVersion', () => {
|
||||
test('should return version from config', () => {
|
||||
globals.config.get.mockReturnValue(3);
|
||||
|
||||
const result = utils.getInfluxDbVersion();
|
||||
|
||||
expect(result).toBe(3);
|
||||
expect(globals.config.get).toHaveBeenCalledWith('Butler-SOS.influxdbConfig.version');
|
||||
});
|
||||
});
|
||||
|
||||
describe('isInfluxDbEnabled', () => {
|
||||
test('should return true when client exists', () => {
|
||||
globals.influx = { write: jest.fn() };
|
||||
|
||||
const result = utils.isInfluxDbEnabled();
|
||||
|
||||
expect(result).toBe(true);
|
||||
});
|
||||
|
||||
test('should return false and log warning when client does not exist', () => {
|
||||
globals.influx = null;
|
||||
|
||||
const result = utils.isInfluxDbEnabled();
|
||||
|
||||
expect(result).toBe(false);
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Influxdb object not initialized')
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('writeToInfluxWithRetry', () => {
|
||||
test('should successfully write on first attempt', async () => {
|
||||
const writeFn = jest.fn().mockResolvedValue();
|
||||
|
||||
await utils.writeToInfluxWithRetry(writeFn, 'Test context', 'v3', '');
|
||||
|
||||
expect(writeFn).toHaveBeenCalledTimes(1);
|
||||
expect(globals.logger.error).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should retry on timeout error and succeed', async () => {
|
||||
const timeoutError = new Error('Request timed out');
|
||||
timeoutError.name = 'RequestTimedOutError';
|
||||
|
||||
const writeFn = jest.fn().mockRejectedValueOnce(timeoutError).mockResolvedValueOnce();
|
||||
|
||||
await utils.writeToInfluxWithRetry(writeFn, 'Test context', 'v3', '', {
|
||||
maxRetries: 3,
|
||||
initialDelayMs: 10,
|
||||
});
|
||||
|
||||
expect(writeFn).toHaveBeenCalledTimes(2);
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('INFLUXDB V3 RETRY: Test context - Retryable')
|
||||
);
|
||||
});
|
||||
|
||||
test('should retry multiple times before succeeding', async () => {
|
||||
const timeoutError = new Error('Request timed out');
|
||||
timeoutError.name = 'RequestTimedOutError';
|
||||
|
||||
const writeFn = jest
|
||||
.fn()
|
||||
.mockRejectedValueOnce(timeoutError)
|
||||
.mockRejectedValueOnce(timeoutError)
|
||||
.mockResolvedValueOnce();
|
||||
|
||||
await utils.writeToInfluxWithRetry(writeFn, 'Test context', 'v3', '', {
|
||||
maxRetries: 3,
|
||||
initialDelayMs: 10,
|
||||
});
|
||||
|
||||
expect(writeFn).toHaveBeenCalledTimes(3);
|
||||
expect(globals.logger.warn).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
test('should throw error after max retries on timeout', async () => {
|
||||
const timeoutError = new Error('Request timed out');
|
||||
timeoutError.name = 'RequestTimedOutError';
|
||||
|
||||
const writeFn = jest.fn().mockRejectedValue(timeoutError);
|
||||
globals.errorTracker = { incrementError: jest.fn().mockResolvedValue() };
|
||||
|
||||
await expect(
|
||||
utils.writeToInfluxWithRetry(writeFn, 'Test context', 'v3', '', {
|
||||
maxRetries: 2,
|
||||
initialDelayMs: 10,
|
||||
})
|
||||
).rejects.toThrow('Request timed out');
|
||||
|
||||
expect(writeFn).toHaveBeenCalledTimes(3); // 1 initial + 2 retries
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('INFLUXDB V3 RETRY: Test context - All')
|
||||
);
|
||||
expect(globals.errorTracker.incrementError).toHaveBeenCalledWith(
|
||||
'INFLUXDB_V3_WRITE',
|
||||
''
|
||||
);
|
||||
});
|
||||
|
||||
test('should throw non-retryable error immediately without retry', async () => {
|
||||
const nonRetryableError = new Error('Connection refused');
|
||||
const writeFn = jest.fn().mockRejectedValue(nonRetryableError);
|
||||
globals.errorTracker = { incrementError: jest.fn().mockResolvedValue() };
|
||||
|
||||
await expect(
|
||||
utils.writeToInfluxWithRetry(writeFn, 'Test context', 'v3', '', {
|
||||
maxRetries: 3,
|
||||
initialDelayMs: 10,
|
||||
})
|
||||
).rejects.toThrow('Connection refused');
|
||||
|
||||
expect(writeFn).toHaveBeenCalledTimes(1);
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('INFLUXDB V3 WRITE: Test context - Non-retryable error')
|
||||
);
|
||||
});
|
||||
|
||||
test('should detect timeout from error message', async () => {
|
||||
const timeoutError = new Error('Request timed out after 10s');
|
||||
|
||||
const writeFn = jest.fn().mockRejectedValueOnce(timeoutError).mockResolvedValueOnce();
|
||||
|
||||
await utils.writeToInfluxWithRetry(writeFn, 'Test context', 'v3', '', {
|
||||
maxRetries: 3,
|
||||
initialDelayMs: 10,
|
||||
});
|
||||
|
||||
expect(writeFn).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
test('should detect timeout from constructor name', async () => {
|
||||
const timeoutError = new Error('Timeout');
|
||||
Object.defineProperty(timeoutError, 'constructor', {
|
||||
value: { name: 'RequestTimedOutError' },
|
||||
});
|
||||
|
||||
const writeFn = jest.fn().mockRejectedValueOnce(timeoutError).mockResolvedValueOnce();
|
||||
|
||||
await utils.writeToInfluxWithRetry(writeFn, 'Test context', 'v3', '', {
|
||||
maxRetries: 3,
|
||||
initialDelayMs: 10,
|
||||
});
|
||||
|
||||
expect(writeFn).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('applyTagsToPoint3', () => {
|
||||
test('should apply tags to point', () => {
|
||||
const mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
const tags = {
|
||||
env: 'production',
|
||||
host: 'server1',
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, tags);
|
||||
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('env', 'production');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('host', 'server1');
|
||||
});
|
||||
|
||||
test('should handle empty tags object', () => {
|
||||
const mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, {});
|
||||
|
||||
expect(mockPoint.setTag).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle null tags', () => {
|
||||
const mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
};
|
||||
|
||||
utils.applyTagsToPoint3(mockPoint, null);
|
||||
|
||||
expect(mockPoint.setTag).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
234
src/lib/influxdb/__tests__/v3-user-events.test.js
Normal file
234
src/lib/influxdb/__tests__/v3-user-events.test.js
Normal file
@@ -0,0 +1,234 @@
|
||||
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
|
||||
|
||||
// Mock globals
|
||||
const mockGlobals = {
|
||||
logger: {
|
||||
info: jest.fn(),
|
||||
verbose: jest.fn(),
|
||||
debug: jest.fn(),
|
||||
error: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
silly: jest.fn(),
|
||||
},
|
||||
config: {
|
||||
get: jest.fn(),
|
||||
has: jest.fn(),
|
||||
},
|
||||
influx: {
|
||||
write: jest.fn(),
|
||||
},
|
||||
errorTracker: {
|
||||
incrementError: jest.fn().mockResolvedValue(),
|
||||
},
|
||||
getErrorMessage: jest.fn((err) => err.message),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../../../globals.js', () => ({
|
||||
default: mockGlobals,
|
||||
}));
|
||||
|
||||
// Mock shared utils
|
||||
const mockUtils = {
|
||||
isInfluxDbEnabled: jest.fn(),
|
||||
writeToInfluxWithRetry: jest.fn(),
|
||||
writeBatchToInfluxV3: jest.fn(),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('../shared/utils.js', () => mockUtils);
|
||||
|
||||
// Mock Point3
|
||||
const mockPoint = {
|
||||
setTag: jest.fn().mockReturnThis(),
|
||||
setField: jest.fn().mockReturnThis(),
|
||||
setStringField: jest.fn().mockReturnThis(),
|
||||
setTimestamp: jest.fn().mockReturnThis(),
|
||||
toLineProtocol: jest.fn().mockReturnValue('user_events'),
|
||||
};
|
||||
|
||||
jest.unstable_mockModule('@influxdata/influxdb3-client', () => ({
|
||||
Point: jest.fn(() => mockPoint),
|
||||
}));
|
||||
|
||||
describe('v3/user-events', () => {
|
||||
let postUserEventToInfluxdbV3;
|
||||
let globals;
|
||||
let utils;
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
globals = (await import('../../../globals.js')).default;
|
||||
utils = await import('../shared/utils.js');
|
||||
const userEvents = await import('../v3/user-events.js');
|
||||
postUserEventToInfluxdbV3 = userEvents.postUserEventToInfluxdbV3;
|
||||
|
||||
// Setup default mocks
|
||||
globals.config.get.mockReturnValue('test-db');
|
||||
utils.isInfluxDbEnabled.mockReturnValue(true);
|
||||
utils.writeToInfluxWithRetry.mockResolvedValue();
|
||||
utils.writeBatchToInfluxV3.mockResolvedValue();
|
||||
});
|
||||
|
||||
describe('postUserEventToInfluxdbV3', () => {
|
||||
test('should return early when InfluxDB is disabled', async () => {
|
||||
utils.isInfluxDbEnabled.mockReturnValue(false);
|
||||
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
};
|
||||
|
||||
await postUserEventToInfluxdbV3(msg);
|
||||
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should warn and return early when required fields are missing', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
// Missing user_directory, user_id, origin
|
||||
};
|
||||
|
||||
await postUserEventToInfluxdbV3(msg);
|
||||
|
||||
expect(globals.logger.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Missing required fields')
|
||||
);
|
||||
expect(utils.writeToInfluxWithRetry).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should successfully write user event with all fields', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
appId: 'app-123',
|
||||
appName: 'Test App',
|
||||
ua: {
|
||||
os: 'Windows',
|
||||
browser: 'Chrome',
|
||||
device: 'Desktop',
|
||||
},
|
||||
};
|
||||
|
||||
await postUserEventToInfluxdbV3(msg);
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('host', 'server1');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('event_action', 'OpenApp');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('userDirectory', 'DOMAIN');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('userId', 'user1');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('origin', 'QlikSense');
|
||||
});
|
||||
|
||||
test('should handle user event without optional fields', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'CreateApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
};
|
||||
|
||||
await postUserEventToInfluxdbV3(msg);
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('host', 'server1');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('event_action', 'CreateApp');
|
||||
});
|
||||
|
||||
test('should sanitize tag values with special characters', async () => {
|
||||
const msg = {
|
||||
host: 'server<1>',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN\\SUB',
|
||||
user_id: 'user 1',
|
||||
origin: 'Qlik Sense',
|
||||
};
|
||||
|
||||
await postUserEventToInfluxdbV3(msg);
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('should handle write errors', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
};
|
||||
|
||||
const writeError = new Error('Write failed');
|
||||
utils.writeBatchToInfluxV3.mockRejectedValue(writeError);
|
||||
|
||||
await postUserEventToInfluxdbV3(msg);
|
||||
|
||||
expect(globals.logger.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Error saving user event to InfluxDB v3')
|
||||
);
|
||||
expect(globals.errorTracker.incrementError).toHaveBeenCalledWith(
|
||||
'INFLUXDB_V3_WRITE',
|
||||
''
|
||||
);
|
||||
});
|
||||
|
||||
test('should handle events with user agent information', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
ua: {
|
||||
browser: {
|
||||
name: 'Chrome',
|
||||
major: '96',
|
||||
},
|
||||
os: {
|
||||
name: 'Windows',
|
||||
version: '10',
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
await postUserEventToInfluxdbV3(msg);
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('uaBrowserName', 'Chrome');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('uaBrowserMajorVersion', '96');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('uaOsName', 'Windows');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('uaOsVersion', '10');
|
||||
});
|
||||
|
||||
test('should handle events with app information', async () => {
|
||||
const msg = {
|
||||
host: 'server1',
|
||||
command: 'OpenApp',
|
||||
user_directory: 'DOMAIN',
|
||||
user_id: 'user1',
|
||||
origin: 'QlikSense',
|
||||
appId: 'abc-123-def',
|
||||
appName: 'Sales Dashboard',
|
||||
};
|
||||
|
||||
await postUserEventToInfluxdbV3(msg);
|
||||
|
||||
expect(utils.writeBatchToInfluxV3).toHaveBeenCalled();
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('appId', 'abc-123-def');
|
||||
expect(mockPoint.setStringField).toHaveBeenCalledWith('appId_field', 'abc-123-def');
|
||||
expect(mockPoint.setTag).toHaveBeenCalledWith('appName', 'Sales Dashboard');
|
||||
expect(mockPoint.setStringField).toHaveBeenCalledWith(
|
||||
'appName_field',
|
||||
'Sales Dashboard'
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
48
src/lib/influxdb/error-metrics.js
Normal file
48
src/lib/influxdb/error-metrics.js
Normal file
@@ -0,0 +1,48 @@
|
||||
/**
|
||||
* Placeholder function for storing error metrics to InfluxDB.
|
||||
*
|
||||
* This function will be implemented in the future to store API error counts
|
||||
* to InfluxDB for historical tracking and visualization.
|
||||
*
|
||||
* @param {object} errorStats - Error statistics object grouped by API type
|
||||
* @param {object} errorStats.apiType - Object containing total count and server breakdown
|
||||
* @param {number} errorStats.apiType.total - Total error count for this API type
|
||||
* @param {object} errorStats.apiType.servers - Object with server names as keys and error counts as values
|
||||
* @returns {Promise<void>}
|
||||
*
|
||||
* @example
|
||||
* const stats = {
|
||||
* HEALTH_API: {
|
||||
* total: 5,
|
||||
* servers: {
|
||||
* 'sense1': 3,
|
||||
* 'sense2': 2
|
||||
* }
|
||||
* },
|
||||
* INFLUXDB_V3_WRITE: {
|
||||
* total: 2,
|
||||
* servers: {
|
||||
* '_no_server_context': 2
|
||||
* }
|
||||
* }
|
||||
* };
|
||||
* await postErrorMetricsToInfluxdb(stats);
|
||||
*/
|
||||
export async function postErrorMetricsToInfluxdb(errorStats) {
|
||||
// TODO: Implement InfluxDB storage for error metrics
|
||||
// This function should:
|
||||
// 1. Check if InfluxDB is enabled in config
|
||||
// 2. Route to appropriate version-specific implementation (v1/v2/v3)
|
||||
// 3. Create data points with:
|
||||
// - Measurement: 'api_error_counts' or similar
|
||||
// - Tags: apiType, serverName
|
||||
// - Fields: errorCount, timestamp
|
||||
// 4. Write to InfluxDB with appropriate error handling
|
||||
//
|
||||
// For now, this is a no-op placeholder
|
||||
|
||||
// Uncomment for debugging during development:
|
||||
// console.log('ERROR METRICS: Would store to InfluxDB:', JSON.stringify(errorStats, null, 2));
|
||||
|
||||
return Promise.resolve();
|
||||
}
|
||||
255
src/lib/influxdb/factory.js
Normal file
255
src/lib/influxdb/factory.js
Normal file
@@ -0,0 +1,255 @@
|
||||
import globals from '../../globals.js';
|
||||
import { getInfluxDbVersion } from './shared/utils.js';
|
||||
|
||||
// Import version-specific implementations
|
||||
import { storeHealthMetricsV1 } from './v1/health-metrics.js';
|
||||
import { storeSessionsV1 } from './v1/sessions.js';
|
||||
import { storeButlerMemoryV1 } from './v1/butler-memory.js';
|
||||
import { storeUserEventV1 } from './v1/user-events.js';
|
||||
import { storeEventCountV1, storeRejectedEventCountV1 } from './v1/event-counts.js';
|
||||
import { storeUserEventQueueMetricsV1, storeLogEventQueueMetricsV1 } from './v1/queue-metrics.js';
|
||||
import { storeLogEventV1 } from './v1/log-events.js';
|
||||
|
||||
import { storeHealthMetricsV2 } from './v2/health-metrics.js';
|
||||
import { storeSessionsV2 } from './v2/sessions.js';
|
||||
import { storeButlerMemoryV2 } from './v2/butler-memory.js';
|
||||
import { storeUserEventV2 } from './v2/user-events.js';
|
||||
import { storeEventCountV2, storeRejectedEventCountV2 } from './v2/event-counts.js';
|
||||
import { storeUserEventQueueMetricsV2, storeLogEventQueueMetricsV2 } from './v2/queue-metrics.js';
|
||||
import { storeLogEventV2 } from './v2/log-events.js';
|
||||
|
||||
import { postHealthMetricsToInfluxdbV3 } from './v3/health-metrics.js';
|
||||
import { postProxySessionsToInfluxdbV3 } from './v3/sessions.js';
|
||||
import { postButlerSOSMemoryUsageToInfluxdbV3 } from './v3/butler-memory.js';
|
||||
import { postUserEventToInfluxdbV3 } from './v3/user-events.js';
|
||||
import { storeEventCountInfluxDBV3, storeRejectedEventCountInfluxDBV3 } from './v3/event-counts.js';
|
||||
import {
|
||||
postUserEventQueueMetricsToInfluxdbV3,
|
||||
postLogEventQueueMetricsToInfluxdbV3,
|
||||
} from './v3/queue-metrics.js';
|
||||
import { postLogEventToInfluxdbV3 } from './v3/log-events.js';
|
||||
|
||||
/**
|
||||
* Factory function that routes health metrics to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @param {string} serverName - The name of the Qlik Sense server
|
||||
* @param {string} host - The hostname or IP of the Qlik Sense server
|
||||
* @param {object} body - The health metrics data from Sense engine healthcheck API
|
||||
* @param {object} serverTags - Tags to associate with the metrics
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postHealthMetricsToInfluxdb(serverName, host, body, serverTags) {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeHealthMetricsV1(serverTags, body);
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeHealthMetricsV2(serverName, host, body, serverTags);
|
||||
}
|
||||
if (version === 3) {
|
||||
return postHealthMetricsToInfluxdbV3(serverName, host, body, serverTags);
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function that routes proxy sessions to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @param {object} userSessions - User session data
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postProxySessionsToInfluxdb(userSessions) {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeSessionsV1(userSessions);
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeSessionsV2(userSessions);
|
||||
}
|
||||
if (version === 3) {
|
||||
return postProxySessionsToInfluxdbV3(userSessions);
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function that routes Butler SOS memory usage to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @param {object} memory - Memory usage data object
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postButlerSOSMemoryUsageToInfluxdb(memory) {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeButlerMemoryV1(memory);
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeButlerMemoryV2(memory);
|
||||
}
|
||||
if (version === 3) {
|
||||
return postButlerSOSMemoryUsageToInfluxdbV3(memory);
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function that routes user events to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @param {object} msg - The user event message
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postUserEventToInfluxdb(msg) {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeUserEventV1(msg);
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeUserEventV2(msg);
|
||||
}
|
||||
if (version === 3) {
|
||||
return postUserEventToInfluxdbV3(msg);
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function that routes event count storage to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeEventCountInfluxDB() {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeEventCountV1();
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeEventCountV2();
|
||||
}
|
||||
if (version === 3) {
|
||||
return storeEventCountInfluxDBV3();
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function that routes rejected event count storage to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeRejectedEventCountInfluxDB() {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeRejectedEventCountV1();
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeRejectedEventCountV2();
|
||||
}
|
||||
if (version === 3) {
|
||||
return storeRejectedEventCountInfluxDBV3();
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function that routes user event queue metrics to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postUserEventQueueMetricsToInfluxdb() {
|
||||
try {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeUserEventQueueMetricsV1();
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeUserEventQueueMetricsV2();
|
||||
}
|
||||
if (version === 3) {
|
||||
return postUserEventQueueMetricsToInfluxdbV3();
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
} catch (err) {
|
||||
globals.logger.error(
|
||||
`INFLUXDB FACTORY: Error in postUserEventQueueMetricsToInfluxdb: ${err.message}`
|
||||
);
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Error stack: ${err.stack}`);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function that routes log event queue metrics to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postLogEventQueueMetricsToInfluxdb() {
|
||||
try {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeLogEventQueueMetricsV1();
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeLogEventQueueMetricsV2();
|
||||
}
|
||||
if (version === 3) {
|
||||
return postLogEventQueueMetricsToInfluxdbV3();
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
} catch (err) {
|
||||
globals.logger.error(
|
||||
`INFLUXDB FACTORY: Error in postLogEventQueueMetricsToInfluxdb: ${err.message}`
|
||||
);
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Error stack: ${err.stack}`);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Factory function that routes log events to the appropriate InfluxDB version implementation.
|
||||
*
|
||||
* @param {object} msg - The log event message
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postLogEventToInfluxdb(msg) {
|
||||
const version = getInfluxDbVersion();
|
||||
|
||||
if (version === 1) {
|
||||
return storeLogEventV1(msg);
|
||||
}
|
||||
if (version === 2) {
|
||||
return storeLogEventV2(msg);
|
||||
}
|
||||
if (version === 3) {
|
||||
return postLogEventToInfluxdbV3(msg);
|
||||
}
|
||||
|
||||
globals.logger.debug(`INFLUXDB FACTORY: Unknown InfluxDB version: v${version}`);
|
||||
throw new Error(`InfluxDB v${version} not supported`);
|
||||
}
|
||||
|
||||
// TODO: Add other factory functions as they're implemented
|
||||
// etc...
|
||||
201
src/lib/influxdb/index.js
Normal file
201
src/lib/influxdb/index.js
Normal file
@@ -0,0 +1,201 @@
|
||||
import { getFormattedTime } from './shared/utils.js';
|
||||
import * as factory from './factory.js';
|
||||
import globals from '../../globals.js';
|
||||
|
||||
/**
|
||||
* Main facade that routes to version-specific implementations via factory.
|
||||
*
|
||||
* All InfluxDB versions (v1, v2, v3) now use refactored modular code.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Calculates and formats the uptime of a Qlik Sense engine.
|
||||
* This function is version-agnostic and always uses the shared implementation.
|
||||
*
|
||||
* @param {string} serverStarted - The server start time in format "YYYYMMDDThhmmss"
|
||||
* @returns {string} A formatted string representing uptime (e.g. "5 days, 3h 45m 12s")
|
||||
*/
|
||||
export { getFormattedTime };
|
||||
|
||||
/**
|
||||
* Posts health metrics data from Qlik Sense to InfluxDB.
|
||||
*
|
||||
* @param {string} serverName - The name of the Qlik Sense server
|
||||
* @param {string} host - The hostname or IP of the Qlik Sense server
|
||||
* @param {object} body - The health metrics data from Sense engine healthcheck API
|
||||
* @param {object} serverTags - Tags to associate with the metrics
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postHealthMetricsToInfluxdb(serverName, host, body, serverTags) {
|
||||
return await factory.postHealthMetricsToInfluxdb(serverName, host, body, serverTags);
|
||||
}
|
||||
|
||||
/**
|
||||
* Posts proxy sessions data to InfluxDB.
|
||||
*
|
||||
* @param {object} userSessions - User session data
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function postProxySessionsToInfluxdb(userSessions) {
|
||||
return await factory.postProxySessionsToInfluxdb(userSessions);
|
||||
}
|
||||
|
||||
/**
|
||||
* Posts Butler SOS's own memory usage to InfluxDB.
|
||||
*
|
||||
* @param {object} memory - Memory usage data object
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function postButlerSOSMemoryUsageToInfluxdb(memory) {
|
||||
return await factory.postButlerSOSMemoryUsageToInfluxdb(memory);
|
||||
}
|
||||
|
||||
/**
|
||||
* Posts user events to InfluxDB.
|
||||
*
|
||||
* @param {object} msg - The user event message
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function postUserEventToInfluxdb(msg) {
|
||||
return await factory.postUserEventToInfluxdb(msg);
|
||||
}
|
||||
|
||||
/**
|
||||
* Posts log events to InfluxDB.
|
||||
*
|
||||
* @param {object} msg - The log event message
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function postLogEventToInfluxdb(msg) {
|
||||
return await factory.postLogEventToInfluxdb(msg);
|
||||
}
|
||||
|
||||
/**
|
||||
* Stores event counts to InfluxDB.
|
||||
*
|
||||
* @param {string} eventsSinceMidnight - Events since midnight data (unused, kept for compatibility)
|
||||
* @param {string} eventsLastHour - Events last hour data (unused, kept for compatibility)
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function storeEventCountInfluxDB(eventsSinceMidnight, eventsLastHour) {
|
||||
return await factory.storeEventCountInfluxDB();
|
||||
}
|
||||
|
||||
/**
|
||||
* Stores rejected event counts to InfluxDB.
|
||||
*
|
||||
* @param {object} rejectedSinceMidnight - Rejected events since midnight (unused, kept for compatibility)
|
||||
* @param {object} rejectedLastHour - Rejected events last hour (unused, kept for compatibility)
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function storeRejectedEventCountInfluxDB(rejectedSinceMidnight, rejectedLastHour) {
|
||||
return await factory.storeRejectedEventCountInfluxDB();
|
||||
}
|
||||
|
||||
/**
|
||||
* Stores user event queue metrics to InfluxDB.
|
||||
*
|
||||
* @param {object} queueMetrics - Queue metrics data (unused, kept for compatibility)
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function postUserEventQueueMetricsToInfluxdb(queueMetrics) {
|
||||
return await factory.postUserEventQueueMetricsToInfluxdb();
|
||||
}
|
||||
|
||||
/**
|
||||
* Stores log event queue metrics to InfluxDB.
|
||||
*
|
||||
* @param {object} queueMetrics - Queue metrics data (unused, kept for compatibility)
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function postLogEventQueueMetricsToInfluxdb(queueMetrics) {
|
||||
return await factory.postLogEventQueueMetricsToInfluxdb();
|
||||
}
|
||||
|
||||
/**
|
||||
* Sets up timers for queue metrics storage.
|
||||
*
|
||||
* @returns {object} Object containing interval IDs for cleanup
|
||||
*/
|
||||
export function setupUdpQueueMetricsStorage() {
|
||||
const intervalIds = {
|
||||
userEvents: null,
|
||||
logEvents: null,
|
||||
};
|
||||
|
||||
// Check if InfluxDB is enabled
|
||||
if (globals.config.get('Butler-SOS.influxdbConfig.enable') !== true) {
|
||||
globals.logger.info(
|
||||
'UDP QUEUE METRICS: InfluxDB is disabled. Skipping setup of queue metrics storage'
|
||||
);
|
||||
return intervalIds;
|
||||
}
|
||||
|
||||
// Set up user events queue metrics storage
|
||||
if (
|
||||
globals.config.get('Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.enable') ===
|
||||
true
|
||||
) {
|
||||
const writeFrequency = globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.writeFrequency'
|
||||
);
|
||||
|
||||
intervalIds.userEvents = setInterval(async () => {
|
||||
try {
|
||||
globals.logger.verbose(
|
||||
'UDP QUEUE METRICS: Timer for storing user event queue metrics to InfluxDB triggered'
|
||||
);
|
||||
await postUserEventQueueMetricsToInfluxdb();
|
||||
} catch (err) {
|
||||
globals.logger.error(
|
||||
`UDP QUEUE METRICS: Error storing user event queue metrics to InfluxDB: ${
|
||||
err && err.stack ? err.stack : err
|
||||
}`
|
||||
);
|
||||
}
|
||||
}, writeFrequency);
|
||||
|
||||
globals.logger.info(
|
||||
`UDP QUEUE METRICS: Set up timer for storing user event queue metrics to InfluxDB (interval: ${writeFrequency} ms)`
|
||||
);
|
||||
} else {
|
||||
globals.logger.info(
|
||||
'UDP QUEUE METRICS: User event queue metrics storage to InfluxDB is disabled'
|
||||
);
|
||||
}
|
||||
|
||||
// Set up log events queue metrics storage
|
||||
if (
|
||||
globals.config.get('Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.enable') ===
|
||||
true
|
||||
) {
|
||||
const writeFrequency = globals.config.get(
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.writeFrequency'
|
||||
);
|
||||
|
||||
intervalIds.logEvents = setInterval(async () => {
|
||||
try {
|
||||
globals.logger.verbose(
|
||||
'UDP QUEUE METRICS: Timer for storing log event queue metrics to InfluxDB triggered'
|
||||
);
|
||||
await postLogEventQueueMetricsToInfluxdb();
|
||||
} catch (err) {
|
||||
globals.logger.error(
|
||||
`UDP QUEUE METRICS: Error storing log event queue metrics to InfluxDB: ${
|
||||
err && err.stack ? err.stack : err
|
||||
}`
|
||||
);
|
||||
}
|
||||
}, writeFrequency);
|
||||
|
||||
globals.logger.info(
|
||||
`UDP QUEUE METRICS: Set up timer for storing log event queue metrics to InfluxDB (interval: ${writeFrequency} ms)`
|
||||
);
|
||||
} else {
|
||||
globals.logger.info(
|
||||
'UDP QUEUE METRICS: Log event queue metrics storage to InfluxDB is disabled'
|
||||
);
|
||||
}
|
||||
|
||||
return intervalIds;
|
||||
}
|
||||
606
src/lib/influxdb/shared/utils.js
Normal file
606
src/lib/influxdb/shared/utils.js
Normal file
@@ -0,0 +1,606 @@
|
||||
import globals from '../../../globals.js';
|
||||
|
||||
const sessionAppPrefix = 'SessionApp';
|
||||
const MIN_TIMESTAMP_LENGTH = 15;
|
||||
|
||||
/**
|
||||
* Calculates and formats the uptime of a Qlik Sense engine.
|
||||
*
|
||||
* This function takes the server start time from the engine healthcheck API
|
||||
* and calculates how long the server has been running, returning a formatted string.
|
||||
*
|
||||
* @param {string} serverStarted - The server start time in format "YYYYMMDDThhmmss"
|
||||
* @returns {string} A formatted string representing uptime (e.g. "5 days, 3h 45m 12s")
|
||||
*/
|
||||
export function getFormattedTime(serverStarted) {
|
||||
// Handle invalid or empty input
|
||||
if (
|
||||
!serverStarted ||
|
||||
typeof serverStarted !== 'string' ||
|
||||
serverStarted.length < MIN_TIMESTAMP_LENGTH
|
||||
) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const dateTime = Date.now();
|
||||
const timestamp = Math.floor(dateTime);
|
||||
|
||||
const str = serverStarted;
|
||||
const year = str.substring(0, 4);
|
||||
const month = str.substring(4, 6);
|
||||
const day = str.substring(6, 8);
|
||||
const hour = str.substring(9, 11);
|
||||
const minute = str.substring(11, 13);
|
||||
const second = str.substring(13, 15);
|
||||
|
||||
// Validate date components
|
||||
if (
|
||||
isNaN(year) ||
|
||||
isNaN(month) ||
|
||||
isNaN(day) ||
|
||||
isNaN(hour) ||
|
||||
isNaN(minute) ||
|
||||
isNaN(second)
|
||||
) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const dateTimeStarted = new Date(year, month - 1, day, hour, minute, second);
|
||||
|
||||
// Check if the date is valid
|
||||
if (isNaN(dateTimeStarted.getTime())) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const timestampStarted = Math.floor(dateTimeStarted);
|
||||
|
||||
const diff = timestamp - timestampStarted;
|
||||
|
||||
// Create a new JavaScript Date object based on the timestamp
|
||||
// multiplied by 1000 so that the argument is in milliseconds, not seconds.
|
||||
const date = new Date(diff);
|
||||
|
||||
const days = Math.trunc(diff / (1000 * 60 * 60 * 24));
|
||||
|
||||
// Hours part from the timestamp
|
||||
const hours = date.getHours();
|
||||
|
||||
// Minutes part from the timestamp
|
||||
const minutes = `0${date.getMinutes()}`;
|
||||
|
||||
// Seconds part from the timestamp
|
||||
const seconds = `0${date.getSeconds()}`;
|
||||
|
||||
// Will display time in 10:30:23 format
|
||||
return `${days} days, ${hours}h ${minutes.substr(-2)}m ${seconds.substr(-2)}s`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Processes app documents and categorizes them as session apps or regular apps.
|
||||
* Returns arrays of app names for both categories.
|
||||
*
|
||||
* @param {string[]} docIDs - Array of document IDs to process
|
||||
* @param {string} logPrefix - Prefix for log messages
|
||||
* @param {string} appState - Description of app state (e.g., 'active', 'loaded', 'in memory')
|
||||
* @returns {Promise<{appNames: string[], sessionAppNames: string[]}>} Object containing sorted arrays of app names
|
||||
*/
|
||||
export async function processAppDocuments(docIDs, logPrefix, appState) {
|
||||
const appNames = [];
|
||||
const sessionAppNames = [];
|
||||
|
||||
/**
|
||||
* Stores a document ID in the appropriate array based on its type.
|
||||
*
|
||||
* @param {string} docID - The document ID to store
|
||||
* @returns {Promise<void>} Promise that resolves when the document ID has been processed
|
||||
*/
|
||||
const storeDoc = (docID) => {
|
||||
return new Promise((resolve, _reject) => {
|
||||
if (docID.substring(0, sessionAppPrefix.length) === sessionAppPrefix) {
|
||||
// Session app
|
||||
globals.logger.debug(`${logPrefix}: Session app is ${appState}: ${docID}`);
|
||||
sessionAppNames.push(docID);
|
||||
} else {
|
||||
// Not session app
|
||||
const app = globals.appNames.find((element) => element.id === docID);
|
||||
|
||||
if (app) {
|
||||
globals.logger.debug(`${logPrefix}: App is ${appState}: ${app.name}`);
|
||||
appNames.push(app.name);
|
||||
} else {
|
||||
appNames.push(docID);
|
||||
}
|
||||
}
|
||||
|
||||
resolve();
|
||||
});
|
||||
};
|
||||
|
||||
const promises = docIDs.map(
|
||||
(docID) =>
|
||||
new Promise(async (resolve, _reject) => {
|
||||
await storeDoc(docID);
|
||||
resolve();
|
||||
})
|
||||
);
|
||||
|
||||
await Promise.all(promises);
|
||||
|
||||
appNames.sort();
|
||||
sessionAppNames.sort();
|
||||
|
||||
return { appNames, sessionAppNames };
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks if InfluxDB is enabled and initialized.
|
||||
*
|
||||
* @returns {boolean} True if InfluxDB is enabled and initialized
|
||||
*/
|
||||
export function isInfluxDbEnabled() {
|
||||
if (!globals.influx) {
|
||||
globals.logger.warn(
|
||||
'INFLUXDB: Influxdb object not initialized. Data will not be sent to InfluxDB'
|
||||
);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets the InfluxDB version from configuration.
|
||||
*
|
||||
* @returns {number} The InfluxDB version (1, 2, or 3)
|
||||
*/
|
||||
export function getInfluxDbVersion() {
|
||||
return globals.config.get('Butler-SOS.influxdbConfig.version');
|
||||
}
|
||||
|
||||
/**
|
||||
* Applies tags from a tags object to an InfluxDB Point3 object.
|
||||
* This is needed for v3 as it doesn't have automatic default tags like v2.
|
||||
*
|
||||
* @param {object} point - The Point3 object to apply tags to
|
||||
* @param {object} tags - Object containing tag key-value pairs
|
||||
* @returns {object} The Point3 object with tags applied (for chaining)
|
||||
*/
|
||||
export function applyTagsToPoint3(point, tags) {
|
||||
if (!tags || typeof tags !== 'object') {
|
||||
return point;
|
||||
}
|
||||
|
||||
// Apply each tag to the point
|
||||
Object.entries(tags).forEach(([key, value]) => {
|
||||
if (value !== undefined && value !== null) {
|
||||
point.setTag(key, String(value));
|
||||
}
|
||||
});
|
||||
|
||||
return point;
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes data to InfluxDB (v1, v2, or v3) with retry logic and exponential backoff.
|
||||
*
|
||||
* This unified function handles writes to any InfluxDB version with configurable retry logic.
|
||||
* If a write fails due to timeout or network issues, it will retry up to maxRetries times
|
||||
* with exponential backoff between attempts.
|
||||
*
|
||||
* @param {Function} writeFn - Async function that performs the write operation
|
||||
* @param {string} context - Description of what's being written (for logging)
|
||||
* @param {string} version - InfluxDB version ('v1', 'v2', or 'v3')
|
||||
* @param {string} errorCategory - Error category for tracking (e.g., server name or component)
|
||||
* @param {object} options - Retry options
|
||||
* @param {number} options.maxRetries - Maximum number of retry attempts (default: 3)
|
||||
* @param {number} options.initialDelayMs - Initial delay before first retry in ms (default: 1000)
|
||||
* @param {number} options.maxDelayMs - Maximum delay between retries in ms (default: 10000)
|
||||
* @param {number} options.backoffMultiplier - Multiplier for exponential backoff (default: 2)
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when write succeeds or rejects after all retries fail
|
||||
*
|
||||
* @throws {Error} The last error encountered after all retries are exhausted
|
||||
*/
|
||||
export async function writeToInfluxWithRetry(
|
||||
writeFn,
|
||||
context,
|
||||
version,
|
||||
errorCategory = '',
|
||||
options = {}
|
||||
) {
|
||||
const {
|
||||
maxRetries = 3,
|
||||
initialDelayMs = 1000,
|
||||
maxDelayMs = 10000,
|
||||
backoffMultiplier = 2,
|
||||
} = options;
|
||||
|
||||
let lastError;
|
||||
let attempt = 0;
|
||||
const versionTag = version.toUpperCase();
|
||||
|
||||
while (attempt <= maxRetries) {
|
||||
try {
|
||||
await writeFn();
|
||||
|
||||
// Log success if this was a retry
|
||||
if (attempt > 0) {
|
||||
globals.logger.info(
|
||||
`INFLUXDB ${versionTag} RETRY: ${context} - Write succeeded on attempt ${attempt + 1}/${maxRetries + 1}`
|
||||
);
|
||||
}
|
||||
|
||||
return; // Success!
|
||||
} catch (err) {
|
||||
lastError = err;
|
||||
attempt++;
|
||||
|
||||
// Check if this is a retryable error (timeout or network issue)
|
||||
const errorName = err.constructor?.name || err.name || '';
|
||||
const errorMessage = err.message || '';
|
||||
const isRetryableError =
|
||||
errorName === 'RequestTimedOutError' ||
|
||||
errorMessage.includes('timeout') ||
|
||||
errorMessage.includes('timed out') ||
|
||||
errorMessage.includes('ETIMEDOUT') ||
|
||||
errorMessage.includes('ECONNREFUSED') ||
|
||||
errorMessage.includes('ENOTFOUND') ||
|
||||
errorMessage.includes('ECONNRESET');
|
||||
|
||||
// Log the error type for debugging
|
||||
globals.logger.debug(
|
||||
`INFLUXDB ${versionTag} RETRY: ${context} - Error caught: ${errorName}, message: ${errorMessage}, isRetryable: ${isRetryableError}`
|
||||
);
|
||||
|
||||
// Don't retry on non-retryable errors - fail immediately
|
||||
if (!isRetryableError) {
|
||||
globals.logger.warn(
|
||||
`INFLUXDB ${versionTag} WRITE: ${context} - Non-retryable error (${errorName}), not retrying: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
|
||||
// Track error immediately for non-retryable errors
|
||||
await globals.errorTracker.incrementError(
|
||||
`INFLUXDB_${versionTag}_WRITE`,
|
||||
errorCategory
|
||||
);
|
||||
|
||||
throw err;
|
||||
}
|
||||
|
||||
// This is a retryable error - check if we have retries left
|
||||
if (attempt <= maxRetries) {
|
||||
// Calculate delay with exponential backoff
|
||||
const delayMs = Math.min(
|
||||
initialDelayMs * Math.pow(backoffMultiplier, attempt - 1),
|
||||
maxDelayMs
|
||||
);
|
||||
|
||||
globals.logger.warn(
|
||||
`INFLUXDB ${versionTag} RETRY: ${context} - Retryable error (${errorName}) on attempt ${attempt}/${maxRetries + 1}, retrying in ${delayMs}ms...`
|
||||
);
|
||||
|
||||
// Wait before retrying
|
||||
await new Promise((resolve) => setTimeout(resolve, delayMs));
|
||||
} else {
|
||||
// All retries exhausted
|
||||
globals.logger.error(
|
||||
`INFLUXDB ${versionTag} RETRY: ${context} - All ${maxRetries + 1} attempts failed. Last error: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
|
||||
// Track error count (final failure after all retries)
|
||||
await globals.errorTracker.incrementError(
|
||||
`INFLUXDB_${versionTag}_WRITE`,
|
||||
errorCategory
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// All retries failed, throw the last error
|
||||
throw lastError;
|
||||
}
|
||||
|
||||
/**
|
||||
* Splits an array into chunks of a specified size.
|
||||
*
|
||||
* @param {Array} array - The array to chunk
|
||||
* @param {number} chunkSize - The size of each chunk
|
||||
*
|
||||
* @returns {Array[]} Array of chunks
|
||||
*/
|
||||
export function chunkArray(array, chunkSize) {
|
||||
if (!Array.isArray(array) || array.length === 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
if (!chunkSize || chunkSize <= 0) {
|
||||
return [array];
|
||||
}
|
||||
|
||||
const chunks = [];
|
||||
for (let i = 0; i < array.length; i += chunkSize) {
|
||||
chunks.push(array.slice(i, i + chunkSize));
|
||||
}
|
||||
return chunks;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates that a field value is non-negative (unsigned).
|
||||
* Logs a warning once per measurement if negative values are found and clamps to 0.
|
||||
*
|
||||
* @param {number} value - The value to validate
|
||||
* @param {string} measurement - Measurement name for logging
|
||||
* @param {string} field - Field name for logging
|
||||
* @param {string} serverContext - Server/context name for logging
|
||||
*
|
||||
* @returns {number} The validated value (clamped to 0 if negative)
|
||||
*/
|
||||
export function validateUnsignedField(value, measurement, field, serverContext) {
|
||||
// Convert to number if string
|
||||
const numValue = typeof value === 'string' ? parseFloat(value) : value;
|
||||
|
||||
// Handle null/undefined/NaN
|
||||
if (numValue == null || isNaN(numValue)) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Check if negative
|
||||
if (numValue < 0) {
|
||||
// Warn once per measurement (using a Set to track)
|
||||
if (!validateUnsignedField._warnedMeasurements) {
|
||||
validateUnsignedField._warnedMeasurements = new Set();
|
||||
}
|
||||
|
||||
if (!validateUnsignedField._warnedMeasurements.has(measurement)) {
|
||||
globals.logger.warn(
|
||||
`Negative value detected for unsigned field: measurement=${measurement}, field=${field}, value=${numValue}, server=${serverContext}. Clamping to 0.`
|
||||
);
|
||||
validateUnsignedField._warnedMeasurements.add(measurement);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
return numValue;
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes data to InfluxDB v1 in batches with progressive retry strategy.
|
||||
* If a batch fails, it will automatically try smaller batch sizes.
|
||||
*
|
||||
* @param {Array} datapoints - Array of datapoint objects to write
|
||||
* @param {string} context - Description of what's being written
|
||||
* @param {string} errorCategory - Error category for tracking
|
||||
* @param {number} maxBatchSize - Maximum batch size from config
|
||||
*
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function writeBatchToInfluxV1(datapoints, context, errorCategory, maxBatchSize) {
|
||||
if (!Array.isArray(datapoints) || datapoints.length === 0) {
|
||||
globals.logger.verbose(`INFLUXDB V1 BATCH: ${context} - No points to write`);
|
||||
return;
|
||||
}
|
||||
|
||||
const progressiveSizes = [maxBatchSize, 500, 250, 100, 10, 1].filter(
|
||||
(size) => size <= maxBatchSize
|
||||
);
|
||||
|
||||
for (const batchSize of progressiveSizes) {
|
||||
const chunks = chunkArray(datapoints, batchSize);
|
||||
let allSucceeded = true;
|
||||
let failedChunks = [];
|
||||
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const chunk = chunks[i];
|
||||
const startIdx = i * batchSize;
|
||||
const endIdx = Math.min(startIdx + chunk.length - 1, datapoints.length - 1);
|
||||
|
||||
try {
|
||||
await writeToInfluxWithRetry(
|
||||
async () => await globals.influx.writePoints(chunk),
|
||||
`${context} (chunk ${i + 1}/${chunks.length}, points ${startIdx}-${endIdx})`,
|
||||
'v1',
|
||||
errorCategory
|
||||
);
|
||||
} catch (err) {
|
||||
allSucceeded = false;
|
||||
failedChunks.push({ index: i + 1, startIdx, endIdx, total: chunks.length });
|
||||
|
||||
globals.logger.error(
|
||||
`INFLUXDB V1 BATCH: ${context} - Chunk ${i + 1} of ${chunks.length} (points ${startIdx}-${endIdx}) failed: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (allSucceeded) {
|
||||
if (batchSize < maxBatchSize) {
|
||||
globals.logger.info(
|
||||
`INFLUXDB V1 BATCH: ${context} - Successfully wrote all data using batch size ${batchSize} (reduced from ${maxBatchSize})`
|
||||
);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// If this wasn't the last attempt, log that we're trying smaller batches
|
||||
if (batchSize !== progressiveSizes[progressiveSizes.length - 1]) {
|
||||
globals.logger.warn(
|
||||
`INFLUXDB V1 BATCH: ${context} - ${failedChunks.length} chunk(s) failed with batch size ${batchSize}, retrying with smaller batches`
|
||||
);
|
||||
} else {
|
||||
// Final attempt failed
|
||||
globals.logger.error(
|
||||
`INFLUXDB V1 BATCH: ${context} - Failed to write data even with batch size 1. ${failedChunks.length} point(s) could not be written.`
|
||||
);
|
||||
throw new Error(`Failed to write batch after trying all progressive sizes`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes data to InfluxDB v2 in batches with progressive retry strategy.
|
||||
* Handles writeApi lifecycle management.
|
||||
*
|
||||
* @param {Array} points - Array of Point objects to write
|
||||
* @param {string} org - InfluxDB organization
|
||||
* @param {string} bucketName - InfluxDB bucket name
|
||||
* @param {string} context - Description of what's being written
|
||||
* @param {string} errorCategory - Error category for tracking
|
||||
* @param {number} maxBatchSize - Maximum batch size from config
|
||||
*
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function writeBatchToInfluxV2(
|
||||
points,
|
||||
org,
|
||||
bucketName,
|
||||
context,
|
||||
errorCategory,
|
||||
maxBatchSize
|
||||
) {
|
||||
if (!Array.isArray(points) || points.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const progressiveSizes = [maxBatchSize, 500, 250, 100, 10, 1].filter(
|
||||
(size) => size <= maxBatchSize
|
||||
);
|
||||
|
||||
for (const batchSize of progressiveSizes) {
|
||||
const chunks = chunkArray(points, batchSize);
|
||||
let allSucceeded = true;
|
||||
let failedChunks = [];
|
||||
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const chunk = chunks[i];
|
||||
const startIdx = i * batchSize;
|
||||
const endIdx = Math.min(startIdx + chunk.length - 1, points.length - 1);
|
||||
|
||||
try {
|
||||
await writeToInfluxWithRetry(
|
||||
async () => {
|
||||
const writeApi = globals.influx.getWriteApi(org, bucketName, 'ns', {
|
||||
flushInterval: 5000,
|
||||
});
|
||||
try {
|
||||
await writeApi.writePoints(chunk);
|
||||
await writeApi.close();
|
||||
} catch (err) {
|
||||
try {
|
||||
await writeApi.close();
|
||||
} catch (closeErr) {
|
||||
// Ignore close errors
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
`${context} (chunk ${i + 1}/${chunks.length}, points ${startIdx}-${endIdx})`,
|
||||
'v2',
|
||||
errorCategory
|
||||
);
|
||||
} catch (err) {
|
||||
allSucceeded = false;
|
||||
failedChunks.push({ index: i + 1, startIdx, endIdx, total: chunks.length });
|
||||
|
||||
globals.logger.error(
|
||||
`INFLUXDB V2 BATCH: ${context} - Chunk ${i + 1} of ${chunks.length} (points ${startIdx}-${endIdx}) failed: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (allSucceeded) {
|
||||
if (batchSize < maxBatchSize) {
|
||||
globals.logger.info(
|
||||
`INFLUXDB V2 BATCH: ${context} - Successfully wrote all data using batch size ${batchSize} (reduced from ${maxBatchSize})`
|
||||
);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// If this wasn't the last attempt, log that we're trying smaller batches
|
||||
if (batchSize !== progressiveSizes[progressiveSizes.length - 1]) {
|
||||
globals.logger.warn(
|
||||
`INFLUXDB V2 BATCH: ${context} - ${failedChunks.length} chunk(s) failed with batch size ${batchSize}, retrying with smaller batches`
|
||||
);
|
||||
} else {
|
||||
// Final attempt failed
|
||||
globals.logger.error(
|
||||
`INFLUXDB V2 BATCH: ${context} - Failed to write data even with batch size 1. ${failedChunks.length} point(s) could not be written.`
|
||||
);
|
||||
throw new Error(`Failed to write batch after trying all progressive sizes`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes data to InfluxDB v3 in batches with progressive retry strategy.
|
||||
* Converts Point3 objects to line protocol and concatenates them.
|
||||
*
|
||||
* @param {Array} points - Array of Point3 objects to write
|
||||
* @param {string} database - InfluxDB database name
|
||||
* @param {string} context - Description of what's being written
|
||||
* @param {string} errorCategory - Error category for tracking
|
||||
* @param {number} maxBatchSize - Maximum batch size from config
|
||||
*
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function writeBatchToInfluxV3(points, database, context, errorCategory, maxBatchSize) {
|
||||
if (!Array.isArray(points) || points.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const progressiveSizes = [maxBatchSize, 500, 250, 100, 10, 1].filter(
|
||||
(size) => size <= maxBatchSize
|
||||
);
|
||||
|
||||
for (const batchSize of progressiveSizes) {
|
||||
const chunks = chunkArray(points, batchSize);
|
||||
let allSucceeded = true;
|
||||
let failedChunks = [];
|
||||
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const chunk = chunks[i];
|
||||
const startIdx = i * batchSize;
|
||||
const endIdx = Math.min(startIdx + chunk.length - 1, points.length - 1);
|
||||
|
||||
try {
|
||||
// Convert Point3 objects to line protocol and concatenate
|
||||
const lineProtocol = chunk.map((p) => p.toLineProtocol()).join('\n');
|
||||
|
||||
await writeToInfluxWithRetry(
|
||||
async () => await globals.influx.write(lineProtocol, database),
|
||||
`${context} (chunk ${i + 1}/${chunks.length}, points ${startIdx}-${endIdx})`,
|
||||
'v3',
|
||||
errorCategory
|
||||
);
|
||||
} catch (err) {
|
||||
allSucceeded = false;
|
||||
failedChunks.push({ index: i + 1, startIdx, endIdx, total: chunks.length });
|
||||
|
||||
globals.logger.error(
|
||||
`INFLUXDB V3 BATCH: ${context} - Chunk ${i + 1} of ${chunks.length} (points ${startIdx}-${endIdx}) failed: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (allSucceeded) {
|
||||
if (batchSize < maxBatchSize) {
|
||||
globals.logger.info(
|
||||
`INFLUXDB V3 BATCH: ${context} - Successfully wrote all data using batch size ${batchSize} (reduced from ${maxBatchSize})`
|
||||
);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// If this wasn't the last attempt, log that we're trying smaller batches
|
||||
if (batchSize !== progressiveSizes[progressiveSizes.length - 1]) {
|
||||
globals.logger.warn(
|
||||
`INFLUXDB V3 BATCH: ${context} - ${failedChunks.length} chunk(s) failed with batch size ${batchSize}, retrying with smaller batches`
|
||||
);
|
||||
} else {
|
||||
// Final attempt failed
|
||||
globals.logger.error(
|
||||
`INFLUXDB V3 BATCH: ${context} - Failed to write data even with batch size 1. ${failedChunks.length} point(s) could not be written.`
|
||||
);
|
||||
throw new Error(`Failed to write batch after trying all progressive sizes`);
|
||||
}
|
||||
}
|
||||
}
|
||||
72
src/lib/influxdb/v1/butler-memory.js
Normal file
72
src/lib/influxdb/v1/butler-memory.js
Normal file
@@ -0,0 +1,72 @@
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV1 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts Butler SOS memory usage metrics to InfluxDB v1.
|
||||
*
|
||||
* This function captures memory usage metrics from the Butler SOS process itself
|
||||
* and stores them in InfluxDB v1.
|
||||
*
|
||||
* @param {object} memory - Memory usage data object
|
||||
* @param {string} memory.instanceTag - Instance identifier tag
|
||||
* @param {number} memory.heapUsedMByte - Heap used in MB
|
||||
* @param {number} memory.heapTotalMByte - Total heap size in MB
|
||||
* @param {number} memory.externalMemoryMByte - External memory usage in MB
|
||||
* @param {number} memory.processMemoryMByte - Process memory usage in MB
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeButlerMemoryV1(memory) {
|
||||
globals.logger.debug(`MEMORY USAGE V1: Memory usage ${JSON.stringify(memory, null, 2)}`);
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const butlerVersion = globals.appVersion;
|
||||
|
||||
const datapoint = [
|
||||
{
|
||||
measurement: 'butlersos_memory_usage',
|
||||
tags: {
|
||||
butler_sos_instance: memory.instanceTag,
|
||||
version: butlerVersion,
|
||||
},
|
||||
fields: {
|
||||
heap_used: memory.heapUsedMByte,
|
||||
heap_total: memory.heapTotalMByte,
|
||||
external: memory.externalMemoryMByte,
|
||||
process_memory: memory.processMemoryMByte,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
globals.logger.silly(
|
||||
`MEMORY USAGE V1: Influxdb datapoint for Butler SOS memory usage: ${JSON.stringify(
|
||||
datapoint,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
|
||||
// Get max batch size from config
|
||||
const maxBatchSize = globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize');
|
||||
|
||||
// Write with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
datapoint,
|
||||
'Memory usage metrics',
|
||||
'INFLUXDB_V1_WRITE',
|
||||
maxBatchSize
|
||||
);
|
||||
|
||||
globals.logger.verbose('MEMORY USAGE V1: Sent Butler SOS memory usage data to InfluxDB');
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', '');
|
||||
globals.logger.error(
|
||||
`MEMORY USAGE V1: Error saving Butler SOS memory data: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
242
src/lib/influxdb/v1/event-counts.js
Normal file
242
src/lib/influxdb/v1/event-counts.js
Normal file
@@ -0,0 +1,242 @@
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV1 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Store event count in InfluxDB v1
|
||||
*
|
||||
* @description
|
||||
* This function reads arrays of log and user events from the `udpEvents` object,
|
||||
* and stores the data in InfluxDB v1. The data is written to a measurement named after
|
||||
* the `Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName` config setting.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeEventCountV1() {
|
||||
// Get array of log events
|
||||
const logEvents = await globals.udpEvents.getLogEvents();
|
||||
const userEvents = await globals.udpEvents.getUserEvents();
|
||||
|
||||
globals.logger.debug(`EVENT COUNT V1: Log events: ${JSON.stringify(logEvents, null, 2)}`);
|
||||
globals.logger.debug(`EVENT COUNT V1: User events: ${JSON.stringify(userEvents, null, 2)}`);
|
||||
|
||||
// Are there any events to store?
|
||||
if (logEvents.length === 0 && userEvents.length === 0) {
|
||||
globals.logger.verbose('EVENT COUNT V1: No events to store in InfluxDB');
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const points = [];
|
||||
|
||||
// Get measurement name to use for event counts
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName'
|
||||
);
|
||||
|
||||
// Get config tags once to avoid repeated config lookups
|
||||
const configTagsArray =
|
||||
globals.config.has('Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags') &&
|
||||
Array.isArray(globals.config.get('Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags'))
|
||||
? globals.config.get('Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags')
|
||||
: null;
|
||||
|
||||
// Loop through data in log events and create datapoints
|
||||
for (const event of logEvents) {
|
||||
const point = {
|
||||
measurement: measurementName,
|
||||
tags: {
|
||||
event_type: 'log',
|
||||
source: event.source,
|
||||
host: event.host,
|
||||
subsystem: event.subsystem,
|
||||
},
|
||||
fields: {
|
||||
counter: event.counter,
|
||||
},
|
||||
};
|
||||
|
||||
// Add static tags from config file
|
||||
if (configTagsArray) {
|
||||
for (const item of configTagsArray) {
|
||||
point.tags[item.name] = item.value;
|
||||
}
|
||||
}
|
||||
|
||||
points.push(point);
|
||||
}
|
||||
|
||||
// Loop through data in user events and create datapoints
|
||||
for (const event of userEvents) {
|
||||
const point = {
|
||||
measurement: measurementName,
|
||||
tags: {
|
||||
event_type: 'user',
|
||||
source: event.source,
|
||||
host: event.host,
|
||||
subsystem: event.subsystem,
|
||||
},
|
||||
fields: {
|
||||
counter: event.counter,
|
||||
},
|
||||
};
|
||||
|
||||
// Add static tags from config file
|
||||
if (configTagsArray) {
|
||||
for (const item of configTagsArray) {
|
||||
point.tags[item.name] = item.value;
|
||||
}
|
||||
}
|
||||
|
||||
points.push(point);
|
||||
}
|
||||
|
||||
// Write with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
points,
|
||||
'Event counts',
|
||||
'',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('EVENT COUNT V1: Sent event count data to InfluxDB');
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', '');
|
||||
globals.logger.error(`EVENT COUNT V1: Error saving data: ${globals.getErrorMessage(err)}`);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Store rejected event counts to InfluxDB v1
|
||||
*
|
||||
* @description
|
||||
* Tracks events that were rejected due to validation failures, rate limiting,
|
||||
* or filtering rules. Particularly important for QIX performance monitoring.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeRejectedEventCountV1() {
|
||||
// Get array of rejected log events
|
||||
const rejectedLogEvents = await globals.rejectedEvents.getRejectedLogEvents();
|
||||
|
||||
globals.logger.debug(
|
||||
`REJECTED EVENT COUNT V1: Rejected log events: ${JSON.stringify(
|
||||
rejectedLogEvents,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
|
||||
// Are there any events to store?
|
||||
if (rejectedLogEvents.length === 0) {
|
||||
globals.logger.verbose('REJECTED EVENT COUNT V1: No events to store in InfluxDB');
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const points = [];
|
||||
|
||||
// Get measurement name to use for rejected events
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.qlikSenseEvents.rejectedEventCount.influxdb.measurementName'
|
||||
);
|
||||
|
||||
// Loop through data in rejected log events and create datapoints
|
||||
// Use counter and process_time as fields
|
||||
for (const event of rejectedLogEvents) {
|
||||
if (event.source === 'qseow-qix-perf') {
|
||||
// For each unique combination of source, appId, appName, method and objectType,
|
||||
// write the counter and processTime properties to InfluxDB
|
||||
const tags = {
|
||||
source: event.source,
|
||||
app_id: event.appId,
|
||||
method: event.method,
|
||||
object_type: event.objectType,
|
||||
};
|
||||
|
||||
// Tags that are empty in some cases. Only add if they are non-empty
|
||||
if (event?.appName?.length > 0) {
|
||||
tags.app_name = event.appName;
|
||||
tags.app_name_set = 'true';
|
||||
} else {
|
||||
tags.app_name_set = 'false';
|
||||
}
|
||||
|
||||
// Add static tags from config file
|
||||
if (
|
||||
globals.config.has(
|
||||
'Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.tags'
|
||||
) &&
|
||||
globals.config.get(
|
||||
'Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.tags'
|
||||
) !== null &&
|
||||
globals.config.get(
|
||||
'Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.tags'
|
||||
).length > 0
|
||||
) {
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.tags'
|
||||
);
|
||||
for (const item of configTags) {
|
||||
tags[item.name] = item.value;
|
||||
}
|
||||
}
|
||||
|
||||
const fields = {
|
||||
counter: event.counter,
|
||||
process_time: event.processTime,
|
||||
};
|
||||
|
||||
const point = {
|
||||
measurement: measurementName,
|
||||
tags,
|
||||
fields,
|
||||
};
|
||||
|
||||
points.push(point);
|
||||
} else {
|
||||
const point = {
|
||||
measurement: measurementName,
|
||||
tags: {
|
||||
source: event.source,
|
||||
},
|
||||
fields: {
|
||||
counter: event.counter,
|
||||
},
|
||||
};
|
||||
|
||||
points.push(point);
|
||||
}
|
||||
}
|
||||
|
||||
// Write with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
points,
|
||||
'Rejected event counts',
|
||||
'',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose(
|
||||
'REJECTED EVENT COUNT V1: Sent rejected event count data to InfluxDB'
|
||||
);
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', '');
|
||||
globals.logger.error(
|
||||
`REJECTED EVENT COUNT V1: Error saving data: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
205
src/lib/influxdb/v1/health-metrics.js
Normal file
205
src/lib/influxdb/v1/health-metrics.js
Normal file
@@ -0,0 +1,205 @@
|
||||
import globals from '../../../globals.js';
|
||||
import {
|
||||
getFormattedTime,
|
||||
processAppDocuments,
|
||||
isInfluxDbEnabled,
|
||||
writeBatchToInfluxV1,
|
||||
} from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts health metrics data from Qlik Sense to InfluxDB v1.
|
||||
*
|
||||
* This function processes health data from the Sense engine's healthcheck API and
|
||||
* formats it for storage in InfluxDB v1. It handles various metrics including:
|
||||
* - CPU usage
|
||||
* - Memory usage
|
||||
* - Cache metrics
|
||||
* - Active/loaded/in-memory apps
|
||||
* - Session counts
|
||||
* - User counts
|
||||
*
|
||||
* @param {object} serverTags - Tags to associate with the metrics (e.g., server_name, host, etc.)
|
||||
* @param {object} body - The health metrics data from Sense engine healthcheck API
|
||||
* @param {object} body.version - Qlik Sense version
|
||||
* @param {string} body.started - Server start time
|
||||
* @param {object} body.mem - Memory metrics
|
||||
* @param {object} body.apps - App metrics including active_docs, loaded_docs, in_memory_docs
|
||||
* @param {object} body.cpu - CPU metrics
|
||||
* @param {object} body.session - Session metrics
|
||||
* @param {object} body.users - User metrics
|
||||
* @param {object} body.cache - Cache metrics
|
||||
* @param {boolean} body.saturated - Saturation status
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeHealthMetricsV1(serverTags, body) {
|
||||
globals.logger.debug(
|
||||
`HEALTH METRICS V1: Processing health data for server: ${serverTags.server_name}`
|
||||
);
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
globals.logger.debug(
|
||||
`HEALTH METRICS V1: Number of apps active: ${body.apps.active_docs.length}`
|
||||
);
|
||||
globals.logger.debug(
|
||||
`HEALTH METRICS V1: Number of apps loaded: ${body.apps.loaded_docs.length}`
|
||||
);
|
||||
globals.logger.debug(
|
||||
`HEALTH METRICS V1: Number of apps in memory: ${body.apps.in_memory_docs.length}`
|
||||
);
|
||||
|
||||
// Process app names for different document types
|
||||
const { appNames: appNamesActive, sessionAppNames: sessionAppNamesActive } =
|
||||
await processAppDocuments(body.apps.active_docs, 'HEALTH METRICS V1', 'active');
|
||||
|
||||
const { appNames: appNamesLoaded, sessionAppNames: sessionAppNamesLoaded } =
|
||||
await processAppDocuments(body.apps.loaded_docs, 'HEALTH METRICS V1', 'loaded');
|
||||
|
||||
const { appNames: appNamesInMemory, sessionAppNames: sessionAppNamesInMemory } =
|
||||
await processAppDocuments(body.apps.in_memory_docs, 'HEALTH METRICS V1', 'in memory');
|
||||
|
||||
// Create datapoint array for v1 - plain objects with measurement, tags, fields
|
||||
const datapoint = [
|
||||
{
|
||||
measurement: 'sense_server',
|
||||
tags: serverTags,
|
||||
fields: {
|
||||
version: body.version,
|
||||
started: body.started,
|
||||
uptime: getFormattedTime(body.started),
|
||||
},
|
||||
},
|
||||
{
|
||||
measurement: 'mem',
|
||||
tags: serverTags,
|
||||
fields: {
|
||||
comitted: body.mem.committed,
|
||||
allocated: body.mem.allocated,
|
||||
free: body.mem.free,
|
||||
},
|
||||
},
|
||||
{
|
||||
measurement: 'apps',
|
||||
tags: serverTags,
|
||||
fields: {
|
||||
active_docs_count: body.apps.active_docs.length,
|
||||
loaded_docs_count: body.apps.loaded_docs.length,
|
||||
in_memory_docs_count: body.apps.in_memory_docs.length,
|
||||
|
||||
active_docs: globals.config.get(
|
||||
'Butler-SOS.influxdbConfig.includeFields.activeDocs'
|
||||
)
|
||||
? body.apps.active_docs
|
||||
: '',
|
||||
active_docs_names:
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
|
||||
? appNamesActive.map((name) => `"${name}"`).join(',')
|
||||
: '',
|
||||
active_session_docs_names:
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
|
||||
? sessionAppNamesActive.map((name) => `"${name}"`).join(',')
|
||||
: '',
|
||||
|
||||
loaded_docs: globals.config.get(
|
||||
'Butler-SOS.influxdbConfig.includeFields.loadedDocs'
|
||||
)
|
||||
? body.apps.loaded_docs
|
||||
: '',
|
||||
loaded_docs_names:
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
|
||||
? appNamesLoaded.map((name) => `"${name}"`).join(',')
|
||||
: '',
|
||||
loaded_session_docs_names:
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
|
||||
? sessionAppNamesLoaded.map((name) => `"${name}"`).join(',')
|
||||
: '',
|
||||
|
||||
in_memory_docs: globals.config.get(
|
||||
'Butler-SOS.influxdbConfig.includeFields.inMemoryDocs'
|
||||
)
|
||||
? body.apps.in_memory_docs
|
||||
: '',
|
||||
in_memory_docs_names:
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
|
||||
? appNamesInMemory.map((name) => `"${name}"`).join(',')
|
||||
: '',
|
||||
in_memory_session_docs_names:
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
|
||||
? sessionAppNamesInMemory.map((name) => `"${name}"`).join(',')
|
||||
: '',
|
||||
calls: body.apps.calls,
|
||||
selections: body.apps.selections,
|
||||
},
|
||||
},
|
||||
{
|
||||
measurement: 'cpu',
|
||||
tags: serverTags,
|
||||
fields: {
|
||||
total: body.cpu.total,
|
||||
},
|
||||
},
|
||||
{
|
||||
measurement: 'session',
|
||||
tags: serverTags,
|
||||
fields: {
|
||||
active: body.session.active,
|
||||
total: body.session.total,
|
||||
},
|
||||
},
|
||||
{
|
||||
measurement: 'users',
|
||||
tags: serverTags,
|
||||
fields: {
|
||||
active: body.users.active,
|
||||
total: body.users.total,
|
||||
},
|
||||
},
|
||||
{
|
||||
measurement: 'cache',
|
||||
tags: serverTags,
|
||||
fields: {
|
||||
hits: body.cache.hits,
|
||||
lookups: body.cache.lookups,
|
||||
added: body.cache.added,
|
||||
replaced: body.cache.replaced,
|
||||
bytes_added: body.cache.bytes_added,
|
||||
},
|
||||
},
|
||||
{
|
||||
measurement: 'saturated',
|
||||
tags: serverTags,
|
||||
fields: {
|
||||
saturated: body.saturated,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
// Write to InfluxDB v1 using node-influx library with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
datapoint,
|
||||
`Health metrics for ${serverTags.server_name}`,
|
||||
serverTags.server_name,
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose(
|
||||
`HEALTH METRICS V1: Stored health data from server: ${serverTags.server_name}`
|
||||
);
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', serverTags.server_name);
|
||||
globals.logger.error(
|
||||
`HEALTH METRICS V1: Error saving health data for ${serverTags.server_name}: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
235
src/lib/influxdb/v1/log-events.js
Normal file
235
src/lib/influxdb/v1/log-events.js
Normal file
@@ -0,0 +1,235 @@
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV1 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Post log event to InfluxDB v1
|
||||
*
|
||||
* @description
|
||||
* Handles log events from 5 different Qlik Sense sources:
|
||||
* - qseow-engine: Engine log events
|
||||
* - qseow-proxy: Proxy log events
|
||||
* - qseow-scheduler: Scheduler log events
|
||||
* - qseow-repository: Repository log events
|
||||
* - qseow-qix-perf: QIX performance metrics
|
||||
*
|
||||
* Each source has specific fields and tags that are written to InfluxDB.
|
||||
*
|
||||
* @param {object} msg - The log event message
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeLogEventV1(msg) {
|
||||
globals.logger.debug(`LOG EVENT V1: ${JSON.stringify(msg)}`);
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Verify the message source is valid
|
||||
if (
|
||||
msg.source !== 'qseow-engine' &&
|
||||
msg.source !== 'qseow-proxy' &&
|
||||
msg.source !== 'qseow-scheduler' &&
|
||||
msg.source !== 'qseow-repository' &&
|
||||
msg.source !== 'qseow-qix-perf'
|
||||
) {
|
||||
globals.logger.warn(`LOG EVENT V1: Unsupported log event source: ${msg.source}`);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
let tags;
|
||||
let fields;
|
||||
|
||||
// Process each source type
|
||||
if (msg.source === 'qseow-engine') {
|
||||
tags = {
|
||||
host: msg.host,
|
||||
level: msg.level,
|
||||
source: msg.source,
|
||||
log_row: msg.log_row,
|
||||
subsystem: msg.subsystem,
|
||||
};
|
||||
|
||||
// Tags that are empty in some cases. Only add if they are non-empty
|
||||
if (msg?.user_full?.length > 0) tags.user_full = msg.user_full;
|
||||
if (msg?.user_directory?.length > 0) tags.user_directory = msg.user_directory;
|
||||
if (msg?.user_id?.length > 0) tags.user_id = msg.user_id;
|
||||
if (msg?.result_code?.length > 0) tags.result_code = msg.result_code;
|
||||
if (msg?.windows_user?.length > 0) tags.windows_user = msg.windows_user;
|
||||
if (msg?.task_id?.length > 0) tags.task_id = msg.task_id;
|
||||
if (msg?.task_name?.length > 0) tags.task_name = msg.task_name;
|
||||
if (msg?.app_id?.length > 0) tags.app_id = msg.app_id;
|
||||
if (msg?.app_name?.length > 0) tags.app_name = msg.app_name;
|
||||
if (msg?.engine_exe_version?.length > 0)
|
||||
tags.engine_exe_version = msg.engine_exe_version;
|
||||
|
||||
fields = {
|
||||
message: msg.message,
|
||||
exception_message: msg.exception_message,
|
||||
command: msg.command,
|
||||
result_code: msg.result_code,
|
||||
origin: msg.origin,
|
||||
context: msg.context,
|
||||
session_id: msg.session_id,
|
||||
raw_event: JSON.stringify(msg),
|
||||
};
|
||||
} else if (msg.source === 'qseow-proxy') {
|
||||
tags = {
|
||||
host: msg.host,
|
||||
level: msg.level,
|
||||
source: msg.source,
|
||||
log_row: msg.log_row,
|
||||
subsystem: msg.subsystem,
|
||||
};
|
||||
|
||||
// Tags that are empty in some cases. Only add if they are non-empty
|
||||
if (msg?.user_full?.length > 0) tags.user_full = msg.user_full;
|
||||
if (msg?.user_directory?.length > 0) tags.user_directory = msg.user_directory;
|
||||
if (msg?.user_id?.length > 0) tags.user_id = msg.user_id;
|
||||
if (msg?.result_code?.length > 0) tags.result_code = msg.result_code;
|
||||
|
||||
fields = {
|
||||
message: msg.message,
|
||||
exception_message: msg.exception_message,
|
||||
command: msg.command,
|
||||
result_code: msg.result_code,
|
||||
origin: msg.origin,
|
||||
context: msg.context,
|
||||
raw_event: JSON.stringify(msg),
|
||||
};
|
||||
} else if (msg.source === 'qseow-scheduler') {
|
||||
tags = {
|
||||
host: msg.host,
|
||||
level: msg.level,
|
||||
source: msg.source,
|
||||
log_row: msg.log_row,
|
||||
subsystem: msg.subsystem,
|
||||
};
|
||||
|
||||
// Tags that are empty in some cases. Only add if they are non-empty
|
||||
if (msg?.user_full?.length > 0) tags.user_full = msg.user_full;
|
||||
if (msg?.user_directory?.length > 0) tags.user_directory = msg.user_directory;
|
||||
if (msg?.user_id?.length > 0) tags.user_id = msg.user_id;
|
||||
if (msg?.task_id?.length > 0) tags.task_id = msg.task_id;
|
||||
if (msg?.task_name?.length > 0) tags.task_name = msg.task_name;
|
||||
|
||||
fields = {
|
||||
message: msg.message,
|
||||
exception_message: msg.exception_message,
|
||||
app_name: msg.app_name,
|
||||
app_id: msg.app_id,
|
||||
execution_id: msg.execution_id,
|
||||
raw_event: JSON.stringify(msg),
|
||||
};
|
||||
} else if (msg.source === 'qseow-repository') {
|
||||
tags = {
|
||||
host: msg.host,
|
||||
level: msg.level,
|
||||
source: msg.source,
|
||||
log_row: msg.log_row,
|
||||
subsystem: msg.subsystem,
|
||||
};
|
||||
|
||||
// Tags that are empty in some cases. Only add if they are non-empty
|
||||
if (msg?.user_full?.length > 0) tags.user_full = msg.user_full;
|
||||
if (msg?.user_directory?.length > 0) tags.user_directory = msg.user_directory;
|
||||
if (msg?.user_id?.length > 0) tags.user_id = msg.user_id;
|
||||
if (msg?.result_code?.length > 0) tags.result_code = msg.result_code;
|
||||
|
||||
fields = {
|
||||
message: msg.message,
|
||||
exception_message: msg.exception_message,
|
||||
command: msg.command,
|
||||
result_code: msg.result_code,
|
||||
origin: msg.origin,
|
||||
context: msg.context,
|
||||
raw_event: JSON.stringify(msg),
|
||||
};
|
||||
} else if (msg.source === 'qseow-qix-perf') {
|
||||
tags = {
|
||||
host: msg.host?.length > 0 ? msg.host : '<Unknown>',
|
||||
level: msg.level?.length > 0 ? msg.level : '<Unknown>',
|
||||
source: msg.source?.length > 0 ? msg.source : '<Unknown>',
|
||||
log_row: msg.log_row?.length > 0 ? msg.log_row : '-1',
|
||||
subsystem: msg.subsystem?.length > 0 ? msg.subsystem : '<Unknown>',
|
||||
method: msg.method?.length > 0 ? msg.method : '<Unknown>',
|
||||
object_type: msg.object_type?.length > 0 ? msg.object_type : '<Unknown>',
|
||||
proxy_session_id: msg.proxy_session_id?.length > 0 ? msg.proxy_session_id : '-1',
|
||||
session_id: msg.session_id?.length > 0 ? msg.session_id : '-1',
|
||||
event_activity_source:
|
||||
msg.event_activity_source?.length > 0 ? msg.event_activity_source : '<Unknown>',
|
||||
};
|
||||
|
||||
// Tags that are empty in some cases. Only add if they are non-empty
|
||||
if (msg?.user_full?.length > 0) tags.user_full = msg.user_full;
|
||||
if (msg?.user_directory?.length > 0) tags.user_directory = msg.user_directory;
|
||||
if (msg?.user_id?.length > 0) tags.user_id = msg.user_id;
|
||||
if (msg?.app_id?.length > 0) tags.app_id = msg.app_id;
|
||||
if (msg?.app_name?.length > 0) tags.app_name = msg.app_name;
|
||||
if (msg?.object_id?.length > 0) tags.object_id = msg.object_id;
|
||||
|
||||
fields = {
|
||||
app_id: msg.app_id,
|
||||
process_time: msg.process_time,
|
||||
work_time: msg.work_time,
|
||||
lock_time: msg.lock_time,
|
||||
validate_time: msg.validate_time,
|
||||
traverse_time: msg.traverse_time,
|
||||
handle: msg.handle,
|
||||
net_ram: msg.net_ram,
|
||||
peak_ram: msg.peak_ram,
|
||||
raw_event: JSON.stringify(msg),
|
||||
};
|
||||
}
|
||||
|
||||
// Add log event categories to tags if available
|
||||
// The msg.category array contains objects with properties 'name' and 'value'
|
||||
if (msg?.category?.length > 0) {
|
||||
msg.category.forEach((category) => {
|
||||
tags[category.name] = category.value;
|
||||
});
|
||||
}
|
||||
|
||||
// Add custom tags from config file to payload
|
||||
if (
|
||||
globals.config.has('Butler-SOS.logEvents.tags') &&
|
||||
globals.config.get('Butler-SOS.logEvents.tags') !== null &&
|
||||
globals.config.get('Butler-SOS.logEvents.tags').length > 0
|
||||
) {
|
||||
const configTags = globals.config.get('Butler-SOS.logEvents.tags');
|
||||
for (const item of configTags) {
|
||||
tags[item.name] = item.value;
|
||||
}
|
||||
}
|
||||
|
||||
const datapoint = [
|
||||
{
|
||||
measurement: 'log_event',
|
||||
tags,
|
||||
fields,
|
||||
},
|
||||
];
|
||||
|
||||
globals.logger.silly(
|
||||
`LOG EVENT V1: Influxdb datapoint: ${JSON.stringify(datapoint, null, 2)}`
|
||||
);
|
||||
|
||||
// Write with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
datapoint,
|
||||
`Log event from ${msg.source}`,
|
||||
msg.host,
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('LOG EVENT V1: Sent log event data to InfluxDB');
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', msg.host);
|
||||
globals.logger.error(
|
||||
`LOG EVENT V1: Error saving log event: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
195
src/lib/influxdb/v1/queue-metrics.js
Normal file
195
src/lib/influxdb/v1/queue-metrics.js
Normal file
@@ -0,0 +1,195 @@
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV1 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Store user event queue metrics to InfluxDB v1
|
||||
*
|
||||
* @description
|
||||
* Retrieves metrics from the user event queue manager and stores them in InfluxDB v1
|
||||
* for monitoring queue health, backpressure, dropped messages, and processing performance.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeUserEventQueueMetricsV1() {
|
||||
try {
|
||||
// Check if queue metrics are enabled
|
||||
if (
|
||||
!globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.enable'
|
||||
)
|
||||
) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get metrics from queue manager
|
||||
const queueManager = globals.udpQueueManagerUserActivity;
|
||||
if (!queueManager) {
|
||||
globals.logger.warn('USER EVENT QUEUE METRICS V1: Queue manager not initialized');
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const metrics = await queueManager.getMetrics();
|
||||
|
||||
// Get configuration
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
);
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.tags'
|
||||
);
|
||||
|
||||
const point = {
|
||||
measurement: measurementName,
|
||||
tags: {
|
||||
queue_type: 'user_events',
|
||||
host: globals.hostInfo.hostname,
|
||||
},
|
||||
fields: {
|
||||
queue_size: metrics.queueSize,
|
||||
queue_max_size: metrics.queueMaxSize,
|
||||
queue_utilization_pct: metrics.queueUtilizationPct,
|
||||
queue_pending: metrics.queuePending,
|
||||
messages_received: metrics.messagesReceived,
|
||||
messages_queued: metrics.messagesQueued,
|
||||
messages_processed: metrics.messagesProcessed,
|
||||
messages_failed: metrics.messagesFailed,
|
||||
messages_dropped_total: metrics.messagesDroppedTotal,
|
||||
messages_dropped_rate_limit: metrics.messagesDroppedRateLimit,
|
||||
messages_dropped_queue_full: metrics.messagesDroppedQueueFull,
|
||||
messages_dropped_size: metrics.messagesDroppedSize,
|
||||
processing_time_avg_ms: metrics.processingTimeAvgMs,
|
||||
processing_time_p95_ms: metrics.processingTimeP95Ms,
|
||||
processing_time_max_ms: metrics.processingTimeMaxMs,
|
||||
rate_limit_current: metrics.rateLimitCurrent,
|
||||
backpressure_active: metrics.backpressureActive,
|
||||
},
|
||||
};
|
||||
|
||||
// Add static tags from config file
|
||||
if (configTags && configTags.length > 0) {
|
||||
for (const item of configTags) {
|
||||
point.tags[item.name] = item.value;
|
||||
}
|
||||
}
|
||||
|
||||
// Write with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
[point],
|
||||
'User event queue metrics',
|
||||
'',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('USER EVENT QUEUE METRICS V1: Sent queue metrics data to InfluxDB');
|
||||
|
||||
// Clear metrics after writing
|
||||
await queueManager.clearMetrics();
|
||||
} catch (err) {
|
||||
globals.logger.error(
|
||||
`USER EVENT QUEUE METRICS V1: Error saving data: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Store log event queue metrics to InfluxDB v1
|
||||
*
|
||||
* @description
|
||||
* Retrieves metrics from the log event queue manager and stores them in InfluxDB v1
|
||||
* for monitoring queue health, backpressure, dropped messages, and processing performance.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeLogEventQueueMetricsV1() {
|
||||
try {
|
||||
// Check if queue metrics are enabled
|
||||
if (
|
||||
!globals.config.get('Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.enable')
|
||||
) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get metrics from queue manager
|
||||
const queueManager = globals.udpQueueManagerLogEvents;
|
||||
if (!queueManager) {
|
||||
globals.logger.warn('LOG EVENT QUEUE METRICS V1: Queue manager not initialized');
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const metrics = await queueManager.getMetrics();
|
||||
|
||||
// Get configuration
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
);
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.tags'
|
||||
);
|
||||
|
||||
const point = {
|
||||
measurement: measurementName,
|
||||
tags: {
|
||||
queue_type: 'log_events',
|
||||
host: globals.hostInfo.hostname,
|
||||
},
|
||||
fields: {
|
||||
queue_size: metrics.queueSize,
|
||||
queue_max_size: metrics.queueMaxSize,
|
||||
queue_utilization_pct: metrics.queueUtilizationPct,
|
||||
queue_pending: metrics.queuePending,
|
||||
messages_received: metrics.messagesReceived,
|
||||
messages_queued: metrics.messagesQueued,
|
||||
messages_processed: metrics.messagesProcessed,
|
||||
messages_failed: metrics.messagesFailed,
|
||||
messages_dropped_total: metrics.messagesDroppedTotal,
|
||||
messages_dropped_rate_limit: metrics.messagesDroppedRateLimit,
|
||||
messages_dropped_queue_full: metrics.messagesDroppedQueueFull,
|
||||
messages_dropped_size: metrics.messagesDroppedSize,
|
||||
processing_time_avg_ms: metrics.processingTimeAvgMs,
|
||||
processing_time_p95_ms: metrics.processingTimeP95Ms,
|
||||
processing_time_max_ms: metrics.processingTimeMaxMs,
|
||||
rate_limit_current: metrics.rateLimitCurrent,
|
||||
backpressure_active: metrics.backpressureActive,
|
||||
},
|
||||
};
|
||||
|
||||
// Add static tags from config file
|
||||
if (configTags && configTags.length > 0) {
|
||||
for (const item of configTags) {
|
||||
point.tags[item.name] = item.value;
|
||||
}
|
||||
}
|
||||
|
||||
// Write with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
[point],
|
||||
'Log event queue metrics',
|
||||
'',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('LOG EVENT QUEUE METRICS V1: Sent queue metrics data to InfluxDB');
|
||||
|
||||
// Clear metrics after writing
|
||||
await queueManager.clearMetrics();
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', '');
|
||||
globals.logger.error(
|
||||
`LOG EVENT QUEUE METRICS V1: Error saving data: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
75
src/lib/influxdb/v1/sessions.js
Normal file
75
src/lib/influxdb/v1/sessions.js
Normal file
@@ -0,0 +1,75 @@
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV1 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts proxy sessions data to InfluxDB v1.
|
||||
*
|
||||
* This function takes user session data from Qlik Sense proxy and formats it for storage
|
||||
* in InfluxDB v1. It writes three types of measurements:
|
||||
* - user_session_summary: Summary with count and user list
|
||||
* - user_session_list: List of users (for compatibility)
|
||||
* - user_session_details: Individual session details for each active session
|
||||
*
|
||||
* @param {object} userSessions - User session data containing information about active sessions
|
||||
* @param {string} userSessions.host - The hostname of the server
|
||||
* @param {string} userSessions.virtualProxy - The virtual proxy name
|
||||
* @param {string} userSessions.serverName - Server name
|
||||
* @param {number} userSessions.sessionCount - Number of sessions
|
||||
* @param {string} userSessions.uniqueUserList - Comma-separated list of unique users
|
||||
* @param {Array} userSessions.datapointInfluxdb - Array of datapoints (plain objects for v1)
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeSessionsV1(userSessions) {
|
||||
globals.logger.debug(`PROXY SESSIONS V1: User sessions: ${JSON.stringify(userSessions)}`);
|
||||
|
||||
globals.logger.silly(
|
||||
`PROXY SESSIONS V1: Data for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}"`
|
||||
);
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
globals.logger.silly(
|
||||
`PROXY SESSIONS V1: Influxdb datapoint for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}": ${JSON.stringify(
|
||||
userSessions.datapointInfluxdb,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
|
||||
// Validate datapoints exist
|
||||
if (!userSessions.datapointInfluxdb || userSessions.datapointInfluxdb.length === 0) {
|
||||
globals.logger.warn('PROXY SESSIONS V1: No datapoints to write to InfluxDB');
|
||||
return;
|
||||
}
|
||||
|
||||
// Data points are already in InfluxDB v1 format (plain objects)
|
||||
// Write array of measurements with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
userSessions.datapointInfluxdb,
|
||||
`Proxy sessions for ${userSessions.host}/${userSessions.virtualProxy}`,
|
||||
userSessions.serverName,
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.debug(
|
||||
`PROXY SESSIONS V1: Session count for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}": ${userSessions.sessionCount}`
|
||||
);
|
||||
globals.logger.debug(
|
||||
`PROXY SESSIONS V1: User list for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}": ${userSessions.uniqueUserList}`
|
||||
);
|
||||
|
||||
globals.logger.verbose(
|
||||
`PROXY SESSIONS V1: Sent user session data to InfluxDB for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}"`
|
||||
);
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', userSessions.host);
|
||||
globals.logger.error(
|
||||
`PROXY SESSIONS V1: Error saving user session data: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
106
src/lib/influxdb/v1/user-events.js
Normal file
106
src/lib/influxdb/v1/user-events.js
Normal file
@@ -0,0 +1,106 @@
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV1 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts a user event to InfluxDB v1.
|
||||
*
|
||||
* User events track user interactions with Qlik Sense, such as opening apps,
|
||||
* starting sessions, creating connections, etc.
|
||||
*
|
||||
* @param {object} msg - The event to be posted to InfluxDB. The object should contain the following properties:
|
||||
* - host: The hostname of the Qlik Sense server that the user event originated from.
|
||||
* - command: The command (e.g. OpenApp, CreateApp, etc.) that the user event corresponds to.
|
||||
* - user_directory: The user directory of the user who triggered the event.
|
||||
* - user_id: The user ID of the user who triggered the event.
|
||||
* - origin: The origin of the event (e.g. Qlik Sense, QlikView, etc.).
|
||||
* - appId: The ID of the app that the event corresponds to (if applicable).
|
||||
* - appName: The name of the app that the event corresponds to (if applicable).
|
||||
* - ua: An object containing user agent information (if available).
|
||||
* @returns {Promise<void>} A promise that resolves when the event has been posted to InfluxDB.
|
||||
*/
|
||||
export async function storeUserEventV1(msg) {
|
||||
globals.logger.debug(`USER EVENT V1: ${JSON.stringify(msg)}`);
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Validate required fields
|
||||
if (!msg.host || !msg.command || !msg.user_directory || !msg.user_id || !msg.origin) {
|
||||
globals.logger.warn(
|
||||
`USER EVENT V1: Missing required fields in user event message: ${JSON.stringify(msg)}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
// First prepare tags relating to the actual user event, then add tags defined in the config file
|
||||
// The config file tags can for example be used to separate data from DEV/TEST/PROD environments
|
||||
const tags = {
|
||||
host: msg.host,
|
||||
event_action: msg.command,
|
||||
userFull: `${msg.user_directory}\\${msg.user_id}`,
|
||||
userDirectory: msg.user_directory,
|
||||
userId: msg.user_id,
|
||||
origin: msg.origin,
|
||||
};
|
||||
|
||||
// Add app id and name to tags if available
|
||||
if (msg?.appId) tags.appId = msg.appId;
|
||||
if (msg?.appName) tags.appName = msg.appName;
|
||||
|
||||
// Add user agent info to tags if available
|
||||
if (msg?.ua?.browser?.name) tags.uaBrowserName = msg?.ua?.browser?.name;
|
||||
if (msg?.ua?.browser?.major) tags.uaBrowserMajorVersion = msg?.ua?.browser?.major;
|
||||
if (msg?.ua?.os?.name) tags.uaOsName = msg?.ua?.os?.name;
|
||||
if (msg?.ua?.os?.version) tags.uaOsVersion = msg?.ua?.os?.version;
|
||||
|
||||
// Add custom tags from config file to payload
|
||||
if (
|
||||
globals.config.has('Butler-SOS.userEvents.tags') &&
|
||||
globals.config.get('Butler-SOS.userEvents.tags') !== null &&
|
||||
globals.config.get('Butler-SOS.userEvents.tags').length > 0
|
||||
) {
|
||||
const configTags = globals.config.get('Butler-SOS.userEvents.tags');
|
||||
for (const item of configTags) {
|
||||
tags[item.name] = item.value;
|
||||
}
|
||||
}
|
||||
|
||||
const datapoint = [
|
||||
{
|
||||
measurement: 'user_events',
|
||||
tags,
|
||||
fields: {
|
||||
userFull: tags.userFull,
|
||||
userId: tags.userId,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
// Add app id and name to fields if available
|
||||
if (msg?.appId) datapoint[0].fields.appId = msg.appId;
|
||||
if (msg?.appName) datapoint[0].fields.appName = msg.appName;
|
||||
|
||||
globals.logger.silly(
|
||||
`USER EVENT V1: Influxdb datapoint: ${JSON.stringify(datapoint, null, 2)}`
|
||||
);
|
||||
|
||||
// Write with retry logic
|
||||
await writeBatchToInfluxV1(
|
||||
datapoint,
|
||||
'User event',
|
||||
msg.host,
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('USER EVENT V1: Sent user event data to InfluxDB');
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V1_WRITE', msg.host);
|
||||
globals.logger.error(
|
||||
`USER EVENT V1: Error saving user event: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
65
src/lib/influxdb/v2/butler-memory.js
Normal file
65
src/lib/influxdb/v2/butler-memory.js
Normal file
@@ -0,0 +1,65 @@
|
||||
import { Point } from '@influxdata/influxdb-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV2 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts Butler SOS memory usage metrics to InfluxDB v2.
|
||||
*
|
||||
* This function captures memory usage metrics from the Butler SOS process itself
|
||||
* and stores them in InfluxDB v2.
|
||||
*
|
||||
* @param {object} memory - Memory usage data object
|
||||
* @param {string} memory.instanceTag - Instance identifier tag
|
||||
* @param {number} memory.heapUsedMByte - Heap used in MB
|
||||
* @param {number} memory.heapTotalMByte - Total heap size in MB
|
||||
* @param {number} memory.externalMemoryMByte - External memory usage in MB
|
||||
* @param {number} memory.processMemoryMByte - Process memory usage in MB
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeButlerMemoryV2(memory) {
|
||||
globals.logger.debug(`MEMORY USAGE V2: Memory usage ${JSON.stringify(memory, null, 2)}`);
|
||||
|
||||
// Check if InfluxDB v2 is enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Validate input
|
||||
if (!memory || typeof memory !== 'object') {
|
||||
globals.logger.warn('MEMORY USAGE V2: Invalid memory data provided');
|
||||
return;
|
||||
}
|
||||
|
||||
const butlerVersion = globals.appVersion;
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
|
||||
// Create point using v2 Point class
|
||||
const point = new Point('butlersos_memory_usage')
|
||||
.tag('butler_sos_instance', memory.instanceTag)
|
||||
.tag('version', butlerVersion)
|
||||
.floatField('heap_used', memory.heapUsedMByte)
|
||||
.floatField('heap_total', memory.heapTotalMByte)
|
||||
.floatField('external', memory.externalMemoryMByte)
|
||||
.floatField('process_memory', memory.processMemoryMByte);
|
||||
|
||||
globals.logger.silly(
|
||||
`MEMORY USAGE V2: Influxdb datapoint for Butler SOS memory usage: ${JSON.stringify(
|
||||
point,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
|
||||
// Write to InfluxDB with retry logic
|
||||
await writeBatchToInfluxV2(
|
||||
[point],
|
||||
org,
|
||||
bucketName,
|
||||
'Memory usage metrics',
|
||||
'',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('MEMORY USAGE V2: Sent Butler SOS memory usage data to InfluxDB');
|
||||
}
|
||||
178
src/lib/influxdb/v2/event-counts.js
Normal file
178
src/lib/influxdb/v2/event-counts.js
Normal file
@@ -0,0 +1,178 @@
|
||||
import { Point } from '@influxdata/influxdb-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV2 } from '../shared/utils.js';
|
||||
import { applyInfluxTags } from './utils.js';
|
||||
|
||||
/**
|
||||
* Posts event counts to InfluxDB v2.
|
||||
*
|
||||
* @description
|
||||
* This function reads arrays of log and user events from the `udpEvents` object,
|
||||
* and stores the data in InfluxDB v2. The data is written to a measurement named after
|
||||
* the `Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName` config setting.
|
||||
*
|
||||
* Aggregates and stores counts for log and user events
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeEventCountV2() {
|
||||
globals.logger.debug('EVENT COUNT V2: Starting to store event counts');
|
||||
|
||||
// Check if InfluxDB v2 is enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get array of log events
|
||||
const logEvents = await globals.udpEvents.getLogEvents();
|
||||
const userEvents = await globals.udpEvents.getUserEvents();
|
||||
|
||||
globals.logger.debug(`EVENT COUNT V2: Log events: ${JSON.stringify(logEvents, null, 2)}`);
|
||||
globals.logger.debug(`EVENT COUNT V2: User events: ${JSON.stringify(userEvents, null, 2)}`);
|
||||
|
||||
// Are there any events to store?
|
||||
if (logEvents.length === 0 && userEvents.length === 0) {
|
||||
globals.logger.verbose('EVENT COUNT V2: No events to store in InfluxDB');
|
||||
return;
|
||||
}
|
||||
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName'
|
||||
);
|
||||
const configTags = globals.config.get('Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags');
|
||||
|
||||
const points = [];
|
||||
|
||||
// Loop through data in log events and create datapoints
|
||||
for (const event of logEvents) {
|
||||
const point = new Point(measurementName)
|
||||
.tag('event_type', 'log')
|
||||
.tag('source', event.source)
|
||||
.tag('host', event.host)
|
||||
.tag('subsystem', event.subsystem)
|
||||
.intField('counter', event.counter);
|
||||
|
||||
// Add static tags from config file
|
||||
applyInfluxTags(point, configTags);
|
||||
points.push(point);
|
||||
}
|
||||
|
||||
// Loop through data in user events and create datapoints
|
||||
for (const event of userEvents) {
|
||||
const point = new Point(measurementName)
|
||||
.tag('event_type', 'user')
|
||||
.tag('source', event.source)
|
||||
.tag('host', event.host)
|
||||
.tag('subsystem', event.subsystem)
|
||||
.intField('counter', event.counter);
|
||||
|
||||
// Add static tags from config file
|
||||
applyInfluxTags(point, configTags);
|
||||
points.push(point);
|
||||
}
|
||||
|
||||
// Write to InfluxDB with retry logic
|
||||
await writeBatchToInfluxV2(
|
||||
points,
|
||||
org,
|
||||
bucketName,
|
||||
'Event count metrics',
|
||||
'',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('EVENT COUNT V2: Sent event count data to InfluxDB');
|
||||
}
|
||||
|
||||
/**
|
||||
* Posts rejected event counts to InfluxDB v2.
|
||||
*
|
||||
* @description
|
||||
* Tracks events that were rejected by Butler SOS due to validation failures,
|
||||
* rate limiting, or filtering rules. Helps monitor data quality and filtering effectiveness.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeRejectedEventCountV2() {
|
||||
globals.logger.debug('REJECTED EVENT COUNT V2: Starting to store rejected event counts');
|
||||
|
||||
// Check if InfluxDB v2 is enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get array of rejected log events
|
||||
const rejectedLogEvents = await globals.rejectedEvents.getRejectedLogEvents();
|
||||
|
||||
globals.logger.debug(
|
||||
`REJECTED EVENT COUNT V2: Rejected log events: ${JSON.stringify(
|
||||
rejectedLogEvents,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
|
||||
// Are there any events to store?
|
||||
if (rejectedLogEvents.length === 0) {
|
||||
globals.logger.verbose('REJECTED EVENT COUNT V2: No events to store in InfluxDB');
|
||||
return;
|
||||
}
|
||||
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.qlikSenseEvents.rejectedEventCount.influxdb.measurementName'
|
||||
);
|
||||
|
||||
const points = [];
|
||||
|
||||
// Loop through data in rejected log events and create datapoints
|
||||
for (const event of rejectedLogEvents) {
|
||||
if (event.source === 'qseow-qix-perf') {
|
||||
// For qix-perf events, include app info and performance metrics
|
||||
const point = new Point(measurementName)
|
||||
.tag('source', event.source)
|
||||
.tag('app_id', event.appId)
|
||||
.tag('method', event.method)
|
||||
.tag('object_type', event.objectType)
|
||||
.intField('counter', event.counter)
|
||||
.floatField('process_time', event.processTime);
|
||||
|
||||
if (event?.appName?.length > 0) {
|
||||
point.tag('app_name', event.appName).tag('app_name_set', 'true');
|
||||
} else {
|
||||
point.tag('app_name_set', 'false');
|
||||
}
|
||||
|
||||
// Add static tags from config file
|
||||
const perfMonitorTags = globals.config.get(
|
||||
'Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.tags'
|
||||
);
|
||||
applyInfluxTags(point, perfMonitorTags);
|
||||
|
||||
points.push(point);
|
||||
} else {
|
||||
const point = new Point(measurementName)
|
||||
.tag('source', event.source)
|
||||
.intField('counter', event.counter);
|
||||
|
||||
points.push(point);
|
||||
}
|
||||
}
|
||||
|
||||
// Write to InfluxDB with retry logic
|
||||
await writeBatchToInfluxV2(
|
||||
points,
|
||||
org,
|
||||
bucketName,
|
||||
'Rejected event count metrics',
|
||||
'',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('REJECTED EVENT COUNT V2: Sent rejected event count data to InfluxDB');
|
||||
}
|
||||
191
src/lib/influxdb/v2/health-metrics.js
Normal file
191
src/lib/influxdb/v2/health-metrics.js
Normal file
@@ -0,0 +1,191 @@
|
||||
import { Point } from '@influxdata/influxdb-client';
|
||||
import globals from '../../../globals.js';
|
||||
import {
|
||||
getFormattedTime,
|
||||
processAppDocuments,
|
||||
isInfluxDbEnabled,
|
||||
writeToInfluxWithRetry,
|
||||
} from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts health metrics data from Qlik Sense to InfluxDB v2.
|
||||
*
|
||||
* This function processes health data from the Sense engine's healthcheck API and
|
||||
* formats it for storage in InfluxDB v2. It handles various metrics including:
|
||||
* - CPU usage
|
||||
* - Memory usage (committed, allocated, free)
|
||||
* - Cache metrics (hits, lookups, additions, replacements)
|
||||
* - Active/loaded/in-memory apps
|
||||
* - Session counts (active, total)
|
||||
* - User counts (active, total)
|
||||
* - Server version and uptime
|
||||
*
|
||||
* @param {string} serverName - The name of the Qlik Sense server
|
||||
* @param {string} host - The hostname or IP of the Qlik Sense server
|
||||
* @param {object} body - Health metrics data from Sense engine
|
||||
* @param {object} serverTags - Server-specific tags to add to datapoints
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function storeHealthMetricsV2(serverName, host, body, serverTags) {
|
||||
globals.logger.debug(`HEALTH METRICS V2: Health data: ${JSON.stringify(body, null, 2)}`);
|
||||
|
||||
// Check if InfluxDB v2 is enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Validate input
|
||||
if (!body || typeof body !== 'object') {
|
||||
globals.logger.warn(`HEALTH METRICS V2: Invalid health data from server ${serverName}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
|
||||
// Process app names for different document types
|
||||
const { appNames: appNamesActive, sessionAppNames: sessionAppNamesActive } =
|
||||
await processAppDocuments(body.apps.active_docs, 'HEALTH METRICS', 'active');
|
||||
const { appNames: appNamesLoaded, sessionAppNames: sessionAppNamesLoaded } =
|
||||
await processAppDocuments(body.apps.loaded_docs, 'HEALTH METRICS', 'loaded');
|
||||
const { appNames: appNamesInMemory, sessionAppNames: sessionAppNamesInMemory } =
|
||||
await processAppDocuments(body.apps.in_memory_docs, 'HEALTH METRICS', 'in memory');
|
||||
|
||||
const formattedTime = getFormattedTime(body.started);
|
||||
|
||||
// Create points using v2 Point class
|
||||
const points = [
|
||||
new Point('sense_server')
|
||||
.stringField('version', body.version)
|
||||
.stringField('started', body.started)
|
||||
.stringField('uptime', formattedTime),
|
||||
|
||||
new Point('mem')
|
||||
.floatField('comitted', body.mem.committed)
|
||||
.floatField('allocated', body.mem.allocated)
|
||||
.floatField('free', body.mem.free),
|
||||
|
||||
new Point('apps')
|
||||
.intField('active_docs_count', body.apps.active_docs.length)
|
||||
.intField('loaded_docs_count', body.apps.loaded_docs.length)
|
||||
.intField('in_memory_docs_count', body.apps.in_memory_docs.length)
|
||||
.stringField(
|
||||
'active_docs',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
|
||||
? body.apps.active_docs
|
||||
: ''
|
||||
)
|
||||
.stringField(
|
||||
'active_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
|
||||
? appNamesActive.toString()
|
||||
: ''
|
||||
)
|
||||
.stringField(
|
||||
'active_session_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
|
||||
? sessionAppNamesActive.toString()
|
||||
: ''
|
||||
)
|
||||
.stringField(
|
||||
'loaded_docs',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
|
||||
? body.apps.loaded_docs
|
||||
: ''
|
||||
)
|
||||
.stringField(
|
||||
'loaded_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
|
||||
? appNamesLoaded.toString()
|
||||
: ''
|
||||
)
|
||||
.stringField(
|
||||
'loaded_session_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
|
||||
? sessionAppNamesLoaded.toString()
|
||||
: ''
|
||||
)
|
||||
.stringField(
|
||||
'in_memory_docs',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
|
||||
? body.apps.in_memory_docs
|
||||
: ''
|
||||
)
|
||||
.stringField(
|
||||
'in_memory_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
|
||||
? appNamesInMemory.toString()
|
||||
: ''
|
||||
)
|
||||
.stringField(
|
||||
'in_memory_session_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
|
||||
? sessionAppNamesInMemory.toString()
|
||||
: ''
|
||||
)
|
||||
.uintField('calls', body.apps.calls)
|
||||
.uintField('selections', body.apps.selections),
|
||||
|
||||
new Point('cpu').floatField('total', body.cpu.total),
|
||||
|
||||
new Point('session')
|
||||
.uintField('active', body.session.active)
|
||||
.uintField('total', body.session.total),
|
||||
|
||||
new Point('users')
|
||||
.uintField('active', body.users.active)
|
||||
.uintField('total', body.users.total),
|
||||
|
||||
new Point('cache')
|
||||
.uintField('hits', body.cache.hits)
|
||||
.uintField('lookups', body.cache.lookups)
|
||||
.intField('added', body.cache.added)
|
||||
.intField('replaced', body.cache.replaced)
|
||||
.intField('bytes_added', body.cache.bytes_added),
|
||||
|
||||
new Point('saturated').booleanField('saturated', body.saturated),
|
||||
];
|
||||
|
||||
// Add server tags to all points
|
||||
if (serverTags && typeof serverTags === 'object') {
|
||||
for (const point of points) {
|
||||
for (const [key, value] of Object.entries(serverTags)) {
|
||||
if (value !== undefined && value !== null) {
|
||||
point.tag(key, String(value));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Write all points to InfluxDB with retry logic
|
||||
await writeToInfluxWithRetry(
|
||||
async () => {
|
||||
const writeApi = globals.influx.getWriteApi(org, bucketName, 'ns', {
|
||||
flushInterval: 5000,
|
||||
maxRetries: 0,
|
||||
});
|
||||
try {
|
||||
await writeApi.writePoints(points);
|
||||
await writeApi.close();
|
||||
} catch (err) {
|
||||
try {
|
||||
await writeApi.close();
|
||||
} catch (closeErr) {
|
||||
// Ignore close errors
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
`Health metrics from ${serverName}`,
|
||||
'v2',
|
||||
serverName
|
||||
);
|
||||
|
||||
globals.logger.verbose(`HEALTH METRICS V2: Stored health data from server: ${serverName}`);
|
||||
}
|
||||
229
src/lib/influxdb/v2/log-events.js
Normal file
229
src/lib/influxdb/v2/log-events.js
Normal file
@@ -0,0 +1,229 @@
|
||||
import { Point } from '@influxdata/influxdb-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV2 } from '../shared/utils.js';
|
||||
import { applyInfluxTags } from './utils.js';
|
||||
|
||||
/**
|
||||
* Store log event to InfluxDB v2
|
||||
*
|
||||
* @description
|
||||
* Handles log events from 5 different Qlik Sense sources:
|
||||
* - qseow-engine: Engine log events
|
||||
* - qseow-proxy: Proxy log events
|
||||
* - qseow-scheduler: Scheduler log events
|
||||
* - qseow-repository: Repository log events
|
||||
* - qseow-qix-perf: QIX performance metrics
|
||||
*
|
||||
* Each source has specific fields and tags that are written to InfluxDB.
|
||||
* Note: Uses _field suffix for fields that conflict with tag names (e.g., result_code_field).
|
||||
*
|
||||
* @param {object} msg - Log event message containing the following properties:
|
||||
* @param {string} msg.host - Hostname of the Qlik Sense server
|
||||
* @param {string} msg.source - Event source (qseow-engine, qseow-proxy, qseow-scheduler, qseow-repository, qseow-qix-perf)
|
||||
* @param {string} msg.level - Log level (e.g., INFO, WARN, ERROR)
|
||||
* @param {string} msg.log_row - Log row identifier
|
||||
* @param {string} msg.subsystem - Subsystem generating the log
|
||||
* @param {string} msg.message - Log message text
|
||||
* @param {string} [msg.exception_message] - Exception message if applicable
|
||||
* @param {string} [msg.command] - Command being executed
|
||||
* @param {string} [msg.result_code] - Result code of operation
|
||||
* @param {string} [msg.origin] - Origin of the event
|
||||
* @param {string} [msg.context] - Context information
|
||||
* @param {string} [msg.session_id] - Session identifier
|
||||
* @param {string} [msg.user_full] - Full user name
|
||||
* @param {string} [msg.user_directory] - User directory
|
||||
* @param {string} [msg.user_id] - User ID
|
||||
* @param {string} [msg.windows_user] - Windows username
|
||||
* @param {string} [msg.task_id] - Task identifier
|
||||
* @param {string} [msg.task_name] - Task name
|
||||
* @param {string} [msg.app_id] - Application ID
|
||||
* @param {string} [msg.app_name] - Application name
|
||||
* @param {string} [msg.engine_exe_version] - Engine executable version
|
||||
* @param {string} [msg.execution_id] - Execution identifier (scheduler)
|
||||
* @param {string} [msg.method] - QIX method (qix-perf)
|
||||
* @param {string} [msg.object_type] - Object type (qix-perf)
|
||||
* @param {string} [msg.proxy_session_id] - Proxy session ID (qix-perf)
|
||||
* @param {string} [msg.event_activity_source] - Event activity source (qix-perf)
|
||||
* @param {number} [msg.process_time] - Process time in ms (qix-perf)
|
||||
* @param {number} [msg.work_time] - Work time in ms (qix-perf)
|
||||
* @param {number} [msg.lock_time] - Lock time in ms (qix-perf)
|
||||
* @param {number} [msg.validate_time] - Validate time in ms (qix-perf)
|
||||
* @param {number} [msg.traverse_time] - Traverse time in ms (qix-perf)
|
||||
* @param {string} [msg.handle] - Handle identifier (qix-perf)
|
||||
* @param {number} [msg.net_ram] - Net RAM usage (qix-perf)
|
||||
* @param {number} [msg.peak_ram] - Peak RAM usage (qix-perf)
|
||||
* @param {string} [msg.object_id] - Object identifier (qix-perf)
|
||||
* @param {Array<{name: string, value: string}>} [msg.category] - Array of category objects
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeLogEventV2(msg) {
|
||||
globals.logger.debug(`LOG EVENT V2: ${JSON.stringify(msg)}`);
|
||||
|
||||
// Only write to InfluxDB if enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Validate source
|
||||
if (
|
||||
msg.source !== 'qseow-engine' &&
|
||||
msg.source !== 'qseow-proxy' &&
|
||||
msg.source !== 'qseow-scheduler' &&
|
||||
msg.source !== 'qseow-repository' &&
|
||||
msg.source !== 'qseow-qix-perf'
|
||||
) {
|
||||
globals.logger.warn(`LOG EVENT V2: Unsupported log event source: ${msg.source}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
|
||||
let point;
|
||||
|
||||
// Process each source type
|
||||
if (msg.source === 'qseow-engine') {
|
||||
point = new Point('log_event')
|
||||
.tag('host', msg.host)
|
||||
.tag('level', msg.level)
|
||||
.tag('source', msg.source)
|
||||
.tag('log_row', msg.log_row)
|
||||
.tag('subsystem', msg.subsystem)
|
||||
.stringField('message', msg.message)
|
||||
.stringField('exception_message', msg.exception_message || '')
|
||||
.stringField('command', msg.command || '')
|
||||
.stringField('result_code_field', msg.result_code || '')
|
||||
.stringField('origin', msg.origin || '')
|
||||
.stringField('context', msg.context || '')
|
||||
.stringField('session_id', msg.session_id || '')
|
||||
.stringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.tag('user_full', msg.user_full);
|
||||
if (msg?.user_directory?.length > 0) point.tag('user_directory', msg.user_directory);
|
||||
if (msg?.user_id?.length > 0) point.tag('user_id', msg.user_id);
|
||||
if (msg?.result_code?.length > 0) point.tag('result_code', msg.result_code);
|
||||
if (msg?.windows_user?.length > 0) point.tag('windows_user', msg.windows_user);
|
||||
if (msg?.task_id?.length > 0) point.tag('task_id', msg.task_id);
|
||||
if (msg?.task_name?.length > 0) point.tag('task_name', msg.task_name);
|
||||
if (msg?.app_id?.length > 0) point.tag('app_id', msg.app_id);
|
||||
if (msg?.app_name?.length > 0) point.tag('app_name', msg.app_name);
|
||||
if (msg?.engine_exe_version?.length > 0)
|
||||
point.tag('engine_exe_version', msg.engine_exe_version);
|
||||
} else if (msg.source === 'qseow-proxy') {
|
||||
point = new Point('log_event')
|
||||
.tag('host', msg.host)
|
||||
.tag('level', msg.level)
|
||||
.tag('source', msg.source)
|
||||
.tag('log_row', msg.log_row)
|
||||
.tag('subsystem', msg.subsystem)
|
||||
.stringField('message', msg.message)
|
||||
.stringField('exception_message', msg.exception_message || '')
|
||||
.stringField('command', msg.command || '')
|
||||
.stringField('result_code_field', msg.result_code || '')
|
||||
.stringField('origin', msg.origin || '')
|
||||
.stringField('context', msg.context || '')
|
||||
.stringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.tag('user_full', msg.user_full);
|
||||
if (msg?.user_directory?.length > 0) point.tag('user_directory', msg.user_directory);
|
||||
if (msg?.user_id?.length > 0) point.tag('user_id', msg.user_id);
|
||||
if (msg?.result_code?.length > 0) point.tag('result_code', msg.result_code);
|
||||
} else if (msg.source === 'qseow-scheduler') {
|
||||
point = new Point('log_event')
|
||||
.tag('host', msg.host)
|
||||
.tag('level', msg.level)
|
||||
.tag('source', msg.source)
|
||||
.tag('log_row', msg.log_row)
|
||||
.tag('subsystem', msg.subsystem)
|
||||
.stringField('message', msg.message)
|
||||
.stringField('exception_message', msg.exception_message || '')
|
||||
.stringField('app_name', msg.app_name || '')
|
||||
.stringField('app_id', msg.app_id || '')
|
||||
.stringField('execution_id', msg.execution_id || '')
|
||||
.stringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.tag('user_full', msg.user_full);
|
||||
if (msg?.user_directory?.length > 0) point.tag('user_directory', msg.user_directory);
|
||||
if (msg?.user_id?.length > 0) point.tag('user_id', msg.user_id);
|
||||
if (msg?.task_id?.length > 0) point.tag('task_id', msg.task_id);
|
||||
if (msg?.task_name?.length > 0) point.tag('task_name', msg.task_name);
|
||||
} else if (msg.source === 'qseow-repository') {
|
||||
point = new Point('log_event')
|
||||
.tag('host', msg.host)
|
||||
.tag('level', msg.level)
|
||||
.tag('source', msg.source)
|
||||
.tag('log_row', msg.log_row)
|
||||
.tag('subsystem', msg.subsystem)
|
||||
.stringField('message', msg.message)
|
||||
.stringField('exception_message', msg.exception_message || '')
|
||||
.stringField('command', msg.command || '')
|
||||
.stringField('result_code_field', msg.result_code || '')
|
||||
.stringField('origin', msg.origin || '')
|
||||
.stringField('context', msg.context || '')
|
||||
.stringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.tag('user_full', msg.user_full);
|
||||
if (msg?.user_directory?.length > 0) point.tag('user_directory', msg.user_directory);
|
||||
if (msg?.user_id?.length > 0) point.tag('user_id', msg.user_id);
|
||||
if (msg?.result_code?.length > 0) point.tag('result_code', msg.result_code);
|
||||
} else if (msg.source === 'qseow-qix-perf') {
|
||||
point = new Point('log_event')
|
||||
.tag('host', msg.host)
|
||||
.tag('level', msg.level)
|
||||
.tag('source', msg.source)
|
||||
.tag('log_row', msg.log_row)
|
||||
.tag('subsystem', msg.subsystem)
|
||||
.tag('method', msg.method)
|
||||
.tag('object_type', msg.object_type)
|
||||
.tag('proxy_session_id', msg.proxy_session_id)
|
||||
.tag('session_id', msg.session_id)
|
||||
.tag('event_activity_source', msg.event_activity_source)
|
||||
.stringField('app_id', msg.app_id || '')
|
||||
.floatField('process_time', parseFloat(msg.process_time))
|
||||
.floatField('work_time', parseFloat(msg.work_time))
|
||||
.floatField('lock_time', parseFloat(msg.lock_time))
|
||||
.floatField('validate_time', parseFloat(msg.validate_time))
|
||||
.floatField('traverse_time', parseFloat(msg.traverse_time))
|
||||
.stringField('handle', msg.handle || '')
|
||||
.intField('net_ram', parseInt(msg.net_ram))
|
||||
.intField('peak_ram', parseInt(msg.peak_ram))
|
||||
.stringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.tag('user_full', msg.user_full);
|
||||
if (msg?.user_directory?.length > 0) point.tag('user_directory', msg.user_directory);
|
||||
if (msg?.user_id?.length > 0) point.tag('user_id', msg.user_id);
|
||||
if (msg?.app_id?.length > 0) point.tag('app_id', msg.app_id);
|
||||
if (msg?.app_name?.length > 0) point.tag('app_name', msg.app_name);
|
||||
if (msg?.object_id?.length > 0) point.tag('object_id', msg.object_id);
|
||||
}
|
||||
|
||||
// Add log event categories to tags if available
|
||||
if (msg?.category?.length > 0) {
|
||||
msg.category.forEach((category) => {
|
||||
point.tag(category.name, category.value);
|
||||
});
|
||||
}
|
||||
|
||||
// Add custom tags from config file
|
||||
const configTags = globals.config.get('Butler-SOS.logEvents.tags');
|
||||
applyInfluxTags(point, configTags);
|
||||
|
||||
globals.logger.silly(`LOG EVENT V2: Influxdb datapoint: ${JSON.stringify(point, null, 2)}`);
|
||||
|
||||
// Write to InfluxDB with retry logic
|
||||
await writeBatchToInfluxV2(
|
||||
[point],
|
||||
org,
|
||||
bucketName,
|
||||
`Log event for ${msg.host}`,
|
||||
msg.host,
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('LOG EVENT V2: Sent log event data to InfluxDB');
|
||||
}
|
||||
176
src/lib/influxdb/v2/queue-metrics.js
Normal file
176
src/lib/influxdb/v2/queue-metrics.js
Normal file
@@ -0,0 +1,176 @@
|
||||
import { Point } from '@influxdata/influxdb-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV2 } from '../shared/utils.js';
|
||||
import { applyInfluxTags } from './utils.js';
|
||||
|
||||
/**
|
||||
* Store user event queue metrics to InfluxDB v2
|
||||
*
|
||||
* @description
|
||||
* Retrieves metrics from the user event queue manager and stores them in InfluxDB v2
|
||||
* for monitoring queue health, backpressure, dropped messages, and processing performance.
|
||||
* After successful write, clears the metrics to start fresh tracking.
|
||||
*
|
||||
* Metrics include:
|
||||
* - Queue size and utilization
|
||||
* - Message counts (received, queued, processed, failed, dropped)
|
||||
* - Processing time statistics (average, p95, max)
|
||||
* - Rate limiting and backpressure status
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeUserEventQueueMetricsV2() {
|
||||
// Check if queue metrics are enabled
|
||||
if (!globals.config.get('Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.enable')) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get metrics from queue manager
|
||||
const queueManager = globals.udpQueueManagerUserActivity;
|
||||
if (!queueManager) {
|
||||
globals.logger.warn('USER EVENT QUEUE METRICS V2: Queue manager not initialized');
|
||||
return;
|
||||
}
|
||||
|
||||
const metrics = await queueManager.getMetrics();
|
||||
|
||||
// Get configuration
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
);
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.tags'
|
||||
);
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
|
||||
const point = new Point(measurementName)
|
||||
.tag('queue_type', 'user_events')
|
||||
.tag('host', globals.hostInfo.hostname)
|
||||
.intField('queue_size', metrics.queueSize)
|
||||
.intField('queue_max_size', metrics.queueMaxSize)
|
||||
.floatField('queue_utilization_pct', metrics.queueUtilizationPct)
|
||||
.intField('queue_pending', metrics.queuePending)
|
||||
.intField('messages_received', metrics.messagesReceived)
|
||||
.intField('messages_queued', metrics.messagesQueued)
|
||||
.intField('messages_processed', metrics.messagesProcessed)
|
||||
.intField('messages_failed', metrics.messagesFailed)
|
||||
.intField('messages_dropped_total', metrics.messagesDroppedTotal)
|
||||
.intField('messages_dropped_rate_limit', metrics.messagesDroppedRateLimit)
|
||||
.intField('messages_dropped_queue_full', metrics.messagesDroppedQueueFull)
|
||||
.intField('messages_dropped_size', metrics.messagesDroppedSize)
|
||||
.floatField('processing_time_avg_ms', metrics.processingTimeAvgMs)
|
||||
.floatField('processing_time_p95_ms', metrics.processingTimeP95Ms)
|
||||
.floatField('processing_time_max_ms', metrics.processingTimeMaxMs)
|
||||
.intField('rate_limit_current', metrics.rateLimitCurrent)
|
||||
.intField('backpressure_active', metrics.backpressureActive);
|
||||
|
||||
// Add static tags from config file
|
||||
applyInfluxTags(point, configTags);
|
||||
|
||||
// Write to InfluxDB with retry logic
|
||||
await writeBatchToInfluxV2(
|
||||
[point],
|
||||
org,
|
||||
bucketName,
|
||||
'User event queue metrics',
|
||||
'user-events-queue',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('USER EVENT QUEUE METRICS V2: Sent queue metrics data to InfluxDB');
|
||||
|
||||
// Clear metrics after successful write
|
||||
await queueManager.clearMetrics();
|
||||
}
|
||||
|
||||
/**
|
||||
* Store log event queue metrics to InfluxDB v2
|
||||
*
|
||||
* @description
|
||||
* Retrieves metrics from the log event queue manager and stores them in InfluxDB v2
|
||||
* for monitoring queue health, backpressure, dropped messages, and processing performance.
|
||||
* After successful write, clears the metrics to start fresh tracking.
|
||||
*
|
||||
* Metrics include:
|
||||
* - Queue size and utilization
|
||||
* - Message counts (received, queued, processed, failed, dropped)
|
||||
* - Processing time statistics (average, p95, max)
|
||||
* - Rate limiting and backpressure status
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeLogEventQueueMetricsV2() {
|
||||
// Check if queue metrics are enabled
|
||||
if (!globals.config.get('Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.enable')) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get metrics from queue manager
|
||||
const queueManager = globals.udpQueueManagerLogEvents;
|
||||
if (!queueManager) {
|
||||
globals.logger.warn('LOG EVENT QUEUE METRICS V2: Queue manager not initialized');
|
||||
return;
|
||||
}
|
||||
|
||||
const metrics = await queueManager.getMetrics();
|
||||
|
||||
// Get configuration
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
);
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.tags'
|
||||
);
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
|
||||
const point = new Point(measurementName)
|
||||
.tag('queue_type', 'log_events')
|
||||
.tag('host', globals.hostInfo.hostname)
|
||||
.intField('queue_size', metrics.queueSize)
|
||||
.intField('queue_max_size', metrics.queueMaxSize)
|
||||
.floatField('queue_utilization_pct', metrics.queueUtilizationPct)
|
||||
.intField('queue_pending', metrics.queuePending)
|
||||
.intField('messages_received', metrics.messagesReceived)
|
||||
.intField('messages_queued', metrics.messagesQueued)
|
||||
.intField('messages_processed', metrics.messagesProcessed)
|
||||
.intField('messages_failed', metrics.messagesFailed)
|
||||
.intField('messages_dropped_total', metrics.messagesDroppedTotal)
|
||||
.intField('messages_dropped_rate_limit', metrics.messagesDroppedRateLimit)
|
||||
.intField('messages_dropped_queue_full', metrics.messagesDroppedQueueFull)
|
||||
.intField('messages_dropped_size', metrics.messagesDroppedSize)
|
||||
.floatField('processing_time_avg_ms', metrics.processingTimeAvgMs)
|
||||
.floatField('processing_time_p95_ms', metrics.processingTimeP95Ms)
|
||||
.floatField('processing_time_max_ms', metrics.processingTimeMaxMs)
|
||||
.intField('rate_limit_current', metrics.rateLimitCurrent)
|
||||
.intField('backpressure_active', metrics.backpressureActive);
|
||||
|
||||
// Add static tags from config file
|
||||
applyInfluxTags(point, configTags);
|
||||
|
||||
// Write to InfluxDB with retry logic
|
||||
await writeBatchToInfluxV2(
|
||||
[point],
|
||||
org,
|
||||
bucketName,
|
||||
'Log event queue metrics',
|
||||
'log-events-queue',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose('LOG EVENT QUEUE METRICS V2: Sent queue metrics data to InfluxDB');
|
||||
|
||||
// Clear metrics after successful write
|
||||
await queueManager.clearMetrics();
|
||||
}
|
||||
92
src/lib/influxdb/v2/sessions.js
Normal file
92
src/lib/influxdb/v2/sessions.js
Normal file
@@ -0,0 +1,92 @@
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeToInfluxWithRetry } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Store proxy session data to InfluxDB v2
|
||||
*
|
||||
* @description
|
||||
* Stores user session data from Qlik Sense proxy to InfluxDB v2. The function writes
|
||||
* pre-formatted session data points that have already been converted to InfluxDB Point objects.
|
||||
*
|
||||
* The userSessions.datapointInfluxdb array typically contains three types of measurements:
|
||||
* - user_session_summary: Summary with session count and user list
|
||||
* - user_session_list: List of users (for compatibility)
|
||||
* - user_session_details: Individual session details for each active session
|
||||
*
|
||||
* @param {object} userSessions - User session data object
|
||||
* @param {string} userSessions.serverName - Name of the Qlik Sense server
|
||||
* @param {string} userSessions.host - Hostname of the Qlik Sense server
|
||||
* @param {string} userSessions.virtualProxy - Virtual proxy name
|
||||
* @param {number} userSessions.sessionCount - Total number of active sessions
|
||||
* @param {string} userSessions.uniqueUserList - Comma-separated list of unique users
|
||||
* @param {Array<Point>} userSessions.datapointInfluxdb - Array of InfluxDB Point objects to write.
|
||||
* Each Point object in the array is already formatted and ready to write.
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeSessionsV2(userSessions) {
|
||||
globals.logger.debug(`PROXY SESSIONS V2: User sessions: ${JSON.stringify(userSessions)}`);
|
||||
|
||||
// Only write to InfluxDB if enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Validate input - ensure datapointInfluxdb is an array
|
||||
if (!Array.isArray(userSessions.datapointInfluxdb)) {
|
||||
globals.logger.warn(
|
||||
`PROXY SESSIONS V2: Invalid data format for host ${userSessions.host} - datapointInfluxdb must be an array`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Find writeApi for the server specified by serverName
|
||||
const writeApi = globals.influxWriteApi.find(
|
||||
(element) => element.serverName === userSessions.serverName
|
||||
);
|
||||
|
||||
if (!writeApi) {
|
||||
globals.logger.warn(
|
||||
`PROXY SESSIONS V2: Influxdb write API object not found for host ${userSessions.host}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
globals.logger.silly(
|
||||
`PROXY SESSIONS V2: Influxdb datapoint for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}": ${JSON.stringify(
|
||||
userSessions.datapointInfluxdb,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
|
||||
// Write array of measurements using retry logic
|
||||
await writeToInfluxWithRetry(
|
||||
async () => {
|
||||
const writeApi = globals.influx.getWriteApi(org, bucketName, 'ns', {
|
||||
flushInterval: 5000,
|
||||
maxRetries: 0,
|
||||
});
|
||||
try {
|
||||
await writeApi.writePoints(userSessions.datapointInfluxdb);
|
||||
await writeApi.close();
|
||||
} catch (err) {
|
||||
try {
|
||||
await writeApi.close();
|
||||
} catch (closeErr) {
|
||||
// Ignore close errors
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
`Proxy sessions for ${userSessions.host}/${userSessions.virtualProxy}`,
|
||||
'v2',
|
||||
userSessions.serverName
|
||||
);
|
||||
|
||||
globals.logger.verbose(
|
||||
`PROXY SESSIONS V2: Sent user session data to InfluxDB for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}"`
|
||||
);
|
||||
}
|
||||
107
src/lib/influxdb/v2/user-events.js
Normal file
107
src/lib/influxdb/v2/user-events.js
Normal file
@@ -0,0 +1,107 @@
|
||||
import { Point } from '@influxdata/influxdb-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeToInfluxWithRetry } from '../shared/utils.js';
|
||||
import { applyInfluxTags } from './utils.js';
|
||||
|
||||
/**
|
||||
* Store user event to InfluxDB v2
|
||||
*
|
||||
* @description
|
||||
* Stores user interaction events from Qlik Sense to InfluxDB v2 for tracking user activity,
|
||||
* including app interactions, user agent information, and custom tags.
|
||||
*
|
||||
* @param {object} msg - User event message containing event details
|
||||
* @param {string} msg.host - Hostname of the Qlik Sense server
|
||||
* @param {string} msg.command - Event action/command (e.g., OpenApp, CreateApp, etc.)
|
||||
* @param {string} msg.user_directory - User directory
|
||||
* @param {string} msg.user_id - User ID
|
||||
* @param {string} msg.origin - Origin of the event (e.g., Qlik Sense, QlikView, etc.)
|
||||
* @param {string} [msg.appId] - Application ID (if applicable)
|
||||
* @param {string} [msg.appName] - Application name (if applicable)
|
||||
* @param {object} [msg.ua] - User agent information object
|
||||
* @param {object} [msg.ua.browser] - Browser information
|
||||
* @param {string} [msg.ua.browser.name] - Browser name
|
||||
* @param {string} [msg.ua.browser.major] - Browser major version
|
||||
* @param {object} [msg.ua.os] - Operating system information
|
||||
* @param {string} [msg.ua.os.name] - OS name
|
||||
* @param {string} [msg.ua.os.version] - OS version
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function storeUserEventV2(msg) {
|
||||
globals.logger.debug(`USER EVENT V2: ${JSON.stringify(msg)}`);
|
||||
|
||||
// Only write to InfluxDB if enabled
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Validate required fields
|
||||
if (!msg.host || !msg.command || !msg.user_directory || !msg.user_id || !msg.origin) {
|
||||
globals.logger.warn(
|
||||
`USER EVENT V2: Missing required fields in user event message: ${JSON.stringify(msg)}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const org = globals.config.get('Butler-SOS.influxdbConfig.v2Config.org');
|
||||
const bucketName = globals.config.get('Butler-SOS.influxdbConfig.v2Config.bucket');
|
||||
|
||||
// Create point using v2 Point class
|
||||
const point = new Point('user_events')
|
||||
.tag('host', msg.host)
|
||||
.tag('event_action', msg.command)
|
||||
.tag('userFull', `${msg.user_directory}\\${msg.user_id}`)
|
||||
.tag('userDirectory', msg.user_directory)
|
||||
.tag('userId', msg.user_id)
|
||||
.tag('origin', msg.origin)
|
||||
.stringField('userFull', `${msg.user_directory}\\${msg.user_id}`)
|
||||
.stringField('userId', msg.user_id);
|
||||
|
||||
// Add app id and name to tags and fields if available
|
||||
if (msg?.appId) {
|
||||
point.tag('appId', msg.appId);
|
||||
point.stringField('appId_field', msg.appId);
|
||||
}
|
||||
if (msg?.appName) {
|
||||
point.tag('appName', msg.appName);
|
||||
point.stringField('appName_field', msg.appName);
|
||||
}
|
||||
|
||||
// Add user agent info to tags if available
|
||||
if (msg?.ua?.browser?.name) point.tag('uaBrowserName', msg?.ua?.browser?.name);
|
||||
if (msg?.ua?.browser?.major) point.tag('uaBrowserMajorVersion', msg?.ua?.browser?.major);
|
||||
if (msg?.ua?.os?.name) point.tag('uaOsName', msg?.ua?.os?.name);
|
||||
if (msg?.ua?.os?.version) point.tag('uaOsVersion', msg?.ua?.os?.version);
|
||||
|
||||
// Add custom tags from config file
|
||||
const configTags = globals.config.get('Butler-SOS.userEvents.tags');
|
||||
applyInfluxTags(point, configTags);
|
||||
|
||||
globals.logger.silly(`USER EVENT V2: Influxdb datapoint: ${JSON.stringify(point, null, 2)}`);
|
||||
|
||||
// Write to InfluxDB with retry logic
|
||||
await writeToInfluxWithRetry(
|
||||
async () => {
|
||||
const writeApi = globals.influx.getWriteApi(org, bucketName, 'ns', {
|
||||
flushInterval: 5000,
|
||||
maxRetries: 0,
|
||||
});
|
||||
try {
|
||||
await writeApi.writePoint(point);
|
||||
await writeApi.close();
|
||||
} catch (err) {
|
||||
try {
|
||||
await writeApi.close();
|
||||
} catch (closeErr) {
|
||||
// Ignore close errors
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
`User event for ${msg.host}`,
|
||||
'v2',
|
||||
msg.host
|
||||
);
|
||||
|
||||
globals.logger.verbose('USER EVENT V2: Sent user event data to InfluxDB');
|
||||
}
|
||||
22
src/lib/influxdb/v2/utils.js
Normal file
22
src/lib/influxdb/v2/utils.js
Normal file
@@ -0,0 +1,22 @@
|
||||
import { Point } from '@influxdata/influxdb-client';
|
||||
|
||||
/**
|
||||
* Applies tags from config to an InfluxDB Point object.
|
||||
*
|
||||
* @param {Point} point - The InfluxDB Point object
|
||||
* @param {Array<{name: string, value: string}>} tags - Array of tag objects
|
||||
* @returns {Point} The Point object with tags applied (for chaining)
|
||||
*/
|
||||
export function applyInfluxTags(point, tags) {
|
||||
if (!tags || !Array.isArray(tags) || tags.length === 0) {
|
||||
return point;
|
||||
}
|
||||
|
||||
for (const tag of tags) {
|
||||
if (tag.name && tag.value !== undefined && tag.value !== null) {
|
||||
point.tag(tag.name, String(tag.value));
|
||||
}
|
||||
}
|
||||
|
||||
return point;
|
||||
}
|
||||
67
src/lib/influxdb/v3/butler-memory.js
Normal file
67
src/lib/influxdb/v3/butler-memory.js
Normal file
@@ -0,0 +1,67 @@
|
||||
import { Point as Point3 } from '@influxdata/influxdb3-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV3 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts Butler SOS memory usage metrics to InfluxDB v3.
|
||||
*
|
||||
* This function captures memory usage metrics from the Butler SOS process itself
|
||||
* and stores them in InfluxDB v3.
|
||||
*
|
||||
* @param {object} memory - Memory usage data object
|
||||
* @param {string} memory.instanceTag - Instance identifier tag
|
||||
* @param {number} memory.heapUsedMByte - Heap used in MB
|
||||
* @param {number} memory.heapTotalMByte - Total heap size in MB
|
||||
* @param {number} memory.externalMemoryMByte - External memory usage in MB
|
||||
* @param {number} memory.processMemoryMByte - Process memory usage in MB
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postButlerSOSMemoryUsageToInfluxdbV3(memory) {
|
||||
// Validate input
|
||||
if (!memory || typeof memory !== 'object') {
|
||||
globals.logger.warn(
|
||||
'MEMORY USAGE V3: Invalid memory data provided. Data will not be sent to InfluxDB'
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
globals.logger.debug(`MEMORY USAGE V3: Memory usage ${JSON.stringify(memory, null, 2)})`);
|
||||
|
||||
// Get Butler version
|
||||
const butlerVersion = globals.appVersion;
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
// Create point for v3
|
||||
const point = new Point3('butlersos_memory_usage')
|
||||
.setTag('butler_sos_instance', memory.instanceTag)
|
||||
.setTag('version', butlerVersion)
|
||||
.setFloatField('heap_used', memory.heapUsedMByte)
|
||||
.setFloatField('heap_total', memory.heapTotalMByte)
|
||||
.setFloatField('external', memory.externalMemoryMByte)
|
||||
.setFloatField('process_memory', memory.processMemoryMByte);
|
||||
|
||||
try {
|
||||
// Convert point to line protocol and write directly with retry logic
|
||||
await writeBatchToInfluxV3(
|
||||
[point],
|
||||
database,
|
||||
'Memory usage metrics',
|
||||
'butler-memory',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
globals.logger.debug(`MEMORY USAGE V3: Wrote data to InfluxDB v3`);
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', '');
|
||||
globals.logger.error(
|
||||
`MEMORY USAGE V3: Error saving memory usage data to InfluxDB v3! ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
|
||||
globals.logger.verbose('MEMORY USAGE V3: Sent Butler SOS memory usage data to InfluxDB');
|
||||
}
|
||||
265
src/lib/influxdb/v3/event-counts.js
Normal file
265
src/lib/influxdb/v3/event-counts.js
Normal file
@@ -0,0 +1,265 @@
|
||||
import { Point as Point3 } from '@influxdata/influxdb3-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV3 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Store event count in InfluxDB v3
|
||||
*
|
||||
* @description
|
||||
* This function reads arrays of log and user events from the `udpEvents` object,
|
||||
* and stores the data in InfluxDB v3. The data is written to a measurement named after
|
||||
* the `Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName` config setting.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeEventCountInfluxDBV3() {
|
||||
// Get array of log events
|
||||
const logEvents = await globals.udpEvents.getLogEvents();
|
||||
const userEvents = await globals.udpEvents.getUserEvents();
|
||||
|
||||
// Debug
|
||||
globals.logger.debug(
|
||||
`EVENT COUNT INFLUXDB V3: Log events: ${JSON.stringify(logEvents, null, 2)}`
|
||||
);
|
||||
globals.logger.debug(
|
||||
`EVENT COUNT INFLUXDB V3: User events: ${JSON.stringify(userEvents, null, 2)}`
|
||||
);
|
||||
|
||||
// Are there any events to store?
|
||||
if (logEvents.length === 0 && userEvents.length === 0) {
|
||||
globals.logger.verbose('EVENT COUNT INFLUXDB V3: No events to store in InfluxDB');
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
try {
|
||||
const points = [];
|
||||
|
||||
// Store data for each log event
|
||||
for (const logEvent of logEvents) {
|
||||
const tags = {
|
||||
butler_sos_instance: globals.options.instanceTag,
|
||||
event_type: 'log',
|
||||
source: logEvent.source,
|
||||
host: logEvent.host,
|
||||
subsystem: logEvent.subsystem,
|
||||
};
|
||||
|
||||
// Add static tags defined in config file, if any
|
||||
if (
|
||||
globals.config.has('Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags') &&
|
||||
Array.isArray(
|
||||
globals.config.get('Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags')
|
||||
)
|
||||
) {
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags'
|
||||
);
|
||||
|
||||
configTags.forEach((tag) => {
|
||||
tags[tag.name] = tag.value;
|
||||
});
|
||||
}
|
||||
|
||||
const point = new Point3(
|
||||
globals.config.get('Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName')
|
||||
)
|
||||
.setTag('event_type', 'log')
|
||||
.setTag('source', logEvent.source)
|
||||
.setTag('host', logEvent.host)
|
||||
.setTag('subsystem', logEvent.subsystem)
|
||||
.setIntegerField('counter', logEvent.counter);
|
||||
|
||||
// Add additional tags to point
|
||||
Object.keys(tags).forEach((key) => {
|
||||
point.setTag(key, tags[key]);
|
||||
});
|
||||
|
||||
points.push(point);
|
||||
}
|
||||
|
||||
// Loop through data in user events and create datapoints
|
||||
for (const event of userEvents) {
|
||||
const tags = {
|
||||
butler_sos_instance: globals.options.instanceTag,
|
||||
event_type: 'user',
|
||||
source: event.source,
|
||||
host: event.host,
|
||||
subsystem: event.subsystem,
|
||||
};
|
||||
|
||||
// Add static tags defined in config file, if any
|
||||
if (
|
||||
globals.config.has('Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags') &&
|
||||
Array.isArray(
|
||||
globals.config.get('Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags')
|
||||
)
|
||||
) {
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.qlikSenseEvents.eventCount.influxdb.tags'
|
||||
);
|
||||
|
||||
configTags.forEach((tag) => {
|
||||
tags[tag.name] = tag.value;
|
||||
});
|
||||
}
|
||||
|
||||
const point = new Point3(
|
||||
globals.config.get('Butler-SOS.qlikSenseEvents.eventCount.influxdb.measurementName')
|
||||
)
|
||||
.setTag('event_type', 'user')
|
||||
.setTag('source', event.source)
|
||||
.setTag('host', event.host)
|
||||
.setTag('subsystem', event.subsystem)
|
||||
.setIntegerField('counter', event.counter);
|
||||
|
||||
// Add additional tags to point
|
||||
Object.keys(tags).forEach((key) => {
|
||||
point.setTag(key, tags[key]);
|
||||
});
|
||||
|
||||
points.push(point);
|
||||
}
|
||||
|
||||
await writeBatchToInfluxV3(
|
||||
points,
|
||||
database,
|
||||
'Event counts',
|
||||
'event-counts',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.debug(`EVENT COUNT INFLUXDB V3: Wrote event data to InfluxDB v3`);
|
||||
|
||||
globals.logger.verbose(
|
||||
'EVENT COUNT INFLUXDB V3: Sent Butler SOS event count data to InfluxDB'
|
||||
);
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', '');
|
||||
globals.logger.error(
|
||||
`EVENT COUNT INFLUXDB V3: Error writing data to InfluxDB: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Store rejected event count in InfluxDB v3
|
||||
*
|
||||
* @description
|
||||
* This function reads an array of rejected log events from the `rejectedEvents` object,
|
||||
* and stores the data in InfluxDB v3. The data is written to a measurement named after
|
||||
* the `Butler-SOS.qlikSenseEvents.rejectedEventCount.influxdb.measurementName` config setting.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function storeRejectedEventCountInfluxDBV3() {
|
||||
// Get array of rejected log events
|
||||
const rejectedLogEvents = await globals.rejectedEvents.getRejectedLogEvents();
|
||||
|
||||
// Debug
|
||||
globals.logger.debug(
|
||||
`REJECTED EVENT COUNT INFLUXDB V3: Rejected log events: ${JSON.stringify(
|
||||
rejectedLogEvents,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
|
||||
// Are there any events to store?
|
||||
if (rejectedLogEvents.length === 0) {
|
||||
globals.logger.verbose('REJECTED EVENT COUNT INFLUXDB V3: No events to store in InfluxDB');
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
try {
|
||||
const points = [];
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.qlikSenseEvents.rejectedEventCount.influxdb.measurementName'
|
||||
);
|
||||
|
||||
rejectedLogEvents.forEach((event) => {
|
||||
globals.logger.debug(`REJECTED LOG EVENT INFLUXDB V3: ${JSON.stringify(event)}`);
|
||||
|
||||
if (event.source === 'qseow-qix-perf') {
|
||||
let point = new Point3(measurementName)
|
||||
.setTag('source', event.source)
|
||||
.setTag('object_type', event.objectType)
|
||||
.setTag('method', event.method)
|
||||
.setIntegerField('counter', event.counter)
|
||||
.setFloatField('process_time', event.processTime);
|
||||
|
||||
// Add app_id and app_name if available
|
||||
if (event?.appId) {
|
||||
point.setTag('app_id', event.appId);
|
||||
}
|
||||
if (event?.appName?.length > 0) {
|
||||
point.setTag('app_name', event.appName);
|
||||
point.setTag('app_name_set', 'true');
|
||||
} else {
|
||||
point.setTag('app_name_set', 'false');
|
||||
}
|
||||
|
||||
// Add static tags defined in config file, if any
|
||||
if (
|
||||
globals.config.has(
|
||||
'Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.tags'
|
||||
) &&
|
||||
Array.isArray(
|
||||
globals.config.get(
|
||||
'Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.tags'
|
||||
)
|
||||
)
|
||||
) {
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.tags'
|
||||
);
|
||||
for (const item of configTags) {
|
||||
point.setTag(item.name, item.value);
|
||||
}
|
||||
}
|
||||
|
||||
points.push(point);
|
||||
} else {
|
||||
let point = new Point3(measurementName)
|
||||
.setTag('source', event.source)
|
||||
.setIntegerField('counter', event.counter);
|
||||
|
||||
points.push(point);
|
||||
}
|
||||
});
|
||||
|
||||
// Write to InfluxDB
|
||||
await writeBatchToInfluxV3(
|
||||
points,
|
||||
database,
|
||||
'Rejected event counts',
|
||||
'rejected-event-counts',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
globals.logger.debug(`REJECT LOG EVENT INFLUXDB V3: Wrote data to InfluxDB v3`);
|
||||
|
||||
globals.logger.verbose(
|
||||
'REJECT LOG EVENT INFLUXDB V3: Sent Butler SOS rejected event count data to InfluxDB'
|
||||
);
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', '');
|
||||
globals.logger.error(
|
||||
`REJECTED LOG EVENT INFLUXDB V3: Error writing data to InfluxDB: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
261
src/lib/influxdb/v3/health-metrics.js
Normal file
261
src/lib/influxdb/v3/health-metrics.js
Normal file
@@ -0,0 +1,261 @@
|
||||
import { Point as Point3 } from '@influxdata/influxdb3-client';
|
||||
import globals from '../../../globals.js';
|
||||
import {
|
||||
getFormattedTime,
|
||||
processAppDocuments,
|
||||
isInfluxDbEnabled,
|
||||
applyTagsToPoint3,
|
||||
writeBatchToInfluxV3,
|
||||
validateUnsignedField,
|
||||
} from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts health metrics data from Qlik Sense to InfluxDB v3.
|
||||
*
|
||||
* This function processes health data from the Sense engine's healthcheck API and
|
||||
* formats it for storage in InfluxDB v3. It handles various metrics including:
|
||||
* - CPU usage
|
||||
* - Memory usage
|
||||
* - Cache metrics
|
||||
* - Active/loaded/in-memory apps
|
||||
* - Session counts
|
||||
* - User counts
|
||||
*
|
||||
* @param {string} serverName - The name of the Qlik Sense server
|
||||
* @param {string} host - The hostname or IP of the Qlik Sense server
|
||||
* @param {object} body - The health metrics data from Sense engine healthcheck API
|
||||
* @param {object} serverTags - Tags to associate with the metrics
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postHealthMetricsToInfluxdbV3(serverName, host, body, serverTags) {
|
||||
// Validate input
|
||||
if (!body || typeof body !== 'object') {
|
||||
globals.logger.warn(
|
||||
`HEALTH METRICS V3: Invalid health data from server ${serverName}. Data will not be sent to InfluxDB`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Calculate server uptime
|
||||
const formattedTime = getFormattedTime(body.started);
|
||||
|
||||
// Build tags structure that will be passed to InfluxDB
|
||||
globals.logger.debug(
|
||||
`HEALTH METRICS TO INFLUXDB V3: Health data: Tags sent to InfluxDB: ${JSON.stringify(
|
||||
serverTags
|
||||
)}`
|
||||
);
|
||||
|
||||
globals.logger.debug(
|
||||
`HEALTH METRICS TO INFLUXDB V3: Number of apps active: ${body.apps.active_docs.length}`
|
||||
);
|
||||
globals.logger.debug(
|
||||
`HEALTH METRICS TO INFLUXDB V3: Number of apps loaded: ${body.apps.loaded_docs.length}`
|
||||
);
|
||||
globals.logger.debug(
|
||||
`HEALTH METRICS TO INFLUXDB V3: Number of apps in memory: ${body.apps.in_memory_docs.length}`
|
||||
);
|
||||
|
||||
// Get active app names
|
||||
const { appNames: appNamesActive, sessionAppNames: sessionAppNamesActive } =
|
||||
await processAppDocuments(body.apps.active_docs, 'HEALTH METRICS TO INFLUXDB V3', 'active');
|
||||
|
||||
// Get loaded app names
|
||||
const { appNames: appNamesLoaded, sessionAppNames: sessionAppNamesLoaded } =
|
||||
await processAppDocuments(body.apps.loaded_docs, 'HEALTH METRICS TO INFLUXDB V3', 'loaded');
|
||||
|
||||
// Get in memory app names
|
||||
const { appNames: appNamesInMemory, sessionAppNames: sessionAppNamesInMemory } =
|
||||
await processAppDocuments(
|
||||
body.apps.in_memory_docs,
|
||||
'HEALTH METRICS TO INFLUXDB V3',
|
||||
'in memory'
|
||||
);
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influxWriteApi object has been initialized
|
||||
if (!globals.influxWriteApi) {
|
||||
globals.logger.warn(
|
||||
'HEALTH METRICS V3: Influxdb write API object not initialized. Data will not be sent to InfluxDB'
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Find writeApi for the server specified by serverName
|
||||
const writeApi = globals.influxWriteApi.find((element) => element.serverName === serverName);
|
||||
|
||||
// Ensure that the writeApi object was found
|
||||
if (!writeApi) {
|
||||
globals.logger.warn(
|
||||
`HEALTH METRICS V3: Influxdb write API object not found for host ${host}. Data will not be sent to InfluxDB`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Get database from config
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
// Create a new point with the data to be written to InfluxDB v3
|
||||
const points = [
|
||||
new Point3('sense_server')
|
||||
.setStringField('version', body.version)
|
||||
.setStringField('started', body.started)
|
||||
.setStringField('uptime', formattedTime),
|
||||
|
||||
new Point3('mem')
|
||||
.setFloatField('comitted', body.mem.committed)
|
||||
.setFloatField('allocated', body.mem.allocated)
|
||||
.setFloatField('free', body.mem.free),
|
||||
|
||||
new Point3('apps')
|
||||
.setIntegerField('active_docs_count', body.apps.active_docs.length)
|
||||
.setIntegerField('loaded_docs_count', body.apps.loaded_docs.length)
|
||||
.setIntegerField('in_memory_docs_count', body.apps.in_memory_docs.length)
|
||||
.setStringField(
|
||||
'active_docs',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
|
||||
? body.apps.active_docs
|
||||
: ''
|
||||
)
|
||||
.setStringField(
|
||||
'active_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
|
||||
? appNamesActive.toString()
|
||||
: ''
|
||||
)
|
||||
.setStringField(
|
||||
'active_session_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.activeDocs')
|
||||
? sessionAppNamesActive.toString()
|
||||
: ''
|
||||
)
|
||||
.setStringField(
|
||||
'loaded_docs',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
|
||||
? body.apps.loaded_docs
|
||||
: ''
|
||||
)
|
||||
.setStringField(
|
||||
'loaded_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
|
||||
? appNamesLoaded.toString()
|
||||
: ''
|
||||
)
|
||||
.setStringField(
|
||||
'loaded_session_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.loadedDocs')
|
||||
? sessionAppNamesLoaded.toString()
|
||||
: ''
|
||||
)
|
||||
.setStringField(
|
||||
'in_memory_docs',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
|
||||
? body.apps.in_memory_docs
|
||||
: ''
|
||||
)
|
||||
.setStringField(
|
||||
'in_memory_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
|
||||
? appNamesInMemory.toString()
|
||||
: ''
|
||||
)
|
||||
.setStringField(
|
||||
'in_memory_session_docs_names',
|
||||
globals.config.get('Butler-SOS.appNames.enableAppNameExtract') &&
|
||||
globals.config.get('Butler-SOS.influxdbConfig.includeFields.inMemoryDocs')
|
||||
? sessionAppNamesInMemory.toString()
|
||||
: ''
|
||||
)
|
||||
.setIntegerField(
|
||||
'calls',
|
||||
validateUnsignedField(body.apps.calls, 'apps', 'calls', serverName)
|
||||
)
|
||||
.setIntegerField(
|
||||
'selections',
|
||||
validateUnsignedField(body.apps.selections, 'apps', 'selections', serverName)
|
||||
),
|
||||
|
||||
new Point3('cpu').setIntegerField(
|
||||
'total',
|
||||
validateUnsignedField(body.cpu.total, 'cpu', 'total', serverName)
|
||||
),
|
||||
|
||||
new Point3('session')
|
||||
.setIntegerField(
|
||||
'active',
|
||||
validateUnsignedField(body.session.active, 'session', 'active', serverName)
|
||||
)
|
||||
.setIntegerField(
|
||||
'total',
|
||||
validateUnsignedField(body.session.total, 'session', 'total', serverName)
|
||||
),
|
||||
|
||||
new Point3('users')
|
||||
.setIntegerField(
|
||||
'active',
|
||||
validateUnsignedField(body.users.active, 'users', 'active', serverName)
|
||||
)
|
||||
.setIntegerField(
|
||||
'total',
|
||||
validateUnsignedField(body.users.total, 'users', 'total', serverName)
|
||||
),
|
||||
|
||||
new Point3('cache')
|
||||
.setIntegerField(
|
||||
'hits',
|
||||
validateUnsignedField(body.cache.hits, 'cache', 'hits', serverName)
|
||||
)
|
||||
.setIntegerField(
|
||||
'lookups',
|
||||
validateUnsignedField(body.cache.lookups, 'cache', 'lookups', serverName)
|
||||
)
|
||||
.setIntegerField(
|
||||
'added',
|
||||
validateUnsignedField(body.cache.added, 'cache', 'added', serverName)
|
||||
)
|
||||
.setIntegerField(
|
||||
'replaced',
|
||||
validateUnsignedField(body.cache.replaced, 'cache', 'replaced', serverName)
|
||||
)
|
||||
.setIntegerField(
|
||||
'bytes_added',
|
||||
validateUnsignedField(body.cache.bytes_added, 'cache', 'bytes_added', serverName)
|
||||
),
|
||||
|
||||
new Point3('saturated').setBooleanField('saturated', body.saturated),
|
||||
];
|
||||
|
||||
// Write to InfluxDB
|
||||
try {
|
||||
for (const point of points) {
|
||||
// Apply server tags to each point
|
||||
applyTagsToPoint3(point, serverTags);
|
||||
}
|
||||
|
||||
await writeBatchToInfluxV3(
|
||||
points,
|
||||
database,
|
||||
`Health metrics for ${host}`,
|
||||
'health-metrics',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.debug(`HEALTH METRICS V3: Wrote data to InfluxDB v3`);
|
||||
} catch (err) {
|
||||
// Track error count
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', serverName);
|
||||
|
||||
globals.logger.error(
|
||||
`HEALTH METRICS V3: Error saving health data to InfluxDB v3! ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
318
src/lib/influxdb/v3/log-events copy.js
Normal file
318
src/lib/influxdb/v3/log-events copy.js
Normal file
@@ -0,0 +1,318 @@
|
||||
import { Point as Point3 } from '@influxdata/influxdb3-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeToInfluxWithRetry } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Clean tag values for InfluxDB v3 line protocol
|
||||
* Remove only characters that are explicitly not supported by the line protocol spec.
|
||||
* According to the spec, newlines are not supported in tag or field values.
|
||||
*
|
||||
* The Point3 class should handle required escaping for tag values:
|
||||
* - Comma (,) → \,
|
||||
* - Equals (=) → \=
|
||||
* - Space ( ) → \
|
||||
*
|
||||
* @param {string} value - The tag value to clean
|
||||
* @returns {string} The cleaned tag value
|
||||
*/
|
||||
function cleanTagValue(value) {
|
||||
if (!value || typeof value !== 'string') {
|
||||
return value;
|
||||
}
|
||||
return value.replace(/[\n\r]/g, ''); // Remove only newlines and carriage returns
|
||||
}
|
||||
|
||||
/**
|
||||
* Post log event to InfluxDB v3
|
||||
*
|
||||
* @description
|
||||
* Handles log events from 5 different Qlik Sense sources:
|
||||
* - qseow-engine: Engine log events
|
||||
* - qseow-proxy: Proxy log events
|
||||
* - qseow-scheduler: Scheduler log events
|
||||
* - qseow-repository: Repository log events
|
||||
* - qseow-qix-perf: QIX performance metrics
|
||||
*
|
||||
* Each source has specific fields and tags that are written to InfluxDB.
|
||||
*
|
||||
* @param {object} msg - The log event message
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function postLogEventToInfluxdbV3(msg) {
|
||||
globals.logger.debug(`LOG EVENT INFLUXDB V3: ${msg})`);
|
||||
|
||||
try {
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Verify the message source is valid
|
||||
if (
|
||||
msg.source !== 'qseow-engine' &&
|
||||
msg.source !== 'qseow-proxy' &&
|
||||
msg.source !== 'qseow-scheduler' &&
|
||||
msg.source !== 'qseow-repository' &&
|
||||
msg.source !== 'qseow-qix-perf'
|
||||
) {
|
||||
globals.logger.warn(
|
||||
`LOG EVENT INFLUXDB V3: Unknown log event source: ${msg.source}. Skipping.`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
let point;
|
||||
|
||||
// Handle each message type with its specific fields
|
||||
if (msg.source === 'qseow-engine') {
|
||||
// Engine fields: message, exception_message, command, result_code_field, origin, context, session_id, raw_event
|
||||
// NOTE: result_code uses _field suffix to avoid conflict with result_code tag
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('level', msg.level)
|
||||
.setTag('source', msg.source)
|
||||
.setTag('log_row', msg.log_row)
|
||||
.setTag('subsystem', msg.subsystem || 'n/a')
|
||||
.setStringField('message', msg.message)
|
||||
.setStringField('exception_message', msg.exception_message || '')
|
||||
.setStringField('command', msg.command || '')
|
||||
.setStringField('result_code_field', msg.result_code || '')
|
||||
.setStringField('origin', msg.origin || '')
|
||||
.setStringField('context', msg.context || '')
|
||||
.setStringField('session_id', msg.session_id || '')
|
||||
.setStringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.result_code?.length > 0)
|
||||
point.setTag('result_code', cleanTagValue(msg.result_code));
|
||||
if (msg?.windows_user?.length > 0)
|
||||
point.setTag('windows_user', cleanTagValue(msg.windows_user));
|
||||
if (msg?.task_id?.length > 0) point.setTag('task_id', cleanTagValue(msg.task_id));
|
||||
if (msg?.task_name?.length > 0) point.setTag('task_name', cleanTagValue(msg.task_name));
|
||||
if (msg?.app_id?.length > 0) point.setTag('app_id', cleanTagValue(msg.app_id));
|
||||
if (msg?.app_name?.length > 0) point.setTag('app_name', cleanTagValue(msg.app_name));
|
||||
if (msg?.engine_exe_version?.length > 0)
|
||||
point.setTag('engine_exe_version', cleanTagValue(msg.engine_exe_version));
|
||||
} else if (msg.source === 'qseow-proxy') {
|
||||
// Proxy fields: message, exception_message, command, result_code_field, origin, context, raw_event
|
||||
// NOTE: result_code uses _field suffix to avoid conflict with result_code tag
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('level', msg.level)
|
||||
.setTag('source', msg.source)
|
||||
.setTag('log_row', msg.log_row)
|
||||
.setTag('subsystem', msg.subsystem || 'n/a')
|
||||
.setStringField('message', msg.message)
|
||||
.setStringField('exception_message', msg.exception_message || '')
|
||||
.setStringField('command', msg.command || '')
|
||||
.setStringField('result_code_field', msg.result_code || '')
|
||||
.setStringField('origin', msg.origin || '')
|
||||
.setStringField('context', msg.context || '')
|
||||
.setStringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.result_code?.length > 0)
|
||||
point.setTag('result_code', cleanTagValue(msg.result_code));
|
||||
} else if (msg.source === 'qseow-scheduler') {
|
||||
// Scheduler fields: message, exception_message, app_name_field, app_id_field, execution_id, raw_event
|
||||
// NOTE: app_name and app_id use _field suffix to avoid conflict with conditional tags
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('level', msg.level)
|
||||
.setTag('source', msg.source)
|
||||
.setTag('log_row', msg.log_row)
|
||||
.setTag('subsystem', msg.subsystem || 'n/a')
|
||||
.setStringField('message', msg.message)
|
||||
.setStringField('exception_message', msg.exception_message || '')
|
||||
.setStringField('app_name_field', msg.app_name || '')
|
||||
.setStringField('app_id_field', msg.app_id || '')
|
||||
.setStringField('execution_id', msg.execution_id || '')
|
||||
.setStringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.task_id?.length > 0) point.setTag('task_id', cleanTagValue(msg.task_id));
|
||||
if (msg?.task_name?.length > 0) point.setTag('task_name', cleanTagValue(msg.task_name));
|
||||
} else if (msg.source === 'qseow-repository') {
|
||||
// Repository fields: message, exception_message, command, result_code_field, origin, context, raw_event
|
||||
// NOTE: result_code uses _field suffix to avoid conflict with result_code tag
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('level', msg.level)
|
||||
.setTag('source', msg.source)
|
||||
.setTag('log_row', msg.log_row)
|
||||
.setTag('subsystem', msg.subsystem || 'n/a')
|
||||
.setStringField('message', msg.message)
|
||||
.setStringField('exception_message', msg.exception_message || '')
|
||||
.setStringField('command', msg.command || '')
|
||||
.setStringField('result_code_field', msg.result_code || '')
|
||||
.setStringField('origin', msg.origin || '')
|
||||
.setStringField('context', msg.context || '')
|
||||
.setStringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.result_code?.length > 0)
|
||||
point.setTag('result_code', cleanTagValue(msg.result_code));
|
||||
} else if (msg.source === 'qseow-qix-perf') {
|
||||
// QIX Performance fields: app_id, process_time, work_time, lock_time, validate_time, traverse_time, handle, net_ram, peak_ram, raw_event
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', cleanTagValue(msg.host || '<Unknown>'))
|
||||
.setTag('level', cleanTagValue(msg.level || '<Unknown>'))
|
||||
.setTag('source', cleanTagValue(msg.source || '<Unknown>'))
|
||||
.setTag('log_row', msg.log_row || '-1')
|
||||
.setTag('subsystem', cleanTagValue(msg.subsystem || '<Unknown>'))
|
||||
.setTag('method', cleanTagValue(msg.method || '<Unknown>'))
|
||||
.setTag('object_type', cleanTagValue(msg.object_type || '<Unknown>'))
|
||||
.setTag('proxy_session_id', msg.proxy_session_id || '-1')
|
||||
.setTag('session_id', msg.session_id || '-1')
|
||||
.setTag(
|
||||
'event_activity_source',
|
||||
cleanTagValue(msg.event_activity_source || '<Unknown>')
|
||||
)
|
||||
.setStringField('app_id_field', msg.app_id || '');
|
||||
|
||||
// Add numeric fields with validation to prevent NaN
|
||||
const processTime = parseFloat(msg.process_time);
|
||||
if (!isNaN(processTime)) {
|
||||
point.setFloatField('process_time', processTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid process_time value: ${msg.process_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const workTime = parseFloat(msg.work_time);
|
||||
if (!isNaN(workTime)) {
|
||||
point.setFloatField('work_time', workTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid work_time value: ${msg.work_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const lockTime = parseFloat(msg.lock_time);
|
||||
if (!isNaN(lockTime)) {
|
||||
point.setFloatField('lock_time', lockTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid lock_time value: ${msg.lock_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const validateTime = parseFloat(msg.validate_time);
|
||||
if (!isNaN(validateTime)) {
|
||||
point.setFloatField('validate_time', validateTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid validate_time value: ${msg.validate_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const traverseTime = parseFloat(msg.traverse_time);
|
||||
if (!isNaN(traverseTime)) {
|
||||
point.setFloatField('traverse_time', traverseTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid traverse_time value: ${msg.traverse_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const handle = parseInt(msg.handle, 10);
|
||||
if (!isNaN(handle)) {
|
||||
point.setIntegerField('handle', handle);
|
||||
} else {
|
||||
globals.logger.debug(`LOG EVENT INFLUXDB V3: Invalid handle value: ${msg.handle}`);
|
||||
}
|
||||
|
||||
const netRam = parseInt(msg.net_ram, 10);
|
||||
if (!isNaN(netRam)) {
|
||||
point.setIntegerField('net_ram', netRam);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid net_ram value: ${msg.net_ram}`
|
||||
);
|
||||
}
|
||||
|
||||
const peakRam = parseInt(msg.peak_ram, 10);
|
||||
if (!isNaN(peakRam)) {
|
||||
point.setIntegerField('peak_ram', peakRam);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid peak_ram value: ${msg.peak_ram}`
|
||||
);
|
||||
}
|
||||
|
||||
// Remove newlines from raw event (not supported in line protocol field values)
|
||||
const cleanedRawEvent = JSON.stringify(msg).replace(/[\n\r]/g, '');
|
||||
point.setStringField('raw_event', cleanedRawEvent);
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.app_id?.length > 0) point.setTag('app_id', cleanTagValue(msg.app_id));
|
||||
if (msg?.app_name?.length > 0) point.setTag('app_name', cleanTagValue(msg.app_name));
|
||||
if (msg?.object_id?.length > 0) point.setTag('object_id', cleanTagValue(msg.object_id));
|
||||
}
|
||||
|
||||
// Add log event categories to tags if available
|
||||
// The msg.category array contains objects with properties 'name' and 'value'
|
||||
if (msg?.category?.length > 0) {
|
||||
msg.category.forEach((category) => {
|
||||
point.setTag(category.name, cleanTagValue(category.value));
|
||||
});
|
||||
}
|
||||
|
||||
// Add custom tags from config file
|
||||
if (
|
||||
globals.config.has('Butler-SOS.logEvents.tags') &&
|
||||
globals.config.get('Butler-SOS.logEvents.tags') !== null &&
|
||||
globals.config.get('Butler-SOS.logEvents.tags').length > 0
|
||||
) {
|
||||
const configTags = globals.config.get('Butler-SOS.logEvents.tags');
|
||||
for (const item of configTags) {
|
||||
point.setTag(item.name, cleanTagValue(item.value));
|
||||
}
|
||||
}
|
||||
|
||||
// Debug logging to troubleshoot line protocol issues
|
||||
console.log('LOG EVENT V3 MESSAGE:', JSON.stringify(msg, null, 2));
|
||||
console.log('LOG EVENT V3 LINE PROTOCOL:', point.toLineProtocol());
|
||||
|
||||
await writeToInfluxWithRetry(
|
||||
async () => await globals.influx.write(point.toLineProtocol(), database),
|
||||
`Log event for ${msg.host}`,
|
||||
'v3',
|
||||
msg.host
|
||||
);
|
||||
|
||||
globals.logger.debug(`LOG EVENT INFLUXDB V3: Wrote data to InfluxDB v3`);
|
||||
|
||||
globals.logger.verbose('LOG EVENT INFLUXDB V3: Sent Butler SOS log event data to InfluxDB');
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', msg.host);
|
||||
globals.logger.error(
|
||||
`LOG EVENT INFLUXDB V3: Error saving log event to InfluxDB! ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
315
src/lib/influxdb/v3/log-events.js
Normal file
315
src/lib/influxdb/v3/log-events.js
Normal file
@@ -0,0 +1,315 @@
|
||||
import { Point as Point3 } from '@influxdata/influxdb3-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV3 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Clean tag values for InfluxDB v3 line protocol
|
||||
* Remove characters not supported by line protocol.
|
||||
*
|
||||
* According to the line protocol spec:
|
||||
* - Newlines (\n) and carriage returns (\r) are NOT supported → remove them
|
||||
* - Comma, equals, space are escaped automatically by Point3
|
||||
*
|
||||
* @param {string} value - The tag value to clean
|
||||
* @returns {string} The cleaned tag value
|
||||
*/
|
||||
function cleanTagValue(value) {
|
||||
if (!value || typeof value !== 'string') {
|
||||
return value;
|
||||
}
|
||||
return value.replace(/[\n\r]/g, ''); // Remove newlines and carriage returns (not supported)
|
||||
}
|
||||
|
||||
/**
|
||||
* Post log event to InfluxDB v3
|
||||
*
|
||||
* @description
|
||||
* Handles log events from 5 different Qlik Sense sources:
|
||||
* - qseow-engine: Engine log events
|
||||
* - qseow-proxy: Proxy log events
|
||||
* - qseow-scheduler: Scheduler log events
|
||||
* - qseow-repository: Repository log events
|
||||
* - qseow-qix-perf: QIX performance metrics
|
||||
*
|
||||
* Each source has specific fields and tags that are written to InfluxDB.
|
||||
*
|
||||
* @param {object} msg - The log event message
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function postLogEventToInfluxdbV3(msg) {
|
||||
globals.logger.debug(`LOG EVENT INFLUXDB V3: ${msg})`);
|
||||
|
||||
try {
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Verify the message source is valid
|
||||
if (
|
||||
msg.source !== 'qseow-engine' &&
|
||||
msg.source !== 'qseow-proxy' &&
|
||||
msg.source !== 'qseow-scheduler' &&
|
||||
msg.source !== 'qseow-repository' &&
|
||||
msg.source !== 'qseow-qix-perf'
|
||||
) {
|
||||
globals.logger.warn(
|
||||
`LOG EVENT INFLUXDB V3: Unknown log event source: ${msg.source}. Skipping.`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
let point;
|
||||
|
||||
// Handle each message type with its specific fields
|
||||
if (msg.source === 'qseow-engine') {
|
||||
// Engine fields: message, exception_message, command, result_code_field, origin, context, session_id, raw_event
|
||||
// NOTE: result_code uses _field suffix to avoid conflict with result_code tag
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('level', msg.level)
|
||||
.setTag('source', msg.source)
|
||||
.setTag('log_row', msg.log_row)
|
||||
.setTag('subsystem', msg.subsystem || 'n/a')
|
||||
.setStringField('message', msg.message)
|
||||
.setStringField('exception_message', msg.exception_message || '')
|
||||
.setStringField('command', msg.command || '')
|
||||
.setStringField('result_code_field', msg.result_code || '')
|
||||
.setStringField('origin', msg.origin || '')
|
||||
.setStringField('context', msg.context || '')
|
||||
.setStringField('session_id', msg.session_id || '')
|
||||
.setStringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.result_code?.length > 0)
|
||||
point.setTag('result_code', cleanTagValue(msg.result_code));
|
||||
if (msg?.windows_user?.length > 0)
|
||||
point.setTag('windows_user', cleanTagValue(msg.windows_user));
|
||||
if (msg?.task_id?.length > 0) point.setTag('task_id', cleanTagValue(msg.task_id));
|
||||
if (msg?.task_name?.length > 0) point.setTag('task_name', cleanTagValue(msg.task_name));
|
||||
if (msg?.app_id?.length > 0) point.setTag('app_id', cleanTagValue(msg.app_id));
|
||||
if (msg?.app_name?.length > 0) point.setTag('app_name', cleanTagValue(msg.app_name));
|
||||
if (msg?.engine_exe_version?.length > 0)
|
||||
point.setTag('engine_exe_version', cleanTagValue(msg.engine_exe_version));
|
||||
} else if (msg.source === 'qseow-proxy') {
|
||||
// Proxy fields: message, exception_message, command, result_code_field, origin, context, raw_event
|
||||
// NOTE: result_code uses _field suffix to avoid conflict with result_code tag
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('level', msg.level)
|
||||
.setTag('source', msg.source)
|
||||
.setTag('log_row', msg.log_row)
|
||||
.setTag('subsystem', msg.subsystem || 'n/a')
|
||||
.setStringField('message', msg.message)
|
||||
.setStringField('exception_message', msg.exception_message || '')
|
||||
.setStringField('command', msg.command || '')
|
||||
.setStringField('result_code_field', msg.result_code || '')
|
||||
.setStringField('origin', msg.origin || '')
|
||||
.setStringField('context', msg.context || '')
|
||||
.setStringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.result_code?.length > 0)
|
||||
point.setTag('result_code', cleanTagValue(msg.result_code));
|
||||
} else if (msg.source === 'qseow-scheduler') {
|
||||
// Scheduler fields: message, exception_message, app_name_field, app_id_field, execution_id, raw_event
|
||||
// NOTE: app_name and app_id use _field suffix to avoid conflict with conditional tags
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('level', msg.level)
|
||||
.setTag('source', msg.source)
|
||||
.setTag('log_row', msg.log_row)
|
||||
.setTag('subsystem', msg.subsystem || 'n/a')
|
||||
.setStringField('message', msg.message)
|
||||
.setStringField('exception_message', msg.exception_message || '')
|
||||
.setStringField('app_name_field', msg.app_name || '')
|
||||
.setStringField('app_id_field', msg.app_id || '')
|
||||
.setStringField('execution_id', msg.execution_id || '')
|
||||
.setStringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.task_id?.length > 0) point.setTag('task_id', cleanTagValue(msg.task_id));
|
||||
if (msg?.task_name?.length > 0) point.setTag('task_name', cleanTagValue(msg.task_name));
|
||||
} else if (msg.source === 'qseow-repository') {
|
||||
// Repository fields: message, exception_message, command, result_code_field, origin, context, raw_event
|
||||
// NOTE: result_code uses _field suffix to avoid conflict with result_code tag
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('level', msg.level)
|
||||
.setTag('source', msg.source)
|
||||
.setTag('log_row', msg.log_row)
|
||||
.setTag('subsystem', msg.subsystem || 'n/a')
|
||||
.setStringField('message', msg.message)
|
||||
.setStringField('exception_message', msg.exception_message || '')
|
||||
.setStringField('command', msg.command || '')
|
||||
.setStringField('result_code_field', msg.result_code || '')
|
||||
.setStringField('origin', msg.origin || '')
|
||||
.setStringField('context', msg.context || '')
|
||||
.setStringField('raw_event', JSON.stringify(msg));
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
if (msg?.result_code?.length > 0)
|
||||
point.setTag('result_code', cleanTagValue(msg.result_code));
|
||||
} else if (msg.source === 'qseow-qix-perf') {
|
||||
// QIX Performance fields: app_id, process_time, work_time, lock_time, validate_time, traverse_time, handle, net_ram, peak_ram, raw_event
|
||||
point = new Point3('log_event')
|
||||
.setTag('host', cleanTagValue(msg.host || '<Unknown>'))
|
||||
.setTag('level', cleanTagValue(msg.level || '<Unknown>'))
|
||||
.setTag('source', cleanTagValue(msg.source || '<Unknown>'))
|
||||
.setTag('log_row', msg.log_row || '-1')
|
||||
.setTag('subsystem', cleanTagValue(msg.subsystem || '<Unknown>'))
|
||||
.setTag('method', cleanTagValue(msg.method || '<Unknown>'))
|
||||
.setTag('object_type', cleanTagValue(msg.object_type || '<Unknown>'))
|
||||
.setTag('proxy_session_id', msg.proxy_session_id || '-1')
|
||||
.setTag('session_id', msg.session_id || '-1')
|
||||
.setTag(
|
||||
'event_activity_source',
|
||||
cleanTagValue(msg.event_activity_source || '<Unknown>')
|
||||
)
|
||||
.setStringField('app_id_field', msg.app_id || '');
|
||||
|
||||
// Add numeric fields with validation to prevent NaN
|
||||
const processTime = parseFloat(msg.process_time);
|
||||
if (!isNaN(processTime)) {
|
||||
point.setFloatField('process_time', processTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid process_time value: ${msg.process_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const workTime = parseFloat(msg.work_time);
|
||||
if (!isNaN(workTime)) {
|
||||
point.setFloatField('work_time', workTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid work_time value: ${msg.work_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const lockTime = parseFloat(msg.lock_time);
|
||||
if (!isNaN(lockTime)) {
|
||||
point.setFloatField('lock_time', lockTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid lock_time value: ${msg.lock_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const validateTime = parseFloat(msg.validate_time);
|
||||
if (!isNaN(validateTime)) {
|
||||
point.setFloatField('validate_time', validateTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid validate_time value: ${msg.validate_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const traverseTime = parseFloat(msg.traverse_time);
|
||||
if (!isNaN(traverseTime)) {
|
||||
point.setFloatField('traverse_time', traverseTime);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid traverse_time value: ${msg.traverse_time}`
|
||||
);
|
||||
}
|
||||
|
||||
const handle = parseInt(msg.handle, 10);
|
||||
if (!isNaN(handle)) {
|
||||
point.setIntegerField('handle', handle);
|
||||
} else {
|
||||
globals.logger.debug(`LOG EVENT INFLUXDB V3: Invalid handle value: ${msg.handle}`);
|
||||
}
|
||||
|
||||
const netRam = parseInt(msg.net_ram, 10);
|
||||
if (!isNaN(netRam)) {
|
||||
point.setIntegerField('net_ram', netRam);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid net_ram value: ${msg.net_ram}`
|
||||
);
|
||||
}
|
||||
|
||||
const peakRam = parseInt(msg.peak_ram, 10);
|
||||
if (!isNaN(peakRam)) {
|
||||
point.setIntegerField('peak_ram', peakRam);
|
||||
} else {
|
||||
globals.logger.debug(
|
||||
`LOG EVENT INFLUXDB V3: Invalid peak_ram value: ${msg.peak_ram}`
|
||||
);
|
||||
}
|
||||
|
||||
// Remove newlines from raw event (not supported in line protocol field values)
|
||||
const cleanedRawEvent = JSON.stringify(msg).replace(/[\n\r]/g, '');
|
||||
point.setStringField('raw_event', cleanedRawEvent);
|
||||
|
||||
// Conditional tags
|
||||
if (msg?.user_full?.length > 0) point.setTag('user_full', cleanTagValue(msg.user_full));
|
||||
if (msg?.user_directory?.length > 0)
|
||||
point.setTag('user_directory', cleanTagValue(msg.user_directory));
|
||||
if (msg?.user_id?.length > 0) point.setTag('user_id', cleanTagValue(msg.user_id));
|
||||
|
||||
if (msg?.app_id?.length > 0) point.setTag('app_id', cleanTagValue(msg.app_id));
|
||||
if (msg?.app_name?.length > 0) point.setTag('app_name', cleanTagValue(msg.app_name));
|
||||
|
||||
if (msg?.object_id?.length > 0) point.setTag('object_id', cleanTagValue(msg.object_id));
|
||||
}
|
||||
|
||||
// Add log event categories to tags if available
|
||||
// The msg.category array contains objects with properties 'name' and 'value'
|
||||
if (msg?.category?.length > 0) {
|
||||
msg.category.forEach((category) => {
|
||||
point.setTag(category.name, cleanTagValue(category.value));
|
||||
});
|
||||
}
|
||||
|
||||
// Add custom tags from config file
|
||||
if (
|
||||
globals.config.has('Butler-SOS.logEvents.tags') &&
|
||||
globals.config.get('Butler-SOS.logEvents.tags') !== null &&
|
||||
globals.config.get('Butler-SOS.logEvents.tags').length > 0
|
||||
) {
|
||||
const configTags = globals.config.get('Butler-SOS.logEvents.tags');
|
||||
for (const item of configTags) {
|
||||
point.setTag(item.name, cleanTagValue(item.value));
|
||||
}
|
||||
}
|
||||
|
||||
await writeBatchToInfluxV3(
|
||||
[point],
|
||||
database,
|
||||
`Log event for ${msg.host}`,
|
||||
'log-events',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.debug(`LOG EVENT INFLUXDB V3: Wrote data to InfluxDB v3`);
|
||||
|
||||
globals.logger.verbose('LOG EVENT INFLUXDB V3: Sent Butler SOS log event data to InfluxDB');
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', msg.host);
|
||||
globals.logger.error(
|
||||
`LOG EVENT INFLUXDB V3: Error saving log event to InfluxDB! ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
195
src/lib/influxdb/v3/queue-metrics.js
Normal file
195
src/lib/influxdb/v3/queue-metrics.js
Normal file
@@ -0,0 +1,195 @@
|
||||
import { Point as Point3 } from '@influxdata/influxdb3-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV3 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Store user event queue metrics to InfluxDB v3
|
||||
*
|
||||
* @description
|
||||
* Retrieves metrics from the user event queue manager and stores them in InfluxDB v3
|
||||
* for monitoring queue health, backpressure, dropped messages, and processing performance.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function postUserEventQueueMetricsToInfluxdbV3() {
|
||||
try {
|
||||
// Check if queue metrics are enabled
|
||||
if (
|
||||
!globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.enable'
|
||||
)
|
||||
) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get metrics from queue manager
|
||||
const queueManager = globals.udpQueueManagerUserActivity;
|
||||
if (!queueManager) {
|
||||
globals.logger.warn(
|
||||
'USER EVENT QUEUE METRICS INFLUXDB V3: Queue manager not initialized'
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const metrics = await queueManager.getMetrics();
|
||||
|
||||
// Get configuration
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
);
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.userEvents.udpServerConfig.queueMetrics.influxdb.tags'
|
||||
);
|
||||
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
const point = new Point3(measurementName)
|
||||
.setTag('queue_type', 'user_events')
|
||||
.setTag('host', globals.hostInfo.hostname)
|
||||
.setIntegerField('queue_size', metrics.queueSize)
|
||||
.setIntegerField('queue_max_size', metrics.queueMaxSize)
|
||||
.setFloatField('queue_utilization_pct', metrics.queueUtilizationPct)
|
||||
.setIntegerField('queue_pending', metrics.queuePending)
|
||||
.setIntegerField('messages_received', metrics.messagesReceived)
|
||||
.setIntegerField('messages_queued', metrics.messagesQueued)
|
||||
.setIntegerField('messages_processed', metrics.messagesProcessed)
|
||||
.setIntegerField('messages_failed', metrics.messagesFailed)
|
||||
.setIntegerField('messages_dropped_total', metrics.messagesDroppedTotal)
|
||||
.setIntegerField('messages_dropped_rate_limit', metrics.messagesDroppedRateLimit)
|
||||
.setIntegerField('messages_dropped_queue_full', metrics.messagesDroppedQueueFull)
|
||||
.setIntegerField('messages_dropped_size', metrics.messagesDroppedSize)
|
||||
.setFloatField('processing_time_avg_ms', metrics.processingTimeAvgMs)
|
||||
.setFloatField('processing_time_p95_ms', metrics.processingTimeP95Ms)
|
||||
.setFloatField('processing_time_max_ms', metrics.processingTimeMaxMs)
|
||||
.setIntegerField('rate_limit_current', metrics.rateLimitCurrent)
|
||||
.setIntegerField('backpressure_active', metrics.backpressureActive);
|
||||
|
||||
// Add static tags from config file
|
||||
if (configTags && configTags.length > 0) {
|
||||
for (const item of configTags) {
|
||||
point.setTag(item.name, item.value);
|
||||
}
|
||||
}
|
||||
|
||||
await writeBatchToInfluxV3(
|
||||
[point],
|
||||
database,
|
||||
'User event queue metrics',
|
||||
'user-events-queue',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose(
|
||||
'USER EVENT QUEUE METRICS INFLUXDB V3: Sent queue metrics data to InfluxDB v3'
|
||||
);
|
||||
|
||||
// Clear metrics after writing
|
||||
await queueManager.clearMetrics();
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', '');
|
||||
globals.logger.error(
|
||||
`USER EVENT QUEUE METRICS INFLUXDB V3: Error posting queue metrics: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Store log event queue metrics to InfluxDB v3
|
||||
*
|
||||
* @description
|
||||
* Retrieves metrics from the log event queue manager and stores them in InfluxDB v3
|
||||
* for monitoring queue health, backpressure, dropped messages, and processing performance.
|
||||
*
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
* @throws {Error} Error if unable to write data to InfluxDB
|
||||
*/
|
||||
export async function postLogEventQueueMetricsToInfluxdbV3() {
|
||||
try {
|
||||
// Check if queue metrics are enabled
|
||||
if (
|
||||
!globals.config.get('Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.enable')
|
||||
) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get metrics from queue manager
|
||||
const queueManager = globals.udpQueueManagerLogEvents;
|
||||
if (!queueManager) {
|
||||
globals.logger.warn(
|
||||
'LOG EVENT QUEUE METRICS INFLUXDB V3: Queue manager not initialized'
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const metrics = await queueManager.getMetrics();
|
||||
|
||||
// Get configuration
|
||||
const measurementName = globals.config.get(
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.measurementName'
|
||||
);
|
||||
const configTags = globals.config.get(
|
||||
'Butler-SOS.logEvents.udpServerConfig.queueMetrics.influxdb.tags'
|
||||
);
|
||||
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
const point = new Point3(measurementName)
|
||||
.setTag('queue_type', 'log_events')
|
||||
.setTag('host', globals.hostInfo.hostname)
|
||||
.setIntegerField('queue_size', metrics.queueSize)
|
||||
.setIntegerField('queue_max_size', metrics.queueMaxSize)
|
||||
.setFloatField('queue_utilization_pct', metrics.queueUtilizationPct)
|
||||
.setIntegerField('queue_pending', metrics.queuePending)
|
||||
.setIntegerField('messages_received', metrics.messagesReceived)
|
||||
.setIntegerField('messages_queued', metrics.messagesQueued)
|
||||
.setIntegerField('messages_processed', metrics.messagesProcessed)
|
||||
.setIntegerField('messages_failed', metrics.messagesFailed)
|
||||
.setIntegerField('messages_dropped_total', metrics.messagesDroppedTotal)
|
||||
.setIntegerField('messages_dropped_rate_limit', metrics.messagesDroppedRateLimit)
|
||||
.setIntegerField('messages_dropped_queue_full', metrics.messagesDroppedQueueFull)
|
||||
.setIntegerField('messages_dropped_size', metrics.messagesDroppedSize)
|
||||
.setFloatField('processing_time_avg_ms', metrics.processingTimeAvgMs)
|
||||
.setFloatField('processing_time_p95_ms', metrics.processingTimeP95Ms)
|
||||
.setFloatField('processing_time_max_ms', metrics.processingTimeMaxMs)
|
||||
.setIntegerField('rate_limit_current', metrics.rateLimitCurrent)
|
||||
.setIntegerField('backpressure_active', metrics.backpressureActive);
|
||||
|
||||
// Add static tags from config file
|
||||
if (configTags && configTags.length > 0) {
|
||||
for (const item of configTags) {
|
||||
point.setTag(item.name, item.value);
|
||||
}
|
||||
}
|
||||
|
||||
await writeBatchToInfluxV3(
|
||||
[point],
|
||||
database,
|
||||
'Log event queue metrics',
|
||||
'log-events-queue',
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.verbose(
|
||||
'LOG EVENT QUEUE METRICS INFLUXDB V3: Sent queue metrics data to InfluxDB v3'
|
||||
);
|
||||
|
||||
// Clear metrics after writing
|
||||
await queueManager.clearMetrics();
|
||||
} catch (err) {
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', '');
|
||||
globals.logger.error(
|
||||
`LOG EVENT QUEUE METRICS INFLUXDB V3: Error posting queue metrics: ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
75
src/lib/influxdb/v3/sessions.js
Normal file
75
src/lib/influxdb/v3/sessions.js
Normal file
@@ -0,0 +1,75 @@
|
||||
import { Point as Point3 } from '@influxdata/influxdb3-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV3 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Posts proxy sessions data to InfluxDB v3.
|
||||
*
|
||||
* This function takes user session data from Qlik Sense proxy and formats it for storage
|
||||
* in InfluxDB v3. It creates three measurements:
|
||||
* - user_session_summary: Summary with count and user list
|
||||
* - user_session_list: List of users (for compatibility)
|
||||
* - user_session_details: Individual session details for each active session
|
||||
*
|
||||
* @param {object} userSessions - User session data containing information about active sessions
|
||||
* @param {string} userSessions.host - The hostname of the server
|
||||
* @param {string} userSessions.virtualProxy - The virtual proxy name
|
||||
* @param {string} userSessions.serverName - Server name
|
||||
* @param {number} userSessions.sessionCount - Number of sessions
|
||||
* @param {string} userSessions.uniqueUserList - Comma-separated list of unique users
|
||||
* @param {Array} userSessions.datapointInfluxdb - Array of datapoints including individual sessions
|
||||
* @returns {Promise<void>} Promise that resolves when data has been posted to InfluxDB
|
||||
*/
|
||||
export async function postProxySessionsToInfluxdbV3(userSessions) {
|
||||
globals.logger.debug(`PROXY SESSIONS V3: User sessions: ${JSON.stringify(userSessions)}`);
|
||||
|
||||
globals.logger.silly(
|
||||
`PROXY SESSIONS V3: Data for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}"`
|
||||
);
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Get database from config
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
// Write all datapoints to InfluxDB
|
||||
// The datapointInfluxdb array contains summary points and individual session details
|
||||
try {
|
||||
if (userSessions.datapointInfluxdb && userSessions.datapointInfluxdb.length > 0) {
|
||||
await writeBatchToInfluxV3(
|
||||
userSessions.datapointInfluxdb,
|
||||
database,
|
||||
`Proxy sessions for ${userSessions.host}/${userSessions.virtualProxy}`,
|
||||
userSessions.host,
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
|
||||
globals.logger.debug(
|
||||
`PROXY SESSIONS V3: Wrote ${userSessions.datapointInfluxdb.length} datapoints to InfluxDB v3`
|
||||
);
|
||||
} else {
|
||||
globals.logger.warn('PROXY SESSIONS V3: No datapoints to write to InfluxDB v3');
|
||||
}
|
||||
} catch (err) {
|
||||
// Track error count
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', userSessions.serverName);
|
||||
|
||||
globals.logger.error(
|
||||
`PROXY SESSIONS V3: Error saving user session data to InfluxDB v3! ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
}
|
||||
|
||||
globals.logger.debug(
|
||||
`PROXY SESSIONS V3: Session count for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}": ${userSessions.sessionCount}`
|
||||
);
|
||||
globals.logger.debug(
|
||||
`PROXY SESSIONS V3: User list for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}": ${userSessions.uniqueUserList}`
|
||||
);
|
||||
|
||||
globals.logger.verbose(
|
||||
`PROXY SESSIONS V3: Sent user session data to InfluxDB for server "${userSessions.host}", virtual proxy "${userSessions.virtualProxy}"`
|
||||
);
|
||||
}
|
||||
128
src/lib/influxdb/v3/user-events.js
Normal file
128
src/lib/influxdb/v3/user-events.js
Normal file
@@ -0,0 +1,128 @@
|
||||
import { Point as Point3 } from '@influxdata/influxdb3-client';
|
||||
import globals from '../../../globals.js';
|
||||
import { isInfluxDbEnabled, writeBatchToInfluxV3 } from '../shared/utils.js';
|
||||
|
||||
/**
|
||||
* Sanitize tag values for InfluxDB line protocol.
|
||||
* Remove or replace characters that cause parsing issues.
|
||||
*
|
||||
* @param {string} value - The value to sanitize
|
||||
* @returns {string} - The sanitized value
|
||||
*/
|
||||
function sanitizeTagValue(value) {
|
||||
if (!value) return value;
|
||||
return String(value)
|
||||
.replace(/[<>\\]/g, '')
|
||||
.replace(/\s+/g, '-');
|
||||
}
|
||||
|
||||
/**
|
||||
* Posts a user event to InfluxDB v3.
|
||||
*
|
||||
* @param {object} msg - The event to be posted to InfluxDB. The object should contain the following properties:
|
||||
* - host: The hostname of the Qlik Sense server that the user event originated from.
|
||||
* - command: The command (e.g. OpenApp, CreateApp, etc.) that the user event corresponds to.
|
||||
* - user_directory: The user directory of the user who triggered the event.
|
||||
* - user_id: The user ID of the user who triggered the event.
|
||||
* - origin: The origin of the event (e.g. Qlik Sense, QlikView, etc.).
|
||||
* - appId: The ID of the app that the event corresponds to (if applicable).
|
||||
* - appName: The name of the app that the event corresponds to (if applicable).
|
||||
* - ua: An object containing user agent information (if available).
|
||||
* @returns {Promise<void>} - A promise that resolves when the event has been posted to InfluxDB.
|
||||
*/
|
||||
export async function postUserEventToInfluxdbV3(msg) {
|
||||
globals.logger.debug(`USER EVENT INFLUXDB V3: ${msg})`);
|
||||
|
||||
// Only write to InfluxDB if the global influx object has been initialized
|
||||
if (!isInfluxDbEnabled()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const database = globals.config.get('Butler-SOS.influxdbConfig.v3Config.database');
|
||||
|
||||
// Validate required fields
|
||||
if (!msg.host || !msg.command || !msg.user_directory || !msg.user_id || !msg.origin) {
|
||||
globals.logger.warn(
|
||||
`USER EVENT INFLUXDB V3: Missing required fields in user event message: ${JSON.stringify(msg)}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Create a new point with the data to be written to InfluxDB v3
|
||||
// NOTE: InfluxDB v3 does not allow the same name for both tags and fields,
|
||||
// unlike v1/v2. Fields use different names with _field suffix where needed.
|
||||
const point = new Point3('user_events')
|
||||
.setTag('host', msg.host)
|
||||
.setTag('event_action', msg.command)
|
||||
.setTag('userFull', `${msg.user_directory}\\${msg.user_id}`)
|
||||
.setTag('userDirectory', msg.user_directory)
|
||||
.setTag('userId', msg.user_id)
|
||||
.setTag('origin', msg.origin)
|
||||
.setStringField('userFull_field', `${msg.user_directory}\\${msg.user_id}`)
|
||||
.setStringField('userId_field', msg.user_id);
|
||||
|
||||
// Add app id and name to tags and fields if available
|
||||
if (msg?.appId) {
|
||||
point.setTag('appId', msg.appId);
|
||||
point.setStringField('appId_field', msg.appId);
|
||||
}
|
||||
if (msg?.appName) {
|
||||
point.setTag('appName', msg.appName);
|
||||
point.setStringField('appName_field', msg.appName);
|
||||
}
|
||||
|
||||
// Add user agent info to tags if available
|
||||
if (msg?.ua?.browser?.name) point.setTag('uaBrowserName', msg?.ua?.browser?.name);
|
||||
if (msg?.ua?.browser?.major) point.setTag('uaBrowserMajorVersion', msg?.ua?.browser?.major);
|
||||
if (msg?.ua?.os?.name) point.setTag('uaOsName', msg?.ua?.os?.name);
|
||||
if (msg?.ua?.os?.version) point.setTag('uaOsVersion', msg?.ua?.os?.version);
|
||||
|
||||
// Add custom tags from config file to payload
|
||||
if (
|
||||
globals.config.has('Butler-SOS.userEvents.tags') &&
|
||||
globals.config.get('Butler-SOS.userEvents.tags') !== null &&
|
||||
globals.config.get('Butler-SOS.userEvents.tags').length > 0
|
||||
) {
|
||||
const configTags = globals.config.get('Butler-SOS.userEvents.tags');
|
||||
for (const item of configTags) {
|
||||
point.setTag(item.name, item.value);
|
||||
}
|
||||
}
|
||||
|
||||
globals.logger.silly(
|
||||
`USER EVENT INFLUXDB V3: Influxdb datapoint for Butler SOS user event: ${JSON.stringify(
|
||||
point,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
|
||||
// Write to InfluxDB
|
||||
try {
|
||||
// Convert point to line protocol and write directly with retry logic
|
||||
await writeBatchToInfluxV3(
|
||||
[point],
|
||||
database,
|
||||
`User event for ${msg.host}`,
|
||||
msg.host,
|
||||
globals.config.get('Butler-SOS.influxdbConfig.maxBatchSize')
|
||||
);
|
||||
globals.logger.debug(`USER EVENT INFLUXDB V3: Wrote data to InfluxDB v3`);
|
||||
} catch (err) {
|
||||
// Track error count
|
||||
await globals.errorTracker.incrementError('INFLUXDB_V3_WRITE', '');
|
||||
|
||||
globals.logger.error(
|
||||
`USER EVENT INFLUXDB V3: Error saving user event to InfluxDB v3! ${globals.getErrorMessage(err)}`
|
||||
);
|
||||
// Log the line protocol for debugging
|
||||
try {
|
||||
const lineProtocol = point.toLineProtocol();
|
||||
globals.logger.debug(`USER EVENT INFLUXDB V3: Failed line protocol: ${lineProtocol}`);
|
||||
} catch (e) {
|
||||
// Ignore errors in debug logging
|
||||
}
|
||||
}
|
||||
|
||||
globals.logger.verbose('USER EVENT INFLUXDB V3: Sent Butler SOS user event data to InfluxDB');
|
||||
}
|
||||
135
src/lib/log-error.js
Normal file
135
src/lib/log-error.js
Normal file
@@ -0,0 +1,135 @@
|
||||
/**
|
||||
* Enhanced error logging utility for Butler SOS
|
||||
*
|
||||
* Provides consistent error logging across the application with different
|
||||
* behavior for SEA (Single Executable Application) vs non-SEA environments.
|
||||
*
|
||||
* In SEA mode: Only the error message is logged (cleaner output for end users)
|
||||
* In non-SEA mode: Both error message and stack trace are logged as separate
|
||||
* entries (better debugging for developers)
|
||||
*/
|
||||
|
||||
import globals from '../globals.js';
|
||||
import sea from './sea-wrapper.js';
|
||||
|
||||
/**
|
||||
* Log an error with appropriate formatting based on execution environment
|
||||
*
|
||||
* This function wraps the global logger and provides enhanced error logging:
|
||||
* - In SEA apps: logs only the error message (cleaner for production)
|
||||
* - In non-SEA apps: logs error message and stack trace separately (better for debugging)
|
||||
*
|
||||
* The function accepts the same parameters as winston logger methods.
|
||||
*
|
||||
* @param {string} level - The log level ('error', 'warn', 'info', 'verbose', 'debug')
|
||||
* @param {string} message - The log message (prefix/context for the error)
|
||||
* @param {Error} error - The error object to log
|
||||
* @param {...unknown} args - Additional arguments to pass to the logger
|
||||
*
|
||||
* @example
|
||||
* // Basic error logging
|
||||
* try {
|
||||
* // some code
|
||||
* } catch (err) {
|
||||
* logError('HEALTH: Error when calling health check API', err);
|
||||
* }
|
||||
*
|
||||
* @example
|
||||
* // With contextual information
|
||||
* try {
|
||||
* // some code
|
||||
* } catch (err) {
|
||||
* logError(`PROXY SESSIONS: Error for server '${serverName}' (${host})`, err);
|
||||
* }
|
||||
*/
|
||||
function logErrorWithLevel(level, message, error, ...args) {
|
||||
// Check if running as SEA app
|
||||
const isSeaApp = globals.isSea !== undefined ? globals.isSea : sea.isSea();
|
||||
|
||||
if (!error) {
|
||||
// If no error object provided, just log the message normally
|
||||
globals.logger[level](message, ...args);
|
||||
return;
|
||||
}
|
||||
|
||||
// Get error message - prefer error.message, fallback to toString()
|
||||
const errorMessage = error.message || error.toString();
|
||||
|
||||
if (isSeaApp) {
|
||||
// SEA mode: Only log the error message (cleaner output)
|
||||
globals.logger[level](`${message}: ${errorMessage}`, ...args);
|
||||
} else {
|
||||
// Non-SEA mode: Log error message first, then stack trace separately
|
||||
// This provides better readability and debugging information
|
||||
|
||||
// Log 1: The error message with context
|
||||
globals.logger[level](`${message}: ${errorMessage}`, ...args);
|
||||
|
||||
// Log 2: The stack trace (if available)
|
||||
if (error.stack) {
|
||||
globals.logger[level](`Stack trace: ${error.stack}`, ...args);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience function for logging errors at 'error' level
|
||||
*
|
||||
* @param {string} message - The log message (prefix/context for the error)
|
||||
* @param {Error} error - The error object to log
|
||||
* @param {...unknown} args - Additional arguments to pass to the logger
|
||||
*
|
||||
* @example
|
||||
* try {
|
||||
* // some code
|
||||
* } catch (err) {
|
||||
* logError('HEALTH: Error when calling health check API', err);
|
||||
* }
|
||||
*/
|
||||
export function logError(message, error, ...args) {
|
||||
logErrorWithLevel('error', message, error, ...args);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience function for logging errors at 'warn' level
|
||||
*
|
||||
* @param {string} message - The log message (prefix/context for the error)
|
||||
* @param {Error} error - The error object to log
|
||||
* @param {...unknown} args - Additional arguments to pass to the logger
|
||||
*/
|
||||
export function logWarn(message, error, ...args) {
|
||||
logErrorWithLevel('warn', message, error, ...args);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience function for logging errors at 'info' level
|
||||
*
|
||||
* @param {string} message - The log message (prefix/context for the error)
|
||||
* @param {Error} error - The error object to log
|
||||
* @param {...unknown} args - Additional arguments to pass to the logger
|
||||
*/
|
||||
export function logInfo(message, error, ...args) {
|
||||
logErrorWithLevel('info', message, error, ...args);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience function for logging errors at 'verbose' level
|
||||
*
|
||||
* @param {string} message - The log message (prefix/context for the error)
|
||||
* @param {Error} error - The error object to log
|
||||
* @param {...unknown} args - Additional arguments to pass to the logger
|
||||
*/
|
||||
export function logVerbose(message, error, ...args) {
|
||||
logErrorWithLevel('verbose', message, error, ...args);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience function for logging errors at 'debug' level
|
||||
*
|
||||
* @param {string} message - The log message (prefix/context for the error)
|
||||
* @param {Error} error - The error object to log
|
||||
* @param {...unknown} args - Additional arguments to pass to the logger
|
||||
*/
|
||||
export function logDebug(message, error, ...args) {
|
||||
logErrorWithLevel('debug', message, error, ...args);
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
import globals from '../globals.js';
|
||||
import { logError } from './log-error.js';
|
||||
|
||||
/**
|
||||
* Categorizes log events based on configured rules.
|
||||
@@ -118,7 +119,7 @@ export function categoriseLogEvent(logLevel, logMessage) {
|
||||
// Return the log event category and the action taken
|
||||
return { category: uniqueCategories, actionTaken: 'categorised' };
|
||||
} catch (err) {
|
||||
globals.logger.error(`LOG EVENT CATEGORISATION: Error processing log event: ${err}`);
|
||||
logError('LOG EVENT CATEGORISATION: Error processing log event', err);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user