Sowmik Sarker

Integrating Prometheus & Grafana to monitor Spring Boot and Nodejs Application

3412

Application Part

JAVA APP: STEP BY STEP

Add prometheus dependency in application’s build.gradle

implementation 'io.micrometer:micrometer-registry-prometheus' Note: Never use specific version. Gradle will auto download spring boot’s compatible version.

Make sure you also have these dependencies in your build.gradle

implementation 'org.springframework.boot:spring-boot-starter-web' implementation 'org.springframework.boot:spring-boot-starter-actuator'

Include following dependencies in application properties file

management.endpoints.web.exposure.include=*
management.endpoints.web.base-path=/actuator
management.endpoint.health.show-details=always
management.endpoint.health.show-components=always
management.endpoint.metrics.enabled=true
management.endpoint.prometheus.enabled=true
management.server.port=9023 //customizable port: here in this port application will expose prometheus metrics

#added for percentiles
management.metrics.web.server.request.autotime.percentiles=0.5, 0.7, 0.8, 0.9, 0.95
management.metrics.web.client.request.autotime.percentiles=0.5, 0.7, 0.8, 0.9, 0.95
management.metrics.web.server.request.autotime.percentiles-histogram=true
management.metrics.web.client.request.autotime.percentiles-histogram=true

#for exposing outbound metrics
management.metrics.web.client.request.autotime.enabled=true
management.metrics.web.client.request.metric-name=outbound_http_client_request
management.metrics.web.server.request.autotime.enabled=true
management.metrics.web.server.request.metric-name=outbound_http_server_request

#added for logback metrics

Prometheus Remote Server Part

Modify prometheus remote server yml configuration like below:

scrape_configs:
- job_name: 'Provide a name here'
  metrics_path: '/actuator/prometheus' //this is the endpoint where we will get metrics
  scrape_interval: 3s
  static_configs:
  - targets: ['10.21.114.70:9023'] //in this ip our application is deployed and 9023 is published only for exposing prometheus metrics. So this remote prometheus will grab our application data from this url
    labels:
    application: 'My Spring Boot Application' //provide here app name
    basic_auth:  //this part is not mandatory but it' better to use
    username: user
    password: password

  //you can also configure here the port from where Grafana will consume prometheus metrics

Grafana Part

Go to grafana dashboard and go to setting->Data Source

Suppose your remote prometheus is serving here in this url: http://10.10.5.130:9090/

Then add prometheus url as data source then click save and test. It’s done!

Now, import and add different type of dashboard and vizualize your data.

@FYI: Search here for different types of dashboard and import for better vizualization: **https://grafana.com/grafana/dashboards/ **

How to integrate Prometheus & Grafana to monitor Node Express Application

Top header filter : Host (server-01, server-02, All)

Http Server (Load average for all process/port)

  • Request handling per seconds
  • Request counts
  • Success, Error Rate
  • Response bytes per seconds
  • Request bytes per seconds
  • Response time percentile
  • Request bytes percentile
  • Response bytes percentile

Http Server (individual process/port)

  • Request handling per seconds
  • Request counts
  • Success, Error Rate
  • Response bytes per seconds
  • Request bytes per seconds
  • Response time percentile
  • Request bytes percentile
  • Response bytes percentile

System Resources

  • Uptime
  • Process CPU Usage
  • Heap Usage, Non Heap Usage,
  • Process open files
  • Load Average
  • Network usage

Node js Misc

  • Memory Pool
  • Garbage collection
  • Eventloop lag
  • Active handler

DB Pool Usages

  • Min pool, Max pool
  • Total connections
  • Request enqued, dequed, failed, timeout, max queue length, queue time sum
  • db pool connection percentiles

integrate api-common package in package.json

Package.json

"api-common": "^2.1.3",

Config.js

const METRICS_PORT = parseInt(process.env.METRICS_PORT)
const METRICS_ENABLED = true
const METRICS_PATH = "/metrics/account"
const APPLICATION_NAME = process.env.name
const INFO_LOG_FILE_PATH = process.env.INFO_LOG_PATH
const ERROR_LOG_PATH = process.env.ERROR_LOG_PATH
const DEBUG_LOG_PATH = process.env.DEBUG_LOG_PATH
const ENABLE_RESP_LOG = process.env.ENABLE_RESP_LOG!==undefined ?
    process.env.ENABLE_RESP_LOG : false;

module.exports = {
    ENV, PORT, ROUTE, ROUTEv1, ROUTEv2, API_URL, SECURED, USEDIRCON, USEDBPOOL, DBNAME, CREATEOPERID,
    REG_FIELD, METRICS_PORT, METRICS_ENABLED, METRICS_PATH, APPLICATION_NAME, INFO_LOG_FILE_PATH,
    ERROR_LOG_PATH, DEBUG_LOG_PATH, ENABLE_RESP_LOG
}

dbSetup.js

const oracledb = require('oracledb');
oracledb.outFormat = oracledb.OBJECT;
oracledb.fetchAsString = [ oracledb.NUMBER ];
import {commonLoggerInstance} from "./index";
import Sequelize from 'sequelize-oracle';
import {getDBObject, dbConf} from '../globalFunctions/globalVariables';
import { DBNAME, METRICS_ENABLED } from './config';

const initDBPool =  async () => {
  let selectedDb = await getDBObject(DBNAME);
  oracledb.createPool({
     poolAlias: DBNAME
    ,user: selectedDb.user
    ,password: selectedDb.password
    ,connectString: '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST='+selectedDb.host+')(PORT='+selectedDb.port+'))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME='+selectedDb.dbname+')))'
    ,queueRequests: true
    ,poolMin: 1
    ,poolMax: 2
    ,poolIncrement: 1
    ,poolTimeout: 10
    ,queueTimeout: 5000
    ,poolPingInterval: 10
	  ,stmtCacheSize: 30
    , _enableStats: METRICS_ENABLED //Remember to enable oracle db status true
  });
}

module.exports = {initDBPool, oracledb}

modify app.js or index.js add below snippets:

import {oracledb} from './dbSetup'
import CommonLogger from 'api-common/CommonLogger';
import CommonFunctions from 'api-common/CommonFunctions';
import {
    SECURED
    METRICS_ENABLED,
    METRICS_PORT,
    DBNAME,
    METRICS_PATH,
    APPLICATION_NAME, INFO_LOG_FILE_PATH,
    ERROR_LOG_PATH, DEBUG_LOG_PATH,
    ENABLE_RESP_LOG
} from './config';
const app = express();
const metricsApp = express();

export const commonLoggerInstance = new CommonLogger(
    APPLICATION_NAME, INFO_LOG_FILE_PATH,
    ERROR_LOG_PATH, DEBUG_LOG_PATH
);
if (commonLoggerInstance && commonLoggerInstance.permission && commonLoggerInstance.permission==="denied" ||
    (commonLoggerInstance.initError && commonLoggerInstance.initError===true)) {
    throw new Error("Error initializing logger! Please make sure you have r/w permission to the provided log path");
}
// add the prometheus middleware to all routes
if (METRICS_ENABLED) {
    metricsApp.use(async (req, res, next) => {
        if (req.url !== METRICS_PATH) {
            res.status(403).send("Access denied");
        } else {
            let responseHeader = Register.contentType;
            let responseBody = await Register.metrics();
            res.setHeader('Content-Type', responseHeader);
            res.send(responseBody);
        }
    })
    metricsApp.use(metricsMiddleware);
    app.use(metricsRequestMiddleware);
    app.use(requestCounters({METRICS_PATH: METRICS_PATH}));
    app.use(responseCounters({RESTApi: false, oracledb: oracledb, DBNAME: DBNAME}));
    //as there is no outbound call in this app so commented below lines
    // patch(http);
    // patch(https);
}
app.use(bodyparser.text({type: 'application/graphql'}));
if(ENABLE_RESP_LOG==='true') app.use(CommonFunctions.setResponseBody);
app.use('/', (req, res, next) => {

    res.startTime = new Date().getTime();
    res.reqTime = moment().format("YYYY-MM-DD HH:mm:ss.SSS");
    req.sessionId = "rnd_" + uuidv1();
    req.clientName = '';

    let requestTraceId = req.get('x-request-trace-id')
    if (requestTraceId && requestTraceId !== '') {
        req.sessionId = requestTraceId;
    }
    let clientName = req.get('x-client-name');
    if (clientName) {
        req.clientName = clientName;
    }

    res.on('finish', function () {
        if (!req.originalUrl.includes("/status")) {
            let ip = CommonFunctions.getIP(req);
            const errMsg = res.get('Errors');
            if (errMsg) {
                commonLoggerInstance.errorLogger("", commonLoggerInstance.prepareLogMsg(req, res, ip, errMsg, false));
            } else {
                commonLoggerInstance.unifiedLogger("", commonLoggerInstance.prepareLogMsg(req, res, ip, null, false));
                if(ENABLE_RESP_LOG==='true') {
                    const respLog = commonLoggerInstance.prepareLogMsg(req, res, ip, errMsg, true);
                    commonLoggerInstance.debugLogger("", respLog);
                }
            }
        }
    });
    next();
});
async function init() {
    try {
        let currentPort = parseInt(process.env.NODE_APP_INSTANCE) + PORT;
        let mPort = parseInt(process.env.NODE_APP_INSTANCE) + METRICS_PORT;
        const time = moment().format("YYYY-MM-DD HH:mm:ss");
        if (SECURED) {
            const server = await https.createServer(options, app).listen(currentPort, () => {
                console.log(
                    `[${time}] [🚀] API server is ready to accept SSL connections on ${currentPort}`
                );
            });
        } else {
            const server = await app.listen(currentPort, () => {
                console.log(
                    `[${time}] [🚀] API server is ready to accept connections on ${currentPort}`
                );
            });
        }
        if (METRICS_ENABLED) {
            metricsApp.listen(mPort, () => {
                console.log(`[${time}][💥] Prometheus is exposing metrics on port: ${mPort}`);
            });
        } else {
            console.log(`[${time}][💥] Prometheus metrics server is disabled in configuration`);
        }
    } catch (err) {
        console.error("initDB error: " + err.message);
    }
}

init();

Here is a sample of GraphQL metrics response:

# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 12.849592

# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 1.215168

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 14.06476

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1707290493

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 86097920

# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0

# HELP nodejs_eventloop_lag_min_seconds The minimum recorded event loop delay.
# TYPE nodejs_eventloop_lag_min_seconds gauge
nodejs_eventloop_lag_min_seconds 0

# HELP nodejs_eventloop_lag_max_seconds The maximum recorded event loop delay.
# TYPE nodejs_eventloop_lag_max_seconds gauge
nodejs_eventloop_lag_max_seconds 0

# HELP nodejs_eventloop_lag_mean_seconds The mean of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_mean_seconds gauge
nodejs_eventloop_lag_mean_seconds 0

# HELP nodejs_eventloop_lag_stddev_seconds The standard deviation of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_stddev_seconds gauge
nodejs_eventloop_lag_stddev_seconds 0

# HELP nodejs_eventloop_lag_p50_seconds The 50th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p50_seconds gauge
nodejs_eventloop_lag_p50_seconds 0

# HELP nodejs_eventloop_lag_p90_seconds The 90th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p90_seconds gauge
nodejs_eventloop_lag_p90_seconds 0

# HELP nodejs_eventloop_lag_p99_seconds The 99th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p99_seconds gauge
nodejs_eventloop_lag_p99_seconds 0

# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name.
# TYPE nodejs_active_handles gauge
nodejs_active_handles{type="Pipe"} 1
nodejs_active_handles{type="Socket"} 1

# HELP nodejs_active_handles_total Total number of active handles.
# TYPE nodejs_active_handles_total gauge
nodejs_active_handles_total 2

# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name.
# TYPE nodejs_active_requests gauge

# HELP nodejs_active_requests_total Total number of active requests.
# TYPE nodejs_active_requests_total gauge
nodejs_active_requests_total 0

# HELP nodejs_heap_size_total_bytes Process heap size from Node.js in bytes.
# TYPE nodejs_heap_size_total_bytes gauge
nodejs_heap_size_total_bytes 57761792

# HELP nodejs_heap_size_used_bytes Process heap size used from Node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 50859584

# HELP nodejs_external_memory_bytes Node.js external memory size in bytes.
# TYPE nodejs_external_memory_bytes gauge
nodejs_external_memory_bytes 749543

# HELP nodejs_heap_space_size_total_bytes Process heap space size total from Node.js in bytes.
# TYPE nodejs_heap_space_size_total_bytes gauge
nodejs_heap_space_size_total_bytes{space="read_only"} 524288
nodejs_heap_space_size_total_bytes{space="new"} 1048576
nodejs_heap_space_size_total_bytes{space="old"} 45846528
nodejs_heap_space_size_total_bytes{space="code"} 2621440
nodejs_heap_space_size_total_bytes{space="map"} 3158016
nodejs_heap_space_size_total_bytes{space="large_object"} 4562944

# HELP nodejs_heap_space_size_used_bytes Process heap space size used from Node.js in bytes.
# TYPE nodejs_heap_space_size_used_bytes gauge
nodejs_heap_space_size_used_bytes{space="read_only"} 35200
nodejs_heap_space_size_used_bytes{space="new"} 581384
nodejs_heap_space_size_used_bytes{space="old"} 43160264
nodejs_heap_space_size_used_bytes{space="code"} 1706016
nodejs_heap_space_size_used_bytes{space="map"} 2185568
nodejs_heap_space_size_used_bytes{space="large_object"} 3201720

# HELP nodejs_heap_space_size_available_bytes Process heap space size available from Node.js in bytes.
# TYPE nodejs_heap_space_size_available_bytes gauge
nodejs_heap_space_size_available_bytes{space="read_only"} 480384
nodejs_heap_space_size_available_bytes{space="new"} 449784
nodejs_heap_space_size_available_bytes{space="old"} 1767248
nodejs_heap_space_size_available_bytes{space="code"} 605568
nodejs_heap_space_size_available_bytes{space="map"} 911432
nodejs_heap_space_size_available_bytes{space="large_object"} 1468956160

# HELP nodejs_version_info Node.js version info.
# TYPE nodejs_version_info gauge
nodejs_version_info{version="v10.24.1",major="10",minor="24",patch="1"} 1

# HELP nodejs_gc_duration_seconds Garbage collection duration by kind, one of major, minor, incremental or weakcb.
# TYPE nodejs_gc_duration_seconds histogram
nodejs_gc_duration_seconds_bucket{le="0.001",kind="minor"} 312
nodejs_gc_duration_seconds_bucket{le="0.01",kind="minor"} 440
nodejs_gc_duration_seconds_bucket{le="0.1",kind="minor"} 440
nodejs_gc_duration_seconds_bucket{le="1",kind="minor"} 440
nodejs_gc_duration_seconds_bucket{le="2",kind="minor"} 440
nodejs_gc_duration_seconds_bucket{le="5",kind="minor"} 440
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="minor"} 440
nodejs_gc_duration_seconds_sum{kind="minor"} 0.3343854669999998
nodejs_gc_duration_seconds_count{kind="minor"} 440
nodejs_gc_duration_seconds_bucket{le="0.001",kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="0.01",kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="0.1",kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="1",kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="2",kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="5",kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="incremental"} 34
nodejs_gc_duration_seconds_sum{kind="incremental"} 0.0018212539999999998
nodejs_gc_duration_seconds_count{kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="0.001",kind="major"} 0
nodejs_gc_duration_seconds_bucket{le="0.01",kind="major"} 33
nodejs_gc_duration_seconds_bucket{le="0.1",kind="major"} 34
nodejs_gc_duration_seconds_bucket{le="1",kind="major"} 34
nodejs_gc_duration_seconds_bucket{le="2",kind="major"} 34
nodejs_gc_duration_seconds_bucket{le="5",kind="major"} 34
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="major"} 34
nodejs_gc_duration_seconds_sum{kind="major"} 0.09746790300000002
nodejs_gc_duration_seconds_count{kind="major"} 34

# HELP http_request_duration_seconds duration histogram of http responses labeled with: status_code, method, path
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.003",status_code="200",method="GET",path="/dso"} 1
http_request_duration_seconds_bucket{le="0.03",status_code="200",method="GET",path="/dso"} 2
http_request_duration_seconds_bucket{le="0.1",status_code="200",method="GET",path="/dso"} 2
http_request_duration_seconds_bucket{le="0.3",status_code="200",method="GET",path="/dso"} 2
http_request_duration_seconds_bucket{le="1.5",status_code="200",method="GET",path="/dso"} 2
http_request_duration_seconds_bucket{le="10",status_code="200",method="GET",path="/dso"} 2
http_request_duration_seconds_bucket{le="+Inf",status_code="200",method="GET",path="/dso"} 2
http_request_duration_seconds_sum{status_code="200",method="GET",path="/dso"} 0.020068134
http_request_duration_seconds_count{status_code="200",method="GET",path="/dso"} 2
http_request_duration_seconds_bucket{le="0.003",status_code="200",method="POST",path="/dso"} 17
http_request_duration_seconds_bucket{le="0.03",status_code="200",method="POST",path="/dso"} 808
http_request_duration_seconds_bucket{le="0.1",status_code="200",method="POST",path="/dso"} 812
http_request_duration_seconds_bucket{le="0.3",status_code="200",method="POST",path="/dso"} 816
http_request_duration_seconds_bucket{le="1.5",status_code="200",method="POST",path="/dso"} 816
http_request_duration_seconds_bucket{le="10",status_code="200",method="POST",path="/dso"} 816
http_request_duration_seconds_bucket{le="+Inf",status_code="200",method="POST",path="/dso"} 816
http_request_duration_seconds_sum{status_code="200",method="POST",path="/dso"} 4.530245155000003
http_request_duration_seconds_count{status_code="200",method="POST",path="/dso"} 816
http_request_duration_seconds_bucket{le="0.003",status_code="400",method="POST",path="/dso"} 0
http_request_duration_seconds_bucket{le="0.03",status_code="400",method="POST",path="/dso"} 0
http_request_duration_seconds_bucket{le="0.1",status_code="400",method="POST",path="/dso"} 1
http_request_duration_seconds_bucket{le="0.3",status_code="400",method="POST",path="/dso"} 1
http_request_duration_seconds_bucket{le="1.5",status_code="400",method="POST",path="/dso"} 1
http_request_duration_seconds_bucket{le="10",status_code="400",method="POST",path="/dso"} 1
http_request_duration_seconds_bucket{le="+Inf",status_code="400",method="POST",path="/dso"} 1
http_request_duration_seconds_sum{status_code="400",method="POST",path="/dso"} 0.037692689
http_request_duration_seconds_count{status_code="400",method="POST",path="/dso"} 1
http_request_duration_seconds_bucket{le="0.003",status_code="404",method="GET",path="/metrics/dso"} 1
http_request_duration_seconds_bucket{le="0.03",status_code="404",method="GET",path="/metrics/dso"} 2
http_request_duration_seconds_bucket{le="0.1",status_code="404",method="GET",path="/metrics/dso"} 2
http_request_duration_seconds_bucket{le="0.3",status_code="404",method="GET",path="/metrics/dso"} 2
http_request_duration_seconds_bucket{le="1.5",status_code="404",method="GET",path="/metrics/dso"} 2
http_request_duration_seconds_bucket{le="10",status_code="404",method="GET",path="/metrics/dso"} 2
http_request_duration_seconds_bucket{le="+Inf",status_code="404",method="GET",path="/metrics/dso"} 2
http_request_duration_seconds_sum{status_code="404",method="GET",path="/metrics/dso"} 0.004232611
http_request_duration_seconds_count{status_code="404",method="GET",path="/metrics/dso"} 2

# HELP up 1 = up, 0 = not up
# TYPE up gauge
up 1

# HELP total_request Total number of request made
# TYPE total_request counter
total_request{method="GET"} 6
total_request{method="POST"} 1634

# HELP gql_function_call_count GraphQL function call count
# TYPE gql_function_call_count counter
gql_function_call_count{path="/dso"} 819
gql_function_call_count{path="/dso/"} 2
gql_function_call_count{path="/dso/__schema"} 807
gql_function_call_count{path="/dso/getUpdateTime"} 8
gql_function_call_count{path="/dso/updateRequestTypeAndAmount"} 2
gql_function_call_count{path="/metrics/dso/"} 2

# HELP success_error_response_total Total success & error response
# TYPE success_error_response_total counter
success_error_response_total{path="/dso/",status="200"} 2
success_error_response_total{path="/dso/__schema",status="200"} 807
success_error_response_total{path="/dso/getUpdateTime",status="200"} 7
success_error_response_total{path="/dso/getUpdateTime",status="400"} 1
success_error_response_total{path="/dso/updateRequestTypeAndAmount",status="200"} 2
success_error_response_total{path="/metrics/dso/",status="404"} 2

# HELP gql_functions_response_summary GraphQL functions response summary
# TYPE gql_functions_response_summary summary
gql_functions_response_summary{quantile="0.01",method="GET",path="/dso/",status="200"} 0.7322179999999999
gql_functions_response_summary{quantile="0.05",method="GET",path="/dso/",status="200"} 0.7322179999999999
gql_functions_response_summary{quantile="0.5",method="GET",path="/dso/",status="200"} 3.7697135
gql_functions_response_summary{quantile="0.9",method="GET",path="/dso/",status="200"} 6.807208999999999
gql_functions_response_summary{quantile="0.95",method="GET",path="/dso/",status="200"} 6.807208999999999
gql_functions_response_summary{quantile="0.99",method="GET",path="/dso/",status="200"} 6.807208999999999
gql_functions_response_summary{quantile="0.999",method="GET",path="/dso/",status="200"} 6.807208999999999
gql_functions_response_summary_sum{method="GET",path="/dso/",status="200"} 7.539426999999999
gql_functions_response_summary_count{method="GET",path="/dso/",status="200"} 2
gql_functions_response_summary{quantile="0.01",method="POST",path="/dso/__schema",status="200"} 2.62362708
gql_functions_response_summary{quantile="0.05",method="POST",path="/dso/__schema",status="200"} 2.83135815
gql_functions_response_summary{quantile="0.5",method="POST",path="/dso/__schema",status="200"} 3.900384283333333
gql_functions_response_summary{quantile="0.9",method="POST",path="/dso/__schema",status="200"} 5.3864410904761915
gql_functions_response_summary{quantile="0.95",method="POST",path="/dso/__schema",status="200"} 6.108391299999999
gql_functions_response_summary{quantile="0.99",method="POST",path="/dso/__schema",status="200"} 9.13532251999999
gql_functions_response_summary{quantile="0.999",method="POST",path="/dso/__schema",status="200"} 48.43206992499918
gql_functions_response_summary_sum{method="POST",path="/dso/__schema",status="200"} 3421.9032729999985
gql_functions_response_summary_count{method="POST",path="/dso/__schema",status="200"} 807
gql_functions_response_summary{quantile="0.01",method="POST",path="/dso/getUpdateTime",status="200"} 22.402758
gql_functions_response_summary{quantile="0.05",method="POST",path="/dso/getUpdateTime",status="200"} 22.402758
gql_functions_response_summary{quantile="0.5",method="POST",path="/dso/getUpdateTime",status="200"} 100.217777
gql_functions_response_summary{quantile="0.9",method="POST",path="/dso/getUpdateTime",status="200"} 121.0150834
gql_functions_response_summary{quantile="0.95",method="POST",path="/dso/getUpdateTime",status="200"} 125.328508
gql_functions_response_summary{quantile="0.99",method="POST",path="/dso/getUpdateTime",status="200"} 125.328508
gql_functions_response_summary{quantile="0.999",method="POST",path="/dso/getUpdateTime",status="200"} 125.328508
gql_functions_response_summary_sum{method="POST",path="/dso/getUpdateTime",status="200"} 644.368372
gql_functions_response_summary_count{method="POST",path="/dso/getUpdateTime",status="200"} 7
gql_functions_response_summary{quantile="0.01",method="POST",path="/dso/getUpdateTime",status="400"} 36.986306
gql_functions_response_summary{quantile="0.05",method="POST",path="/dso/getUpdateTime",status="400"} 36.986306
gql_functions_response_summary{quantile="0.5",method="POST",path="/dso/getUpdateTime",status="400"} 36.986306
gql_functions_response_summary{quantile="0.9",method="POST",path="/dso/getUpdateTime",status="400"} 36.986306
gql_functions_response_summary{quantile="0.95",method="POST",path="/dso/getUpdateTime",status="400"} 36.986306
gql_functions_response_summary{quantile="0.99",method="POST",path="/dso/getUpdateTime",status="400"} 36.986306
gql_functions_response_summary{quantile="0.999",method="POST",path="/dso/getUpdateTime",status="400"} 36.986306
gql_functions_response_summary_sum{method="POST",path="/dso/getUpdateTime",status="400"} 36.986306
gql_functions_response_summary_count{method="POST",path="/dso/getUpdateTime",status="400"} 1
gql_functions_response_summary{quantile="0.01",method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 7.481459999999999
gql_functions_response_summary{quantile="0.05",method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 7.481459999999999
gql_functions_response_summary{quantile="0.5",method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 44.5061445
gql_functions_response_summary{quantile="0.9",method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 81.530829
gql_functions_response_summary{quantile="0.95",method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 81.530829
gql_functions_response_summary{quantile="0.99",method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 81.530829
gql_functions_response_summary{quantile="0.999",method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 81.530829
gql_functions_response_summary_sum{method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 89.012289
gql_functions_response_summary_count{method="POST",path="/dso/updateRequestTypeAndAmount",status="200"} 2
gql_functions_response_summary{quantile="0.01",method="GET",path="/metrics/dso/",status="404"} 0.34775999999999996
gql_functions_response_summary{quantile="0.05",method="GET",path="/metrics/dso/",status="404"} 0.34775999999999996
gql_functions_response_summary{quantile="0.5",method="GET",path="/metrics/dso/",status="404"} 1.631666
gql_functions_response_summary{quantile="0.9",method="GET",path="/metrics/dso/",status="404"} 2.915572
gql_functions_response_summary{quantile="0.95",method="GET",path="/metrics/dso/",status="404"} 2.915572
gql_functions_response_summary{quantile="0.99",method="GET",path="/metrics/dso/",status="404"} 2.915572
gql_functions_response_summary{quantile="0.999",method="GET",path="/metrics/dso/",status="404"} 2.915572
gql_functions_response_summary_sum{method="GET",path="/metrics/dso/",status="404"} 3.263332
gql_functions_response_summary_count{method="GET",path="/metrics/dso/",status="404"} 2

# HELP request_in_bytes_summary Request size in bytes Summary
# TYPE request_in_bytes_summary summary
request_in_bytes_summary{quantile="0.01",method="POST",path="/dso/__schema"} 1468
request_in_bytes_summary{quantile="0.05",method="POST",path="/dso/__schema"} 1468
request_in_bytes_summary{quantile="0.5",method="POST",path="/dso/__schema"} 1468
request_in_bytes_summary{quantile="0.9",method="POST",path="/dso/__schema"} 1468
request_in_bytes_summary{quantile="0.95",method="POST",path="/dso/__schema"} 1468
request_in_bytes_summary{quantile="0.99",method="POST",path="/dso/__schema"} 1468
request_in_bytes_summary{quantile="0.999",method="POST",path="/dso/__schema"} 1468
request_in_bytes_summary_sum{method="POST",path="/dso/__schema"} 1184676
request_in_bytes_summary_count{method="POST",path="/dso/__schema"} 807
request_in_bytes_summary{quantile="0.01",method="POST",path="/dso/getUpdateTime"} 94
request_in_bytes_summary{quantile="0.05",method="POST",path="/dso/getUpdateTime"} 94
request_in_bytes_summary{quantile="0.5",method="POST",path="/dso/getUpdateTime"} 1913
request_in_bytes_summary{quantile="0.9",method="POST",path="/dso/getUpdateTime"} 2417
request_in_bytes_summary{quantile="0.95",method="POST",path="/dso/getUpdateTime"} 2503
request_in_bytes_summary{quantile="0.99",method="POST",path="/dso/getUpdateTime"} 2503
request_in_bytes_summary{quantile="0.999",method="POST",path="/dso/getUpdateTime"} 2503
request_in_bytes_summary_sum{method="POST",path="/dso/getUpdateTime"} 13734
request_in_bytes_summary_count{method="POST",path="/dso/getUpdateTime"} 8
request_in_bytes_summary{quantile="0.01",method="POST",path="/dso/updateRequestTypeAndAmount"} 537
request_in_bytes_summary{quantile="0.05",method="POST",path="/dso/updateRequestTypeAndAmount"} 537
request_in_bytes_summary{quantile="0.5",method="POST",path="/dso/updateRequestTypeAndAmount"} 584
request_in_bytes_summary{quantile="0.9",method="POST",path="/dso/updateRequestTypeAndAmount"} 631
request_in_bytes_summary{quantile="0.95",method="POST",path="/dso/updateRequestTypeAndAmount"} 631
request_in_bytes_summary{quantile="0.99",method="POST",path="/dso/updateRequestTypeAndAmount"} 631
request_in_bytes_summary{quantile="0.999",method="POST",path="/dso/updateRequestTypeAndAmount"} 631
request_in_bytes_summary_sum{method="POST",path="/dso/updateRequestTypeAndAmount"} 1168
request_in_bytes_summary_count{method="POST",path="/dso/updateRequestTypeAndAmount"} 2

# HELP request_in_bytes_hist Request size in bytes histogram
# TYPE request_in_bytes_hist histogram
request_in_bytes_hist_bucket{le="200",method="POST",path="/dso/__schema"} 0
request_in_bytes_hist_bucket{le="400",method="POST",path="/dso/__schema"} 0
request_in_bytes_hist_bucket{le="600",method="POST",path="/dso/__schema"} 0
request_in_bytes_hist_bucket{le="800",method="POST",path="/dso/__schema"} 0
request_in_bytes_hist_bucket{le="1000",method="POST",path="/dso/__schema"} 0
request_in_bytes_hist_bucket{le="+Inf",method="POST",path="/dso/__schema"} 807
request_in_bytes_hist_sum{method="POST",path="/dso/__schema"} 1184676
request_in_bytes_hist_count{method="POST",path="/dso/__schema"} 807
request_in_bytes_hist_bucket{le="200",method="POST",path="/dso/getUpdateTime"} 1
request_in_bytes_hist_bucket{le="400",method="POST",path="/dso/getUpdateTime"} 1
request_in_bytes_hist_bucket{le="600",method="POST",path="/dso/getUpdateTime"} 1
request_in_bytes_hist_bucket{le="800",method="POST",path="/dso/getUpdateTime"} 1
request_in_bytes_hist_bucket{le="1000",method="POST",path="/dso/getUpdateTime"} 1
request_in_bytes_hist_bucket{le="+Inf",method="POST",path="/dso/getUpdateTime"} 8
request_in_bytes_hist_sum{method="POST",path="/dso/getUpdateTime"} 13734
request_in_bytes_hist_count{method="POST",path="/dso/getUpdateTime"} 8
request_in_bytes_hist_bucket{le="200",method="POST",path="/dso/updateRequestTypeAndAmount"} 0
request_in_bytes_hist_bucket{le="400",method="POST",path="/dso/updateRequestTypeAndAmount"} 0
request_in_bytes_hist_bucket{le="600",method="POST",path="/dso/updateRequestTypeAndAmount"} 1
request_in_bytes_hist_bucket{le="800",method="POST",path="/dso/updateRequestTypeAndAmount"} 2
request_in_bytes_hist_bucket{le="1000",method="POST",path="/dso/updateRequestTypeAndAmount"} 2
request_in_bytes_hist_bucket{le="+Inf",method="POST",path="/dso/updateRequestTypeAndAmount"} 2
request_in_bytes_hist_sum{method="POST",path="/dso/updateRequestTypeAndAmount"} 1168
request_in_bytes_hist_count{method="POST",path="/dso/updateRequestTypeAndAmount"} 2

# HELP response_in_bytes_summary Response size in bytes Summary
# TYPE response_in_bytes_summary summary
response_in_bytes_summary{quantile="0.01",method="POST",path="/dso/__schema"} 43111
response_in_bytes_summary{quantile="0.05",method="POST",path="/dso/__schema"} 43111
response_in_bytes_summary{quantile="0.5",method="POST",path="/dso/__schema"} 43111
response_in_bytes_summary{quantile="0.9",method="POST",path="/dso/__schema"} 43111
response_in_bytes_summary{quantile="0.95",method="POST",path="/dso/__schema"} 43111
response_in_bytes_summary{quantile="0.99",method="POST",path="/dso/__schema"} 43111
response_in_bytes_summary{quantile="0.999",method="POST",path="/dso/__schema"} 43111
response_in_bytes_summary_sum{method="POST",path="/dso/__schema"} 34790577
response_in_bytes_summary_count{method="POST",path="/dso/__schema"} 807
response_in_bytes_summary{quantile="0.01",method="POST",path="/dso/getUpdateTime"} 68
response_in_bytes_summary{quantile="0.05",method="POST",path="/dso/getUpdateTime"} 68
response_in_bytes_summary{quantile="0.5",method="POST",path="/dso/getUpdateTime"} 758.3333333333334
response_in_bytes_summary{quantile="0.9",method="POST",path="/dso/getUpdateTime"} 1859.2000000000003
response_in_bytes_summary{quantile="0.95",method="POST",path="/dso/getUpdateTime"} 2320
response_in_bytes_summary{quantile="0.99",method="POST",path="/dso/getUpdateTime"} 2320
response_in_bytes_summary{quantile="0.999",method="POST",path="/dso/getUpdateTime"} 2320
response_in_bytes_summary_sum{method="POST",path="/dso/getUpdateTime"} 6774
response_in_bytes_summary_count{method="POST",path="/dso/getUpdateTime"} 8
response_in_bytes_summary{quantile="0.01",method="POST",path="/dso/updateRequestTypeAndAmount"} 166
response_in_bytes_summary{quantile="0.05",method="POST",path="/dso/updateRequestTypeAndAmount"} 166
response_in_bytes_summary{quantile="0.5",method="POST",path="/dso/updateRequestTypeAndAmount"} 285
response_in_bytes_summary{quantile="0.9",method="POST",path="/dso/updateRequestTypeAndAmount"} 404
response_in_bytes_summary{quantile="0.95",method="POST",path="/dso/updateRequestTypeAndAmount"} 404
response_in_bytes_summary{quantile="0.99",method="POST",path="/dso/updateRequestTypeAndAmount"} 404
response_in_bytes_summary{quantile="0.999",method="POST",path="/dso/updateRequestTypeAndAmount"} 404
response_in_bytes_summary_sum{method="POST",path="/dso/updateRequestTypeAndAmount"} 570
response_in_bytes_summary_count{method="POST",path="/dso/updateRequestTypeAndAmount"} 2
response_in_bytes_summary{quantile="0.01",method="GET",path="/metrics/dso/"} 150
response_in_bytes_summary{quantile="0.05",method="GET",path="/metrics/dso/"} 150
response_in_bytes_summary{quantile="0.5",method="GET",path="/metrics/dso/"} 150
response_in_bytes_summary{quantile="0.9",method="GET",path="/metrics/dso/"} 150
response_in_bytes_summary{quantile="0.95",method="GET",path="/metrics/dso/"} 150
response_in_bytes_summary{quantile="0.99",method="GET",path="/metrics/dso/"} 150
response_in_bytes_summary{quantile="0.999",method="GET",path="/metrics/dso/"} 150
response_in_bytes_summary_sum{method="GET",path="/metrics/dso/"} 300
response_in_bytes_summary_count{method="GET",path="/metrics/dso/"} 2

# HELP response_in_bytes_hist Response size in bytes histogram
# TYPE response_in_bytes_hist histogram
response_in_bytes_hist_bucket{le="300",method="POST",path="/dso/__schema"} 0
response_in_bytes_hist_bucket{le="500",method="POST",path="/dso/__schema"} 0
response_in_bytes_hist_bucket{le="1000",method="POST",path="/dso/__schema"} 0
response_in_bytes_hist_bucket{le="2000",method="POST",path="/dso/__schema"} 0
response_in_bytes_hist_bucket{le="3000",method="POST",path="/dso/__schema"} 0
response_in_bytes_hist_bucket{le="+Inf",method="POST",path="/dso/__schema"} 807
response_in_bytes_hist_sum{method="POST",path="/dso/__schema"} 34790577
response_in_bytes_hist_count{method="POST",path="/dso/__schema"} 807
response_in_bytes_hist_bucket{le="300",method="POST",path="/dso/getUpdateTime"} 1
response_in_bytes_hist_bucket{le="500",method="POST",path="/dso/getUpdateTime"} 1
response_in_bytes_hist_bucket{le="1000",method="POST",path="/dso/getUpdateTime"} 7
response_in_bytes_hist_bucket{le="2000",method="POST",path="/dso/getUpdateTime"} 7
response_in_bytes_hist_bucket{le="3000",method="POST",path="/dso/getUpdateTime"} 8
response_in_bytes_hist_bucket{le="+Inf",method="POST",path="/dso/getUpdateTime"} 8
response_in_bytes_hist_sum{method="POST",path="/dso/getUpdateTime"} 6774
response_in_bytes_hist_count{method="POST",path="/dso/getUpdateTime"} 8
response_in_bytes_hist_bucket{le="300",method="POST",path="/dso/updateRequestTypeAndAmount"} 1
response_in_bytes_hist_bucket{le="500",method="POST",path="/dso/updateRequestTypeAndAmount"} 2
response_in_bytes_hist_bucket{le="1000",method="POST",path="/dso/updateRequestTypeAndAmount"} 2
response_in_bytes_hist_bucket{le="2000",method="POST",path="/dso/updateRequestTypeAndAmount"} 2
response_in_bytes_hist_bucket{le="3000",method="POST",path="/dso/updateRequestTypeAndAmount"} 2
response_in_bytes_hist_bucket{le="+Inf",method="POST",path="/dso/updateRequestTypeAndAmount"} 2
response_in_bytes_hist_sum{method="POST",path="/dso/updateRequestTypeAndAmount"} 570
response_in_bytes_hist_count{method="POST",path="/dso/updateRequestTypeAndAmount"} 2
response_in_bytes_hist_bucket{le="300",method="GET",path="/metrics/dso/"} 2
response_in_bytes_hist_bucket{le="500",method="GET",path="/metrics/dso/"} 2
response_in_bytes_hist_bucket{le="1000",method="GET",path="/metrics/dso/"} 2
response_in_bytes_hist_bucket{le="2000",method="GET",path="/metrics/dso/"} 2
response_in_bytes_hist_bucket{le="3000",method="GET",path="/metrics/dso/"} 2
response_in_bytes_hist_bucket{le="+Inf",method="GET",path="/metrics/dso/"} 2
response_in_bytes_hist_sum{method="GET",path="/metrics/dso/"} 300
response_in_bytes_hist_count{method="GET",path="/metrics/dso/"} 2

# HELP db_pool_name DB pool statistics
# TYPE db_pool_name gauge
db_pool_name{pool_alias="mdsitdb"} 1

# HELP db_pool_up_time DB pool statistics(number of milliseconds since this pool was created)
# TYPE db_pool_up_time gauge
db_pool_up_time{pool_alias="mdsitdb"} 2158156

# HELP db_min_pool DB pool statistics
# TYPE db_min_pool gauge
db_min_pool{pool_alias="mdsitdb"} 1

# HELP db_max_pool DB pool statistics
# TYPE db_max_pool gauge
db_max_pool{pool_alias="mdsitdb"} 2

# HELP db_pool_total_connection_req DB pool statistics
# TYPE db_pool_total_connection_req gauge
db_pool_total_connection_req{pool_alias="mdsitdb"} 104

# HELP db_pool_connection_req_enqued DB pool statistics
# TYPE db_pool_connection_req_enqued gauge
db_pool_connection_req_enqued{pool_alias="mdsitdb"} 80

# HELP db_pool_connection_req_dequed DB pool statistics
# TYPE db_pool_connection_req_dequed gauge
db_pool_connection_req_dequed{pool_alias="mdsitdb"} 80

# HELP db_pool_connection_req_failed DB pool statistics
# TYPE db_pool_connection_req_failed gauge
db_pool_connection_req_failed{pool_alias="mdsitdb"} 0

# HELP db_pool_connection_req_timeout DB pool statistics
# TYPE db_pool_connection_req_timeout gauge
db_pool_connection_req_timeout{pool_alias="mdsitdb"} 0

# HELP db_pool_max_queue_len DB pool statistics
# TYPE db_pool_max_queue_len gauge
db_pool_max_queue_len{pool_alias="mdsitdb"} 15

# HELP db_pool_sum_of_time_in_queue DB pool statistics
# TYPE db_pool_sum_of_time_in_queue gauge
db_pool_sum_of_time_in_queue{pool_alias="mdsitdb"} 3543

# HELP db_pool_summary_connection_status DB pool statistics
# TYPE db_pool_summary_connection_status summary
db_pool_summary_connection_status{quantile="0.01",pool_alias="mdsitdb",connection_use="0",connection_open="1"} 2.62654356
db_pool_summary_connection_status{quantile="0.05",pool_alias="mdsitdb",connection_use="0",connection_open="1"} 2.82303485
db_pool_summary_connection_status{quantile="0.5",pool_alias="mdsitdb",connection_use="0",connection_open="1"} 3.878806
db_pool_summary_connection_status{quantile="0.9",pool_alias="mdsitdb",connection_use="0",connection_open="1"} 5.445018
db_pool_summary_connection_status{quantile="0.95",pool_alias="mdsitdb",connection_use="0",connection_open="1"} 6.375987099999998
db_pool_summary_connection_status{quantile="0.99",pool_alias="mdsitdb",connection_use="0",connection_open="1"} 12.401405339999993
db_pool_summary_connection_status{quantile="0.999",pool_alias="mdsitdb",connection_use="0",connection_open="1"} 80.66427213400038
db_pool_summary_connection_status_sum{pool_alias="mdsitdb",connection_use="0",connection_open="1"} 2468.6458650000004
db_pool_summary_connection_status_count{pool_alias="mdsitdb",connection_use="0",connection_open="1"} 549
db_pool_summary_connection_status{quantile="0.01",pool_alias="mdsitdb",connection_use="0",connection_open="2"} 2.5437123699999997
db_pool_summary_connection_status{quantile="0.05",pool_alias="mdsitdb",connection_use="0",connection_open="2"} 2.83419215
db_pool_summary_connection_status{quantile="0.5",pool_alias="mdsitdb",connection_use="0",connection_open="2"} 3.931933
db_pool_summary_connection_status{quantile="0.9",pool_alias="mdsitdb",connection_use="0",connection_open="2"} 5.674174
db_pool_summary_connection_status{quantile="0.95",pool_alias="mdsitdb",connection_use="0",connection_open="2"} 6.7537448
db_pool_summary_connection_status{quantile="0.99",pool_alias="mdsitdb",connection_use="0",connection_open="2"} 102.91710351000006
db_pool_summary_connection_status{quantile="0.999",pool_alias="mdsitdb",connection_use="0",connection_open="2"} 125.328508
db_pool_summary_connection_status_sum{pool_alias="mdsitdb",connection_use="0",connection_open="2"} 1730.5443330000023
db_pool_summary_connection_status_count{pool_alias="mdsitdb",connection_use="0",connection_open="2"} 271
db_pool_summary_connection_status{quantile="0.01",pool_alias="mdsitdb",connection_use="2",connection_open="2"} 3.8828009999999997
db_pool_summary_connection_status{quantile="0.05",pool_alias="mdsitdb",connection_use="2",connection_open="2"} 3.8828009999999997
db_pool_summary_connection_status{quantile="0.5",pool_alias="mdsitdb",connection_use="2",connection_open="2"} 3.8828009999999997
db_pool_summary_connection_status{quantile="0.9",pool_alias="mdsitdb",connection_use="2",connection_open="2"} 3.8828009999999997
db_pool_summary_connection_status{quantile="0.95",pool_alias="mdsitdb",connection_use="2",connection_open="2"} 3.8828009999999997
db_pool_summary_connection_status{quantile="0.99",pool_alias="mdsitdb",connection_use="2",connection_open="2"} 3.8828009999999997
db_pool_summary_connection_status{quantile="0.999",pool_alias="mdsitdb",connection_use="2",connection_open="2"} 3.8828009999999997
db_pool_summary_connection_status_sum{pool_alias="mdsitdb",connection_use="2",connection_open="2"} 3.8828009999999997
db_pool_summary_connection_status_count{pool_alias="mdsitdb",connection_use="2",connection_open="2"} 1

# HELP db_pool_connection_use DB pool statistics
# TYPE db_pool_connection_use gauge
db_pool_connection_use{pool_alias="mdsitdb"} 0

# HELP db_pool_connection_open DB pool statistics
# TYPE db_pool_connection_open gauge
db_pool_connection_open{pool_alias="mdsitdb"} 2

# HELP outbound_api_call_summary Outbound Api Call Metrics
# TYPE outbound_api_call_summary counter

To Run Application Locally with Prometheus and Grafana follow below steps:

prometheus.yml

scrape_configs:
  - job_name: 'Spring Boot Application input'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 3s
    static_configs:
      - targets: ['10.11.110.10:8080']
        labels:
          application: 'My Spring Boot Application'
    basic_auth:
      username: user
      password: password

docker-compose.yaml

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - ./prometheus:/etc/prometheus
      - prom_data:/prometheus
  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - 3000:3000
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=grafana
    volumes:
      - ./grafana:/etc/grafana/provisioning/datasources
volumes:
  prom_data:

Steps to follow:

1. Keep these 2 file in same directory
2. update prometheus.yaml : update below 2 (mandatory) and others fields are not mandatory to update for basic installation/run
 - targets: ['10.22.110.10:8080'] //replace ip port with u'r application ip port from where prometheus will listen the metrics
 - metrics_path: '/actuator/prometheus' //update this as per u'r application path where u expose metrics
4. Execute command "docker compose up -d" from terminal
5. Grafana can be access from browser in port localhost:3000 and login user: "admin", password: "grafana"
6. Prometheus will expose metrics in localhost:9090
7. To stop docker use: docker compose down -v from terminal


Share this post on...

0
APMMonitoringSpring-BootNodejs