Вы здесь:

Grafana demo

Prometheus - инструмент сбора метрик приложений.
Grafana - отображение собранных статистических данных о работе приложений.
На картинке ниже отображены результат работы двух приложений, расположенных на разных хостах.

Запущенный процесс Prometheus:

1289 ?        Ssl    2:41 /usr/bin/prometheus-node-exporter
    --collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvmed+nd+p)d+$
    --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/)
    --collector.netdev.ignored-devices=^lo$
    --collector.textfile.directory=/var/lib/prometheus/node-exporter
21144 ?        Ssl    3:02 /usr/local/bin/prometheus
    --config.file=/etc/prometheus/prometheus.yml
    --storage.tsdb.path=/var/lib/prometheus
    --web.console.templates=/etc/prometheus/consoles
    --web.console.libraries=/etc/prometheus/console_libraries
    --web.listen-address=0.0.0.0:9090
    --storage.tsdb.retention.time=2d
    --storage.tsdb.retention.size=512MB

Параметры --storage.tsdb.retention.time=2d и --storage.tsdb.retention.size=512MB ограничивают количество сохраненных данных(метрик). Если не ограничить, то их накапливается ОГРОМНОЕ(ПРОСТО НЕРЕАЛЬНОЕ) количество. Заданы в сервисе старта prometheus (/etc/init.d/prometheus см. в конце заметки полный текст):

HELPER_ARGS="--name=$NAME --output=$LOGFILE  --pidfile=$PIDFILE --user=$USER --storage.tsdb.retention=2d"

Просмотр Prometheus.

Prometheus

Grafana - инструмент мониторинга приложений. Получение метрик из spring-boot приложения делается с помощью io.spring.dependency-management.
Пример проекта shop_kotlin

Установка Grafana.

Загрузить Grafana с https://grafana.com/grafana/download
или так:

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_10.1.0_amd64.deb
sudo dpkg -i grafana-enterprise_10.1.0_amd64.deb


TODO:

  • Обстрелять yandex-tankом проект shop_kotlin

Ссылки:
Set up and observe a Spring Boot application with Grafana Cloud, Prometheus, and OpenTelemetry
Установка prometheus
Осваиваем мониторинг с Prometheus. Часть 3. Настройка Prometheus server (!!!)


Файл /etc/init.d/prometheus:

#!/bin/sh
# kFreeBSD do not accept scripts as interpreters, using #!/bin/sh and sourcing.
if [ true != "$INIT_D_SCRIPT_SOURCED" ] ; then
    set "$0" "$@"; INIT_D_SCRIPT_SOURCED=true . /lib/init/init-d-script
fi
### BEGIN INIT INFO
# Provides:          prometheus
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Monitoring system and time series database
# Description:       Prometheus is a systems and services monitoring system. It
#                    collects metrics from configured targets at given
#                    intervals, evaluates rule expressions, displays the
#                    results, and can trigger alerts if some condition is
#                    observed to be true.
### END INIT INFO

# Author: Martín Ferrari <Адрес электронной почты защищен от спам-ботов. Для просмотра адреса в браузере должен быть включен Javascript.>

DESC="monitoring system and time series database"
DAEMON=/usr/bin/prometheus
NAME=prometheus
USER=prometheus
PIDFILE=/var/run/prometheus/prometheus.pid
LOGFILE=/var/log/prometheus/prometheus.log
CFGFILE=/etc/prometheus/prometheus.yml

HELPER=/usr/bin/daemon
HELPER_ARGS="--name=$NAME --output=$LOGFILE  --pidfile=$PIDFILE --user=$USER --storage.tsdb.retention=2d"

ARGS=""
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

config_check()
{
    retcode=0
    errors="$(/usr/bin/promtool check config $CFGFILE 2>&1)" || retcode=$?
    if [ $retcode -ne 0 ]; then
        log_failure_msg
        echo "Configuration test failed. Output of config test was:" >&2
        echo "$errors" >&2
        return $retcode
    fi
}

do_start_prepare()
{
    mkdir -p `dirname $PIDFILE` || true
    chown -R $USER: /var/lib/prometheus/metrics2
    chown -R $USER: `dirname $LOGFILE`
    chown -R $USER: `dirname $PIDFILE`
}

do_start_cmd()
{
    # Return
    #   0 if daemon has been started
    #   1 if daemon was already running
    #   2 if daemon could not be started
    $HELPER $HELPER_ARGS --running && return 1
    config_check || return 2
    $HELPER $HELPER_ARGS -- $DAEMON $ARGS || return 2
    return 0
}

do_stop_cmd()
{
    # Return
    #   0 if daemon has been stopped
    #   1 if daemon was already stopped
    #   2 if daemon could not be stopped
    #   other if a failure occurred
    $HELPER $HELPER_ARGS --running || return 1
    $HELPER $HELPER_ARGS --stop || return 2
    # wait for the process to really terminate
    for n in 1 2 3 4 5; do
        sleep $n
        $HELPER $HELPER_ARGS --running || break
    done
    $HELPER $HELPER_ARGS --running || return 0
    return 2
}

do_reload()
{
    log_daemon_msg "Reloading $DESC configuration files" "$NAME"
    $HELPER $HELPER_ARGS --running || return 1
    config_check || return 2
    helper_pid=$(cat $PIDFILE)
    if [ -z "$helper_pid" ]; then
        log_failure_msg "Unable to find PID"
        return 1
    fi
    start-stop-daemon --stop --signal 1 --quiet \
        --ppid "$helper_pid" --exec "$DAEMON"
    log_end_msg $?
}

Для Prometheus нужно настроить задания для сбора метрик. Пример заданий в /etc/prometheus/prometheus.yml:

global:
scrape_interval:     15s
evaluation_interval: 15s
external_labels:
monitor: 'example'

rule_files:
- "rules.yml"

scrape_configs:
- job_name: 'prometheus'
  scrape_interval: 5s
  scrape_timeout: 5s
  static_configs:
    - targets: ['localhost:9090']

- job_name: "camel_kafka_consumer_extdto(1.57)"
  scrape_interval: 5s
  metrics_path: "/camel_kafka_consumer_extdto/api/actuator/prometheus/"
  static_configs:
    - targets: ["192.168.1.57:8992"]

- job_name: "shop_kotlin(1.20)"
  scrape_interval: 5s
  metrics_path: "/shop_kotlin/api/actuator/prometheus"
  static_configs:
    - targets: ["192.168.1.20:8988"]

- job_name: "camel_kafka_consumer_extdto(1.20)"
  scrape_interval: 5s
  metrics_path: "/camel_kafka_consumer_extdto/api/actuator/prometheus"
  static_configs:
    - targets: ["192.168.1.20:8992"]

- job_name: "shop_kafka_consumer(1.20)"
  scrape_interval: 5s
  metrics_path: "/camel_kafka_consumer_extdto/api/actuator/prometheus"
  static_configs:
    - targets: ["192.168.1.20:8992"]

- job_name: "shop_kafka_consumer(1.57:8698)"
  scrape_interval: 5s
  metrics_path: "/shop_kafka_consumer/api/actuator/prometheus"
  static_configs:
    - targets: ["192.168.1.57:8698"]

- job_name: "shop_kotlin(1.57:8988)"
  scrape_interval: 5s
  metrics_path: "/shop_kotlin/api/actuator/prometheus"
  static_configs:
    - targets: ["192.168.1.57:8988"]

- job_name: "vacancy_backend(1.57:8988)"
  scrape_interval: 5s
  metrics_path: "/vacancy/api/actuator/prometheus"
  static_configs:
    - targets: ["192.168.1.57:8988"]

- job_name: "vacancy_backend(1.20:8988)"
  scrape_interval: 5s
  metrics_path: "/vacancy/api/actuator/prometheus"
  static_configs:
    - targets: ["192.168.1.20:8988"]

Путь к файлу заданий указан в параметре "CFGFILE" файла /etc/init.d/prometheus (см.выше "CFGFILE=/etc/prometheus/prometheus.yml")