Loki對標(biāo)EFK/ELK，由于其輕量的設(shè)計(jì)，備受歡迎，Loki相比EFK/ELK，它不對原始日志進(jìn)行索引，只對日志的標(biāo)簽進(jìn)行索引，而日志通過壓縮進(jìn)行存儲(chǔ)，通常是文件系統(tǒng)存儲(chǔ)，所以其操作成本更低，數(shù)量級(jí)效率更高
由于Loki的存儲(chǔ)都是基于文件系統(tǒng)的，所以它的日志搜索時(shí)基于內(nèi)容即日志行中的文本，所以它的查詢支持LogQL，在搜索窗口中通過過濾標(biāo)簽的方式進(jìn)行搜索和查詢。
Loki分兩部分，Loki是日志引擎部分，Promtail是收集日志端，然后通過Grafana進(jìn)行展示.
1.安裝grafana
wget https://dl.grafana.com/oss/release/grafana-8.2.5.linux-amd64.tar.gz
tar -zxvf grafana-8.2.5.linux-amd64.tar.gz
mv grafana-8.2.5 /usr/local/grafana創(chuàng)建Systemd服務(wù)
cat>/usr/lib/systemd/system/grafana-server.service<<EOF
[Unit]
Description=Grafana Server
After=network.target
[Service]
Type=simple
User=root
WorkingDirectory=/usr/local/grafana
ExecStart=/usr/local/grafana/bin/grafana-server
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF服務(wù)開機(jī)自啟
systemctl daemon-reload
systemctl enable grafana-server.service && systemctl start grafana-server.service/usr/local/grafana/conf/defaults.ini為默認(rèn)配置文件
工作端口，默認(rèn)為3000。
Loki的官方文檔
https://grafana.com/docs/loki/latest/installation
Loki·安裝
從官方文檔看，Loki支持源碼安裝、Docker、Helm、Local、Tanka
我選擇local，就是編譯好的二進(jìn)制可執(zhí)行文件
安裝步驟：
鏡像網(wǎng)址
https://hub.fastgit.org/grafana
1.下載二進(jìn)制可執(zhí)行文件
https://hub.fastgit.org/grafana/loki/releases/download/v2.4.1/loki-linux-amd64.zip
unzip loki-linux-amd64.zip && mv loki-linux-amd64 /usr/bin/loki
chmod o+x /usr/bin/loki
創(chuàng)建工作目錄并下載配置文件
mkdir -p /data/loki
參考配置文件
https://hub.fastgit.org/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml
最終lok.yml如下
vim /data/loki/lok.yml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /data/loki
storage:
filesystem:
chunks_directory: /data/loki/chunks
rules_directory: /data/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
##報(bào)警地址
ruler:
alertmanager_url: http://localhost:9093
#啟用Redis緩存配置
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /data/loki
storage:
filesystem:
chunks_directory: /data/loki/chunks
rules_directory: /data/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
frontend:
compress_responses: true
query_range:
split_queries_by_interval: 24h
results_cache:
cache:
redis:
endpoint: 192.168.1.6:6379
expiration: 10s
db: 1
cache_results: true
chunk_store_config:
chunk_cache_config:
redis:
endpoint: 192.168.1.6:6379
expiration: 10s
db: 1
write_dedupe_cache_config:
redis:
endpoint: 192.168.1.6:6379
expiration: 1h
db: 2
ruler:
alertmanager_url: http://localhost:9093創(chuàng)建相關(guān)目錄
mkdir -p /data/loki/{chunks,rules,logs}
啟動(dòng)
nohup loki --config.file=/data/loki/loki.yml > /data/loki/logs/loki.log 2>&1 &啟動(dòng)腳本
#!/bin/bash
CONFILE='/data/loki/loki.yml'
PIDFILE='/data/loki/loki.pid'
LOGFILE='/data/loki/loki.log'
OPTS='/usr/bin/loki --config.file='
PID=`ps -ef | grep ${CONFILE} | grep 'config.file' | awk '{print $2}'`
if [ ! ${PID} ]; then
nohup ${OPTS}${CONFILE} >${LOGFILE} 2>&1 & echo $! >${PIDFILE} &
sleep 3
echo "loki PID:`ps -ef | grep ${CONFILE} | grep 'config.file' | awk '{print $2}'`"
PID=`ps -ef | grep ${CONFILE} | grep 'config.file' | awk '{print $2}'`
if [ ! ${PID} ] ; then echo "loki config error, tail -f ${LOGFILE}!"
fi
else
echo "loki is running,PID:`cat ${PIDFILE}`"
fi安裝promtail進(jìn)行收集日志
Promtail是收集日志端
下載安裝
wget https://hub.fastgit.org/grafana/loki/releases/download/v2.4.1/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip && mv promtail-linux-amd64 /usr/bin/promtail && chmod o+x /usr/bin/promtail配置參考文件
官網(wǎng)配置文件文檔
https://hub.fastgit.org/grafana/loki/blob/main/clients/cmd/promtail/promtail-local-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log參數(shù)說明
server部分定義監(jiān)聽端口，positions定義讀取的文件偏移量存儲(chǔ)位置，clients定義loki接口地址，最后一部分scrape_configs是重點(diǎn)部分
promtail通過scrape_configs部分配置收集日志的相關(guān)信息，以測試配置文件為例：
job_name 用來區(qū)分日志組
static_configs 收集日志的靜態(tài)配置
targets 收集日志的節(jié)點(diǎn)，這個(gè)參數(shù)其實(shí)是在自動(dòng)發(fā)現(xiàn)的時(shí)候使用的
labels 定義一個(gè)要收集的日志文件和一組可選的附加標(biāo)簽
job 標(biāo)簽名稱，在grafana索引的時(shí)候用到的標(biāo)簽名稱
__path__ 定義日志收集的文件或路徑，支持正則
配置文件修改完成后，就可以啟動(dòng)promtail了，和loki啟動(dòng)方法一樣，通過--config.file指定配置文件啟動(dòng)promtail，類似于tail，它只監(jiān)聽新增日志，不會(huì)像filebeat一樣，讀取日志所有內(nèi)容，這是和filebeat的一個(gè)區(qū)別
最終配置，啟用靜態(tài)與文件自動(dòng)發(fā)現(xiàn)
mkdir -p /data/promtail
vim /data/promtail/promtail.yml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://192.168.1.6:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
file_sd_configs:
- files:
- /data/promtail/log_file/*.json
refresh_interval: 1m啟動(dòng)
創(chuàng)建相關(guān)文件
mkdir -p /data/promtail/log_file/
文件自動(dòng)發(fā)現(xiàn)配置
vim /data/promtail/log_file/mail.json
[
{
"targets": [ "localhost" ],
"labels": {
"__path__": "/var/log/maillog",
"job": "mailserver"
}
}
]開機(jī)自啟動(dòng)
nohup promtail --config.file=/data/promtail/promtail.yml >/data/promtail/promtail.log 2>&1 &啟動(dòng)腳本
vim /data/promtail/start.sh
#!/bin/bash
CONFILE='/data/promtail/promtail.yml'
PIDFILE='/data/promtail/promtail.pid'
LOGFILE='/data/promtail/promtail.log'
OPTS='/usr/bin/promtail --config.file='
PID=`ps -ef | grep ${CONFILE} | grep 'config.file' | awk '{print $2}'`
if [ ! ${PID} ]; then
nohup ${OPTS}${CONFILE} >${LOGFILE} 2>&1 & echo $! >${PIDFILE} &
sleep 3
echo "promtail PID:`ps -ef | grep ${CONFILE} | grep 'config.file' | awk '{print $2}'`"
PID=`ps -ef | grep ${CONFILE} | grep 'config.file' | awk '{print $2}'`
if [ ! ${PID} ] ; then echo "promtail config error, tail -f ${LOGFILE}!"
fi
else
echo "promtail is running,PID:`cat ${PIDFILE}`"
fichmod o+x /data/promtail/start.sh
查看是否工作
[root@openrestry promtail]# netstat -ntpl | grep loki
tcp6 0 0 :::9096 :::* LISTEN 1721/loki
tcp6 0 0 :::3100 :::* LISTEN 1721/loki
[root@openrestry promtail]# netstat -ntpl | grep promtail
tcp6 0 0 :::40210 :::* LISTEN 1833/promtail
tcp6 0 0 :::9080 :::* LISTEN 1833/promtail在Grafana中添加顯示輸出
登錄http://Grafana:3000，添加loki數(shù)據(jù)源
京公網(wǎng)安備 11010802030320號(hào)