ceph高可用分布式存储集群08cephnautilus14.2.19安装cephmgrdashboard

 Ceph Dashboard介绍
 
Ceph 的监控可视化界面方案很多----grafana、Kraken。但是从Luminous开始，Ceph 提供了原生的Dashboard功能，通过Dashboard可以获取Ceph集群的各种基本状态信息。
 (nautilus版) 需要安装 ceph-mgr-dashboard
ceph nautilus dashboard 已经具有一些管理功能了，可以在WEB界面创建一些内容
 
1.    Install the appropriate package from the below link
# ceph -v
ceph version 14.2.9 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)
# yum install ceph-mgr-dashboard
Note: it will ask for few dependencies, install with yum/apt package manager and then execute the above command.
注：所有mgr节点都安装上ceph-mgr-dashboard包，
2.    Enable the ceph mgr dashboard
# ceph mgr module enable dashboard --force 
# ceph mgr module ls | grep dashboard
 
3.    Create a user for Dashboard
Example: [ceph dashboard ac-user-create (username) (password) administrator]
# ceph dashboard ac-user-create admin admin administrator
{"username": "admin", "lastUpdate": 1581346477, "name": null, "roles": ["administrator"], "password": "$2b$12$55BR7an1KU9Rp3pBcc7HiOWNmI45kNzZlAJVK1c/Tk7krnGEEfY4S", "email": null}
 
------------------------------------------------------------------------
注：新点的版本比如14.2.19,创建用户的命令已经更改，操作如下
echo '123456' > password.txt
ceph dashboard ac-user-create admin  administrator -i password.txt
{"username": "admin", "lastUpdate": 1619334864, "name": null, "roles": ["administrator"], "password": "$2b$12$3XoNkgAD19mGLgjX28s4Z.9os8kMJoRglLuMWejXVINGxNiMNQA2W", "email": null}
------------------------------------------------------------------------
 
[root@ceph-host-01 mgr-dashboard]# ceph mgr services
{
    "dashboard": "https://ceph-host-01:8443/"
}
Note: Here you can access with IP address of ceph-mgr node, instead of hostname.
5.    Make sure firewall port is open
# firewall-cmd --add-port=8443/tcp --permanent 
# firewall-cmd --reload 
6.    Open the dashboard url in any browser
https://ceph-host-01:8443/or https://10.30.1.221:8443
7.    Enter the username: cent and password: password
配置好之后就可以登录，页面如下：


 
 
注：当mgr主机重启后，https://10.30.1.221:8443无法打开页面来了，解决办法是在admin节点重新把那个mgr节点加入进来，或者查看到现在的active的mgr节点是哪个，如果active的mgr节点有安装ceph-mgr-dashboard，则直接使用active的mgr节点IP登陆就好
[root@ceph-host-01 ceph-cluster]# ceph -s
  cluster:
    id:     272905d2-fd66-4ef6-a772-9cd73a274683
    health: HEALTH_WARN
            application not enabled on 1 pool(s)
            8 daemons have recently crashed
  services:
    mon: 3 daemons, quorum ceph-host-01,ceph-host-02,ceph-host-03 (age 14m)
    mgr: ceph-host-03(active, since 15m), standbys: ceph-host-02, ceph-host-01
    mds: nova:1 {0=ceph-host-02=up:active} 1 up:standby
    osd: 20 osds: 20 up (since 13m), 20 in (since 19h)
  data:
    pools:   6 pools, 512 pgs
    objects: 1.10k objects, 7.6 GiB
    usage:   35 GiB used, 1.5 TiB / 1.5 TiB avail
    pgs:     512 active+clean
[root@ceph-host-01 ceph-cluster]# ceph-deploy mgr create ceph-host-01
 
 
总结ceph nautilus开启dashboard的步骤
----------------------------------------------------------------------------
 
mgr节点操作：
 
yum -y install ceph-mgr-dashboard
pip uninstall urllib3
yum install python-urllib3 -y
 
ceph管理节点操作：
ceph mgr module enable dashboard
ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
ceph config set mgr mgr/dashboard/server_port 7000
ceph config set mgr mgr/dashboard/ssl false
ceph dashboard ac-user-create admin password administrator
 
修改默认配置命令#
 
指定集群dashboard的访问端口
# ceph config set mgr mgr/dashboard/server_port 7000
指定集群 dashboard的访问IP
# ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
关闭ssl
# ceph config set mgr mgr/dashboard/ssl false
 
开启Object Gateway管理功能#
1、创建rgw用户
radosgw-admin user create --uid=dashboard --display-name='dashboard rgw'   --system
# radosgw-admin user info --uid=dashboard
2、提供Dashboard证书
# ceph dashboard set-rgw-api-access-key $access_key
# ceph dashboard set-rgw-api-secret-key $secret_key
3、配置rgw主机名和端口
# ceph dashboard set-rgw-api-host 0.0.0.0
# ceph dashboard set-rgw-api-port 7480
4、刷新web页面
 
-----------------------------------------------------------------------------------
使用 Prometheus 监控 Ceph
 
ceph的可视化方案很多，本篇介绍的是比较简单的一种方式，并且对包都进行了二次封装，所以能够在极短的时间内构建出一个可视化的监控系统
本系统组件如下：
* ceph-nautilus版本
* ceph_exporter的nautilus版本
* prometheus的2.20.1版本
* grafana的grafana-7.1.3版本
* Ceph grafana的插件- Clusterby Cristian Calin
适配的系统为centos7
资源如下：
https://github.com/digitalocean/ceph_exporter（国内源https://github.com.cnpmjs.org/digitalocean/ceph_exporter）
https://nodejs.org/dist/v10.13.0/node-v10.13.0-linux-x64.tar.xz
http://static.zybuluo.com/zphj1987/1nu2k4cpcery94q2re3u6s1t/ceph-cluster_rev1.json
https://github.com/prometheus/prometheus/releases/download/v2.24.1/prometheus-2.24.1.linux-amd64.tar.gz
https://dl.grafana.com/oss/release/grafana-8.0.2-1.x86_64.rpm
 
注：一些包的来源地址
grafana下载源
https://github.com/grafana/grafana/tags
https://grafana.com/grafana/download/8.0.2
prometheus下载源
https://github.com/prometheus/prometheus/tags
ceph_exporter下载源
https://github.com/digitalocean/ceph_exporter/tags
 
 
以上资源均可以直接用wget进行下载，然后直接安装
监控的架构介绍
通过ceph_exporter抓取的ceph相关的数据并且在本地监听端口9128端口
prometheus抓取ceph_exporter的9128的端口的数据存储在本地的/usr/local/prometheus/目录下面
grafana抓取prometheus的数据进行渲染成web页面
页面的模板就是使用的grafana的ceph模板插件
那么我们就根据上面的架构去一步步的把系统配置起来
配置监控系统
安装ceph_exporter
方法1：
https://docs.ceph.com/docs/master/mgr/prometheus/
ceph mgr module enable prometheus
端口 9283
 
方法2：
　　　　1）安装需要的软件golang
　　　　　　#yum install golang git librados2-devel librbd1-devel -y
　　　　2）设置go的环境变量
　　　　　　# /etc/profile.d/go.sh
　　　　　　　　export GOROOT=/usr/lib/golang
　　　　　　　　export GOBIN=$GOROOT/bin
　　　　　　　　export GOPATH=/home/golang
　　　　　　　　export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
　　　　　　#source /etc/profile.d/go.sh
　　　　3）安装ceph_exporter
　　　　　　#cd /home/golang
　　　　　　#go get -u github.com/digitalocean/ceph_exporter    （由于被墙我们只能放弃这个办法而改用下面的命令）
                    # mkdir -pv src/github.com/digitalocean/
                    # cd src/github.com/digitalocean/ceph_exporter/
                    # git clone https://github.com.cnpmjs.org/digitalocean/ceph_exporter
                    # go build
　　　　4）运行ceph_exporter
　　　　　　#cd /usr/lib/golang/bin/
　　　　　　#nohup ./ceph_exporter &
　　　　5）检验结果
　　　　　　#curl 127.0.0.1:9128
可以看到端口起来了就是安装成功了，这个ceph_exporter建议是安装在管理节点上，也就是能够执行出ceph -s的节点上面的
 
安装prometheus
admin节点上部署prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.24.1/prometheus-2.24.1.linux-amd64.tar.gz
tar -xvf prometheus-2.24.1.linux-amd64.tar.gz
cd prometheus-2.24.1.linux-amd64/
cp prometheus /usr/bin/   
cp promtool /usr/bin/
 
 
# vim /lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus: the monitoring system
Documentation=http://prometheus.io/docs/
 
[Service]
ExecStart=/usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --web.listen-address=0.0.0.0:9090 --web.external-url=
Restart=always
StartLimitInterval=0
RestartSec=10
 
[Install]
WantedBy=multi-user.target
mkdir /etc/prometheus/
cp -R consoles console_libraries prometheus.yml /etc/prometheus/
mkdir /var/lib/prometheus/
 
systemctl daemon-reload
systemctl enable prometheus.service
systemctl start prometheus.service
 
vim /etc/prometheus/prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
 
# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
 
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
 
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'
 
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
 
    static_configs:
    - targets: ['localhost:9090']
 
  - job_name: 'ceph'
    static_configs:
    - targets: ['10.2.36.196:9283']
systemctl restart prometheus.service
 
这个地方默认是认为prometheus和ceph_exporter在一台机器上面，所以配置文件的/etc/ceph/prometheus.yml里面的targets写的是127.0.0.1，根据需要修改成ceph_exporter的ip地址即可
 
prometheus的默认监听端口为9090，到这个时候直接去web 上面就可以看到prometheus的抓取的数据了
 
到这里是数据到prometheus的已经完成了，下面就去做跟grafana相关的配置了
安装grafana
wget https://dl.grafana.com/oss/release/grafana-8.0.2-1.x86_64.rpm
yum install grafana-8.0.2-1.x86_64.rpm
systemctl enable grafana-server.service
systemctl start grafana-server.service
 
grafana默认监听的3000的端口
默认登陆的用户名密码为admin admin,登陆成功后会强制修改密码
配置grafana
 
首先增加数据源


 
这里如果能上网就直接输入id 917 ，如果不能上网就把上面的ceph-cluster_rev1.json文件弄到本地去，导入进去即可

 

把ceph-cluster_rev1.json内容粘贴进去就行
 
 
到这里就完成了配置了

 

 

 
--------------------------------------------------------------------------------------------------------------------------------------------------
 
扩展：编译N版ceph rpm包支持中文dashboard
一、前言
N版ceph dashboard已经支持中文了，但是用默认是英语，对于有中文需求的人可以参考以下步骤，重新编译ceph RPM包，支持dashboard中文显示。
二，环境准备
需要一个大内存和多核的主机可以是虚拟机，磁盘空间要足够大（80GB），安装centos7.6
三、下载ceph源码包
用git clone方式下载ceph源码需要下载ceph的子模块，这个过程要连接国外的网站，下载速度非常的慢，因此下载ceph的tar源码包，下载地址：https://download.ceph.com/tarballs/。
四、编译dashboard angular
ceph的dashboard使用的angular框架，默认情况下只编译了英文包，因此需要自己重新编译自己需要的语言包
1、解压
下载了ceph 源码包后，解压
wget https://download.ceph.com/tarballs/ceph_14.2.19.orig.tar.gz
tar -xvf ceph_14.2.19.orig.tar.gz
 
2、创建临时文件
TEMP_DIR=`mktemp -d`
3、创建python虚拟机环境
cd ceph-14.2.19/
./src/tools/setup-virtualenv.sh --python=python2.7 $TEMP_DIR
4、安装包
$TEMP_DIR/bin/pip install nodeenv
$TEMP_DIR/bin/nodeenv -p --node=10.13.0
5、进入dashboard目录
cd  ./src/pybind/mgr/dashboard/frontend
6、激活python虚拟环境  
. $TEMP_DIR/bin/activate
7、安装js相关包
npm ci
 
8、编译angular
npx npm-run-all --print-label --parallel --max-parallel 2 "build:*  - - -- --prod"     #编译支持的所有语言也可以用:zh-CN编译中文
 
------------------------------------------------------------------------
报错：Error: Cannot find module 'node-sass'
解决办法：
npm install -g cnpm --registry=https://registry.npm.taobao.org
cnpm install node-sass
 
报错：Error:Node Sass version 6.0.0 is incompatible with ^4.0.0
卸载 node-sass 
npm uninstall node-sass 
然后安装最新版本（5.0之前） 
npm install node-sass@4.14.1
---------------------------------------------------------------------------
 
编译成功后查看dist目录下是否有对应的语言文件
编译成功后退出python虚拟环境
deactivate
9、重新打ceph tar包
tar czvf ceph_14.2.19.tar.gz ceph-14.2.19/
五、编译ceph rpm包
1、安装ceph编译依赖包
执行ceph源码目录下的./install-deps.sh
2、安装gcc 7.3版本
yum install centos-release-scl scl-utils-build
yum list all --enablerepo='centos-sclo-rh' 列出scl有哪些源可以用
yum install devtoolset-7-gcc.x86_64 devtoolset-7-gcc-c++.x86_64 devtoolset-7-gcc-gdb-plugin.x86_64     安装gcc-7.3编译工具链
scl enable devtoolset-7 bash     切换到gcc-7.3编译环境
3、设置编译环境
yum install rpm-build rpmdevtools 安装rpm包编译工具
rpmdev-setuptree 设置编译环境
cp ceph_14.2.19.tar.gz rpmbuild/SOURCES     将ceph 源码压缩包拷贝到rpmbuild目录下
tar --strip-compOnents=1 -C ~/rpmbuild/SPECS/ --no-anchored -xvzf ~/rpmbuild/SOURCES/ceph_14.2.19.tar.gz "ceph.spec" "ceph.spec" 提取ceph.sepc文件
修改文件rpmbuild/SPECS/ceph.spec  编辑Source0:        http://ceph.com/download/ceph-行，
将ceph-改为rpmbuild/SOURCES下的文件名
4、编译ceph rpm包
执行  rpmbuild -ba ~/rpmbuild/SPECS/ceph.spec编译ceph rpm包
 
注：编译的过程会报大量错，主要是缺失一些包导致的
解决办法：
把所有报错中提到的缺失的包都安装上
yum install -y gperf cmake3 cryptsetup fuse-devel devtoolset-8-gcc-c++ gperftools-develleveldb-devellibaio-devel libblkid-devel libcurl-devel libcap-ng-devel libudev-devel libnl3-devel liboath-devel libtool libxml2-devel ncurses-devel snappy-devel valgrind-devel xfsprogs-devel xmlstarlet yasm librabbitmq-devel librdkafka-devel boost-random nss-devel keyutils-libs-devel libibverbs-devel librdmacm-devel openldap-devel openssl-devel CUnit-devel redhat-lsb-core python2-Cython python36-Cython python-prettytable python-sphinx lz4-devel lttng-ust-devel libbabeltrace-devel expat-devel junit gperftools-devel
 
查看结果
# find /root/rpmbuild/ -name "*.rpm"
/root/rpmbuild/SRPMS/ceph-14.2.19-0.el7.src.rpm
/root/rpmbuild/RPMS/x86_64/python3-rbd-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-test-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librados2-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-mon-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python3-rgw-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-mgr-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librgw-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librbd1-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-radosgw-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libcephfs_jni1-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-rgw-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-common-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-osd-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/rados-objclass-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librgw2-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-rados-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/cephfs-java-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/rbd-mirror-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-rbd-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-cephfs-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/rbd-nbd-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-base-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python3-rados-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-ceph-argparse-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python3-ceph-argparse-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-mds-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libcephfs2-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-ceph-compat-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/rbd-fuse-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python3-cephfs-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libcephfs-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-selinux-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-fuse-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libradosstriper1-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libcephfs_jni-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librbd-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libradospp-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librados-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libradosstriper-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-resource-agents-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-debuginfo-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-rook-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-diskprediction-local-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-ssh-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-diskprediction-cloud-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-k8sevents-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-dashboard-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-grafana-dashboards-14.2.19-0.el7.noarch.rpm
六、遇到的问题
1、编译rpm包过程中可能会报需要python3
yum install python36u -y
yum install python36u-devel -y
2、编译rpm包过程中出现以下错误
 
在ceph.spec文件中加入
%define _binaries_in_noarch_packages_terminate_build 0
 
七、安装方法
rpm -Uvh  ceph-mgr-dashboard-14.2.19-0.el7.noarch.rpm --force
 
我们再次打开dashboard后发现支持中文了

 
 
 
作者：Dexter_Wang   工作岗位：某互联网公司资深云计算与存储工程师  联系邮箱：993852246@qq.com