热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

基于【CentOS7+Ambari2.7.0+HDP3.0】搭建HAWQ数据仓库01——准备环境,搭建本地仓库,安装ambari...

一、集群软硬件环境准备:操作系统:centos7x86_64.1804Ambari版本:2.7.0HDP版本:3.0.0HA

一、集群软硬件环境准备:

操作系统:  centos 7 x86_64.1804

Ambari版本:2.7.0

HDP版本:3.0.0

HAWQ版本:2.3.0
5台PC作为工作站:

ep-bd01ep-bd02ep-bd03ep-bd04ep-bd05

  其中ep-bd01作为主节点,用于安装ambari-server。

二、配置操作系统,安装必备软件

1,安装CentOS 7操作系统:环境配置,安装必备软件。

2,安装配置NTP服务,保证集群时间保持同步,以防止由于时间不同而造成掉线故障。

详细看随笔:基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库之——安装配置NTP服务,保证集群时间保持同步
见《安装配置NTP服务》

3,安装MariaDB Server用于Ambari server以及Hue和Hive

详细过程,参见:基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库之一 —— MariaDB 安装配置

 4,安装yum priorities plugin

yum install yum-plugin-priorities -y

 

三、搭建本地仓库:

1,下载软件包:

cd /root
mkdir downloads
cd downloads
wget http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.0.0.0/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/hdp.repo
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari.repo
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-centos7-rpm.tar.gz
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari-2.7.0.0-centos7.tar.gz

 

2、搭建本地仓库:

安装并开启Apache HTTP服务

yum install httpd -y
systemctl enable httpd
systemctl start httpd

安装HTTPD服务

确保/var/www/html目录存在,没有的话创建。

mkdir -p /var/www/html

 创建HDP,HDF子目录

cd /var/www/html
mkdir hdp hdf

解开下载的软件包:

cd /var/www/html
tar -zxvf /root/downloads/ambari-2.7.0.0-centos7.tar.gz -C .
tar -zxvf /root/downloads/HDP-3.0.0.0-centos7-rpm.tar.gz -C ./hdp
tar -zxvf /root/downloads/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz -C ./hdp
tar -zxvf /root/downloads/HDP-UTILS-1.1.0.22-centos7.tar.gz -C ./hdp

 修改下载的ambari.repo,

vim ambari.repo

安装如下内容修改,[注意版本号,需要根据具体下载的版本不同修改,解压后自己查看一下]:

#VERSION_NUMBER=2.7.0.0-897

[ambari-2.7.0.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.0.0
baseurl=http://ep-bd01/ambari/centos7/2.7.0.0-897
gpgcheck=1
gpgkey=http://ep-bd01/ambari/centos7/2.7.0.0-897/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

 

复制到/etc/yum.repos.d

cp ambari.repo /etc/yum.repos.d/ambari.repo

修改下载的hdp.repo,

vim hdp.repo

安装如下内容修改,[注意版本号,需要根据具体下载的版本不同修改,解压后自己查看一下]:

#VERSION_NUMBER=3.0.0.0-1634
[HDP-3.0]
name=HDP Version - HDP-3.0.0.0
baseurl=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-3.0-GPL]
name=HDP GPL Version - HDP-GPL-3.0.0.0
baseurl=http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

保存退出,复制到/etc/yum.repos.d/

cp hdp.repo /etc/yum.repos.d/hdp.repo

四、主节点安装ambari server

1,使用刚才配置好的本地仓库,直接yum命令安装。

yum install ambari-server -y

2,查看ambari server 状态

systemctl status ambari-server ● ambari-server.service - LSB: ambari-server daemonLoaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)Active: inactive (dead)Docs: man:systemd-sysv-generator(8)

看到ambari server已成功安装了。

3,配置mariadb,建立用户和数据库供ambari使用

建立数据库用户ambari

mysql -uroot -p

MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mysql]> grant all privileges on *.* to 'ambari'@'%' identified by 'ambari';
Query OK, 0 rows affected (0.06 sec)

MariaDB [mysql]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

MariaDB [mysql]>

建立数据库ambari,并运行ambari数据库建表sql命令文件。

MariaDB [mysql]> create database ambari;
Query OK,
1 row affected (0.01 sec)MariaDB [mysql]> use ambari;
Database changed
MariaDB [ambari]
> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
Query OK,
0 rows affected (0.01 sec)Query OK, 0 rows affected (0.00 sec)Query OK, 0 rows affected (0.00 sec)Query OK, 0 rows affected (0.06 sec)Query OK, 0 rows affected (0.00 sec)
Statement prepared
... ...

4,配置ambari server

执行命令

ambari-server setup

回答选择项,其中JDK选择“”Custom“”,给出系统安装目录,数据库一定要选择高级配置,指定mariadb数据库和用户,本人系统中详细过程如下:

Using python /usr/bin/python
Setup ambari
-server
Checking SELinux...
SELinux status is
'disabled'
Customize user account
for ambari-server daemon [y/n] (n)? n
Adjusting ambari
-server permissions and ownership...
Checking firewall status...
Checking JDK...
[
1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[
2] Custom JDK
==============================================================================
Enter choice (
1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http:
//www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? ^C
Aborting ... Keyboard Interrupt.
[root@ep
-bd01 downloads]# ambari-server setup
Using python
/usr/bin/python
Setup ambari
-server
Checking SELinux...
SELinux status is
'disabled'
Customize user account
for ambari-server daemon [y/n] (n)? n
Adjusting ambari
-server permissions and ownership...
Checking firewall status...
Checking JDK...
[
1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[
2] Custom JDK
==============================================================================
Enter choice (
1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required
for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME:
^C
Aborting ... Keyboard Interrupt.
[root@ep
-bd01 downloads]# echo $JAVA_HOME
/usr/java/jdk1.8.0_181-amd64
[root@ep
-bd01 downloads]# $JAVA_HOME/bin/java -version
java version
"1.8.0_181"
Java(TM) SE Runtime Environment (build
1.8.0_181-b13)
Java HotSpot(TM)
64-Bit Server VM (build 25.181-b13, mixed mode)
[root@ep
-bd01 downloads]# ambari-server setup
Using python
/usr/bin/python
Setup ambari
-server
Checking SELinux...
SELinux status is
'disabled'
Customize user account
for ambari-server daemon [y/n] (n)? n
Adjusting ambari
-server permissions and ownership...
Checking firewall status...
Checking JDK...
[
1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[
2] Custom JDK
==============================================================================
Enter choice (
1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required
for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME:
^C
Aborting ... Keyboard Interrupt.
[root@ep
-bd01 downloads]# ambari-server setup
Using python
/usr/bin/python
Setup ambari
-server
Checking SELinux...
SELinux status is
'disabled'
Customize user account
for ambari-server daemon [y/n] (n)? n
Adjusting ambari
-server permissions and ownership...
Checking firewall status...
Checking JDK...
[
1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[
2] Custom JDK
==============================================================================
Enter choice (
1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http:
//www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y
Downloading JDK from http:
//public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz
ERROR: Exiting with exit code 1.
REASON: Downloading or installing JDK failed:
'Fatal exception: Failed to download JDK: . Please check that the JDK is available at http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz. Also you may specify JDK file location in local filesystem using --jdk-location command line argument., exit code 1'. Exiting.
[root@ep
-bd01 downloads]# ambari-server setup
Using python
/usr/bin/python
Setup ambari
-server
Checking SELinux...
SELinux status is
'disabled'
Customize user account
for ambari-server daemon [y/n] (n)? n
Adjusting ambari
-server permissions and ownership...
Checking firewall status...
Checking JDK...
[
1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[
2] Custom JDK
==============================================================================
Enter choice (
1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required
for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME:
/usr/java/jdk1.8.0_181-amd64
Validating JDK on Ambari Server...
done.
Check JDK version
for Ambari Server...
JDK version found:
8
Minimum JDK version is
8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License
for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y
/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[
1] - PostgreSQL (Embedded)
[
2] - Oracle
[
3] - MySQL / MariaDB
[
4] - PostgreSQL
[
5] - Microsoft SQL Server (Tech Preview)
[
6] - SQL Anywhere
[
7] - BDB
==============================================================================
Enter choice (
1): 3
Hostname (localhost): ep
-bd01
Port (
3306):
Database name (ambari): ambari
Username (ambari): ambari
Enter Database Password (bigdata):
Re
-enter password:
Configuring ambari database...
Should ambari use existing default jdbc
/usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema:
/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y
/n] (y)? y
Extracting system views...
ambari
-admin-2.7.0.0.897.jar
....
Ambari repo
file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server
'setup' completed successfully.

5,命令行方式设置mysql数据库连接库,用于oozie和ranger连接mariadb时使用(上面设置后不起作用,必须如下操作):

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

 

6,配置ambari-server自启动,启动ambari-server

[root@ep-bd01 downloads]# systemctl enable ambari-server
[root@ep
-bd01 downloads]# systemctl start ambari-server
[root@ep
-bd01 downloads]# systemctl status ambari-server
● ambari
-server.service - LSB: ambari-server daemonLoaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)Active: active (running) since Tue 2018-08-14 11:06:16 CST; 2min 36s agoDocs: man:systemd-sysv-generator(8)Process: 323056 ExecStart=/etc/rc.d/init.d/ambari-server start (code=exited, status=0/SUCCESS)CGroup: /system.slice/ambari-server.service└─323080 /usr/java/jdk1.8.0_181-amd64/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -...Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Organizing resource files at /var/lib/ambari-server/resources...
Aug
14 11:05:58 ep-bd01 ambari-server[323056]: Ambari database consistency check started...
Aug
14 11:05:58 ep-bd01 ambari-server[323056]: Server PID at: /var/run/ambari-server/ambari-server.pid
Aug
14 11:05:58 ep-bd01 ambari-server[323056]: Server out at: /var/log/ambari-server/ambari-server.out
Aug
14 11:05:58 ep-bd01 ambari-server[323056]: Server log at: /var/log/ambari-server/ambari-server.log
Aug
14 11:06:16 ep-bd01 ambari-server[323056]: Waiting for server start.....................
Aug
14 11:06:16 ep-bd01 ambari-server[323056]: Server started listening on 8080
Aug
14 11:06:16 ep-bd01 ambari-server[323056]: DB configs consistency check: no errors and warnings were found.
Aug
14 11:06:16 ep-bd01 ambari-server[323056]: Ambari Server 'start' completed successfully.
Aug
14 11:06:16 ep-bd01 systemd[1]: Started LSB: ambari-server daemon.systemctl status ambari-server
● ambari
-server.service - LSB: ambari-server daemonLoaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)Active: active (running) since Tue 2018-08-14 11:06:16 CST; 2min 36s agoDocs: man:systemd-sysv-generator(8)Process: 323056 ExecStart=/etc/rc.d/init.d/ambari-server start (code=exited, status=0/SUCCESS)CGroup: /system.slice/ambari-server.service└─323080 /usr/java/jdk1.8.0_181-amd64/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -...Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Organizing resource files at /var/lib/ambari-server/resources...
Aug
14 11:05:58 ep-bd01 ambari-server[323056]: Ambari database consistency check started...
Aug
14 11:05:58 ep-bd01 ambari-server[323056]: Server PID at: /var/run/ambari-server/ambari-server.pid
Aug
14 11:05:58 ep-bd01 ambari-server[323056]: Server out at: /var/log/ambari-server/ambari-server.out
Aug
14 11:05:58 ep-bd01 ambari-server[323056]: Server log at: /var/log/ambari-server/ambari-server.log
Aug
14 11:06:16 ep-bd01 ambari-server[323056]: Waiting for server start.....................
Aug
14 11:06:16 ep-bd01 ambari-server[323056]: Server started listening on 8080
Aug
14 11:06:16 ep-bd01 ambari-server[323056]: DB configs consistency check: no errors and warnings were found.
Aug
14 11:06:16 ep-bd01 ambari-server[323056]: Ambari Server 'start' completed successfully.
Aug
14 11:06:16 ep-bd01 systemd[1]: Started LSB: ambari-server daemon.

ok,Ambari-server安装完成。浏览器访问ambari-server服务:

http://ep-bd01:8080

默认用户名/密码为:admin/admin,登陆后界面如下:

 五、所有主机节点,安装ambari-agent,并配置自启动

yum install ambari-agent -y
systemctl enable ambari-agent
systemctl restart ambari-agent && systemctl status ambari-agent

 


转载于:https://www.cnblogs.com/dajianshi/p/9455980.html


推荐阅读
  • Spring Security基础配置详解
    本文详细介绍了Spring Security的基础配置方法,包括如何搭建Maven多模块工程以及具体的安全配置步骤,帮助开发者更好地理解和应用这一强大的安全框架。 ... [详细]
  • 本文探讨了在AspNetForums平台中实施基于角色的权限控制系统的方法,旨在为不同级别的用户提供合适的访问权限,确保系统的安全性和可用性。 ... [详细]
  • Gradle 是 Android Studio 中默认的构建工具,了解其基本配置对于开发效率的提升至关重要。本文将详细介绍如何在 Gradle 中定义和使用共享变量,以确保项目的一致性和可维护性。 ... [详细]
  • 本文探讨了Linux环境下线程私有数据(Thread-Specific Data, TSD)的概念及其重要性,介绍了如何通过TSD技术避免多线程间全局变量冲突的问题,并提供了具体的实现方法和示例代码。 ... [详细]
  • C/C++ 应用程序的安装与卸载解决方案
    本文介绍了如何使用Inno Setup来创建C/C++应用程序的安装程序,包括自动检测并安装所需的运行库,确保应用能够顺利安装和卸载。 ... [详细]
  • 本文介绍了Tomcat的基本操作,包括启动、关闭及首次访问的方法,并详细讲解了如何在IDEA中创建Web项目,配置Servlet及其映射,以及如何将项目部署到Tomcat。 ... [详细]
  • 【MySQL】frm文件解析
    官网说明:http:dev.mysql.comdocinternalsenfrm-file-format.htmlfrm是MySQL表结构定义文件,通常frm文件是不会损坏的,但是如果 ... [详细]
  • 本文探讨了互联网服务提供商(ISP)如何可能篡改或插入用户请求的数据流,并提供了有效的技术手段来防止此类劫持行为,确保网络环境的安全与纯净。 ... [详细]
  • 数据输入验证与控件绑定方法
    本文提供了多种数据输入验证函数及控件绑定方法的实现代码,包括电话号码、数字、传真、邮政编码、电子邮件和网址的验证,以及报表绑定和自动编号等功能。 ... [详细]
  • Asynchronous JavaScript and XML (AJAX) 的流行很大程度上得益于 Google 在其产品如 Google Suggest 和 Google Maps 中的应用。本文将深入探讨 AJAX 在 .NET 环境下的工作原理及其实现方法。 ... [详细]
  • 本文详细介绍了如何在Java Swing中使用`JButton.registerKeyboardAction()`方法来为按钮设置键盘快捷键,并提供了多个实用的代码示例。 ... [详细]
  • 汇总了2023年7月7日最新的网络安全新闻和技术更新,包括最新的漏洞披露、工具发布及安全事件。 ... [详细]
  • 本文详细介绍如何在SSM(Spring + Spring MVC + MyBatis)框架中实现分页功能。包括分页的基本概念、数据准备、前端分页栏的设计与实现、后端分页逻辑的编写以及最终的测试步骤。 ... [详细]
  • WebBenchmark:强大的Web API性能测试工具
    本文介绍了一款名为WebBenchmark的Web API性能测试工具,该工具不仅支持HTTP和HTTPS服务的测试,还提供了丰富的功能来帮助开发者进行高效的性能评估。 ... [详细]
  • 探讨了在HTML表单中使用元素代替进行表单提交的方法。 ... [详细]
author-avatar
儒雅的天麟
这个家伙很懒,什么也没留下!
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有