官网：
http://kylin.apache.org/
社区：
https://github.com/KylinOLAP/Kylin/issues
http://apache-kylin.74782.x6.nabble.com/
源码：
https://github.com/apache/kylin
博客：
逐层（By Level）算法 VS 逐块(By Split) 算法
一、工具准备
zookeeper3.4.6 （hadoop、hbase 管理工具）Hadoop.2.7.1Hbase1.1.4Kylin1.5.0-HBase1.1.3Jdk1.7.80Hive 2.0.0
二、虚拟主机
192.168.200.165 master1192.168.200.166 master2192.168.200.167 slave1192.168.200.168 slave2
三、安装mysql
查看是否安装了mysql（master1）
[root@master1 ~]# ps -aux | grep mysqlMysql 3632 0.0 0.0 115348 1648?SsApr01 0:00 /bin/sh /wdcloud/app/mysql/bin/mysqld_safemysql 4519 0.5 19.8 13895940 1591664 ? Sl Apr01 29:55/wdcloud/app/mysql/bin/mysqld--basedir=/wdcloud/app/mysql--datadir=/wdcloud/data/mysql/data--plugin-dir=/wdcloud/app/mysql/lib/mysql/plugin--log-error=/wdcloud/data/mysql/data/mysql-error.log--open-files-limit=20000--pid-file=/wdcloud/data/mysql/data/localhost.localdomain.pid --socket=/tmp/mysql.sock--port=3306
查看mysql版本
[root@master1 ~]# mysql --versionmysql Ver 14.14 Distrib 5.6.29-76.2, for Linux (x86_64) using 6.2
登录mysql
[root@master1 ~]# mysql -uroot -pEnter password:
四、安装jdk
查看安装版本
[root@master1 ~]# java -versionjava version "1.7.0_80"Java(TM) SE Runtime Environment (build 1.7.0_80-b15)Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
查看安装位置
[root@master1 ~]# which java/jdk1.7.0_80/bin/java
五、安装zookeeper
1.解压缩zookeeper到根目录下，进入目录，创建文件夹data、datalog、logs，配置环境变量
export ZOOKEEPER_HOME=/zookeeper-3.4.6export PATH=$PATH:$ZOOKEEPER_HOME/bin
2.进入conf文件夹，复制zoo_sample.cfg 为zoo.cfg
3.修改zoo.cfg，增加红色内容，保存退出：
tickTime=2000initLimit=10syncLimit=5dataDir=/zookeeper-3.4.6/datadataLogDir=/zookeeper-3.4.6/datalogclientPort=2181server.0=master1:2888:3888server.1=master2:2888:3888server.2=slave1:2888:3888server.3=slave2:2888:3888
修改conf下的log4j.properties文件，配置log文件生成位置
zookeeper.log.dir=/zookeeper-3.4.6/logszookeeper.log.file=zookeeper.logzookeeper.tracelog.dir=/zookeeper-3.4.6/logszookeeper.tracelog.file=zookeeper_trace.log
4.进入data文件夹创建文件“myid”，添加内容 0 保存退出
5.分发zookeeper文件夹到各个虚拟主机上的根目录上
scp –r /zookeeper-3.4.6 hadoop@master2:/scp –r /zookeeper-3.4.6 hadoop@slave1:/scp –r /zookeeper-3.4.6 hadoop@ slave2:/
6.修改每台主机上的myid ，按照顺序 master1 的myid 为0 master2 的myid 1 以此类推。
7.启动zookeeper
分别进入各个虚拟主机的zookeeper目录下启动zk服务
bin/zkServer.sh start
8.分别查询zookeeper的状态
bin/zkServer.sh status
[hadoop@master1 ~]$ zkServer.sh statusJMX enabled by defaultUsing config: /zookeeper-3.4.6/bin/../conf/zoo.cfgMode: follower
Leader 是 zookeeper 主机启动了
Follower是zookeeper 从机启动了
9.停止zookeeper
Bin/zkServer stop
六、hadoop 高可用部署
1.解压缩hadoop2.7.1到hadoop家目录下，进入目录，创建文件夹tmp，hdfs/name，hdfs/data
2.进入~/hadoop/etc/hadoop，该文件夹包含了hadoop的大部分配置文件。
3.修改hadoop各配置文件如下：
core-site.xml
fs.default.name hdfs://master1:9000 true hadoop.tmp.dir /home/hadoop/hadoop/tmp A base for other tempory directories io.file.buffer.size 131702 fs.checkpoint.period 3600 多长时间记录一次hdfs的镜像，默认一小时 fs.checkpoint.size 67108864 一次记录多大的size，默认64M
hadoop.env.sh
#设置JAVA_HOMEexport JAVA_HOME=/jdk1.7.0_80#设置HADOOP_CONF_DIRexport HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"} hbase-site.xml
hbase.rootdir hdfs://master1:9000/hbase hbase.cluster.distributed true hbase.zookeeper.quorum master1,master2,slave1, slave2 hbase.zookeeper.property.dataDir /zookeeper-3.4.6 /data hbase.zookeeper.property.clientPort 2181 hbase.coprocessor.user.region.classes org.apache.hadoop.hbase.coprocessor.AggregateImplementation hbase.regionserver.wal.codec org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec hbase.master.loadbalancer.class org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer hbase.coprocessor.master.classes org.apache.phoenix.hbase.index.master.IndexMasterObserver phoenix.query.maxServerCacheBytes 1073741824 hbase.client.scanner.caching 5000 HBase客户端扫描缓存，对查询性能有很大帮助 hbase.rpc.timeout 360000000
hdfs-site.xml
dfs.namenode.name.dir /home/hadoop/hadoop/hdfs/name dfs.datanode.data.dir /home/hadoop/hadoop/hdfs/data dfs.replication 4 dfs.namenode.secondary.http-address master2:9001 dfs.webhdfs.enabled true dfs.client.read.shortcircuit false
mapred-site.xml
mapreduce.framework.name yarn mapreduce.jobtracker.http.address NameNode:50030 mapreduce.jobhistory.address master1:10020 mapreduce.jobhistory.webapp.address master1:19888 mapred.compress.map.output true
新增masters文件，以部署高可用的hadoop，将master2作为Secondary Name Node
masters
master2
slaves
master1master2slave1slave2
yarn-env.sh
export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"export JAVA_HOME=/jdk1.7.0_80JAVA=$JAVA_HOME/bin/javaJAVA_HEAP_MAX=-Xmx4096m yarn-site.xml
yarn.resourcemanager.zk-address master1:2181,master2:2181,slave1:2181,slave2:2181 yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.auxservices.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.resourcemanager.address master1:8032 yarn.resourcemanager.scheduler.address master1:8030 yarn.resourcemanager.resource-tracker.address master1:8031 yarn.resourcemanager.admin.address master1:8033 yarn.resourcemanager.webapp.address master1:8088 yarn.nodemanager.resource.memory-mb 2048
4.配置hadoop环境变量
export HADOOP_HOME=/home/hadoop/hadoopexport PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
5.查看hadoop是否安装配置成功
[hadoop@master1 ~]$ hadoop versionHadoop 2.7.1Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git-r 15ecc87ccf4a0228f35af08fc56de536e6ce657aCompiled by jenkins on 2015-06-29T06:04ZCompiled with protoc 2.5.0From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58aThis command was run using/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar
6.分发hadoop文件夹到各个虚拟主机上的hadoop家目录上
scp -r ~/hadoop/ hadoop@master2:~scp -r ~/hadoop/ hadoop@slave1:~scp -r ~/hadoop/ hadoop@slave2:~
7.重新格式化hdfs系统
如果集群刚配置从没启动过，直接执行格式化操作。
如果集群已经格式化了后启动过，则先执行删除旧数据的操作后，在执行格式化操作。
1）删除旧数据
在 hdfs-ste.xml 配置了
dfs.name.dir = /home/hadoop/hdfs/name (namenode上存储hdfs名字空间元数据)
dfs.data.dir = /home/hadoop/hdfs/data (datanode上数据块的物理存储位置)
在core-site.xml中配置了
hadoop.tmp.dir = /home/hadoop/hadoop/tmp（namenode上本地的hadoop临时文件夹）
将各个集群节点这三个文件夹下面的文件和目录全部删除
2）执行格式化命令
hadoop namenode -format
3）格式化日志
DEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.16/04/05 02:02:28 INFO namenode.NameNode: STARTUP_MSG:/***********************************************************STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = master1/192.168.200.165STARTUP_MSG: args = [-format]STARTUP_MSG: version = 2.7.1STARTUP_MSG: classpath =/home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/hadoop/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.7.1-tests.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-nfs-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1-tests.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.jar:/home/hadoop/hadoop/contrib/capacity-scheduler/*.jar:/home/hadoop/hadoop/contrib/capacity-scheduler/*.jarSTARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git-r 15ecc87ccf4a0228f35af08fc56de536e6ce657a; compiled by 'jenkins' on 2015-06-29T06:04ZSTARTUP_MSG: java = 1.7.0_80************************************************************/16/04/05 02:02:28 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]16/04/05 02:02:28 INFO namenode.NameNode: createNameNode [-format]Formatting using clusterid: CID-070fc765-1b22-4453-83ba-7635ea906e1d16/04/05 02:02:29 INFO namenode.FSNamesystem: No KeyProvider found.16/04/05 02:02:29 INFO namenode.FSNamesystem: fsLock is fair:true16/04/05 02:02:29 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=100016/04/05 02:02:29 INFO blockmanagement.DatanodeManager:dfs.namenode.datanode.registration.ip-hostname-check=true16/04/05 02:02:29 INFO blockmanagement.BlockManager:dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.00016/04/05 02:02:29 INFO blockmanagement.BlockManager: The block deletion will start around 2016 Apr 05 02:02:2916/04/05 02:02:29 INFO util.GSet: Computing capacity for map BlocksMap16/04/05 02:02:29 INFO util.GSet: VM type= 64-bit16/04/05 02:02:29 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB16/04/05 02:02:29 INFO util.GSet: capacity = 2^21 = 2097152 entries16/04/05 02:02:30 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false16/04/05 02:02:30 INFO blockmanagement.BlockManager: defaultReplication= 316/04/05 02:02:30 INFO blockmanagement.BlockManager: maxReplication= 51216/04/05 02:02:30 INFO blockmanagement.BlockManager: minReplication= 116/04/05 02:02:30 INFO blockmanagement.BlockManager: maxReplicationStreams= 216/04/05 02:02:30 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks= false16/04/05 02:02:30 INFO blockmanagement.BlockManager: replicationRecheckInterval = 300016/04/05 02:02:30 INFO blockmanagement.BlockManager: encryptDataTransfer= false16/04/05 02:02:30 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 100016/04/05 02:02:30 INFO namenode.FSNamesystem: fsOwner= hadoop (auth:SIMPLE)16/04/05 02:02:30 INFO namenode.FSNamesystem: supergroup= supergroup16/04/05 02:02:30 INFO namenode.FSNamesystem: isPermissionEnabled = true16/04/05 02:02:30 INFO namenode.FSNamesystem: HA Enabled: false16/04/05 02:02:30 INFO namenode.FSNamesystem: Append Enabled: true16/04/05 02:02:30 INFO util.GSet: Computing capacity for map INodeMap16/04/05 02:02:30 INFO util.GSet: VM type= 64-bit16/04/05 02:02:30 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB16/04/05 02:02:30 INFO util.GSet: capacity= 2^20 = 1048576 entries16/04/05 02:02:30 INFO namenode.FSDirectory: ACLs enabled? false16/04/05 02:02:30 INFO namenode.FSDirectory: XAttrs enabled? true16/04/05 02:02:30 INFO namenode.FSDirectory: Maximum size of an xattr: 1638416/04/05 02:02:30 INFO namenode.NameNode: Caching file names occuring more than 10 times16/04/05 02:02:30 INFO util.GSet: Computing capacity for map cachedBlocks16/04/05 02:02:30 INFO util.GSet: VM type= 64-bit16/04/05 02:02:30 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB16/04/05 02:02:30 INFO util.GSet: capacity= 2^18 = 262144 entries16/04/05 02:02:30 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.999000012874603316/04/05 02:02:30 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 016/04/05 02:02:30 INFOnamenode.FSNamesystem: dfs.namenode.safemode.extension= 3000016/04/05 02:02:30 INFO metrics.TopMetrics: NNTop conf:dfs.namenode.top.window.num.buckets = 1016/04/05 02:02:30 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 1016/04/05 02:02:30 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,2516/04/05 02:02:30 INFO namenode.FSNamesystem: Retry cache on namenode is enabled16/04/05 02:02:30 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis16/04/05 02:02:30 INFO util.GSet: Computing capacity for map NameNodeRetryCache16/04/05 02:02:30 INFO util.GSet: VM type= 64-bit16/04/05 02:02:30 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB16/04/05 02:02:30 INFO util.GSet: capacity= 2^15 = 32768 entries16/04/05 02:02:30 INFO namenode.FSImage: Allocated new BlockPoolId: BP-464058956-192.168.200.165-145983615035616/04/05 02:02:30 INFO common.Storage: Storage directory /home/hadoop/hadoop/hdfs/name has been successfully formatted.16/04/05 02:02:30 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 016/04/05 02:02:30 INFO util.ExitUtil: Exiting with status 016/04/05 02:02:30 INFO namenode.NameNode: SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NameNode at master1/192.168.200.165************************************************************/
8.启停hadoop
#启动hadoop集群
注意事项：
1）各个集群节点之间能免密码登录彼此
2）确保各节点配置的myid和zoo.cfg里的一致，并先启动zookeeper集群
启动命令：sbin/start-all.sh 或者依次执行 start-dfs.sh 和 start-yarn.sh
[hadoop@master1 conf]$ start-all.shStarting namenodes on [master1]master1: starting namenode, logging to/home/hadoop/hadoop/logs/hadoop-hadoop-namenode-master1.outslave2: starting datanode, logging to/home/hadoop/hadoop/logs/hadoop-hadoop-datanode-slave2.outmaster2: starting datanode,logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-master2.outmaster1: starting datanode, logging to/home/hadoop/hadoop/logs/hadoop-hadoop-datanode-master1.outslave1: starting datanode, logging to/home/hadoop/hadoop/logs/hadoop-hadoop-datanode-slave1.outStarting secondary namenodes [master2]master2: starting secondarynamenode, logging to/home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-master2.outstarting yarn daemonsstarting resourcemanager, logging to/home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-master1.outslave2: starting nodemanager, logging to/home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-slave2.outmaster2: starting nodemanager, logging to/home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-master2.outmaster1: starting nodemanager, logging to/home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-master1.outslave1: starting nodemanager, logging to/home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-slave1.out
查看相关守护进程
NAMENODE（master1）
[hadoop@master1 logs]$ jps12892 NameNode13003 DataNode13295 ResourceManager13408 NodeManager13826 QuorumPeerMain
SECONDARY NAMENODE（master2）
[hadoop@master2 ~]$ jps10162 SecondaryNameNode10052 DataNode10245 NodeManager4045 QuorumPeerMain
DATANODE（slave1/slave2）
[hadoop@slave1 ~]$ jps13902 NodeManager13789 DataNode9331 QuorumPeerMain [hadoop@slave2 ~]$ jps13697 QuorumPeerMain18324 DataNode18440 NodeManager
#停止hadoop集群
sbin /stop-all.sh
启动成功后可以访问web控制台查看集群信息：
NAMENODE：
SECONDARY NAMENODE：
Nodes Of Cluster（YARN作业管理界面）:
9.启动jobhistoryserver
因为kylin中需要MapReduce任务调度，所以需要启动jobhistoryserver
[hadoop@master1 logs]$ mr-jobhistory-daemon.sh start historyserver
查看jobhistoryserver守护进程
[hadoop@master1 conf]$ jps24419 JobHistoryServer
七、hbase 部署
1.解压缩hbase-1.1.4到hadoop家目录下，进入目录hbase
2.进入conf文件夹
3.修改hbase各配置文件如下：
hbase-env.sh
export JAVA_HOME=/jdk1.7.0_80
regionservers
master1master2slave1slave2
hbase-site.xml（已优化）
hbase.rootdir hdfs://master1:9000/hbase hbase.cluster.distributed true hbase.zookeeper.quorum master1,master2,slave1,slave2 hbase.zookeeper.property.dataDir /zookeeper-3.4.6/data hbase.zookeeper.property.clientPort 2181 hbase.master.info.bindAddress master1 hbase.master.info.port 60010 hbase.master.maxclockskew 200000 Time difference of regionserver from master hbase.coprocessor.user.region.classes org.apache.hadoop.hbase.coprocessor.AggregateImplementation hbase.regionserver.wal.codec org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec hbase.master.loadbalancer.class org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer hbase.coprocessor.master.classes org.apache.phoenix.hbase.index.master.IndexMasterObserver phoenix.query.maxServerCacheBytes 1073741824 phoenix.query.maxGlobalMemoryPercentage 70 hbase.client.scanner.caching 5000 HBase客户端扫描缓存，对查询性能有很大帮助 hbase.rpc.timeout 360000000 zookeeper.session.timeout 60000 zk超时时间 hbase.regionserver.handler.count 50 用户表接受外来请求的线程数 hbase.hregion.max.filesize 107374182400 单个ColumnFamily的region大小，若按照ConstantSizeRegionSplitPolicy策略，超过设置的该值则自动split>（100G） perf.hfile.block.cache.size 0.2 设置读写平衡 hbase.regionserver.global.memstore.size 0.3 RegionServer进程block进行flush触发条件：该节点上所有region的memstore之和达到upperLimit*heapsize hbase.regionserver.global.memstore.lowerLimit 0.3 RegionServer进程触发flush的一个条件：该节点上所有region的memstore之和达到lowerLimit*heapsize hbase.zookeeper.property.tickTime 6000 Client端与zk发送心跳的时间间隔(6秒) hbase.hstore.blockingStoreFiles 10 设置读写平衡 hbase.hstore.blockingWaitTime 90000 block的等待时间（90s） hbase.hregion.memstore.flush.size 104857600 memstore大小，当达到该值则会flush到外存设备（100M） hbase.hregion.memstore.mslab.enabled true 是否开启mslab方案，减少因内存碎片导致的Full GC，提高整体性能 hbase.regionserver.region.split.policy org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy split操作默认的策略 hbase.client.write.buffer 8388608 客户端写buffer，设置autoFlush为false时，当客户端写满buffer才flush（8m） hbase.hregion.memstore.block.multiplier 4 如果memstores超过了flushsize的multiplier倍则会阻塞客户端的写 hbase.regionserver.regionSplitLimit 150 单台RegionServer上region数上限 hbase.regionserver.maxlogs 16 如果memstores超过了flushsize的multiplier倍则会阻塞客户端的写
4.配置环境变量，优化内存
export HBASE_HOME=/home/hadoop/hbaseexport PATH=$PATH:$HBASE_HOME/binexport HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xms1536m -Xmx2048m -Xmn1024m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70-XX:PermSize=512m -XX:MaxPermSize=512m"export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS-Xms1536m -Xmx2048m -Xmn1024m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=70 -XX:PermSize=512m -XX:MaxPermSize=512m"export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xms1024m -Xmx2048m"
5.分发hbase文件夹到各个虚拟主机上的hadoop家目录上
scp -r ~/hbase/ hadoop@master2:~scp -r ~/hbase/ hadoop@slave1:~scp -r ~/hbase/ hadoop@slave2:~
6.启停
bin/start-hbase.shbin/stop-hbase.sh
启动成功信息
[hadoop@master1 hadoop]$ start-hbase.shmaster1: starting zookeeper, logging to/home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-master1.outslave2: starting zookeeper, logging to/home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-slave2.outmaster2: starting zookeeper, logging to/home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-master2.outslave1: starting zookeeper, logging to/home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-slave1.outstarting master, logging to /home/hadoop/hbase/logs/hbase-hadoop-master-master1.outslave2: starting regionserver, logging to/home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-slave2.outmaster1: starting regionserver, logging to/home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-master1.outmaster2: starting regionserver, logging to/home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-master2.outslave1: starting regionserver, logging to/home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-slave1.out
查看相关守护进程
HMASTER（master1）
[hadoop@master1 hadoop]$ jps14230 HMaster14379 HRegionServer
HREGIONSERVER（master2、slave1、slave2）
[hadoop@ master2 ~]$ jps10574 HRegionServer[hadoop@slave1 ~]$ jps14230 HRegionServer[hadoop@slave2 ~]$ jps18753 HRegionServer
出现的问题：集群节点的时间不同步
org.apache.hadoop.hbase.ClockOutOfSyncException
设置时间同步
#yum install ntpdate
# ntpdate 0.asia.pool.ntp.org
#rm -rf /etc/localtime
#ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
查看时间
date +%Y-%m-%d-%H:%M:%S
八、HIVE部署
仅在master1上部署即可
- 解压缩hive-2.0.0到hadoop家目录下，进入目录hive
- 进入conf文件夹
- 在mysql上创建元数据库hive
在master1上创建数据库hive(编码选latin，如果不选择latin，会出现问题)
#为hive数据库授权
grant all on hive.* to 'root'@'%' IDENTIFIED BY 'weidong' with grant option;flush privilege;
#设置mysql数据库为任意IP可连接
update user set host='%' where host='localhost';
4．修改hive各配置文件如下：
hive-site.xml
javax.jdo.option.ConnectionURL jdbc:mysql://master1:3306/hive?createDatabaseIfNotExist=true javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver 驱动名 javax.jdo.option.ConnectionUserName root 用户名 javax.jdo.option.ConnectionPassword weidong 密码 datanucleus.schema.autoCreateTables true hive.metastore.warehouse.dir hdfs://master1:9000/home/hadoop/hive/warehouse 数据路径（相对hdfs） hive.exec.scratchdir hdfs://master1:9000/home/hadoop/hive/warehouse hive.querylog.location /home/hadoop/hive/logs hive.aux.jars.path file:///home/hadoop/hbase/lib hive.metastore.uris thrift://master1:9083 运行hive得主机地址及端口
将日志配置打开
cp hive-log4j2.properties. template hive-log4j2.propertiescp hive-exec-log4j2.properties.template hive-exec-log4j2.properties
5．启动Hive
1）首先需要先启动元数据库
hive --service metastore &
启动成功信息
2016-04-06T11:46:10,157 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:main(5876)) - Starting hive metastore on port 90832016-04-06T11:46:10,210 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(499)) - 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore2016-04-06T11:46:10,299 INFO [main]:metastore.ObjectStore (ObjectStore.java:initialize(318)) - ObjectStore, initialize called2016-04-06T11:46:12,284 INFO [main]:metastore.ObjectStore (ObjectStore.java:getPMF(402)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"2016-04-06T11:46:15,345 INFO [main]:metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:(142)) - Using direct SQL, underlying DB is MYSQL2016-04-06T11:46:15,352 INFO [main]:metastore.ObjectStore (ObjectStore.java:setConf(301)) - Initialized ObjectStore2016-04-06T11:46:16,051 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles_core(586)) - Added admin role in metastore2016-04-06T11:46:16,058 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles_core(595)) - Added public role in metastore2016-04-06T11:46:16,239 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:addAdminUsers_core(635)) - No user is added in admin role, since config is empty2016-04-06T11:46:16,715 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6020)) - Starting DB backed MetaStore Server with SetUGI enabled2016-04-06T11:46:16,729 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6077)) - Started the new metaserver on port [9083]...2016-04-06T11:46:16,729 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6079)) - Options.minWorkerThreads = 2002016-04-06T11:46:16,729 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6081)) - Options.maxWorkerThreads = 10002016-04-06T11:46:16,730 INFO [main]:metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6083)) - TCP keepalive = true
2）启动hive客户端
输入hive命令即可
[hadoop@master conf]$ hiveSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in[jar:file:/home/hadoop/hive/lib/hive-jdbc-2.1.0-SNAPSHOT-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in[jar:file:/home/hadoop/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in[jar:file:/home/hadoop/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.Logging initialized using configuration in file:/home/hadoop/hive/conf/hive-log4j2.propertiesHive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.hive>
检验hive客户端下能否查看/创建表
6．停止HIVE
ps –aux | grep hive 查看hive目前的进程PID，用kill杀掉即可。
7．常用命令
hive> show databases;OKdefaultTime taken: 1.881 seconds, Fetched: 1 row(s)hive> use default;OKTime taken: 0.081 secondshive> create table kylin_test(test_count int);OKTime taken: 2.9 secondshive> show tables;OKkylin_testTime taken: 0.151 seconds, Fetched: 1 row(s)hive> select * from kylin_test;OKTime taken: 0.318 seconds
在hive数据库里查询
九、kylin部署
仅在master1上部署即可
1．了解kylin的两种二进制包
预打包的二进制安装包：
特别二进制包：
说明：特别二进制包是一个在HBase 1.1+环境上编译的Kylin快照二进制包；安装它需要HBase 1.1.3或更高版本，否则之前版本中有一个已知的关于fuzzy key过滤器的缺陷，会导致Kylin查询结果缺少记录:。此外还需注意的是，这不是一个正式的发布版(每隔几周rebase KYLIN 1.3.x 分支上最新的改动)，没有经过完整的测试。
2．解压缩apache-kylin-1.5.0-HBase1.1.3-bin.tar.gz到hadoop家目录下，进入目录kylin
3．在/etc/profile里配置KYLIN环境变量和一个名为hive_dependency的变量
export KYLIN_HOME=/home/hadoop/kylinexport PATH=$PATH:$ KYLIN_HOME/bin export hive_dependency=/home/hadoop/hive/conf:/home/hadoop/hive/lib/*:/home/hadoop/hive/hcatalog/share/hcatalog/hive-hcatalog-core-2.0.0.jar
这个配置需要在从节点master2,slave1,slave2上同时配置，因为kylin提交的任务交给mr后，hadoop集群将任务分发给从节点时，需要hive的依赖信息，如果不配置，则mr任务将报错为: hcatalogXXX找不到。
4.修改kylin的启动脚本kylin.sh
1）显式声明 KYLIN_HOMEexport KYLIN_HOME=/home/Hadoop/kylin 2）在HBASE_CLASSPATH_PREFIX中显示增加$hive_dependency依赖 export HBASE_CLASSPATH_PREFIX=${tomcat_root}/bin/bootstrap.jar:${tomcat_root}/bin/tomcat-juli.jar:${tomcat_root}/lib/*:$hive_dependency:$HBASE_CLASSPATH_PREFIX 5．检查环境是否设置成功
[hadoop@master1 conf]$ check-env.shKYLIN_HOME is set to /home/hadoop/kylin
6．进入conf文件夹，修改kylin各配置文件如下：
kylin.properties
kylin.owner=wdcloud@kylin.apache.orgkylin.rest.servers=master1:7070kylin.hdfs.working.dir=/home/hadoop/kylin/kylin_hdfs_working_dirkylin.job.remote.cli.working.dir=/home/hadoop/kylin/kylin_job_working_dir #定义kylin用于MR jobs的job.jar包和hbase的协处理jar包，用于提升性能。kylin.job.jar=/home/hadoop/kylin/lib/kylin-job-1.5.0-SNAPSHOT.jarkylin.coprocessor.local.jar=/home/hadoop/kylin/lib/kylin-coprocessor-1.5.0-SNAPSHOT.jar
将kylin_hive_conf.xml和kylin_job_conf.xml的副本数设置为4
dfs.replication 4 Block replication
7．启动和停止kylin
#确认必须启动的服务：
1)hadoop2的hdfs/yarn/jobhistory服务
start-dfs.sh
start-yarn.sh
mr-jobhistory-daemon.sh start historyserver
2)hive 元数据库：hive --service metastore &
3)zookeeper
4)hbase ：start-hbase.sh
#检查hive和hbase的依赖
[hadoop@master1 kylin]$ find-hive-dependency.sh[hadoop@master1 kylin]$ find-hbase-dependency.sh
#启动和停止kylin的命令如下：
[hadoop@master1 kylin]$ kylin.sh start[hadoop@master1 kylin]$ kylin.sh stop
Web访问地址
默认的登录username/password 是 ADMIN/KYLIN.
十、kylin测试
1．测试Kylin自带的sample
Kylin提供一个自动化脚本来创建测试CUBE，这个脚本也会自动创建出相应的hive数据表。
运行sample例子的步骤：
① 运行${KYLIN_HOME}/bin/sample.sh脚本
[hadoop@master1 ~]$ sample.sh
关键提示信息：
KYLIN_HOME is set to /home/hadoop/kylinGoing to create sample tables in hive...Sample hive tables are created successfully; Going to create sample cube...Sample cube is created successfully in project 'learn_kylin'; Restart Kylin server or reload the metadata from web UI to see the change.
#在MYSQL中查看此sample创建了哪几张表
# select DB_ID,OWNER,SD_ID,TBL_NAME from TBLS;
#在hive客户端查看创建的表和数据量（1w条）
hive> show tables;OKkylin_cal_dtkylin_category_groupingskylin_salesTime taken: 1.835 seconds, Fetched: 3 row(s)hive> select count(*) from kylin_sales;OK10000Time taken: 65.351 seconds, Fetched: 1 row(s)
② 重启kylin server 刷新缓存
[hadoop@master1 ~]$ kylin.sh stop[hadoop@master1 ~]$ kylin.sh start
③ 使用默认的用户名密码ADMIN/KYLIN访问192.168.200.165:7070/kylin
进入控制台后选择project为learn_kylin的那个项目。
④ 选择测试cube “kylin_sales_cube”，点击“Action”-“Build”，选择一个2014-01-01以后的日期，这是为了选择全部的10000条测试记录。
选择一个生成日期
点击提交会出现重建任务成功提交的提示
⑤ 在监控台查看这个任务的执行进度，直到这个任务100%完成。
任务完成
切换到model控制台会发现cube的状态成为了ready，表示可以执行sql查询了
执行过程中，在hive里会生成临时表，待任务100%完成后，这张表会自动删除
kylin_intermediate_kylin_sales_cube_desc_20120201000000_20120201000000
执行过程中，在hbase里会生成永久的计算结果表，如：
KYLIN_PTQIXMC64A
如果build了两个以上的segment。还可以执行merge操作：
完成Merge任务
这时候HBASE里面不同的Segment表示的多张表也同时合并成了一张表，以节省磁盘空间
在Build过程中出现的问题：
当任务执行到第五步：创建HTable的时候，报错说创建的表不可用。
最终导致整个任务的失败 ERROR
| 2016-04-07 12:40:57,823 ERROR [pool-7-thread-5] steps.CubeHTableUtil:135 : Failed to create HTable java.lang.IllegalArgumentException: table KYLIN_9USQAHQQXC created, but is not available due to some reasons at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92) at org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:132) at org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:104) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
|
在社区提问：
http://apache-kylin.74782.x6.nabble.com/an-error-occurred-when-build-a-sample-cube-at-step-5-create-HTable-td4102.html
出错的原因：
因为在kylin中默认使用了snappy压缩算法导致的。
HDFS报错日志：
| 2016-04-12 12:05:05,726 ERROR [RS_OPEN_REGION-slave2:16020-0] handler.OpenRegionHandler: Failed open of region=KYLIN_VKRC32OKFP,,1460433926913.73fb906719a75b2733f046e87fbe8105., starting to roll back the global memstore size. org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'snappy' previously failed test. at org.apache.hadoop.hbase.util.CompressionTest.testCompression (CompressionTest.java:91) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs (HRegion.java:6300) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6251) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6218) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6189) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6145) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6096) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion (OpenRegionHandler.java:362) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process (OpenRegionHandler.java:129) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-04-12 12:05:05,727 INFO [RS_OPEN_REGION-slave2:16020-0] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => 73fb906719a75b2733f046e87fbe8105, NAME => 'KYLIN_VKRC32OKFP,,1460433926913.73fb906719a75b2733f046e87fbe8105.', STARTKEY => '', ENDKEY => '\x00\x01'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 1 2016-04-12 12:05:05,775 INFO [PriorityRpcServer.handler=18,queue=0,port=16020] regionserver.RSRpcServices: Open KYLIN_VKRC32OKFP,\x00\x01,1460433926913.06978b9fb1e423563a5aae7e1df044d8. |
解决方法：禁用压缩或者使用LZO作为压缩算法
官网给出的禁用压缩算法的方法如下：
To disable compressing MR jobs you need to modify
$KYLIN_HOME/conf/kylin_job_conf.xml by removing all configuration entries related to compression(Just grep the keyword “compress”). To disable compressing hbase tables you need to open $KYLIN_HOME/conf/kylin.properties and remove the line starting with kylin.hbase.default.compression.codec.
⑥ 切换到Insight 窗口执行SQL语句，例如：
| select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by part_dt order by part_dt; |
在Kylin中执行如上的sql统计只用了0.46s (十次取平均值)
在Hive里执行同一条sql统计语句，花费时间高达136秒
hive> select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by part_dt order by part_dt;
Time taken: 136.489 seconds, Fetched: 731 row(s)
可见：kylin执行这条sql明显提升了效率。
其他测试语句：
① select * from kylin_sales;(1s内)
② 各个时间段内的销售额及购买量（0.39秒）
select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers
from kylin_sales
group by part_dt
order by part_dt;
③ 查询某一时间的销售额及购买量（0.40秒）
select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales
where part_dt = '2014-01-01'
group by part_dt;
发现报错：
Error while compiling generated Java code:
public static class Record3_0 implements java.io.Serializable {
public java.math.BigDecimal f0;
public boolean f1;
public org.apache.kylin.common.hll.HyperLogLogPlusCounter f2;
public Record3_0(java.math.BigDecimal f0, boolean f1, ...
这是因为part_dt是date类型，在解析string到date的时候出问题，应将sql语句改为：
select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers
from kylin_sales
where part_dt between '2014-01-01' and '2014-01-01'
group by part_dt;
或者
select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers
from kylin_sales
where part_dt = date '2014-01-01'
group by part_dt;
④上面查询只用到了fact table，而没有用到lookup table。如果查询各个时间段所有二级商品类型的销售额，则需要fact table与lookup table做inner join（1.36s）
select fact.part_dt, lookup.CATEG_LVL2_NAME, count(distinct seller_id) as sellers
from kylin_sales fact
inner join KYLIN_CATEGORY_GROUPINGS lookup
on fact.LEAF_CATEG_ID = lookup.LEAF_CATEG_ID and fact.LSTG_SITE_ID = lookup.SITE_ID
group by fact.part_dt, lookup.CATEG_LVL2_NAME
order by fact.part_dt desc