CDH6.2.0安装并使用基于HBase的Geomesa

V2AS问路

CDH6.2.0安装并使用基于HBase的Geomesa

阅读原文时间：2023年07月09日阅读：5

1. 查看CDH 安装的hadoop 和 hbase 对应的版本

具体可以参考以下博客：

https://www.cxyzjd.com/article/spark_Streaming/108762904

直接给出答案

hadoop 版本 3.0.0
hbase 版本 2.1.0

2. 在 github 找到其hbase版本对应的geomesa的版本

github 地址:

https://github.com/locationtech/geomesa/releases

这里我使用 3.1.1 的版本进行安装

进入安装了 hbase 的服务器（假设hbase已安装完毕），执行以下命令：

# 如果没有这个目录，自行创建
cd /opt/software
# 拉取安装包
wget "https://github.com/locationtech/geomesa/releases/download/geomesa_2.11-3.1.1/geomesa-hbase_2.11-3.1.1-bin.tar.gz"

# 解压缩， 如果没有这个目录，自行创建
tar -xvf geomesa-hbase_2.11-3.1.1-bin.tar.gz  -C /opt/module

3. 将解压缩目录同步到各个hbase大数据节点

cd /opt/module
# xsync 脚本 之前的博客里有，自行copy，或者scp -r也可以
xsync geomesa-hbase_2.11-3.1.1

4. 配置相关环境变量，修改/etc/profile，增加以下内容

export GEOMESA_HBASE_HOME=/opt/module/geomesa-hbase_2.11-3.1.1
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export HBASE_HOME=/opt/cloudera/parcels/CDH/lib/hbase
export PATH="${PATH}:${GEOMESA_HBASE_HOME}/bin"

记得最后source /etc/profile 一下

5. 安装 GeoMesa HBase分布式运行时JAR（重要一步）

cp $GEOMESA_HBASE_HOME/dist/hbase/geomesa-hbase-distributed-runtime-hbase2_2.11-3.1.1.jar  ${HBASE_HOME}/lib

每台hbase节点都要操作

PS: 官网上的方法我没有跑通，最后推测原因是我用了最新版的tutorial，与运行时的jar版本不匹配导致，但最后没有再验证，这里仅记录我跑通的方案

6. 注册协处理器

修改hbase配置文件hbase-site.xml

位置

/opt/cloudera/parcels/CDH/lib/hbase/conf/hbase-site.xml

在configuration标签中添加：

<property>
    <name>hbase.coprocessor.user.region.classes</name>
    <value>org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor</value>
</property>

每台hbase节点都要操作，或者xsync分发该配置文件到每个节点

PS: 官网上的方法我没有跑通，最后推测原因是我用了最新版的tutorial，与运行时的jar版本不匹配导致，但最后没有再验证，这里仅记录我跑通的方案

7. 验证

command line 验证

geomesa-hbase

GeoMesa tools version: 3.1.1
Commit ID: d24fbb48458fa609dcea5b2f95d987674d19eefa
Branch: d24fbb48458fa609dcea5b2f95d987674d19eefa
Build date: 2021-04-26T12:21:01+0000

$GEOMESA_HBASE_HOME/bin/geomesa-hbase ingest -c example-csv -s example-csv -C example-csv $GEOMESA_HBASE_HOME/examples/ingest/csv/example.csv

#执行ok

hbase shell 后 list 可以看到表已创建

TABLE
example-csv
example-csv_example_2dcsv_attr_name_geom_lastseen_v8
example-csv_example_2dcsv_id_v4
example-csv_example_2dcsv_z2_geom_v5
example-csv_example_2dcsv_z3_geom_lastseen_v6
代码验证
1. 拉取官方的tutorial代码
 
 地址：git clone https://github.com/geomesa/geomesa-tutorials.git
 
 这里有两个小坑
 
 【1】首先要切换到和自己运行时JAR对应的版本上去（没找到3.1.1，切到tag 3.1.0上）
 
 git checkout geomesa-tutorials-3.1.0
 
 【2】pom里指定的hadoop版本和hbase版本不对，需要改成我们对应的版本
 
 【3】为了加快拉取依赖，注释掉不需要的module
2. 构建项目
 
 mvn clean install -pl geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart -am
3. 将jar包丢到hbase对应的服务器上，执行
 
 java -cp geomesa-tutorials-hbase-quickstart-3.1.0.jar org.geomesa.example.hbase.HBaseQuickStart --hbase.zookeepers hadoop1,hadoop2,hadoop3 --hbase.catalog geomesa_grady
可以查看到正确日志，以下为日志：

[hadoop@hadoop1 test]# java -cp geomesa-tutorials-hbase-quickstart-3.1.0.jar org.geomesa.example.hbase.HBaseQuickStart --hbase.zookeepers hadoop1,hadoop2,hadoop3 --hbase.catalog geomesa_grady
Loading datastore
Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

Creating schema: GLOBALEVENTID:String,Actor1Name:String,Actor1CountryCode:String,Actor2Name:String,Actor2CountryCode:String,EventCode:String,NumMentions:Integer,NumSources:Integer,NumArticles:Integer,ActionGeo_Type:Integer,ActionGeo_FullName:String,ActionGeo_CountryCode:String,dtg:Date,geom:Point:srid=4326
The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.
Table geomesa_grady not enabled, it is not exists
Table geomesa_grady not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_z3_geom_dtg_v6 not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_z3_geom_dtg_v6 not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_z2_geom_v5 not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_z2_geom_v5 not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_z2_geom_v5 not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_id_v4 not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_id_v4 not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_attr_EventCode_geom_dtg_v8 not enabled, it is not exists
Table geomesa_grady_gdelt_2dquickstart_attr_EventCode_geom_dtg_v8 not enabled, it is not exists

Generating test data

Writing test data
Wrote 2356 features

Running query BBOX(geom, -120.0,30.0,-75.0,55.0) AND dtg DURING 2017-12-31T00:00:00+00:00/2018-01-02T00:00:00+00:00
01 719025288=719025288|CANADA|CAN|||190|4|1|4|4|Penticton, British Columbia, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-119.583 49.5)
02 719025945=719025945|DEPARTMENT OF EDUCATION||OBAMA|USA|090|20|2|20|2|North Dakota, United States|US|2018-01-01T00:00:00.000Z|POINT (-99.793 47.5362)
03 719025853=719025853|PRESIDENT||MEXICO|MEX|0874|3|3|3|2|Minnesota, United States|US|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)
04 719025856=719025856|PRESIDENT||MEXICO|MEX|131|3|3|3|2|Minnesota, United States|US|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)
05 719026960=719026960|UNITED STATES|USA|||071|6|1|6|2|Minnesota, United States|US|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)
06 719024925=719024925|||CORPORATION||172|2|1|2|4|Saskatoon, Saskatchewan, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)
07 719025289=719025289|CANADA|CAN|CORPORATION||115|6|1|6|4|Saskatoon, Saskatchewan, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)
08 719025290=719025290|CANADA|CAN|CORPORATION||172|8|1|8|4|Saskatoon, Saskatchewan, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)
09 719025276=719025276|SASKATCHEWAN|CAN|||020|20|1|10|4|Saskatchewan, Manitoba, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-97.7167 52.15)
10 719025946=719025946|PREMIER||||015|6|1|6|4|Saskatchewan, Manitoba, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-97.7167 52.15)

Returned 669 total features

Running query BBOX(geom, -120.0,30.0,-75.0,55.0) AND dtg DURING 2017-12-31T00:00:00+00:00/2018-01-02T00:00:00+00:00
Returning attributes [GLOBALEVENTID, dtg, geom]
01 719025288=719025288|2018-01-01T00:00:00.000Z|POINT (-119.583 49.5)
02 719025945=719025945|2018-01-01T00:00:00.000Z|POINT (-99.793 47.5362)
03 719025853=719025853|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)
04 719025856=719025856|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)
05 719026960=719026960|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)
06 719024925=719024925|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)
07 719025289=719025289|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)
08 719025290=719025290|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)
09 719025276=719025276|2018-01-01T00:00:00.000Z|POINT (-97.7167 52.15)
10 719025946=719025946|2018-01-01T00:00:00.000Z|POINT (-97.7167 52.15)

Returned 669 total features

Running query EventCode = '051'
01 719025634=719025634|FIJI|FJI|||051|2|1|2|1|Fiji|FJ|2018-01-01T00:00:00.000Z|POINT (178 -18)
02 719027116=719027116|UNITED STATES|USA|KING||051|8|1|8|3|San Diego, California, United States|US|2018-01-01T00:00:00.000Z|POINT (-117.157 32.7153)
03 719027117=719027117|LOS ANGELES|USA|KING||051|26|3|26|2|California, United States|US|2018-01-01T00:00:00.000Z|POINT (-119.746 36.17)
04 719025036=719025036|||SENATE||051|5|1|5|2|Alabama, United States|US|2018-01-01T00:00:00.000Z|POINT (-86.8073 32.799)
05 719026318=719026318|SENATE||UNITED STATES|USA|051|2|1|2|2|Alabama, United States|US|2018-01-01T00:00:00.000Z|POINT (-86.8073 32.799)
06 719026296=719026296|CHAMBER||||051|20|1|20|3|Springfield, South Carolina, United States|US|2018-01-01T00:00:00.000Z|POINT (-81.2793 33.4968)
07 719025744=719025744|CIVIL SERVANT||||051|20|1|10|3|White House, District of Columbia, United States|US|2018-01-01T00:00:00.000Z|POINT (-77.0364 38.8951)
08 719026946=719026946|INDIANA|USA|||051|12|1|12|2|Indiana, United States|US|2018-01-01T00:00:00.000Z|POINT (-86.2604 39.8647)
09 719025279=719025279|EDMONTON|CAN|||051|5|1|5|4|Dufferin County, Ontario, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-80.1667 44.0833)
10 719027252=719027252|AMERICAN CIVIL LIBERTIES UNION|USA|||051|2|1|2|2|Pennsylvania, United States|US|2018-01-01T00:00:00.000Z|POINT (-77.264 40.5773)

Returned 138 total features

Running query EventCode = '051' AND dtg DURING 2017-12-31T00:00:00+00:00/2018-01-02T00:00:00+00:00
Returning attributes [GLOBALEVENTID, dtg, geom]
01 719025248=719025248|2018-01-01T00:00:00.000Z|POINT (135 -25)
02 719026319=719026319|2018-01-01T00:00:00.000Z|POINT (-97.6475 31.106)
03 719027258=719027258|2018-01-01T00:00:00.000Z|POINT (-97.6475 31.106)
04 719026945=719026945|2018-01-01T00:00:00.000Z|POINT (-119.746 36.17)
05 719027115=719027115|2018-01-01T00:00:00.000Z|POINT (-121.494 38.5816)
06 719025112=719025112|2018-01-01T00:00:00.000Z|POINT (-97.3375 37.6922)
07 719025110=719025110|2018-01-01T00:00:00.000Z|POINT (-81.9296 33.7896)
08 719026297=719026297|2018-01-01T00:00:00.000Z|POINT (-77.0364 38.8951)
09 719026943=719026943|2018-01-01T00:00:00.000Z|POINT (-87.6501 41.85)
10 719026364=719026364|2018-01-01T00:00:00.000Z|POINT (-77.264 40.5773)

Returned 138 total features

Done