-
Notifications
You must be signed in to change notification settings - Fork 103
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[INLONG-989][Doc] Add HTTP report usage demo in Quick Start (#1001)
- Loading branch information
1 parent
632202f
commit 45312c8
Showing
13 changed files
with
114 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
{ | ||
"label": "HTTP Report", | ||
"position": 4 | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
--- | ||
title: HTTP Report Example | ||
sidebar_position: 1 | ||
--- | ||
|
||
In the following content, we will use a complete example to introduce how to use HTTP to report data, quickly verify whether the applied {groupId, streamId} is effective, and whether the data is accepted by InLong DataProxy and correctly written to the MQ cluster. | ||
|
||
## Prepare resources | ||
### Apply for InLong group and stream | ||
We need to apply for {groupId, streamId} in InLong Manager first. As shown in the following figure, we have applied for {test_http, test_stream} information and the administrator has approved it: | ||
![prepare group and stream](img/http_group_stream_en.png) | ||
|
||
In the application report stream, we defined that the data of this report stream is reported in CSV format. The data content consists of three fields (ID, Name, Desc) separated by vertical bars ("|"): | ||
![define report stream](img/http_stream_define_en.png) | ||
|
||
### Find the IP and port of the DataProxy node that supports HTTP access | ||
InLong supports direct data reporting via HTTP. In this reporting example, we directly select a DataProxy that supports HTTP reporting from the resource details page of the InLong group to report the message. In the demonstration environment, the HTTP receiving port opened by DataProxy is 47805, as shown below: | ||
![DataProxy information](img/http_dataproxy_en.png) | ||
|
||
At this point, we have obtained the InLong group and stream information required for data reporting, as well as the DataProxy node IP and port information to be reported by HTTP reporting. Next, we can report data through HTTP to verify whether the requested InLong group and stream, pipeline are available. | ||
|
||
## Report data via HTTP | ||
According to the HTTP reporting protocol requirements of InLong, we use curl tool to construct an HTTP instruction as shown below for execution. In the body part, we construct a record containing three field values according to the format definition of test_stream. {dataproxy_ip:dataproxy_httpport} is the DataProxy IP and port for receiving the reported message. You can replace it with the corresponding information in your environment: | ||
|
||
```bash | ||
curl -X POST -d 'groupId=test_http&streamId=test_stream&dt=data_time&body=1|name_1|desc_record_one&cnt=1' http://{dataproxy_ip:dataproxy_httpport}/dataproxy/message | ||
``` | ||
- Parameter Description: | ||
|
||
| parameter | meaning | Remark | | ||
|-----------|---------------------------------------|---------| | ||
| groupId | Data stream group id | | | ||
| streamId | Data stream ID | | | ||
| body | Data content to be pushed | | | ||
| dt | Data time to be pushed |timestamp in millisecond | | ||
| cnt | The count of data pieces to be pushed | | | ||
|
||
- Return Value: | ||
|
||
| return value | meaning | | ||
|--------------|---------| | ||
| 0 | Success | | ||
| !=0 | Failure | | ||
|
||
## Check the reported data | ||
We use the data preview function of the data stream test_http:test_stream to view the HTTP reporting status. This function directly samples the latest data from the MQ cluster corresponding to the data stream: | ||
![data_preview](img/http_data_preview_en.png) | ||
|
||
We can see that the data just reported has been successfully written to the MQ cluster: | ||
![viewed_data](img/http_data_view_en.png) | ||
|
||
At this point, we quickly and clearly know that the requested InLong group and stream, as well as the pipeline resources, are all available. | ||
|
||
## FAQ | ||
The return code and error message in the HTTP response will clearly indicate the specific cause of the error, such as the group or stream does not exist, the reporting protocol format is not equal, etc., and the problem can be quickly solved by adjusting according to the corresponding error prompt or aligning with the system administrator. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
55 changes: 55 additions & 0 deletions
55
...usaurus-plugin-content-docs/current/quick_start/data_http_report/http_report.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
--- | ||
title: HTTP 上报示例 | ||
sidebar_position: 1 | ||
--- | ||
|
||
在下面的内容中,我们将通过一个完整的示例介绍如何使用 HTTP 上报数据,快速验证申请的 {groupId, streamId} 是否已生效,数据是否被 InLong DataProxy 接纳并正确地写入到 MQ 集群。 | ||
|
||
## 准备上报资源 | ||
### 申请 InLong group 和 stream | ||
我们需要先在 InLong 的 Manager 里申请上报所需的 {groupId, streamId} 信息,如下图示,我们已经申请了 {test_http, test_stream} 信息并且管理员已审批通过: | ||
![group 和 stream 准备](img/http_group_stream_cn.png) | ||
|
||
在申请的上报流里,我们定义了该上报流的数据采用 CSV 格式上报数据,数据内容由三个字段(ID,Name,Desc)组成,并以竖线("|")分隔: | ||
![数据流定义](img/http_stream_define_cn.png) | ||
|
||
### 查找支持 HTTP 接入的 DataProxy 节点 IP 及端口号 | ||
InLong 支持通过 HTTP 直接上报数据,这个上报示例里我们直接从 InLong group 的资源详情页面选一台支持 HTTP 上报的 DataProxy 进行消息上报。在演示环境里 DataProxy 开启的 HTTP 接收端口是 47805,如下图示: | ||
![DataProxy 信息](img/http_dataproxy_cn.png) | ||
|
||
至此,我们已经获得了数据上报所需要的 InLong group 和 stream 信息,以及 HTTP 上报所要上报的 DataProxy 节点 IP 及端口信息,接下来我们就可以通过 HTTP 上报数据,验证申请的 InLong group 和 stream,以及管道是否可用。 | ||
|
||
## 通过 HTTP 上报数据 | ||
按照 InLong 的 HTTP 上报协议要求,我们用 curl 构造一个如下示例的 HTTP 指令进行执行,其中 body 部分我们按照 test_stream 的格式定义构造包含 3 个字段值的记录, {dataproxy_ip:dataproxy_httpport} 为接收上报消息的 DataProxy IP 及端口,大家可以替换为各自环境里的对应信息: | ||
```bash | ||
curl -X POST -d 'groupId=test_http&streamId=test_stream&dt=data_time&body=1|name_1|desc_record_one&cnt=1' http://{dataproxy_ip:dataproxy_httpport}/dataproxy/message | ||
``` | ||
|
||
- 参数说明: | ||
|
||
| 参数 | 含义 | 备注 | | ||
|----------|----------|-----| | ||
| groupId | 数据流组 id | | | ||
| streamId | 数据流 ID | | | ||
| body | 推送的数据内容 | | | ||
| dt | 推送的数据时间 |毫秒为单位的时间戳 | | ||
| cnt | 推送条数 | | | ||
|
||
- 返回值: | ||
|
||
| 返回码 | 含义 | | ||
|-----|-----| | ||
| 0 | 成功 | | ||
| 非0 | 失败 | | ||
|
||
## 检查数据上报情况 | ||
我们通过数据流 test_http:test_stream 的数据预览功能查看 HTTP 上报情况,该功能直接从数据流对应的 MQ 集群抽检最新数据: | ||
![数据预览](img/http_data_preview_cn.png) | ||
|
||
我们可以看到刚才上报的数据已经成功写入 MQ 集群: | ||
![数据呈现](img/http_data_view_cn.png) | ||
|
||
到这里,我们就可以快速、清楚地知道申请的 InLong group 和 stream,以及管道资源已经可用。 | ||
|
||
## 常见问题 | ||
HTTP 上报的返回结果里返回码及错误信息会很清楚指出具体错误原因,比如 group 或 stream 不存在,上报协议格式不对等,按照对应的错误提示进行调整或者与系统管理员进行对齐即可快速解决问题。 |
Binary file added
BIN
+62.1 KB
...-content-docs/current/quick_start/data_http_report/img/http_data_preview_cn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+20.9 KB
...gin-content-docs/current/quick_start/data_http_report/img/http_data_view_cn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+28.4 KB
...gin-content-docs/current/quick_start/data_http_report/img/http_dataproxy_cn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+48 KB
...-content-docs/current/quick_start/data_http_report/img/http_group_stream_cn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+38.8 KB
...content-docs/current/quick_start/data_http_report/img/http_stream_define_cn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.