Skip to content

Commit

Permalink
[INLONG-989][Doc] Add HTTP report usage demo in Quick Start (#1001)
Browse files Browse the repository at this point in the history
  • Loading branch information
gosonzhang authored Sep 28, 2024
1 parent 632202f commit 45312c8
Show file tree
Hide file tree
Showing 13 changed files with 114 additions and 0 deletions.
4 changes: 4 additions & 0 deletions docs/quick_start/data_http_report/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"label": "HTTP Report",
"position": 4
}
55 changes: 55 additions & 0 deletions docs/quick_start/data_http_report/http_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
title: HTTP Report Example
sidebar_position: 1
---

In the following content, we will use a complete example to introduce how to use HTTP to report data, quickly verify whether the applied {groupId, streamId} is effective, and whether the data is accepted by InLong DataProxy and correctly written to the MQ cluster.

## Prepare resources
### Apply for InLong group and stream
We need to apply for {groupId, streamId} in InLong Manager first. As shown in the following figure, we have applied for {test_http, test_stream} information and the administrator has approved it:
![prepare group and stream](img/http_group_stream_en.png)

In the application report stream, we defined that the data of this report stream is reported in CSV format. The data content consists of three fields (ID, Name, Desc) separated by vertical bars ("|"):
![define report stream](img/http_stream_define_en.png)

### Find the IP and port of the DataProxy node that supports HTTP access
InLong supports direct data reporting via HTTP. In this reporting example, we directly select a DataProxy that supports HTTP reporting from the resource details page of the InLong group to report the message. In the demonstration environment, the HTTP receiving port opened by DataProxy is 47805, as shown below:
![DataProxy information](img/http_dataproxy_en.png)

At this point, we have obtained the InLong group and stream information required for data reporting, as well as the DataProxy node IP and port information to be reported by HTTP reporting. Next, we can report data through HTTP to verify whether the requested InLong group and stream, pipeline are available.

## Report data via HTTP
According to the HTTP reporting protocol requirements of InLong, we use curl tool to construct an HTTP instruction as shown below for execution. In the body part, we construct a record containing three field values according to the format definition of test_stream. {dataproxy_ip:dataproxy_httpport} is the DataProxy IP and port for receiving the reported message. You can replace it with the corresponding information in your environment:

```bash
curl -X POST -d 'groupId=test_http&streamId=test_stream&dt=data_time&body=1|name_1|desc_record_one&cnt=1' http://{dataproxy_ip:dataproxy_httpport}/dataproxy/message
```
- Parameter Description:

| parameter | meaning | Remark |
|-----------|---------------------------------------|---------|
| groupId | Data stream group id | |
| streamId | Data stream ID | |
| body | Data content to be pushed | |
| dt | Data time to be pushed |timestamp in millisecond |
| cnt | The count of data pieces to be pushed | |

- Return Value:

| return value | meaning |
|--------------|---------|
| 0 | Success |
| !=0 | Failure |

## Check the reported data
We use the data preview function of the data stream test_http:test_stream to view the HTTP reporting status. This function directly samples the latest data from the MQ cluster corresponding to the data stream:
![data_preview](img/http_data_preview_en.png)

We can see that the data just reported has been successfully written to the MQ cluster:
![viewed_data](img/http_data_view_en.png)

At this point, we quickly and clearly know that the requested InLong group and stream, as well as the pipeline resources, are all available.

## FAQ
The return code and error message in the HTTP response will clearly indicate the specific cause of the error, such as the group or stream does not exist, the reporting protocol format is not equal, etc., and the problem can be quickly solved by adjusting according to the corresponding error prompt or aligning with the system administrator.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
title: HTTP 上报示例
sidebar_position: 1
---

在下面的内容中,我们将通过一个完整的示例介绍如何使用 HTTP 上报数据,快速验证申请的 {groupId, streamId} 是否已生效,数据是否被 InLong DataProxy 接纳并正确地写入到 MQ 集群。

## 准备上报资源
### 申请 InLong group 和 stream
我们需要先在 InLong 的 Manager 里申请上报所需的 {groupId, streamId} 信息,如下图示,我们已经申请了 {test_http, test_stream} 信息并且管理员已审批通过:
![group 和 stream 准备](img/http_group_stream_cn.png)

在申请的上报流里,我们定义了该上报流的数据采用 CSV 格式上报数据,数据内容由三个字段(ID,Name,Desc)组成,并以竖线("|")分隔:
![数据流定义](img/http_stream_define_cn.png)

### 查找支持 HTTP 接入的 DataProxy 节点 IP 及端口号
InLong 支持通过 HTTP 直接上报数据,这个上报示例里我们直接从 InLong group 的资源详情页面选一台支持 HTTP 上报的 DataProxy 进行消息上报。在演示环境里 DataProxy 开启的 HTTP 接收端口是 47805,如下图示:
![DataProxy 信息](img/http_dataproxy_cn.png)

至此,我们已经获得了数据上报所需要的 InLong group 和 stream 信息,以及 HTTP 上报所要上报的 DataProxy 节点 IP 及端口信息,接下来我们就可以通过 HTTP 上报数据,验证申请的 InLong group 和 stream,以及管道是否可用。

## 通过 HTTP 上报数据
按照 InLong 的 HTTP 上报协议要求,我们用 curl 构造一个如下示例的 HTTP 指令进行执行,其中 body 部分我们按照 test_stream 的格式定义构造包含 3 个字段值的记录, {dataproxy_ip:dataproxy_httpport} 为接收上报消息的 DataProxy IP 及端口,大家可以替换为各自环境里的对应信息:
```bash
curl -X POST -d 'groupId=test_http&streamId=test_stream&dt=data_time&body=1|name_1|desc_record_one&cnt=1' http://{dataproxy_ip:dataproxy_httpport}/dataproxy/message
```

- 参数说明:

| 参数 | 含义 | 备注 |
|----------|----------|-----|
| groupId | 数据流组 id | |
| streamId | 数据流 ID | |
| body | 推送的数据内容 | |
| dt | 推送的数据时间 |毫秒为单位的时间戳 |
| cnt | 推送条数 | |

- 返回值:

| 返回码 | 含义 |
|-----|-----|
| 0 | 成功 |
| 非0 | 失败 |

## 检查数据上报情况
我们通过数据流 test_http:test_stream 的数据预览功能查看 HTTP 上报情况,该功能直接从数据流对应的 MQ 集群抽检最新数据:
![数据预览](img/http_data_preview_cn.png)

我们可以看到刚才上报的数据已经成功写入 MQ 集群:
![数据呈现](img/http_data_view_cn.png)

到这里,我们就可以快速、清楚地知道申请的 InLong group 和 stream,以及管道资源已经可用。

## 常见问题
HTTP 上报的返回结果里返回码及错误信息会很清楚指出具体错误原因,比如 group 或 stream 不存在,上报协议格式不对等,按照对应的错误提示进行调整或者与系统管理员进行对齐即可快速解决问题。
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 45312c8

Please sign in to comment.