Data Processing Introduction
Implementation engineers can use custom integrations to ingest data from cloud services or external monitoring systems into Bonree ONE. After creating an ingestion task, check or adjust its data flow to process incoming data for the required queries and monitoring scenarios.
Use Integrations to select an ingestion method and configure the data source. Use Data Processing to manage ingestion tasks and their processing flows. Custom integrations connect these two stages: configure ingestion on the Integrations page, then manage the corresponding task and data flow on the Data Processing page.
Decide When to Use Data Processing
Identify the data source and processing requirements before choosing an ingestion method and processing flow.
Ingest monitoring metrics from cloud services
To monitor cloud services such as AWS EC2, find the corresponding integration and configure a collection task. Then check the data flow and collected metrics to confirm that data from the target resources is available for subsequent queries.
Prepare data from external monitoring systems
After connecting an external system, configure components in the task's data flow to parse raw fields, transform field values, or adjust the output. Check that the output meets the requirements of subsequent analysis.
Change how an existing task processes data
If an existing ingestion task does not meet your processing requirements, edit or copy its data flow, publish the required version, and verify how it processes subsequent data.
Understand the Relationship Between Ingestion and Processing
| Concept | Role in the workflow |
|---|---|
| Data source | Provides raw observability data, such as Prometheus, Zabbix, or a cloud service. |
| Custom integration | Provides an ingestion method for an external data source or a specific data governance scenario. |
| Data task | Manages data ingestion and processing, and provides access to the data flow. |
| Data flow | Parses, transforms, and outputs incoming data. Only one version can be deployed for a task at a time. |
| Integration resources | Metrics, dashboards, alert templates, and other resources provided by an integration. The available resources depend on the integration. |
Start Ingestion and Processing with a Custom Integration
For a new integration, follow these steps from selecting a data source to verifying the data. For an existing task, go directly to data flow management.
- Select an ingestion method: Identify the target integration, collection requirements, and available resources.
- Create an ingestion task: Enter the task and collection settings, save the configuration, and check the data task list.
- Check or adjust the data flow: Verify the processing components and their execution order. Publish the required version if changes are needed.
- Verify the collected data: Check the task status, deployed version, and target data before using the associated monitoring resources.
For the complete procedure, see Create a Data Task with a Custom Integration. For installation and ongoing maintenance of integration resources, see Manage Custom Integration Resources.
Continue Based on Your Goal
Choose the next document based on whether you need to plan, configure, or troubleshoot.
- Best Practices: Read Plan Ingestion and Data Flow Changes to understand metric dependencies and how to decide which version to use.
- User Guide: For initial ingestion, read Create a Data Task. For existing tasks, read Task Management. For preset resources, read Manage Custom Integration Resources.
- FAQ: Read Troubleshoot Task Creation and Data Flow Changes to choose the next check based on the symptoms you observe.