---
title: How data is processed
description: Follow how Coralogix ingests data, transforms and routes it with DataPrime rules, and writes it into dataspaces and datasets.
url: https://coralogix.com/docs/user-guides/data-layer/data-processing/
product: dataengine
feature: Data Engine Overview
feature_id: dataengine-general
feature_identifier: dataengine/dataengine-general
last_updated: '2026-11-02'
---

# Processing and routing

When data enters Coralogix, it goes through a structured lifecycle: received from shippers or agents, transformed with DataPrime rules, routed based on attributes like region or team, and directed into the appropriate dataspace and dataset. If a dataset doesn't already exist, it's created automatically and inherits configuration from the parent dataspace.

## High-level flow

<!-- -->

1. **Ingress**

   Data enters the platform through a shipper or agent. Customers can pre-define a `targetDatabase` (dataspace) and `targetDataset` via the shipper config.

2. **Pre-processing**

   Coralogix applies DataPrime transformation rules:

   For example, fields can be removed or recalculated before finalizing the data structure:

   ```
   remove derived_metric

   | replace raw_value with normalized_value

   | create derived_metric from quantity * 232
   ```

3. **Routing**

   A set of conditions (e.g., region, team, environment) determine where data goes:

   For example, different regions or teams in the data can determine the target dataspace or dataset.

   ```
   <region == 'us2'>       ->      [targetDataspace = bu1, targetDataset = logs-us]

   <team == 'neptune'>     ->      [targetDataspace = planet, targetDataset = gassy]

   <team == 'venus'>       ->      [targetDataspace = planet, targetDataset = rocky]
   ```

   Routing is fully **data-driven** and can include dynamic elements:

   ```
   <region>                 ->     [targetDataspace = bu2, targetDataset = logs-{$l.applicationname}]
   ```

4. **Dataset creation**

   If a dataset does not already exist, it will be created automatically under the target dataspace.

5. **Configuration inheritance**

   The dataset inherits configuration from its dataspace, including:

   * Storage prefix (e.g., `s3://bucket/my-dataspace/logs-regionX`)
   * Retention and archive rules
   * Access control policies
   * Metadata enrichment

6. **Final storage & query**

   Once routed and processed, the data is written to object storage and made available for querying.

[![data ingestion](/docs/assets/images/ingestion-dec6a190e860148feba562d3dfdc23ef.webp)](https://coralogix.com/docs/assets/images/ingestion-dec6a190e860148feba562d3dfdc23ef.webp)

***

## Example dataset structure

```
default/

  └── logs

  └── spans



business-unit1/

  └── logs



business-unit2/

  └── logs-cx510

  └── logs-euprod2

  └── logs-production

  └── ...

  └── <datasets created dynamically as data arrives>



security/

  └── ...
```

***

## Querying across datasets

You can **query multiple datasets at once** by listing them as comma-separated sources:

```
source dataset-1, dataset-2 | ...
```

The query reads every dataset you list and concatenates the results, which makes summarization, comparison, and rollups across logical groups a single query rather than a chain of unions. Every dataset has to be in the same region, and a query can reference up to 128 sources. See [`source`](https://coralogix.com/docs/dataprime/language-reference/commands-reference/sources/source.md) for the full syntax, including the `teamId` and `version` parameters.

<!-- -->

<!-- -->

***

## Handling quota and duplication

* Duplicating data across datasets (e.g., routing the same event to multiple targets) **will count against your quota**.
* You can monitor dataset-level usage in **Dataset Management**.
* Dataset quotas can be enforced per team, space, or workload.
* The [data usage](https://coralogix.com/docs/user-guides/account-management/payment-and-billing/data-usage.md) page shows detailed breakdowns to help you understand where and how your data is being consumed.