# Search Query Language

Logpoint’s Query Language is extensive, intuitive, and user-friendly. It covers all the search commands, functions, arguments, and clauses. You can search the log messages in various formats depending on the query you use.

Logpoint also supports chaining of commands and multi-line queries. Use a pipe (|) to chain the commands and press **Shift + Enter** to add a new line in the query. The search keywords are not case-sensitive. You can also add comments when entering a search query to provide descriptive notes for your query.

Syntax:

```
/* Comment */
```

### Simple Search

You can use the following types of simple queries to familiarize yourself with the Logpoint Query Language.

{% hint style="info" %}
The order of the words is not essential.
{% endhint %}

#### Single word

Single word search is the most basic search that you can run in Logpoint. Enter a **single word** in the *Query Bar* to retrieve the logs containing the word.

```
login
```

This query searches for all logs containing the word **login**.

#### Multiple words

Searching with multiple words lets you search the original logs using a combination of words. For multi-word searches, only logs containing all the words are displayed.&#x20;

```
account locked
```

This query searches for all logs that contain both the search terms **account** and **locked**.

#### Phrases

**Phrase Search** lets you search for the exact phrase in the logs. You must enclose the words inside double-quotes (” “).

```
"account locked"
```

This query searches for all logs containing the exact phrase **account locked**.

#### Field values

The normalized logs contain information in key-value pairs. You can use these pairs directly in the log search. To see all the logs from the user **Bob**, use the following query:

```
user = Bob
```

This query retrieves all logs for the user **Bob**.

```
device_ip = 192.168.2.1
```

This query searches for all the logs coming from the device with the IP Address **192.168.2.1**.\
You can combine multiple field value pairs as:

```
device_ip = 192.168.2.1 sig_id = 10051
```

You can also combine this with a simple query as:

```
login device_ip = 192.168.2.1 sig_id = 10051
```

#### Values with Quotes

When searching for values that are wrapped in quotation marks (” or ‘), add extra quotes around your search term. This is necessary because the system strips away the outermost quotes during processing.

For example, if you want to find logs containing the exact phrase **value**, including the quotes, writing the query as follows will search for the term **value**, without quotes.

```
field = "value"
```

To search for a value including quotes, add extra quotes as shown below:

```
field = ""value""
```

This instructs the system to treat the quotation marks as part of the value.

{% hint style="info" %}
If the value in the normalized field contains a combination of single (‘) and double (“) quotes, the above method will not return a match. In such cases, the system cannot interpret mixed-quote patterns using the above search syntax. For example, field = ""val'ue"" won’t search for logs with "val'ue".
{% endhint %}

### Logical Operators

Use the following keywords to perform logical operations in the Logpoint search query.

#### And

Use the logical AND operator to search for slogs that contain both specified parameters.

```
login and successful
```

This query searches for all logs containing the words "login" and "successful".

The **and** operator can also be used for key-value search queries as follows:

```
login and device_ip=192.168.2.2
```

#### Or

Use the logical operator **or** to search for the logs containing either of the specified parameters.

```
login or logout
```

This query searches for all logs containing either the word *login* **or** the word *logout*.

This operator can also be used with the key-value search query as follows:

```
device_ip = 192.168.2.1 or device_ip = 127.0.0.1
```

#### Not

Use the hyphen (-) symbol for the logical negation in your searches.

```
login -Bob
```

This query searches for logs containing the word **login** but not the word **Bob**.

```
-device_ip = 192.168.2.243
```

This query returns logs that contain all **device\_ip**s except **192.168.2.243**.

{% hint style="info" %}

* While searching with field-names, you can also use **!=** and **NOT** to denote negation.

  ```
  device_ip != 192.168.2.243
  ```

  ```
  NOT device_ip = 192.168.2.243
  ```
* By default, the **or** operator binds strongly than the **and** operator. When performing the **login or logout and MSWinEventLog**, Logpoint returns logs containing either **login** or **logout** with **MsWinEventLog**.
  {% endhint %}

#### Parentheses

In Logpoint, the **or** operator has a higher precedence by default. Use parentheses to override the default binding when using the logical operators in the search query.

```
"login failed" or (denied and locked)
```

This query returns logs containing *login failed* or both *denied* **and** *locked*.

#### Wildcards

Use wildcards to replace parts of the query string. Use the following characters as wildcards:

* **?** - Replacement for a single character.
* **\*** - Replacement for multiple characters.

For logs containing the word **login** or **logon**, use:

```
log?n
```

This query also searches for the log messages containing other variations such as **logan**, **logbn**, and **logcn**.\
The following query returns the logs containing the words starting with **log,** such as **logpoint**, **logout**, and **login**.

```
log*
```

You can also use *Wildcards* while forming a search query with field names. To retrieve all usernames ending in **t**, use:

```
username = *t
```

#### Step

Use the step function to group numeric fields into ranges. You can also use it to group results by multiple fields.

For example, to view logs grouped by destination\_port in intervals of 100:

| **destination\_port** | **count** |
| --------------------- | --------- |
| 0 - 100               | 50        |
| 100 - 200             | 32        |

Use the following query:

```
step(destination_port,100) = 0 | chart count() by destination_port
```

This query searches for logs that contain the destination\_port field and groups them into ranges of 100. The value **0** at the end of the step function defines the start of the grouping.

#### Lower and Upper

Change the letter case of field values using case-conversion functions. Use the **lower** function to convert values to lowercase, and the **upper** function to convert values to uppercase. These functions normalize field values, ensuring consistent results when searching, grouping, or comparing data.

Use the **upper** and **lower** functions with the *chart* and *timechart* commands.

```
| chart count() by upper(action)

| timechart count() by lower(action)
```

### Time Functions

Time functions extract specified values from time-based fields. The following time functions are supported in the **Simple Search Query**:

* second
* minute
* hour
* day
* day\_of\_week
* month

These functions take numeric arguments and parse values as Unix timestamps. Unix time is a system for describing instants in time, defined as the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970, not counting leap seconds. It is used widely in Unix-like and many other operating systems and file formats.

Example: 1384409898 is the Unix time equivalent of Thu, 14 Nov 2013 06:18:18 GMT

In Logpoint, **col\_ts** and **log\_ts** carry Unix timestamps. However, create your own fields containing Unix timestamps using the **rex** or **norm** commands.

{% hint style="info" %}
Use the relational operators (**>**, **<**, **=,** and **!=**) with the time commands to create a sensible time range for your search queries.
{% endhint %}

#### second

Use the **second** function to search for logs generated or collected in seconds.

The generic syntax for second is:

```
second(field) = value
```

The value for the second ranges from 0 to 59.

```
second(log_ts) = 23
```

This query searches for the logs generated during the twenty-third second.

#### minute

Use the **minute** function to search for logs generated or collected in minutes. The minute values range from 0 to 59.

```
minute(col_ts) = 2
```

This query searches for the logs generated during the second minute.

**Minute ()** can also be used in aggregation functions.

#### hour

Use the **hour** function to search for logs generated or collected in hours. The hour values range from 1 to 24.

Example:

```
hour(col_ts) = 1
```

This query displays the logs generated during the first hour.

#### day

You can use the **day** function to search for logs generated or collected in days.

Example:

```
day(col_ts) = 4
```

This query displays the logs of the 4th day.

#### day of week

Use the **day of week** function to search logs for a specific day of the week. The day\_of\_week value ranges from 1 (Sunday) to 7 (Saturday).

Example:

```
day_of_week(col_ts) = 7 OR day_of_week(col_ts) = 1
```

This query displays the logs on off days, i.e, Saturday and Sunday.

#### month

Use the **month** function to search logs generated or collected in months. The value of the month ranges from 1 (January) to 12 (December).

Example:

```
month(col_ts) = 6
```

This query displays the log activity for June.

### Aggregators&#x20;

Aggregators are used with the chart and timechart commands to group and aggregate field values in search results. They summarize large volumes of data into structured output, such as counts, averages, or distributions. Search results processed by aggregators are formatted using the fields, chart, or timechart commands.

**Result Limits**

* Aggregators display **40 results by default.**
* Each aggregator has an internal limit of **500,000 results.**&#x20;
* A single aggregator forwards no more than 500K results to subsequent aggregators or processes commands. Use the [limit \<number>](#limit-less-than-number-greater-than) command to forward more results.
* Queries using an aggregator that produces large groupings can yield incomplete search results. To get complete results, use the [order by](#order-by) to sort the search results in ascending or descending order. There are also audit logs you can use to verify the results.
* Using free-text queries within an aggregator results in raw log processing, which is resource-intensive. Only use free text within an aggregator when necessary.

#### chart

The chart command aggregates search results and presents them in a chart format.&#x20;

To group all logs containing **login** by **device\_ip**, use:

```
login device_ip = * | chart count() by device_ip
```

This query searches for logs containing the term **login**, and groups them by **device\_ip**, and displays the number of logs per device IP.

You can also count by multiple fields. The log count is then displayed for each field.

```
login | chart count() by destination_address, destination_port
```

In this case, the count of the log for every combination of **destination\_address** and **destination\_port** is grouped, and the corresponding count is displayed.

You can use other aggregation functions, such as max and min, in place of count.

```
connection | chart max(datasize) by source_address
```

```
datasize=*| chart max(datasize) as mx, min(datasize) as mn,
sum(datasize) as sm by source_address limit 15
```

You can also display the chart in different forms, such as Column, Bar, Line, and Area.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FwQ1GhsYJcORwryTTZhbn%2Fimage.png?alt=media&#x26;token=1d3d1cd5-c6ac-440b-bccc-2ddea993ed90" alt=""><figcaption><p>Column Chart</p></figcaption></figure>

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FMJPsIcRYECzM0fDeoDms%2Fimage.png?alt=media&#x26;token=79cdb536-06ac-4a38-9c38-747756d27007" alt=""><figcaption><p>Bar Chart</p></figcaption></figure>

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FhQIlPSdHb0fBLxtdcvba%2Fimage.png?alt=media&#x26;token=f49b3602-6a10-4e09-92bb-fd9cba902e87" alt=""><figcaption><p>Line Chart</p></figcaption></figure>

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FxpSFEslmfGY7s36LPyS6%2Fimage.png?alt=media&#x26;token=07f7fee1-f11e-4d34-9584-487729a8eadb" alt=""><figcaption><p>Area Chart</p></figcaption></figure>

You can also modify aggregation functions as follows:

```
object = connection | chart count(action=permitted) by source_address
```

In this query, only logs containing **action=permitted** are counted. You can write the same query as:

```
action = permitted object = connection | chart count() by source_address
```

Multiple counts or other aggregators can be used in a single query string.

```
object = connection | chart count(action=permitted), count(action=blocked)
by source_address
```

This query displays two columns. The first is the count of the connections with the permitted action, and the second is the count of blocked actions.

Use [order by](#order-by) to sort the search results within a single query, without separating it with a pipe (|).

```
device_name=* | chart count() by email, log_ts order by log_ts asc
```

#### timechart

You can use **timechart** to chart logs as a time series. It first displays logs in chronological order by the time they were collected or generated. Then it returns the log results based on the collection timestamp (col\_ts) or the log generation timestamp (log\_ts). You cannot use [order by](#order-by) with timechart.

The terms log\_ts and col\_ts have different functions.

| **log\_ts**                               | **col\_ts**                                       |
| ----------------------------------------- | ------------------------------------------------- |
| Denotes the time present in log messages. | Denotes the time when Logpoint collected the log. |

For example, you can timechart all the logs with **login** shown below.

```
login | timechart count()
```

This plots the count of all the logs containing the word **login** into a graph with the horizontal axis as time. The total time span is the time selected for the search query.

```
| timechart on log_ts count()
```

This query plots the count of logs by the **log\_ts** field.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FmAOK5XGv1KFlsurKDvIJ%2Fimage.png?alt=media&#x26;token=9714f742-49c5-4e5c-9d9f-3ae4ce701206" alt=""><figcaption></figcaption></figure>

You can also use the **timechart** command to plot the data on a fixed time interval. To have a **timechart** with bars for every 20 minutes, use the following query:

```
login | timechart count() every 20 minutes
```

You can use every x minutes, every x hours, or every x days with the **timechart**.

{% hint style="info" %}
When the limit of the **timechart()** is not specified, the number of bars of the **timechart** depends on the nature of the query.

* The number is always equal to 30 if the time range is less than 30 units. For example, if you set a 10-minute time range, Logpoint displays 30 bars over 20 seconds.
* If the time range is greater than 30 units, the number of bars is equal to the time range. This holds true until the upper limit of the number of bars is reached, which is 59.
* There are also some exceptional cases for the number of graphs. The number of bars is equal to the number of seconds specified, and the time span of 1 day displays 24 bars in the span of one hour.
  {% endhint %}

### Available Aggregators

Aggregators are used with the **chart** and the **timechart** commands by joining them with the **|** symbol.

{% hint style="info" %}

* The aggregators are pluggable from Logpoint 5.2.4. This means Logpoint can create such functions on request.
* Use **+**, **-**, **\***, **/**, and **^** to add, subtract, multiply, divide, and to raise the power in the **min()**, **max()**, **sum()**, **avg()**, and **var()** functions.
  {% endhint %}

#### avg()

Use **avg()** to calculate the average of all the values of the specified field.

{% hint style="info" %}
When using avg(), apply a filter to exclude logs that do not contain the specified fields.
{% endhint %}

Example:

```
| chart count(), avg(response_time, response_time=*)
```

This query calculates the average response time.

#### count()

Use **count** to get the total number of logs in the search results.

Example:

```
| chart count()
```

This query displays the total number of logs in the search results.

```
login | chart count() by device_ip
```

This query searches for all logs containing the word **login**. It then groups the logs by device\_ip and shows the count of logs for each device\_ip.

You can also pass filters to count(), as:

```
login | chart count(event_id = 528) by device_ip
```

This query searches for all logs containing the word "login". It then groups them by device IP and shows the count of logs with the event\_id = 528 field value.

{% hint style="info" %}

* If logs are present in the repo but do not match the query when using **chart count()**, a row with a count of **0** is displayed.
* If no logs are available in the repo, the page displays **No results found.** message.
  {% endhint %}

#### distinct\_count()

Use **distinct\_count ()** to get the number of unique values in a list or dataset. It is used to know how many unique elements there are in a particular set of data. If you have a list of users, the distinct count returns the number of unique users, regardless of how many times each user appears in the list.

It is used to identify unique unauthorized access attempts, the activities associated with those attempts, and the measures taken to prevent them.

For example, a sudden increase in the number of unique source-destination IP connections involving uncommon IP addresses might indicate security threats or unauthorized network activity.&#x20;

```
| chart distinct_count(Source_destination)
```

In this case, **distinct\_count()** returns the count of the distinct Source\_destination regardless of the number of times it appears in the list.

If the search results for Source\_destination include the following data, the distinct\_count() result for Source\_destination is 3.

{% hint style="warning" %}
If there are many unique values in the data, the search can be slow.
{% endhint %}

| **Source\_destination** | **Target\_destination** |
| ----------------------- | ----------------------- |
| 10.145.6.78             | 14.48.7.58              |
| 25.7.32.16              | 14.48.7.58              |
| 25.7.32.16              | 15.6.7.89               |
| 90.56.2.156             | 14.48.7.58              |
| 10.145.6.78             | 15.6.7.89               |

#### distinct\_list()

Use **distinct\_list()** to return the list of all the distinct values of the field.

Example:

To view all the distinct values of the field *action* in the system, use:

```
| chart distinct_list(action)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FkJj2bjAGrgz5NXLIRLYs%2Fimage.png?alt=media&#x26;token=2bdc4d34-bb64-444f-b6bb-4b9750db6150" alt=""><figcaption></figcaption></figure>

Use a grouping parameter to group the distinct list.

Example:

```
| chart distinct_list(action) as actions by user
```

This query returns a list of all distinct values of the action field in the actions column, grouped by the user grouping parameter. You can use this example to view all the actions performed and machines used by every user in your system.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FAUE8Jq06Ji671wFJnNI3%2Fimage.png?alt=media&#x26;token=c6b04f04-9003-482f-8c6b-58f781f4dcd5" alt=""><figcaption></figcaption></figure>

You can also use these aggregators with other aggregation commands.

Example:

```
user=Jolly | chart distinct_list(action) as actions, distinct_count(action) as actions_count by user
```

This query returns a list of all distinct actions and their counts for the user **Jolly**.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FyQ2EYnydApLsmh5kfiIo%2Fimage.png?alt=media&#x26;token=abdeb448-f181-4d6e-b77e-4215228c914b" alt=""><figcaption></figcaption></figure>

#### list()

list() takes a field as a parameter and returns the field values as a list in the search result. The duplicate field values are also included in the list.

Syntax:

```
| chart list (field name) as string
```

```
| timechart list (field name) as string
```

Example:

```
| chart list (actual_mps) as number

```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2Faa4VObr2honp2EwMbsmd%2Fimage.png?alt=media&#x26;token=73c853ca-44e0-4851-8705-05fc32cd8dd7" alt=""><figcaption></figcaption></figure>

This query returns the list of actual\_mps field values and stores it in the **number** field.

Example:

```
| chart list (action) as actions by user
```

This query gives the list of the **action** field values grouped by the **user** grouping parameter and returns the list in the **actions** field.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F5oRsZBzwGylCRCiIyWxy%2Fimage.png?alt=media&#x26;token=48afec1e-ec45-4c6d-ba08-a366997a0529" alt=""><figcaption></figcaption></figure>

#### max() and min()

max() and min() aggregators are used to find the maximum or minimum value of a specified field.

{% hint style="info" %}
When using avg(), apply a filter to exclude logs that do not contain the specified fields.
{% endhint %}

Example:

```
| chart max(severity) by device_ip
```

This query displays the maximum **severity** value for each **device\_ip**.

Example:

```
login | chart count(), max(col_ts) by device_ip, col_type
```

This query retrieves all logs containing the word "login". Then it groups the search results by device\_ips and col\_type and shows the log count and the latest col\_ts for each group.

The **max()** and **min()** also support filter expressions as:

```
| chart max(severity, severity < 5)
```

This query shows the maximum **severity** that is less than 5.

#### sum()

Use the sum() aggregator to sum the values of specified fields.

Example:

```
| chart sum(datasize) by device_ip
```

This query displays the sum of all the **datasize** fields for each **device\_ip**.

You can also give filters to the **sum()** function.

Example:

```
| chart sum(datasize, datasize > 500)
```

This query only sums a datasize if it is greater than 500. The expression can be any valid query string, but must not contain any view modifiers.

#### var()

Use var() to calculate the variance of the field values. Variance quantifies how far values are spread from the mean.

Execute the following query to visualize how the data fluctuates around the average value.

```
severity = * | chart count(),avg(severity),var(severity) by device_ip
```

### One-to-One Commands

One-to-one commands take a single input value and return a single output value. They operate on individual logs without changing the number of results returned by a query.

Use one-to-one commands to extract or transform data within logs during a search. For example, the rex and norm commands extract specific parts of a log and assign them to ad-hoc field names. This enables search-time normalization, allowing you to work with structured values without modifying the underlying data.\
\
The rex and norm commands do not filter logs. They list all logs returned by the query and, if possible, add the specified ad hoc key-value pairs.

{% hint style="danger" %}
Using the rex and norm commands or the msg field on large volume of logs may severely impact system performance. If a field you are processing already contains the required information and only needs further processing, we recommend you use norm on or rex on instead.
{% endhint %}

#### rex

Use the rex command to recognize regex patterns in the re2 format. The extracted variable is retained only for the current search scope. The result also shows the logs that do not match the rex expression.

Example Log:

> `Oct 15 20:33:02 WIN-J2OVISWBB31.immuneaps.nfsserver.com MSWinEventLog   1   Security    169978  Sat Oct 15 20:33:01 2011        5156    Microsoft-Windows-Security-Auditing N/A N/A Success Audit   WIN-J2OVISWBB31.immuneaps.nfsserver.com Filtering Platform Connection   The Windows Filtering Platform has allowed a connection.    Application Information:    Process ID:  4   Application Name: System   Network Information:    Direction:  Inbound Source Address:  192.168.2.255   Source Port:  138   Destination Address: 192.168.2.221   Destination Port:  138   Protocol:  17    Filter Information: Filter Run-Time ID: 67524   Layer Name:  Receive/Accept   Layer Run-Time ID: 44 169765`

Use the **rex** command to extract the protocol ID into a field protocol\_id with the following syntax:-

```
| rex Protocol:\s*(?P<protocol_id>\d+)
```

The query format is similar to the following:

```
| rex any regular expression:\s+(?P<field_name>expression to capture to field)
```

{% hint style="warning" %}
The (?P< >) expression is part of the rex syntax to specify the field name.
{% endhint %}

You can also extract multiple fields from a single rex operation as shown below.

```
| rex Source Address:\s*(?P<src_address>\d+\.\d+\.\d+\.\d+)
```

The extracted values can be used to chart your results. For example,

```
| rex Protocol:\s+(?P<protocol_id>\d+) | chart count() by protocol_id
```

Since the rex command acts on the search results, you can add it to a query string as shown below:

```
Windows Filtering AND allowed | rex Protocol:\s+(?P<protocol_id>\d+)
```

```
user=* | rex on user:\s+(?P<account>\S+)@(?P<domain>\S+) |
chart count() by account, domain | search account=*
```

#### norm

Use the **norm** command to extract variables from the search results into a field. It supports both normalization syntax and re2 syntax.

Example Log:

> `Dec 17 05:00:14 ubuntu sshd[7596]: Invalid user Bob from 110.44.116.194`

To extract the value of the user into the field *user*, use:

```
| norm Invalid user <user:word>
```

This can also be used to chart the graph as follows:

```
| norm Invalid user <user:word>| chart count() by user
```

You can also use the **norm** command to extract multiple key-value pairs, as:

```
| norm Invalid user <user:word> from <source_ip:ip>
| chart count() by my_user, msg | search my_user=*
```

{% hint style="info" %}
If you are using square brackets (\[,]) in a search command, wrap them in single quotes (’) to escape them, as they are reserved for stream syntax in correlation queries.

Example:

`| norm on user <my_user:'['A-Z']'> | chart count() by my_user`

However, to extract the literal square bracket from the log, use [signatures](https://docs.logpoint.com/siem/log-management/log-managementdevices/signatures#signatures).

Refer to the [List of Definers](https://docs.logpoint.com/docs/data-integration-guide/en/latest/Configuration/List%20of%20Definers.html) for the built-in regex syntax available in norm commands.
{% endhint %}

#### fields

Use the fields command to display the search results in a tabular form. The table is constructed with headers according to the field names you specify. Logpoint returns **null** if the logs do not contain the specified fields.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FtrpXWFqY5sKyNZBGuHKi%2Fimage.png?alt=media&#x26;token=d9a39662-9efa-4942-b389-84045ddb2881" alt=""><figcaption></figcaption></figure>

```
| fields source_address, source_port, destination_address, destination_port
```

Here, the fields **source\_address**, **source\_port**, **destination\_address**, and **destination\_port** are displayed in a tabular form as shown above. Any logs without the destination\_port field have a corresponding row with the destination\_port column value set to -N/A-.

#### rename

Use the rename command to rename the original field names.

Example:

```
| rename device_ip as host
```

When multiple fields in a log share the same name, the rightmost field takes precedence, and only that field is renamed.

Example:

```
| rename source_address as ip, destination_address as ip
```

Here, if both the *source\_address* and *destination\_address* fields are present in a log, only the *destination\_address* field is renamed as *ip* in search results.

The logs after normalization may have different field names for information that carries similar values. For example, different logs may have **name**, **username**, **u\_name**, or **user\_name** as keys for the same field username. To aggregate all the results and analyze them properly, you can use the rename command.

```
| rename target_user as user, caller_user as user | chart count() by user
```

In some cases, the field names can be more informative with the use of rename command as below:

```
label = Attack | rename source_address as attacking_ip |
chart count() by attacking_ip
```

### Process Commands

Use the process command to execute different one-to-one functions that produce a single output for a given input.

#### AsciiConverter

Converts hexadecimal (hex) value and decimal (dec) value of various keys to their corresponding readable ASCII values. It supports the Extended ASCII Table for processing decimal values.

**Hexadecimal to ASCII**

Syntax:

```
| process  ascii_converter(fieldname,hex) as string
```

Example:

```
| process ascii_converter(sig_id,hex) as alias_name
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FZLn1pnla0wURMvpSxsSj%2Fimage.png?alt=media&#x26;token=e6f5d910-5a55-4b04-bb83-23194e6b77b7" alt=""><figcaption></figcaption></figure>

**Decimal to ASCII**

Syntax:

```
| process  ascii_converter(fieldname,dec) as string
```

Example:

```
| process ascii_converter(sig_id,dec) as alias_name
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FMHpxwtvge3fB39hHcEMW%2Fimage.png?alt=media&#x26;token=bf093eed-26a9-497f-9884-9bac5e98fbf6" alt=""><figcaption></figcaption></figure>

#### Clean Char

Removes all the alphanumeric characters present in a field-value.

Syntax:

```
| process clean_char(<field_name>) as <string_1>, <string_2>
```

Example:

```
| process clean_char(msg) as special, characters
| chart count() by special, characters
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FJncB6QgPKa8llg3v5NxF%2Fimage.png?alt=media&#x26;token=b0f90c8f-eef8-4cdb-bc0a-17a558f0799e" alt=""><figcaption></figcaption></figure>

#### Codec

A codec is a compression technology that includes an encoder to compress files and a decoder to decompress them. This process command encodes field values as ASCII characters or decodes ASCII characters back into their text using Base64 encoding/decoding. Base64 encoding converts binary data into text format so a user can securely handle it over a communication channel.

Syntax:

```
| process codec(<encode/decode function>, <field to be encoded/decoded>) as <attribute_name>
```

Example:

```
| process codec(encode, name) as encoded_name
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FRaFZo5eTOAYsHKcK0CU2%2Fimage.png?alt=media&#x26;token=b2ca52e0-7470-4948-9778-89c460d02ff8" alt=""><figcaption></figcaption></figure>

Here, the “| process codec(encode, name) as encoded\_name” query encodes the value of the name field by applying the encode function and displays the encoded value in encoded\_name.

#### Compare

Compares two values to determine whether they match.

Syntax:

```
| process compare(fieldname1,fieldname2) as string
```

Example:

```
| process compare(source_address, destination_address) as match
| chart count() by match, source_address, destination address
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FXpfhUx4QrUCtCdPlFMGx%2Fimage.png?alt=media&#x26;token=fb302f75-bff8-44b6-b329-2664efdfab00" alt=""><figcaption></figcaption></figure>

#### Compare Network

Takes a list of *IP addresses* as input and checks whether they are on the same network or on different networks. It also checks whether the networks are public or private. The comparison is performed using either the default or the custom CIDR values.

Syntax:

```
| process compare_network(fieldname1,fieldname2)
```

Example: (Using default CIDR value)

```
source_address=* destination_address=*
| process compare_network (source_address, destination_address)
| chart count() by  source_address_public, destination_address_public,
same_network, source_address, destination_address
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FHxc19E4BU63UPXvohpEP%2Fimage.png?alt=media&#x26;token=9186b907-d4e0-497b-8409-6d4443a3bcb3" alt=""><figcaption></figcaption></figure>

#### Count Char

Counts the number of characters present in a field-value.

Syntax:

```
| process count_char(fieldname) as int
```

Example:

```
| process count_char(msg) as total_chars
| search total_chars >= 100
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FvaWoiW6VvMzClAb6X84e%2Fimage.png?alt=media&#x26;token=6016848d-a353-4b3b-a989-032d1e168cf2" alt=""><figcaption></figcaption></figure>

#### CountOf

Takes a field as a parameter and counts the number of times the element(s) occurred in the field’s value.

Syntax:

```
| process count_of (source field name, string, kind)
```

Here, the **source** and **search** parameters are required.

Example:

```
| process count_of (device_address, "127") as cnt
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F1pWcFl8C6rZsgl7mQNZ3%2Fimage.png?alt=media&#x26;token=c8579580-0125-437e-97f0-5faffb3c4e06" alt=""><figcaption></figcaption></figure>

Here, the query counts the occurrences of the **127** string in the **device\_address** field value and displays it in **cnt**.

Example:

```
| process count_of (collected_at, "L") as occur
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F93wo9oNZOAI7Ofziml2F%2Fimage.png?alt=media&#x26;token=343d0cd6-6857-49e6-a17e-c700ce5b620a" alt=""><figcaption></figcaption></figure>

Here, the query counts the occurrences of **L** string in the value of **collected\_at** field and displays it in **occur**.

Example:

```
|process count_of (device_ip, ".0." , "regex") as nonrepeat

```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FkNIWeYmNQgShDiO1VAVU%2Fimage.png?alt=media&#x26;token=e1a8f3a4-2b9c-42bd-9935-b7915454b887" alt=""><figcaption></figcaption></figure>

Here, the query counts the occurrence of the **.0.** string by applying a regex pattern in the value of the device\_ip field and displays it in **nonrepeat**.

#### Current Time

Gets the current time from the user and adds it as a new field to all the logs. This information can be used to compare, compute, and operate the timestamp fields in the log message.

Syntax:

```
| process current_time(a) as string
```

Example:

```
source_address=* | process current_time(a) as time_ts
| chart count() by time_ts, log_ts, source_address
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FsUPQzgTFjWXCSM6R9h4X%2Fimage.png?alt=media&#x26;token=c91a906a-e469-4ab3-8908-e292805f659f" alt=""><figcaption></figcaption></figure>

#### DatetimeDiff

Processes two lists, calculates the difference between them, and returns the absolute value of the difference as the delta. The two lists must contain timestamps. It requires two first and second input parameters that are mandatory and can either be a list or a single field. The third parameter is mandatory and represents the required difference between the two input fields. This difference must be specified in either seconds, minutes or hours. The purpose of the third parameter is to determine how the difference between the two input fields can be represented. For instance, if the difference is specified in seconds, the output will show the absolute difference in seconds.

Syntax:

```
| process datetime_diff("seconds", ts_list1, ts_list2) as delta
```

Example:

```
| process datetime_diff("seconds",log_ts, col_ts) as diff | chart count() by diff
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FmNKDaWD4WCFXMXQynTg9%2Fimage.png?alt=media&#x26;token=19aa4a37-98bf-47a2-b177-cfe3612efc6f" alt=""><figcaption></figcaption></figure>

#### Difference

Calculates the difference between two numerical field values of a search.

Syntax:

```
| process diff(fieldname1,fieldname2) as string
```

Example:

```
| process diff(sent_datasize,received_datasize) as difference
| chart count() by sent_datasize, received_datasize,difference
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FMUegUkweKXMRaN4YI7G2%2Fimage.png?alt=media&#x26;token=a6271fed-bbbc-4e48-8271-b303573f6a35" alt=""><figcaption></figcaption></figure>

#### DNS Cleanup

Converts a DNS from an unreadable format to a readable format.

Syntax:

```
| process dns_cleanup(fieldname) as string
```

Example:

```
col_type=syslog | norm dns=<DNS.string>| search DNS=*
|process dns_cleanup(DNS) as cleaned_dns
| norm on cleaned_dns .<dns:.*>.
| chart count() by DNS, cleaned_dns, dns
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FKTqRhsT7tOGNMhIFThDP%2Fimage.png?alt=media&#x26;token=baf016cd-a447-4a50-8f68-03046ccc2e71" alt=""><figcaption></figcaption></figure>

#### DNS Process

Returns the domain name assigned to an IP address and vice-versa. It takes an **IP address** or a **Domain Name** and a **Field Name** as input. The plugin then verifies the value of the field. If the input is an *IP Address*, it resolves the address to a *hostname* and if the input is a *Domain Name*, it resolves the address to an *IP Address*. The output value is stored in the *Field Name* provided.

Syntax:

```
| process dns(IP Address or Hostname)
```

Example:

```
destination_address=* | process dns(destination_address) as domain
| chart count() by domain
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FB8QJSgxeW96McEZysJlN%2Fimage.png?alt=media&#x26;token=0b300c41-eb7f-42c8-97a8-95bfbca776ab" alt=""><figcaption></figcaption></figure>

#### Domain Lookup

Provides the domain name from a URL.

Syntax:

```
| process domain(url) as domain_name
```

Example:

```
url=* | process domain(url) as domain_name |
chart count() by domain_name, url
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F2NSe4QOfUQamFqnp1w72%2Fimage.png?alt=media&#x26;token=a13f6bdc-1a5b-4c0f-8585-af3007424e42" alt=""><figcaption></figcaption></figure>

#### Entropy

Entropy measures the degree of randomness in a set of data. This process command computes the entropy of a field using the Shanon entropy formula and displays the results in the provided field. A higher entropy value indicates a data set with greater randomness, which increases the likelihood that the values were artificially generated and could lead to a malicious conclusion.

Syntax:

```
| process entropy (field) as field_entropy
```

Example:

```
device_address = *| process entropy (device_address) as test
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FsacsR7ueYBcuPyyov6WS%2Fimage.png?alt=media&#x26;token=b53d5a77-28bd-416b-affa-062bc4c641fc" alt=""><figcaption></figcaption></figure>

Here, the “| process entropy (device\_address) as test” query calculates the entropy of the **device\_address** field and displays it in **test**.

Example:

```
| process entropy (url_address, url) as entropy_url
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FY27FqP1Sib1I5q3GFoIi%2Fimage.png?alt=media&#x26;token=df42f7bf-1da7-4851-bbbb-9073177096b8" alt=""><figcaption></figcaption></figure>

Here, the “| process entropy (url\_address, url) as entropy\_url” query takes **url** as an optional parameter and extracts the domain name from the **url\_address** to perform entropy calculation on it and displays it in **entropy\_url**.

Example:

```
| process entropy ("google.com", string) as en
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FUckKQsVHzqUFOS2pn6Wu%2Fimage.png?alt=media&#x26;token=f06790d6-66b4-4143-bf23-d509fb65bb9a" alt=""><figcaption></figcaption></figure>

Here, the “| process entropy (“google.com”, string) as en” query takes **string** as an optional parameter and calculates the entropy of the **google.com** raw string field and displays it in **en**.

#### Eval

Evaluates mathematical, boolean, and string expressions. It stores the evaluation result in an identifier as a new field.

Syntax:

```
| process eval("identifier=expression")
```

Example:

```
| process eval("Revenue=unit_sold*Selling_price")
```

For more information, go to the [Evaluation Process Plugin Manual](https://docs.logpoint.com/docs/evaluation-process-plugin/en/latest/).

#### Experimental Median Quartile Quantile

Performs statistical analysis (median, quartile, and quantile) of events based on fields. All these commands take numerical field values as input.

**Median**

Syntax:

```
| chart median(fieldname) as string
```

Example:

```
doable_mps=* |chart median(doable_mps)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2Fsnwk9faraJHT7qApoHLQ%2Fimage.png?alt=media&#x26;token=8180f2a6-ca57-4fff-9985-b1e098e122a0" alt=""><figcaption></figcaption></figure>

**Quartile**

Syntax:

```
| chart quartile(fieldname) as string1, string2, string3
```

Example:

```
doable_mps=* |chart quartile(doable_mps)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FlwbrMBbM0Het0m7orxHe%2Fimage.png?alt=media&#x26;token=29d731dc-8729-455f-ba6b-abc0a93d985f" alt=""><figcaption></figcaption></figure>

**Quantile**

Syntax:

```
| process quantile(fieldname)
```

Example:

```
doable_mps=* | process quantile(doable_mps)
|search quantile>0.99
|chart count() by doable_mps order by doable_mps desc
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F8IedkPkLlnPZM51nlyzc%2Fimage.png?alt=media&#x26;token=d6c9a90d-8726-44e9-94b8-3c5721ccf41d" alt=""><figcaption></figcaption></figure>

#### GEOIP

Gives the geographical information of a public IP address. It adds a new value “internal” to all the fields generated for the private IP, supporting the RFC 1918 Address Allocation for Private Internets.

Syntax:

```
| process geoip (fieldname)
```

Example:

```
| process geoip (source_address)
```

**For the Private IP:**

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2Fd8TSaRyCGYxNTO8F6256%2Fimage.png?alt=media&#x26;token=a2cc5be9-e1b6-4b7c-a8c8-af524097e34a" alt=""><figcaption></figcaption></figure>

**For the Public IP:**

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FNXnZWUxkhCCuw1nKksDQ%2Fimage.png?alt=media&#x26;token=fc10365f-ecaa-4395-8567-7b15cb476e5e" alt=""><figcaption></figcaption></figure>

#### Grok

Extracts key-value pairs from logs during query runtime using Grok patterns. Grok patterns are regular expressions that match words, numbers, IP addresses, and other data formats.

Refer to [Grok Patterns](https://docs.logpoint.com/siem/readme/a_work-with-your-log-data/search_your_log_data/grok-patterns) and find a list of all the Grok patterns and their corresponding regular expressions.

Syntax:

```
| process grok("<signature>")
```

A **signature** can contain one or more Grok patterns.

Example:

To extract the IP address, method, and URL from the log message:

```
192.168.3.10 GET /index.html
```

Use the command:

```
| process grok("%{IP:ip_address_in_log} %{WORD:method_in_log} %{URIPATHPARAM:url_in_log}")
```

Using this command adds the **ip\_address\_in\_log**, **method\_in\_log**, and **url\_in\_log** fields and their respective values to the log if it matches the signature pattern.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F5w2gEmdpwSOq8kIHIf55%2Fimage.png?alt=media&#x26;token=8b51945f-d800-42f9-b616-6209773dea04" alt=""><figcaption></figcaption></figure>

#### InRange

Determines whether a certain field-value falls within the range of two given values. The processed query returns TRUE if the value is in the range.

Syntax:

```
| process in_range(endpoint1, endpoint2, field, result, inclusion)

where,

endpoint1 and endpoint2 are the endpoint fields for the range,
the field is the fieldname to check whether its value falls within the given range,
result is the user provided field to assign the result (TRUE or FALSE),
inclusion is the parameter to specify whether the range is inclusive or exclusive of
given endpoint values. When this parameter is TRUE, the endpoints will be included for
the query and if it is FALSE, the endpoints will be excluded.
```

Example:

```
| process in_range(datasize, sig_id, duration,Result, True)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FJEQpOdX7wBC5sCWOXviD%2Fimage.png?alt=media&#x26;token=8326b17b-336e-4270-ba98-e852cab25247" alt=""><figcaption></figcaption></figure>

#### IP Lookup

Enriches the log messages with the Classless Inter-Domain Routing (CIDR) address details. A list of CIDRs is uploaded in CSV format during the configuration of the plugin. For any *IP Address* type in the logs, it matches the IP to the user-defined Lookup table and enriches the search results by adding CIDR details.

Syntax:

```
| process ip_lookup(IP_lookup_table, column, fieldname)
 where IP_lookup_table is the lookup table configured in the plugin,
 Column is the column name of the table which is to be matched
  with the fieldname of the log message.
```

Example:

```
| process ip_lookup(lookup_table_A, IP, device_ip)
```

Here, the command compares the IP column of the lookup\_table\_A with the device\_ip field of the log and if matched, the search result is enriched.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FRkgSKI7s2F0BVOM1fJtr%2Fimage.png?alt=media&#x26;token=23f7386d-7f2f-47ff-a5a2-133d512b1138" alt=""><figcaption></figcaption></figure>

#### JSON Expand

Takes the field with a valid JSON array value and:

1. Splits the array into individual JSON objects.
2. Creates multiple log instances for each array value of that field.
3. Assigns the original field name to each array value.

Syntax:

```
| process json_expand (field name)
```

**Use Case 1**

For example, a normalized **events** field with a valid JSON array value.

events = \[

{“source\_ip”: “192.168.0.1”, “action”: “login”, “status”: “success”},

{“source\_ip”: “192.168.0.2”, “action”: “login”, “status”: “failure”},

{“source\_ip”: “192.168.0.3”, “action”: “login”, “status”: “success”}

]

**Query:**

```
| process json_expand(events)
```

**Output:**

Index Log 1:

events = {“source\_ip”: “192.168.0.1”, “action”: “login”, “status”: “success”}

Index Log 2:

events = {“source\_ip”: “192.168.0.2”, “action”: “login”, “status”: “failure”}

Index Log 3:

events = {“source\_ip”: “192.168.0.3”, “action”: “login”, “status”: “success”}

You can further normalize JSON array values (JSON objects) into indexed logs.

Syntax:

```
| process json_expand (field name, x)
```

**x** enables normalizing the key-value pair of the JSON array into the indexed logs. The normalization occurs up to two levels, and a JSON field is ignored if it conflicts with the existing normalized field.

**Use Case 2**

For example, a normalized **evidence** field with a valid JSON array value.

evidence = \[

{ “odata\_type”:”userAccount”, “userAccount”: {“accountName”: “Logpoint”,”azureUserId”: “177”, “details” :{“department”: “IT”, “accountName”: “Logpoint\_IT”} } } },

{ “odata\_type”:”ipAddress”,”ipAddress”: “192.168.1.1” }

]

**Query:**

```
| process json_expand(evidence, x)
```

**Output:**

Index Log 1:

evidence = {“odata\_type”:”userAccount”, “userAccount”: {“accountName”: “Logpoint”,”azureUserId”: “177”, “details” :{“department”: “IT”, “accountName”: “Logpoint\_IT”} } } } // preserving the original field

odata\_type = userAccount

userAccount\_accountName = Logpoint

userAccount\_azureUserId = 177

Here,

* In level 1, **odata\_type** and **userAccount** fields are extracted, and in level 2, **userAccount\_accountName** and **userAccount\_azureUserId** fields.
* **userAccount\_details\_department** and **userAccount\_details\_accountName** fields can be extracted manually.
* **userAccount\_details\_accountName** field is ignored since **userAccount\_accountName** already exists.

Index Log 2:

evidence = {“odata\_type”:”ipAddress”,”ipAddress”:”192.168.1.1”} // preserving the original field

odata\_type = ipAddress

ipAddress = 192.169.1.1

**Example:**

```
| process json_expand (policy)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FVgDzjbao7dUgvPkgQsSZ%2Fimage.png?alt=media&#x26;token=e06a016c-b6b2-490e-948b-09ce1d032282" alt=""><figcaption><p>The log before applying the JSON Expand query</p></figcaption></figure>

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FKB6j5aHIBGpjJaHpTm8c%2Fimage.png?alt=media&#x26;token=00a4f721-2a24-4ce4-a3b3-f23938a72567" alt=""><figcaption><p>Four log instances after applying the JSON Expand query</p></figcaption></figure>

Here, the “| process json\_expand (policy)” query expands the **policy** field into four log instances. After expansion, each array value takes the **policy** as a field name.

#### JSON Parser

The JavaScript Object Notation (JSON) Parser reads JSON data and extract key values from valid JSON fields of normalized logs. A string filter is applied to the provided field, which defines a path for extracting values from it. The filter contains a key, which can be alphanumeric and special characters except square brackets (\[]), backtick (\`) and tilde (\~). These exceptional characters are reserved for essential use cases, such as mapping the list and selecting a condition in JSON Parser.

**Syntax:**

```
| process json_parser (field name, "filter") as field name
```

The supported filter formats are:

1. Chaining for nested JSON
   1. Example: .fields.user.Username
2. Array access
   1. Example: .\[1]

The supported filters are:

1. flatten

   It collapses nested arrays into a single, flat array and simplifies the structure of deeply nested arrays by combining all their items into a single level.\
   **Example:**

   arr = \[“John”, “Doe”, \[“Hello”, “World”], “Welcome”]

   **Query:**

   process json\_parser(arr, “flatten”) as flattened\_arr

   **Output:**

   flattened\_arr = \[“John”, “Doe”, “Hello”, “World”, “Welcome”]
2. reverse

   It reverses the order of items in an array, simplifying operations that require reversing the order, such as sorting or searching.\
   **Example:**

   jsn\_list = {“severity”: \[5,4,1,8]}

   **Query:**

   process json\_parser(jsn\_list, “.severity | reverse”) as reversed\_severity

   **Output:**

   reversed\_severity = \[8, 1, 4, 5]<br>

   <figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FuG4ynkuJgo9yMYUtvGBW%2Fimage.png?alt=media&#x26;token=5411bed9-32e6-4579-bb27-d1c15892129a" alt=""><figcaption><p>Using Reverse Filter with Flatten and Sort Filters</p></figcaption></figure>
3. sort

   It arranges the items of an array in ascending order.

   The items of an array are sorted in the following order:

   1. null
   2. false
   3. true
   4. numbers
   5. strings, lexicographically
   6. objects

   **Example 1:**

   jsn\_list = {“severity”: \[5,4,1,8]}

   **Query:**

   process json\_parser(jsn\_list, “.severity | sort”) as sorted\_severity

   **Output:**

   sorted\_severity = \[1, 4, 5, 8]

   **Example 2:**

   x: \[true, false, null, “abc”, “ab”, 0, 5, 1]

   **Query:**

   process eval (“x=’\[true, false, null, “abc”, “ab”, 0, 5, 1]’”)process json\_parser(x, “sort”) as sorted\_result

   **Output:**

   sorted\_result = \[null, false, true, 0, 1, 5, “ab”, “abc”]
4. to\_entries\
   It transforms a JSON object into an array of object, where each key-value pair becomes an object with key and value fields. This helps process key-value pairs or convert an object into a more manageable structure in a query result.

   **Example:**

   jsn\_obj = {“name”: “John Doe”, “age”: 30, “email”: “[john@example.com](mailto:john%40example.com)”}

   **Query:**

   process json\_parser(jsn\_obj, “to\_entries”) as key\_fields

   **Output:**

   key\_fields = \[ {“key”: “name”,”value”: “John Doe”}, {“key”: “age”,”value”: 30}, {“key”: “email”,”value”: “[john@example.com](mailto:john%40example.com)”} ]

   <figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FVwYRotzFZ4Qhtzj3hSr3%2Fimage.png?alt=media&#x26;token=7d8f416e-1500-4425-bc05-6b4d8e57cbca" alt=""><figcaption><p>Using to_entries Filter to get Key Fields</p></figcaption></figure>
5. unique\
   It removes duplicate items from an array. After applying the filter, the resulting array contains only one instance of each unique item from the original array, ensuring that all values in the returned array are distinct.\
   **Example:**

   jsn\_list = {“files”: \[“1.exe”, “1.exe”, “2.exe”]}\
   **Query:**

   process json\_parser(jsn\_list, “.files | unique”) as unique\_file

   **Output:**

   unique\_file = \[“1.exe”, “2.exe”]

JSON Parser supports **map** and **select** functions for applying filters with true conditional statements. The supported conditional operators are: =, !=, >, < , >= and <=.

**General syntax to use map and select functions:**

```
| process json_parser(field name, ".[condition]") as field name
```

JSON Parser converts a non-JSON value to a list with a single item when dot (.) filter is applied.

**Use Case:**

For example, a normalized **col\_type** field with a non-JSON **filesystem** value.

**Query:**

```
| process json_parser(col_type, ".") as result
```

**Output:**

```
result=['filesystem']
```

JSON Parser supports negative indexes, enabling the return of the last element of a data set.

**Use Case:**

For example, a normalized IP field appearing as a multiple-value array.

ip = \[10,20,30,40]

**Query:**

```
| process json_parser(ip, ".[-1]") as last_item
```

**Output:**

```
last_item = 40
```

**Example 1:**

```
| process json_parser (msg, ".AzureLogAnalytics") as analytics
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F6d744FZbBqVHztoqFs4T%2Fimage.png?alt=media&#x26;token=17de4b3c-86fc-4f66-a5e3-08981d6bbc4f" alt=""><figcaption></figcaption></figure>

Here, the “| process json\_parser (msg, “.AzureLogAnalytics”) as analytics” query applies the **AzureLogAnalytics** filter to the **msg** field and extracts the key values to the **analytics** field

**Example 2:**

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FnWGgnFs6k5Mcm4uXRcXW%2Fimage.png?alt=media&#x26;token=58416314-7293-4cbf-b83b-b1e1d166d8a2" alt=""><figcaption></figcaption></figure>

In filter, the backslash escaped the period before *type* and query applies the filter to the **evidence** field and extracts the key value to the **data\_type** field.

**Example 3:**

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2Fryk8SbnKOBKVkccSvUL3%2Fimage.png?alt=media&#x26;token=6123bca4-c96f-428c-bef8-bf0d69e0417f" alt=""><figcaption></figcaption></figure>

In the **.\[.severity>50]** filter, a conditional statement *severity>50* is used and the “| process json\_parser(detail, “.\[.severity > 50]”) as listWithSeverityGreaterThan50” query applies the filter to the **detail** field and extracts the list of key values with the true condition to the **listWithSeverityGreaterThan50** field.

#### ListLength

For an indexed field with a list of values, such as destination\_port = \[67, 23, 45, 12, 10], the command returns the number of elements in the list. The command does not support the static or dynamic [lists](https://docs.logpoint.com/siem/readme/a_work-with-your-log-data/build-a-search-query#lists).

Syntax:

```
| process list_length(list) as field name
```

Use Case:

For example, an indexed usernames field appears as a multiple-value array.

usernames = \[‘Alice’, ‘Bob’, ‘Carol’, ‘John’]

Query:

```
| process eval("usernames=split('Alice,Bob,Carol,John',',')") | process list_length(usernames) as field_len
```

Output:

```
4
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F1CdgToWZvcobjKM2NjPC%2Fimage.png?alt=media&#x26;token=481cc509-c8d0-49f7-9e86-07a90656b190" alt=""><figcaption></figcaption></figure>

Here, the “| process eval(“usernames=split(‘Alice,Bob,Carol,John’,’,’)”)” query splits the string **‘Alice,Bob,Carol,John’** into a list of strings and assigns the list to **usernames** field. Then, the “| process list\_length(usernames) as field\_len” query calculates the length of the **usernames** field values and returns the result in the **field\_len** field.

Example:

```
| chart distinct_list(actual_mps) as lst | process list_length(lst) as lst_length

```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FjDnRah11xATt81AZ69yQ%2Fimage.png?alt=media&#x26;token=fd8528ba-dff0-425b-bd18-14a4ccfd9460" alt=""><figcaption></figcaption></figure>

Here, the “| chart distinct\_list(actual\_mps) as lst” query creates a distinct list of values from the **actual\_mps** field and assigns it to the **lst** field. Then, the “| process list\_length(lst) as lst\_length” query calculates the length of the **lst** field values and returns the result in the **lst\_length** field.

#### ListPercentile

Calculates the percentile value of a given list. It requires at least two input parameters. The first parameter is mandatory and must be a list. This command can also accept up to five additional parameters. The second parameter must be an alias used with the percentile to determine the required percentile. The alias is concatenated with the percentile percentage to store the required percentile value.\
Syntax:

```
| process list_percentile(list, 25, 75, 95, 99) as x

Result: x_25th_percentile = respective_value
    x_75th_percentile = respective_value
    x_95th_percentile = respective_value
    x_99th_percentile = respective_value

General:
| process list_percentile(list,p) as aliasalias_pth_percentile
```

Example:

```
| actual_mps=* chart distinct_list(actual mps) as a | process list_percentile(a, 50, 95,99) as x | chart count() by a,
x_50th_percentile, x_95th_percentile, x_99th_percentile

```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F2nJv9tfmC0CdKSF8ACWA%2Fimage.png?alt=media&#x26;token=09d5cf6c-485d-4470-a145-ae20b9ad25bb" alt=""><figcaption></figcaption></figure>

#### Next

Takes a list and an offset as input parameters and returns a new list where the elements of the original list are shifted to the left by the specified offset. The maximum allowable value for the offset is 1024. For example, if the original list is \[1, 2, 3, 4, 5, 6] and the offset is 1, the resulting list would be \[2, 3, 4, 5, 6]. Similarly, if the offset is 2, the resulting list would be \[3, 4, 5, 6]. This command requires two parameters as input. The first is mandatory and must be a list. The second parameter is mandatory and represents the offset value. An alias of 1 must be provided as input.

Syntax:

```
| process next(list, 1) as next_list|  process next(list, 2) as next_list_2
```

Example:

```
| chart list(user) as list | process next(list, 1) as next_list | chart count() by list next_list
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FXycIIBnwmU92q9lZZ2Y5%2Fimage.png?alt=media&#x26;token=fa04809c-ba0c-4c4b-98e5-d3f5d0cfcc4d" alt=""><figcaption></figcaption></figure>

#### Percentile

Percentiles are numbers below which a portion of data is found. This process command calculates the statistical percentile from the provided field and informs whether the field’s value is high, medium or low compared to the rest of the data set.

Syntax:

```
| chart percentile (field name, percentage)
```

Example:

```
doable_mps = *| chart percentile (doable_mps, 99)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F3GcZoZjDJs19TyUONzVm%2Fimage.png?alt=media&#x26;token=8b3f1941-31b7-4b2a-9d98-b7242cfba702" alt=""><figcaption></figcaption></figure>

Here, the “| chart percentile (doable\_mps, 99)” command calculates the percentile for the value of the **doable\_mps** field.

#### Process lookup

Looks up related data from a user-defined table.

Syntax:

```
| process lookup(table,field)
```

Example:

```
| process lookup(lookup_table, device_ip)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F343Qlu2FIp2M8mVFPCh3%2Fimage.png?alt=media&#x26;token=5199abee-7034-484b-b91a-01c6cae1db07" alt=""><figcaption></figcaption></figure>

#### Regex

Extracts specific parts of the log messages into custom field names based on a regex pattern applied.

Syntax:

```
| process regex("_regexpattern", _fieldname)
| process regex("_regexpattern", "_fieldname")

Both syntaxes are valid.
```

Add a **filter** parameter in the syntax with true or false values to apply a filter to the extracted value. The filter allows only the values matching the regex pattern to be included in the output.

If the **filter** value is **true**, the filter is applied to the extracted values and the **regex** command output matches only those values. If the value is **false**, the filter is not applied, and the output includes matched and non-matched values.

If a filter is not specified, the default value is **false**.

Syntax:

```
| process regex("_regexpattern", _fieldname, "filter = true")
```

Use Case 1:

For example, in the indexed log below,

Log Entry:

```
1. user=John Smith, log_id=log_1
2. user=Ketty Perry, log_id=log_2
3. user=Keth Vendict, log_id=log_3
4. user=Keegan Pele, log_id=log_4
```

Query:

```
| process regex("(?P<fst_name>Ket\S+)", user, "filter=true")
```

Regex pattern:

In **(?P\<fst\_name>KetS+)**,

* **(?)** indicates the start of a capture group. A capture group is a way to define and extract specific parts of a text or pattern in a regular expression. It’s like putting parentheses around the part of the text you want to focus on.
* **P** indicates literal character.
* **\<fst\_name>** is the field name assigned to the capture group. It is the custom field name.
* **Ket** matches the literal characters *Ket*.
* **S+** matches one or more non-whitespace characters after *Ket* in the log.

Process:

* The regex pattern searches for sequences that start with **Ket** followed by non-whitespace characters in each log’s **user** field.
* The pattern matches *Ketty* and *Keth* from log entries 2 and 3.
* The pattern captures the matched values, and the filter is applied before assigning the values to the **fst\_name** field.

Output:

```
1. user=Ketty Perry, log_id=log_2, fst_name=Ketty
2. user=Keth Vendict, log_id=log_3, fst_name=Keth
```

These are the first names extracted from the **user** field where the user’s name starts with *Ket*.

Use Case 2:

For the same log entries above,

Query:

```
| process regex("(?P<fst_name>Ket\S+)", user, "filter=false")
```

or

process regex(“(?P\<fst\_name>KetS+)”, user)

When the **filter** value is **false** or the **filter** is not specified, the pattern captures the matched values. It assigns matched and non-matched **user** field values to the **fst\_name** field.

Output:

```
1. user=John Smith, log_id=log_1
2. user=Ketty Perry, log_id=log_2, fst_name=Ketty
3. user=Keth Vendict, log_id=log_3, fst_name=Keth
4. user=Keegan Pele, log_id=log_4
```

Example 1:

```
| process regex("(?P<type>\S*)",msg)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FbM1l3GAl3kvDXEv0i98Y%2Fimage.png?alt=media&#x26;token=148bf004-c67c-4a07-8180-5249f89a0a50" alt=""><figcaption></figcaption></figure>

Here, the “| process regex(“(?P\<type>S\*)”,msg)” query extracts any sequence of non-whitespace characters from the **msg** field and assigns it to a capture grouped **type** field.

**(?P\<type>S\*)** is the regular expression pattern where,

* **(?P\<type>…)** is a capture group, marked by **(?P<…>)**. It assigns the **type** field name to the matched pattern.
* **S** matches zero or more non-whitespace characters.
* ‘\*’ specifies zero or more occurrences of non-whitespace characters.
* **msg** is the data field to which the regular expression will be applied.

Example 2:

```
| process regex("(?P<usr>John@)", User,"filter=true")
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FUPumMvW2pTiE1lJrXAEw%2Fimage.png?alt=media&#x26;token=df1fc6d0-0a9f-4a34-82e1-a42cd755748e" alt=""><figcaption></figcaption></figure>

Here, the “| process regex(“(?P\<usr>John@)”, User,”filter=true”)” query captures the text in the **User** field that matches the **John@** pattern and assigns it to the captured grouped **usr** field.

#### SortList

Sorts a list in ascending or descending order. By default, the command sorts a list in ascending order. The first parameter is mandatory and must be a list. The second parameter desc is optional.

Syntax:

```
| process sort_list(list) as sorted_list
| process sort_list(list, "desc") as sorted_list
```

Example:

```
chart distinct list(actual_mps) as lst | process sort_list(lst) as LP_KB_Dynamictable_Populate_Values | chart count by lst, sorted list

```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FNkruPbwxj2S2MA0mbbHB%2Fimage.png?alt=media&#x26;token=2b20eb48-7ef1-4f8f-9d81-f71273abb24b" alt=""><figcaption></figcaption></figure>

#### String Concat

Joins multiple field values of the search results.

Syntax:

```
| process concat(fieldname1, fieldname2, ...., fieldnameN) as string
```

Example:

```
| process concat(device_name,device_ip) as device
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FwTVlTZ8sSfL0925W9kNB%2Fimage.png?alt=media&#x26;token=f4ade45c-8942-471e-9bbe-881a3403462f" alt=""><figcaption></figcaption></figure>

#### Summation

Calculates the sum between two numerical field values of a search.

Syntax:

```
| chart sum(fieldname)
```

Example:

```
label = Memory | chart sum(used) as Memory_Used by col_ts
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FnwQ0u3TgzO0E74qYAspy%2Fimage.png?alt=media&#x26;token=a49c22ea-e50d-446b-9d8e-d294f83e967a" alt=""><figcaption></figcaption></figure>

#### toList

Populates the dynamic list with the field values of the search result. To learn more, go to [Dynamic List](https://docs.logpoint.com/siem/readme/a_work-with-your-log-data/build-a-search-query#dynamic-list).

Syntax:

```
| process toList (list_name, field_name)
```

Example:

```
device_ip=* | process toList(device_ip_list, device_ip)

```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FPqVefRgGCNAP2l7eOAri%2Fimage.png?alt=media&#x26;token=9889f401-3c12-4748-9304-7595a18c68f1" alt=""><figcaption></figcaption></figure>

#### toTable

Populates the dynamic table with the fields and field values of the search result. To learn more, go to Dynamic [Table](https://docs.logpoint.com/siem/readme/log-management/enrichment/tables).

Syntax:

```
| process toTable (table_name, field_name1, field_name2,...., field_name9)
```

Example:

```
device_ip=* | process toTable(device_ip_table, device_name, device_ip, action)
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2Fq5rDanrMmAi4NL0NHEqk%2Fimage.png?alt=media&#x26;token=d0e57919-6624-4290-985f-0a48cb3574ed" alt=""><figcaption></figcaption></figure>

#### WhoIsLookup

Enriches the search result with the information related to the given field name from the WHOIS database. The WHOIS database consists of information about the registered users of an Internet resource, such as registrar, IP address, registry expiry date, updated date, name server information, and other information. If the specified field name and its corresponding value are matched with the equivalent field values of the WHOIS database, the process command enriches the search result; however, note that the extracted values are not saved.

Syntax:

```
| process whoislookup(field_name)
```

Example:

```
| chart distinct_list(log_ts) as log_ts_list, distinct_list(col_ts) as col_ts_list
| process datetime_diff("seconds", log_ts_list, col_ts_list) as delta
| chart count() by log_ts_list, col_ts_list, delta`

    domain =* | process whoislookup(domain)

```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2F3XhQNahdG47XZVd6q3bp%2Fimage.png?alt=media&#x26;token=3d617d33-8297-4403-b7b1-2fc8a843c39c" alt=""><figcaption></figcaption></figure>

### Filtering Commands

Filtering commands help you filter the search results.

#### search

To conduct searches on search results, use the **search** command. It searches on dynamic fields returned from the norm, rex, and the table commands.

{% hint style="info" %}
Logpoint does not advise using the search command unless absolutely necessary. This is because the search command consumes significant resources. It is always better to apply a filter before running the search command.
{% endhint %}

To search for users who have logged in more than 5 times:

```
login user = * | chart count() as count_user by user | search count_user > 5
```

If you create a dynamic field **new field** using the norm command as,

```
| norm actual_mps = < new_field:int >
```

To view the logs that have **100** as the value of the new field, use the search command as:

```
| norm actual_mps = < new_field:int >|search new_field = 100
```

We recommend you to use the **search** command only in the following cases:

* To filter the results for a simple search (non-key-value search).

For example:

```
| search error
```

* To filter the results using the **or** logical operator.

For example:

```
| search device_name=localhost or col_type=filesystem
```

#### filter

The filter command is used to drill down on the search results. The **search** command is more efficient as it does not index intermediate fields. The filter command can also filter results based on dynamic fields generated by the **norm**, **rex**, and **table** commands. It supports only expressions that use the =, >, <, >=, and <= operators. To apply multiple conditions, you must chain several **filter** expressions.

Syntax:

```
<search query> | filter <condition>
```

For example, to display only the domains that have more than 10 events associated with them in the search results, use the following query:

```
norm_id=*Firewall url=* | process domain(url) as domain | chart count() as events by domain | filter
events>10
```

The query searches for all the logs containing the fields **url** and **norm\_id** with the value of **norm\_id** having *Firewall* at the end. It then adds a new field **domain** to the logs based on the respective URLs and groups the results by their domains. Finally, the **filter** command limits the results to only those domains that have more than 10 events associated with them.

#### latest

The latest command finds the most recent log messages for every unique combination of provided field values.

```
| latest by device_ip | timechart count() by device_ip
```

This query searches for the latest logs of all the devices.

```
status = down port = 80 | latest on log_ts by device_ip
```

This query searches for all the latest devices based on the **log\_ts** field whose web server running on the port number 80 is down.

#### order by

Use order by to sort the search results based on a numeric field in either *ascending* or *descending* order. The sorting order of the search results is inconsistent when a search query does not contain an order sorting command. Use the **order by** command to make it consistent.

For simple searches that do not contain aggregation or correlation queries, the command can sort the search results based on only timestamp fields such as **log\_ts** and **col\_ts**. However, for other searches, all fields are supported.

Examples:

```
device_name= "John Doe" and col_type="syslog" | order by col_ts asc
```

This query searches for all the syslog messages generated from the device named **John Doe** and sorts them in ascending order of their **col\_ts** values.

```
device_name=* | order by log_ts desc
```

This query searches for the logs from all the devices in the system and sorts them in descending order of their **log\_ts** values.

```
device_name=* | chart count() by email, log_ts order by log_ts asc
```

This query filters logs where device\_name exists, then counts occurrences grouped by email and log\_ts, and sorts the results in ascending order of log\_ts.

#### limit \<number>

Use the limit \<number> command to limit the number of results displayed. Additionally, you can add the **other** keyword at the end of the query to display the aggregation of the rest of the results. While using the **limit \<number>** command to retrieve a large volume of logs, make sure that your system has enough resources to load and render the data.

{% hint style="info" %}
The feature to display the *Top-10 and the Rest* graphs is supported for the aggregation queries.
{% endhint %}

Example:

```
destination_address = * | chart count() by source_address limit 10 other
```

This query searches for all the logs having a **destination address**, filters the top 10 results by their **source address** and rolls-up all the remaining results in the eleventh line. The **source\_address** field displays the word *other* in the table as shown in the figure below.

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FgKQRMOyEbEIhKCMOw6Ul%2Fimage.png?alt=media&#x26;token=922e7735-f65c-43dd-8a1c-3878f30b02fc" alt=""><figcaption></figcaption></figure>

Other examples:

```
device_ip=*| chart count() by action, source_address limit 5 other

| chart sum(actual_mps) by service limit 20 other

| chart count() by action limit 10 other
```

### Pattern Finding

Pattern finding is a method of finding one or multiple streams and patterns of data to correlate a particular event. For example: five failed logins, followed by a successful login. It can be performed on the basis of the count and the time of occurrence of the stream. Use the *Pattern Finding* rules to detect complex event patterns in a large number of logs.

*Correlation* is the ability to track multiple types of logs and deduce meanings from them. It lets you look for a collection of events that make up a suspicious behavior and investigate further.

#### Single Stream

A stream consists of a count or occurrence of a query. The query can be a simple search query or an aggregating query. The stream can consist of having the same or a within expression. Stream has a notion of time.

| **Syntax**  | **Description**                                                 |
| ----------- | --------------------------------------------------------------- |
| \[ ]        | For single streams, square brackets contain a stream of events. |
| within      | Keyword to denote the notion of time frame                      |
| having same | Keyword                                                         |

Following are the working examples for pattern finding using single stream:

To find 5 login attempts:

```
[5 action = "logged on"]

[5 login]
```

To find 5 login attempts within a timeframe of 2 minutes:

```
[5 action = "logged on" within 2 minutes]

[5 login within 2 minutes]
```

To find 5 login attempts by the same user:

```
[5 action = "logged on" having same user]

[5 login having same user]
```

To find 10 login attempts by the same user from the same source\_address (multiple fields) within 5 minutes:

```
[10 action = "logged on" having same user, source_address within 5 minutes]
```

The time format for specifying timeframe are: second(s), minute(s), hour(s) and day(s).

```
[error] as E
```

This query finds the logs with errors. It then aliases the result as E and displays the fields prefixed with E such as E.severity, and E.device\_ip. You can then use the aliased fields as shown below:

```
[error] as E | rename E.device_ip as DIP | search DIP = "127.0.0.1"
```

<figure><img src="https://2590616924-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZJI3Yx2mS6Pn4JEd0IKM%2Fuploads%2FjwYsfHDBSj5XanVIERzi%2Fimage.png?alt=media&#x26;token=788ab18c-2f50-4197-b075-af8f414f6b77" alt=""><figcaption></figcaption></figure>

Pattern finding queries for different conditions:

10 login to localhost (source\_address) by the same user for the last 15 minutes.

```
[10 login source_address = 127.0.0.1 having same user_name within 15 minutes]
```

The field of a log file with a norm command.

```
[2 login | norm <username:word> login successful having same username within 10 seconds]
```

#### Multiple Streams

You can join multiple patterns by using **Pattern Finding by Joining Streams** and **Pattern Finding by Following Streams**.

#### Left Join

You can use a left join to return all the values from the table or stream on the left, and only the common values from the table or stream on the right.

Example:

```
[table event_prob] as s1
left join [event = * | chart count() by event] as s2
on s1.event = s2.event
```

#### Right Join

You can use a right join to return all the values from the table or stream on the right and only the common values from the table or stream on the left.

Example:

```
[5 transaction error having same user within 30 seconds] as s1
right join [transaction successful] as s2
on s1.user=s2.user
```

#### Join

Join queries are used to link the results from different sources. The link between two streams must have an **on** condition. The link between two lookup sources or any of the lookup and stream does not require a time-range. Join as a part of a search string, can link one data-set to another based on one or more common fields. For instance, two completely different data-sets can be linked together based on a username or event ID field present in both the data-sets.

The syntax for joining multiple patterns is as follows:

\[stream 1] \<aliased as s1> \<JOIN> \[stream 2] \<aliased as s2> on \<Join\_conditions> | additional filter query.

```
[action = locked] as locked
join
[action = unlocked] as unlocked
on
locked.target_user = unlocked.target_user
| chart count() by locked.target_user, locked.caller_computer,
unlocked.caller_user
```

```
[login] as l join [table User] as u on l.user = u.user
```

To find the events where a reserved port of an Operating System (inside the PORT\_MACHINE table) is equal to the blocked port (inside the BLOCKED\_PORT table):

```
[table PORT_MACHINE port<1024] as s1 join [table BLOCKED_PORT] as s2
on s1.port=s2.port
```

To find 5 login attempts by the same user within 1 minute followed by 5 failed login attempts by the same user within 1 minute

```
[5 login having same user within 1 minute] as s1
followed by
[5 failed having same user within 1 minute]
```

To find 5 login attempts by the same user within 1 minute followed by 5 failed attempts by the same user within 1 minute and users from both result are same

```
[5 login having same user within 1 minute] as s1
followed by
[5 failed having same username within 1 minute] as s2
on
s1.username = s2.username
```

```
[2 login success having same user] AS stream1
followed by
[login failure] as stream2
ON
stream1.user = stream2.user
```

#### Followed by

Pattern Finding by **followed by** is useful when two sequential streams are connected to an action.

For example:

```
[2 login success having same user] AS stream1
followed by
[login failure] as stream2
ON
stream1.user = stream2.user
```

Here,

| **Syntax**                  | **Description**                                   |
| --------------------------- | ------------------------------------------------- |
| \[ ] AS stream1             | A simple pattern finding query aliased as stream1 |
| followed by                 | Keyword                                           |
| \[ ] AS stream2             | A simple search aliased as stream2                |
| ON                          | Keyword                                           |
| stream1.user = stream2.user | Matching field from the 2 streams                 |

The syntax for joining multiple patterns is as follows:

* \[stream 1] \<aliased as s1> \<followed by> \[stream 2] \<aliased as s2> \<within time limit> on \<Join\_conditions>| additional filter query.
* \[stream 1] as s1 followed by \[stream2] as s2 within time\_interval on s1.field = s2.field
* \[stream 1] as s1 followed by \[stream2] as s2 on s1.field = s2.field
* \[stream 1] as s1 followed by \[stream2] as s2 within time\_interval

The inference derived from the above queries:

* Streams can be labeled using alias. Here, the first stream is labeled as s1. This labeling is useful while setting the join conditions in the join query.
* The operation between multiple streams is carried out using “followed by” or “join”.
* Use the **followed by** keyword to connect two sequential streams anticipating an action, e.g., multiple login attempts followed by successful login.
* Use the **join** keyword to view additional information in the final search. The **join** syntax is mostly used with tables for enriching the data.
* Time limit for occurrence can also be specified.
* If you use the **join** keyword, then specify the **on** condition.
* Join conditions are simple mathematical operations between the data-sets of two streams.
* Use additional filter query to mitigate false positives which are generally created while joining a stream and a table. Searching the query with a distinct key from the table displays an error-less result.

```
[| chart count() by device_ip] AS lookup
JOIN
[device_ip=*] AS log ON lookup.device_ip = log.device_ip
```

This query does not display histogram but displays the log table.

```
[device_ip=*] as log join  [| chart count() by device_ip] as lookup on
log.device_ip=lookup.device_ip
```

This query displays both the histogram and the log table.

{% hint style="info" %}

* The **Latest** command is supported in pattern finding queries.
* All the reserved keywords such as **on**, **join**, **as**, and **chart** are not case-sensitive.
* If you want to use reserved keywords in simple search or some other contexts, put them in quotes.

  ```
  login | chart count() by device_ip | search "count()" > 5
  ```

{% endhint %}

### Chaining of commands

Chain multiple commands into a single query by using the pipe (|) character. Any command except **fields** can appear before or after any other command. The **fields** command must always appear at the end of the command chain.

Example:

```
| chart count() as cnt by device_name | search cnt > 1000
```

This query returns the number of logs with the same **device\_name** that appear more than 1000 times.

```
(label = logoff) AND hour (log_ts) > 8 AND hour (log_ts) <16 |
latest by user |
timechart count() by user
```

This query captures all the log messages labeled as **logoff** and those collected between 8 AM and 4 PM. It then displays the timechart of the recent users for the selected time frame.
