Ingest Logs in SIEM

Prerequisites

• Logpoint: v7.8.1 or later

• AWS Access: Read access to the Amazon S3 bucket and its objects

• AWS Credentials: Valid AWS Access Key ID and Secret Access Key

• Network Access: Connectivity to AWS S3 endpoints (proxy support available)

Installation

Install S3Fetcher

1. Download the .pak file from the Marketplace.

2. Go to Settings >> System Settings from the navigation bar of Logpoint.

3. Click Applications.

4. Click Import.

5. Browse to the downloaded .pak file.

6. Click Upload.

After installation, verify the integration appears under Settings >> System Settings >> Plugins

Configuration

Before configuring S3Fetcher in Logpoint, obtain AWS access credentials (Access Key ID and Secret Access Key). Go to the AWS Documentation for instructions on retrieving them.

Configure S3Fetcher using two methods:

1. Log Source, recommended as it provides a centralized interface for all integrations

2. Devices

Method 1: Configure via Log Source

1. Go to Settings >> Log Sources and click + Add Log Source.

2. Click + Create New and select S3 Fetcher.

Source

Configure the log source settings:

1. Click Source.

2. Enter the Log Source's Name.

3. Select the Fetch Interval (min) to set how frequently the logs are retrieved.

4. Select the Charset and Time Zone.

Connector

Configure the connection to AWS:

1. Click Connector.

2. EndPoint URL:

   1. For Amazon S3: https://s3.amazonaws.com (default).

   2. For third-party services: Enter the service URL.

3. Enter your AWS Access Key ID and AWS Secret Access Key.

4. Enter the AWS Bucket Name. S3Fetcher fetches logs from this bucket.

5. In Filter by Prefix, enter a prefix name to fetch a specific file or folder from the bucket. Leave this field empty to fetch the entire bucket.

6. Select the AWS Region.

7. Select Initial Fetch Date. S3Fetcher fetches logs from the specified date. To change the Initial Fetch Date and re-fetch logs from the new date, enable Reset Last Fetch Date and select the new date.

8. If you are using a Distributed Logpoint, select Distributed Collectors from the drop-down.

9. Select a Parser to parse the logs.

10. Enable Proxy to use a proxy server.

    1. Select either HTTP or HTTPS protocol.

    2. Enter the proxy server IP address and the PORT number.

Routing

Set up log storage and routing:

Create Repository:

1. Click Routing and + Create Repo.

2. Enter a Repo name.

3. In Path, specify the location to store incoming logs.

4. In Retention (Days), set how long logs are kept before automatic deletion.

5. In Availability, select the Remote logpoint and Retention (Days).

6. Click Create Repo.

7. Select the created repo in Repo.

Create Routing Criteria

1. Click + Add row.

2. Enter a Key and Value for log filtering.

3. Select log handling options:

   1. Store raw message: Store both incoming and normalized logs.

   2. Discard raw message: Keep only normalized logs.

   3. Discard entire event: Discard both incoming and normalized logs.

4. Select the target Repository.

Normalization

Set up log normalization:

1. Click Normalization.

2. Either:

   1. Select a previously created normalization policy from the dropdown, or

   2. Select a normalizer from the list and click the swap icon.

Enrichment

Configure log enrichment:

1. Click Enrichment.

2. Select an Enrichment Policy.

After the above configurations, click Create Log Source to save all configurations.

Method 2: Configure via Devices

Configuring the S3Fetcher via Devices

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon for the localhost device.

3. Click S3 Fetcher.

4. Click Add and enter a Name.

5. Enter your AWS Access Key ID and Secret Access Key.

6. Enter the AWS Bucket Name. S3Fetcher fetches logs from this bucket.

7. In Filter by Prefix, enter a prefix name to fetch a specific file or folder from the bucket. Leave this field empty to fetch the entire bucket.

8. EndPoint URL:

   1. For Amazon S3: https://s3.amazonaws.com (default).

   2. For third-party services: Enter the service URL.

9. Select the AWS Region.

10. Select the frequency at which logs are retrieved in Fetch Interval (minutes).

11. Select the Logs From date. S3Fetcher fetches logs from the specified date.

12. Select a Processing Policy and Charset.

13. Select a Parser to parse the logs.

14. Enable Proxy to use a proxy server.

    1. Select either HTTP or HTTPS protocol.

    2. Enter the proxy server IP address and the PORT number.

15. Click Test to check if the configuration is working correctly.

16. Click Submit.