Ingest Logs

Prerequisites

• Logpoint: v7.4.0 or later

• AWS Access: Read access to the Amazon S3 bucket and its objects

• AWS Credentials: Valid AWS Access Key ID and Secret Access Key

• Network Access: Connectivity to AWS S3 endpoints (proxy support available)

Install Cloud Trail

1. Download the .pak file, from the Service Desk.

2. Install the Package

   1. Go to Settings >> System Settings from the navigation bar.

   2. Click Applications.

   3. Click Import.

   4. Browse to the downloaded .pak file.

   5. Click Upload.

3. After installation, verify the integration appears under Settings >> System Settings >> Plugins

Configure Cloud Trail

Before configuring Cloud Trail in Logpoint, obtain AWS access credentials (Access Key ID and Secret Access Key). Go to the AWS Documentation for instructions on retrieving them.

You can configure CloudTrail using two methods:

1. Log Source, recommended as it provides a centralized interface for all integrations

2. Devices

Method 1: Configure via Log Source

For Logpoint v7.4.0:

1. Go to Settings >> Log Sources from the navigation bar.

2. Click Browse Log Source Templates and select Cloud Trail.

For Logpoint v7.5.0 and above:

1. Go to Settings >> Log Sources and click + Add Log Source.

2. Select Cloud Trail from the search bar.

Source

Configure the log source settings:

1. Click Source.

2. Enter the Log Source's Name.

3. Select the Fetch Interval (min) to set how frequently the logs are retrieved.

4. Select the Time Zone.

Connector

Configure the connection to AWS:

1. Click Connector.

2. EndPoint URL:

   • For Amazon S3: https://s3.amazonaws.com (default).

   • For third-party services: Enter the service URL.

3. Enter your AWS Access Key ID.

4. Enter your AWS Secret Access Key.

5. Enter the Bucket Name where CloudTrail logs are stored.

6. Select the AWS Region.

Routing

Set up log storage and routing:

Create Repository

1. Click Routing and + Create Repo.

2. Enter a Repo name.

3. In Path, specify the location to store incoming logs.

4. In Retention (Days), set how long logs are kept before automatic deletion.

5. In Availability, select the Remote logpoint and Retention (Days).

6. Click Create Repo.

7. Select the created repo in Repo.

Create Routing Criteria

1. Click + Add row.

2. Enter a Key and Value for log filtering.

3. Select log handling options:

   • Store raw message: Store both incoming and normalized logs.

   • Discard raw message: Keep only normalized logs.

   • Discard entire event: Discard both incoming and normalized logs.

4. Select the target Repository.

Normalization

Set up log normalization:

1. Click Normalization.

2. Either:

   • Select a previously created normalization policy from the dropdown, or

   • Select CloudTrailCompiledNormalizer from the list and click the swap icon.

Enrichment

Configure log enrichment:

1. Click Enrichment.

2. Select an Enrichment Policy.

Finalize Configuration: Click Create Log Source to save all configurations.

Method 2: Configure via Devices

Adding a Normalization Policy

1. Go to Settings >> Configuration >> Normalization Policies.

2. Click Add.

3. Enter a Policy Name.

4. Select CloudTrailCompiledNormalizer.

5. Click Submit.

Adding a Processing Policy

1. Go to Settings >> Configuration >> Processing Policies.

2. Click Add.

3. Enter a Policy Name.

4. Select the previously created normalization policy.

5. Select Enrichment Policy and Routing Policy.

Configuring the CloudTrail Log Fetcher

1. Go to Settings >> Configuration >> Devices.

2. Click the Add collectors/fetchers icon for the localhost device.

3. Click CloudTrail Log Fetcher.

4. Click Add and enter a Name.

5. Enter your AWS Access Key ID and Secret Access Key.

6. Enter the Bucket Name from which Cloud Trail fetches the logs.

7. Select the AWS Region.

8. Select the frequency at which logs are retrieved in Fetch Interval (minutes).

9. Select the previously created processing policy.

10. Select the Charset.

Filter Configuration:

• Base Path: Specify the AWS directory path. Leave the field empty to fetch from all directories.

• Logs From: Set the starting date for log retrieval.

Proxy Configuration (if needed):

1. Select Enable Proxy.

2. Enter proxy server IP address and Port number.

3. Select HTTP or HTTPS protocol.

4. Click Submit.

Verify Ingestion

Check Log Ingestion

Use the following query to verify CloudTrail logs are being ingested:

col_type = cloudtrail

Verify Data Flow

1. Check Fetcher Status: Ensure the CloudTrail fetcher is running without errors

2. Monitor Log Volume: Verify expected log volumes are being processed

3. Validate Normalization: Confirm logs are correctly parsed and normalized. Refer to the CloudTrail field mapping to understand how AWS CloudTrail logs correspond to Logpoint fields.

4. Test Dashboards: Access CloudTrail dashboards to verify data visualization