Ingest Logs

Prerequisites

• Logpoint: v7.8.0 or later

• AWS Account: Active AWS account with CloudWatch service enabled

• IAM Permissions: Ability to create IAM users or roles with CloudWatch read permissions

• CloudWatch Logs: Log Groups and Log Streams configured in AWS CloudWatch

• Network Access: Logpoint must be able to reach AWS CloudWatch endpoints (typically *.amazonaws.com)

Install CloudWatch

1. Download the .pak file from the Release Notes or Help Center.

2. Install the Package

   1. Go to Settings >> System Settings from the navigation bar.

   2. Click Applications.

   3. Click Import.

   4. Browse to the downloaded .pak file.

   5. Click Upload.

3. Verify Installation After installation, verify the integration appears under Settings >> System Settings >> Plugins.

---

Pre-Configuration in AWS

Before configuring the integration in Logpoint, you must complete the following setup in AWS to enable log collection.

Note: While we provide AWS-specific instructions in this guide, it's important to be aware that the AWS interface may change over time. To ensure you have the most up-to-date information and to navigate any potential changes in the AWS interface, we recommend referring to the official AWS documentation or AWS Support resources.

Step 1: Creating AWS Access Keys

1. Go to the AWS Console and log in with your credentials.

2. Click your username in the top-right corner and select My Security Credentials from the dropdown menu.

3. Click Continue to Security Credentials (if prompted).

4. Expand Access Keys (Access Key ID and Secret Access Key).

5. Click Create New Access Key.

6. Select Show Access Key to view the Access Key ID and Secret Access Key, or click Download Key File to download a CSV file containing them.

Important: Write down the Access Key ID and Secret Access Key or download the file containing them, as they cannot be retrieved later. If you close the window without saving them, you must create new ones.

Step 2: Configuring IAM Permissions

The AWS user or role associated with the access keys must have the following permissions to read CloudWatch logs:

Required IAM Permissions:

• logs:DescribeLogGroups

• logs:DescribeLogStreams

• logs:GetLogEvents

• logs:FilterLogEvents

Recommended IAM Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents",
        "logs:FilterLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

You can create a custom IAM policy with these permissions or use the AWS managed policy CloudWatchLogsReadOnlyAccess.

Step 3: Identifying Log Groups and Log Streams

1. In AWS Console, click Services from the navigation panel.

2. Click CloudWatch (or search for "CloudWatch").

3. In the left sidebar, click Logs > Log groups.

4. Note the Log Group Name(s) you want to ingest into Logpoint.

5. Click on a Log Group to view its Log Streams.

6. Note the Log Stream Name(s) if you want to ingest specific streams (optional - if not specified, all streams will be ingested).

Log Group Naming Examples:

• /aws/lambda/my-function

• /aws/rds/instance/my-database/error

• /aws/ec2/instance-id

• /aws/ecs/container-name

---

Configure AWS CloudWatch

You can configure AWS CloudWatch using two methods:

1. Log Source Template (recommended), which provides a centralized interface for all integrations and minimizes setup requirements

2. Devices

Method 1: Configure via Log Source Template

For Logpoint v7.8.0 and above:

You must create a log source using the log source template to receive the normalized AWS CloudWatch logs.

1. Go to Settings >> Log Sources from the navigation bar.

2. Click Browse Log Source Templates.

3. Click Create Log Source and select CloudWatch Fetcher.

Source Configuration

Configure the log source settings:

1. Click Source.

2. Enter the Log Source's Name.

3. Select the Fetch Interval (min) - the frequency at which data is retrieved from AWS CloudWatch.

4. Select the Charset (typically UTF-8).

5. Select the Time Zone.

Connector Configuration

Configure the connection to AWS CloudWatch:

1. Click Connector.

2. Enter the AWS Access Key ID (from pre-configuration Step 1).

3. Enter the AWS Secret Access Key (from pre-configuration Step 1).

4. Select the AWS Region (e.g., us-east-1, eu-west-1, ap-southeast-1). The EndPoint URL is auto-generated after you select the AWS Region.

5. Enter the AWS Log Group name (from pre-configuration Step 3).

6. Enter the Log Stream name(s). You can add multiple log streams separated by commas.

   • Note: If you do not enter any log stream, CloudWatch retrieves logs from all the Log Group's log streams.

7. Select a Start Date - CloudWatch starts fetching logs from the specified date.

8. If you are using a Distributed Logpoint, select Distributed Collectors from the dropdown.

Routing Configuration

Set up log storage and routing:

Create Repository

1. Click Routing and + Create Repo.

2. Enter a Repo name.

3. In Path, specify the location to store incoming logs.

4. In Retention (Days), set how long logs are kept before automatic deletion.

5. In Availability, select the Remote logpoint and Retention (Days).

6. Click Create Repo.

7. Select the created repo in Repo.

Create Routing Criteria

1. Click + Add row.

2. Enter a Key and Value for log filtering (optional).

3. Select log handling options:

   • Store raw message: Store both incoming and normalized logs.

   • Discard raw message: Keep only normalized logs.

   • Discard entire event: Discard both incoming and normalized logs.

4. Select the target Repository.

Normalization Configuration

Set up log normalization:

1. Click Normalization.

2. Either:

   • Select a previously created normalization policy from the Select Normalization Policy dropdown, or

   • Select CloudWatchCompiledNormalizer from the list and click the swap icon.

Enrichment Configuration

Configure log enrichment:

1. Click Enrichment.

2. Select an Enrichment Policy (if available).

Finalize Configuration: Click Create Log Source to save all configurations.

---

Method 2: Configure via Devices

Adding a Normalization Policy for CloudWatch

Normalization policies normalize and standardize logs for efficient storage, analysis, and retrieval. They can also be used in the processing policy to process CloudWatch logs.

1. Go to Settings >> Configuration from the navigation bar and click Normalization Policies.

2. Click Add.

3. Enter a Policy Name.

4. Select CloudWatchCompiledNormalizer.

5. Click Submit.

Adding a Processing Policy for CloudWatch

Processing policy dictates how CloudWatch logs are handled, processed, and stored to enhance their usability and accessibility for monitoring, reporting, and alerting purposes.

1. Go to Settings >> Configuration from the navigation bar and click Processing Policies.

2. Click Add.

3. Enter a Policy Name.

4. Select the previously created normalization policy.

5. Select the Enrichment Policy (if available).

6. Select the Routing Policy.

7. Click Submit.

Configuring the CloudWatch Fetcher

To fetch CloudWatch logs, an initial setup where details about AWS environment, such as its Access Key ID and Secret Access Key must be configured in Logpoint. These details are necessary to establish a connection with AWS and retrieve logs.

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions of the localhost device.

3. Click CloudWatch Fetcher.

4. Click Add.

5. Enter a Name for the configuration.

6. Enter the AWS Access Key ID and Secret Access Key (from pre-configuration Step 1).

7. Select the AWS Region. The EndPoint URL is auto-generated after you select the AWS Region.

8. Enter the AWS Log Group and Log Stream. You can add multiple log streams separated by commas.

   • Note: If you do not enter any log stream, CloudWatch retrieves logs from all the Log Group's log streams.

9. Select a Start Date - CloudWatch starts fetching logs from the specified date.

10. Select the Fetch Interval (minutes) - how often Logpoint retrieves new logs.

11. Select the previously created Processing Policy.

12. Select the Charset (typically UTF-8).

13. Click Submit.

---

Verify Ingestion

Check Log Ingestion

Use the following query to verify AWS CloudWatch logs are being ingested:

norm_id = "cloudwatch"

Or check for any CloudWatch-related data:

col_type = "aws" OR norm_id = "cloudwatch"

Verify Data Flow

1. Check CloudWatch Fetcher Status: Ensure the CloudWatch fetcher is running without errors under Settings >> Configuration >> Devices.

2. Monitor Log Volume: Verify expected log volumes are being processed.

3. Validate Normalization: Confirm logs are correctly parsed and normalized using the CloudWatchCompiledNormalizer.

4. Check AWS CloudWatch Console: Verify logs are present in the specified Log Groups and Log Streams.

5. Review Fetch Interval: Ensure the fetch interval is appropriate for your log volume and latency requirements.