Ingest Logs via Director Console UI

Prerequisites

• Logpoint: v7.8.0 and later

• Director: v2.9.0 and later

• AWS Access: Read access to the Amazon S3 bucket and its objects

• AWS Credentials: Valid AWS Access Key ID and Secret Access Key

• Network Access: Connectivity to AWS S3 endpoints (proxy support available)

Installation

Install S3Fetcher

1. Log in to Director Console.

2. Click ASSETS in the navigation bar.

3. Select Plugin from the Assets drop-down.

4. Upload the S3Fetcher .pak file. Select Replace Existing? to replace the old file with new one.

5. Click UPLOAD.

Once uploaded, the .pak file is displayed in the list of available packages.

1. Select the S3Fetcher .pak file from the list.

2. Click INSTALL.

3. Select the pool and Logpoint to install S3Fetcher. You can select multiple Logpoint from different pools.

4. Click NEXT.

5. Review your changes. You can go BACK to make necessary changes.

6. Click INSTALL and click OK to confirm.

You are redirected to TASKS, which displays the installation progress.

Uninstall S3Fetcher

You must first remove S3Fetcher configurations from Logpoint and then uninstall it.

To remove S3Fetcher configurations:

1. Click CONFIGURE in the navigation bar.

2. Under Entities, click LOG SOURCES.

3. Click the More icon next to the S3Fetcher log source and click Delete Log Source.

4. Click Delete.

To uninstall S3Fetcher:

1. Click ASSETS in the navigation bar.

2. Click UNINSTALL.

3. Select the Logpoint where S3Fetcher is installed. You can select multiple Logpoint from different pools.

4. Select S3Fetcher from the list of available packages.

5. Click NEXT.

6. Review your changes. You can go BACK to make necessary changes.

7. Click UNINSTALL and click OK to confirm.

Configuration

Before configuring S3Fetcher in Logpoint, obtain AWS access credentials (Access Key ID and Secret Access Key). Go to the AWS Documentation for instructions on retrieving them.

1. Click CONFIGURE in the navigation bar.

2. Under ENTITIES, click LOG SOURCES.

3. Click Create Log Source.

4. Select S3Fetcher.

5. Select a Pool and Logpoint to configure the fetcher.

6. Click NEXT.

Source

Configure the log source settings:

1. Click Source.

2. Enter the Log Source's Name.

3. Select the Fetch Interval (min) to set how frequently the logs are retrieved.

4. Select the Charset and Time Zone.

Connector

Configure the connection to AWS:

1. Click Connector.

2. EndPoint URL:

   1. For Amazon S3: https://s3.amazonaws.com (default).

   2. For third-party services: Enter the service URL.

3. Enter your AWS Access Key ID and AWS Secret Access Key.

4. Enter the AWS Bucket Name. S3Fetcher fetches logs from this bucket.

5. In Filter by Prefix, enter a prefix name to fetch a specific file or folder from the bucket. Leave this field empty to fetch the entire bucket.

6. Select the AWS Region.

7. Select Initial Fetch Date. S3Fetcher fetches logs from the specified date. To change the Initial Fetch Date and re-fetch logs from the new date, enable Reset Last Fetch Date and select the new date.

8. If you are using a Distributed Logpoint, select Distributed Collectors from the drop-down.

9. Select a Parser to parse the logs.

10. Enable Proxy to use a proxy server.

    1. Select either HTTP or HTTPS protocol.

    2. Enter the proxy server IP address and the PORT number.

Routing

Set up log storage and routing:

Create Repository:

1. Click Routing and + Create Repo.

2. Enter a Repo name.

3. In Path, specify the location to store incoming logs.

4. In Retention (Days), set how long logs are kept before automatic deletion.

5. In Availability, select the Remote logpoint and Retention (Days).

6. Click Create Repo.

7. Select the created repo in Repo.

Create Routing Criteria

1. Click + Add row.

2. Enter a Key and Value for log filtering.

3. Select log handling options:

   1. Store raw message: Store both incoming and normalized logs.

   2. Discard raw message: Keep only normalized logs.

   3. Discard entire event: Discard both incoming and normalized logs.

4. Select the target Repository.

Normalization

Set up log normalization:

1. Click Normalization.

2. Either:

   1. Select a previously created normalization policy from the dropdown, or

   2. Select a normalizer from the list and click the swap icon.

Enrichment

Configure log enrichment:

1. Click Enrichment.

2. Select an Enrichment Policy.

1. Click Create Log Source to save the configurations of Source, Connector, Routing, Normalization, and Enrichment.