Ingest Logs via Director Console API

Installation

Uploading S3Fetcher to the Fabric Storage

Private Location

You can upload S3Fetcher to a private location in the Fabric Storage using the Upload - Upload API.

Endpoint URL:

https://api-server-host-name/configapi/Uploads/{pool_UUID}/

Method:

POST

Header:

FIELD	DESCRIPTION
file_name	Name of the S3fetcher .pak file.
Content-Type	Mandatory value: (application/octet-stream).

Parameters:

FIELD	TYPE	DESCRIPTION	REQUIRED
file	[Object]	The S3fetcher .pak file to upload.	Mandatory

Success Response:

{
  "status": "Success",
  "message": "S3fetcher_3.0.0.pak successfully uploaded in private storage."
  }

Public Location

You can upload S3Fetcher to a public location in the Fabric Storage using the Upload - UploadPublic files API.

Endpoint URL:

https://api-server-host-name/configapi/Uploads/PublicUpload

Method:

POST

Header:

FIELD	DESCRIPTION
file_name	Name of the S3Fetcher .pak file.
Content-Type	Mandatory value: (application/octet-stream).

Parameters:

FIELD	TYPE	DESCRIPTION	REQUIRED
file	[Object]	The S3Fetcher .pak file to upload.	Mandatory

Success Response:

{
  "status": "Success",
  "message": "S3fetcher_3.0.0.pak successfully uploaded in public storage. "
  }

Installing S3Fetcher in Logpoint

You can install S3Fetcher in a Fabric-enabled Logpoint using the Upload - Install API. It must be uploaded to the Fabric Storage before the installation.

Endpoint URL:

https://api-server-host-name/configapi/Uploads/{pool_UUID}/{logpoint_identifier}/install

Method:

POST

Parameters:

FIELD	LABEL IN UI	TYPE	DESCRIPTION	REQUIRED
application_type	–	String	Mandatory value: Application.	Mandatory
file_name	–	String	Name of the application file. Obtain it from the public location using the Upload - ListPublic API and the private location using the Upload - List API.	Mandatory
file_location	–	String	Location of the application file. Can be either private or public.	Mandatory

Request Example:

 {
 "data": {
     "application_type": "Application",
     "file_name": "S3fetcher_3.0.0.pak",
     "file_location": "private"
 }
}

Success Response:

{
    "status": "Success",
    "message": "monitorapi/{pool_UUID}/{logpoint_identifier}/orders/{request_id}"
}

Uninstalling S3Fetcher

You can uninstall S3Fetcher in a Fabric-enabled Logpoint using the Plugins - Uninstall API. You must first remove the S3Fetcher configuration to uninstall it.

Endpoint URL:

https://api-server-host-name/configapi/{pool_UUID}/{logpoint_identifier}/Plugins/{id}

Method

Delete

Parameters:

FIELD	LABEL IN UI	TYPE	DESCRIPTION	REQUIRED
id	–	String	S3Fetcher application ID. Obtain it using the Plugins - List API.	Mandatory

Success Response:

 {
   "status": "Success",
   "message": "monitorapi/{pool_UUID}/{logpoint_identifier}/orders/{request_id}"
}

Configuration

Before configuring S3Fetcher, obtain AWS access credentials (Access Key ID and Secret Access Key). Go to the AWS Documentation for instructions on retrieving them.

Configure S3Fetcher in a Fabric-enabled Logpoint using the LogSources - Create API.

Endpoint URL:

https://api-server-host-name/configapi/{pool_UUID}/{logpoint_identifier}/LogSources

Method:

POST

Parameters:

Field	Label in UI	Type	Description
dc_metadata	-	json	Information related to S3Fetcher template created in the Director Console. Optional Field.
description	Description	String	Additional information about S3Fetcher. Optional Field.
documentation_link	Documentation Link	String	URL or hyperlink that points to external documentation or reference materials associated to S3Fetcher. Optional Field.
logo	Logo	String	Base64 encoded logo image. Optional Field.
name	Name	String	Name of the S3Fetcher log source in Logpoint. Mandatory Field.
type	-	String	Type or category of S3Fetcher. Mandatory Field.
vendor_name	Vendor Name	String	Name of vendor where the log data originates. Optional Field.
config	-	json	Configuration of S3Fetcher. Source, connector, routing, normalization and enrichment must be configured for S3Fetcher to fetch logs. Mandatory Field.

Source

Configure the log source settings:

Parameters:

Field	Label in UI	Type	Description
name	Name	String	Name for the S3Fetcher Log Source. Mandatory Field.
interval	Fetch Interval (min)	Integer	The frequency at which data is retrieved in minutes. Mandatory Field.
charset	Charset	String	Existing Logpoint charset. Obtain it using the Charsets - List API. Mandatory Field.
timezone	Time Zone	String	S3 Bucket timezone. Mandatory Field.

Connector

Configure the connection to AWS:

Field	Label in UI	Type	Description
endpoint_url	Endpoint URL	String	Endpoint URL for AWS. If you use Amazon S3’s services, the URL is https://s3.amazonaws.com by default. If you are using a third-party service that stores logs in the Amazon S3 bucket, enter the URL of the service that you are using.
access_key	Access Key ID	String	S3 Bucket Access Key ID. Mandatory Field.
secret_key	Secret Key ID	String	S3 Bucket Secret Key ID. Mandatory Field.
bucket_name	Bucket Name	String	Name of the bucket from where logs are fetched. Mandatory Field.
region_name	Region	String	AWS Region. Mandatory Field.
filter_date	Logs From	String	The date from which S3Fetcher starts fetching logs. Mandatory Field.
parser	Parser	String	The type of parser to be used for log processing which defines how the logs will be interpreted and extracted.
enable_proxy	Proxy Configuration	json	Proxy configuration of S3Fetcher server: status: Parameter to enable or disable the proxy server. IP: IP of the proxy server. port: Port of the proxy server. protocol: “HTTP” or “HTTPS” protocol used by the proxy server. Optional Field.

Routing

Set up log storage and routing:

Parameters:

Field	Label in UI	Type	Description
repo_name	Repo name	String	Name of the repo where incoming logs are stored. Mandatory Field.
path	Path	String	Location to store incoming logs. Mandatory Field.
retention	Retention (Days)	String	Number of days logs are kept in a repository before they are automatically deleted. Mandatory Field.
remote_logpoint	Remote logpoint	String	Remote Logpoint where you can replicate the repo as a backup or for high availability. Optional Field.
key and value	Key and Value	String	The key-value pair is used to apply routing criteria to logs. Optional Field.
operation	Operation	String	Operation for logs that have the key-value pair. Optional Field.
repository	Repository	String	Repo to store logs. Optional Field.

Normalization

Set up log normalization:

Parameters:

Field	Label in UI	Type	Description
normalizers	Normalizer	String	Enter the name of the normalizer for the incoming logs. Mandatory Field.
type	Type	String	Type of the normalizer. Mandatory Field.

Enrichment

Configure log enrichment:

Parameters:

Field	Label in UI	Type	Description
EnrichmentPolicy	Enrichment Policy	String	Enter the name of the enrichment policy for the incoming logs. Optional Field.

Request Example:

{
  "data": {
   "name": "S3",
    "type": "S3Fetcher",
    "vendor_name": "AWS",
    "logo": " ",
    "description": "Fetches various AWS logs stored in Amazon S3 buckets, including but not limited to CloudTrail logs for AWS API activity, VPC Flow Logs for network traffic analysis, and MySQL RDS logs.",
    "config": {
      "Source": {
        "name": "S3",
        "interval": 60,
        "charset": "utf_8",
        "timezone": "UTC"
      },
      "Connector": {
        "endpoint_url": "https://s3.amazonaws.com",
        "access_key": "AKIATOEJXHJKHDVGQZTZ5E",
        "secret_key": "ihBavc5Tu383dkjhcT1Z+gmWD3VNWewGbXw3Dohm",
        "bucket_name": "lp-plugins-bucket",
        "region_name": "eu-west-1",
        "filter_date":"2024-07-17T10:56:04.142Z",
        "parser": "LineParser",
        "enable_proxy": false,
        "protocol": "http"
      },
      "RoutingPolicy": {
        "routing_criterion": [],
        "catch_all": "_logpoint"
      },
      "NormalizationPolicy": {
        "normalizers": []
      }
    },
    "documentation_link": "https://docs.logpoint.com/docs/s3fetcher/en/latest/index.html",
    "template_vid": "LOGSOURCE_87",
    "dc_metadata": {
      "template": "0f83c6d3-7958-4c31-8b6b-e5743e2d7ee9",
      "templateVersion": "2"
    }
  }
}

Success Response:

{
  "status": "Success",
  "message": "monitorapi/{pool_UUID}/{logpoint_identifier}/orders/{request_id}"
  }

Editing a S3Fetcher Configuration

You can edit a S3Fetcher configuration in a Fabric-enabled Logpoint using the PluginConfiguration - Edit API.

Endpoint URL:

https://api-server-host-name/configapi/{pool_UUID}/{logpoint_identifier}/LogSources/{id}

Method:

PUT

Parameters:

Field	Label in UI	Type	Description
id	–	String	S3Fetcher UUID. Obtain it using the Devices - List API. Mandatory

Request Example:

{
  "data": {
   "name": "S3Changed",
    "type": "S3Fetcher",
    "vendor_name": "AWS",
    "logo": " ",
    "description": "Fetches various AWS logs stored in Amazon S3 buckets, including but not limited to CloudTrail logs for AWS API activity, VPC Flow Logs for network traffic analysis, and MySQL RDS logs.",
    "config": {
      "Source": {
        "name": "S3Changed",
        "interval": 60,
        "charset": "utf_8",
        "timezone": "UTC"
      },
      "Connector": {
        "endpoint_url": "https://s3.amazonaws.com",
        "access_key": "AKIATOEJXHJKHDVGQZTZ5E",
        "secret_key": "ihBavc5Tu383dkjhcT1Z+gmWD3VNWewGbXw3Dohm",
        "bucket_name": "lp-plugins-bucket",
        "region_name": "eu-west-1",
        "filter_date":"2024-07-17T10:56:04.142Z",
        "parser": "LineParser",
        "enable_proxy": false,
        "protocol": "http"
      },
      "RoutingPolicy": {
        "routing_criterion": [],
        "catch_all": "_logpoint"
      },
      "NormalizationPolicy": {
        "normalizers": []
      }
    },
    "documentation_link": "https://docs.logpoint.com/docs/s3fetcher/en/latest/index.html",
    "id": "LogSources/102e3201-3b0c-4d46-9b49-886e652e4c79"
  }
}

Success Response:

 {
   "status": "Success",
   "message": "monitorapi/{pool_UUID}/{logpoint_identifier}/orders/{request_id}"
}

Deleting a S3Fetcher Configuration

You can delete an S3Fetcher configuration in a Fabric-enabled Logpoint using the PluginConfiguration - Trash API.

Endpoint URL:

https://api-server-host-name/configapi/{pool_UUID}/{logpoint_identifier}/LogSources/{id}

Method:

Delete

Parameters:

FIELD	LABEL IN UI	TYPE	DESCRIPTION	REQUIRED
id	–	String	S3Fetcher UUID. Obtain it using the Devices - List API.	Mandatory

Success Response:

 {
   "status": "Success",
   "message": "monitorapi/{pool_UUID}/{logpoint_identifier}/orders/{request_id}"
}