Other Collectors and Fetchers

Other Collectors and Fetchers

While the most widely used collector or fetcher is the SysLog Collector, there are other out-of-the-box Collectors and Fetchers.

Collectors	Fetchers
Snare Collector	Windows Management Instrumentation (WMI)
FTP Collector	FTP Fetcher
SNMP Trap Collector	SCP Fetcher
SFlow Collector	SNMP2 Fetcher
File System Collector	SDEE Fetcher

---

Snare Collector

Snare Collector collects and analyzes logs from the Windows Snare agent.

Using Snare Collector

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click Snare Collector.

4. Select a Parser, a Processing Policy, and a Charset from the dropdowns.

5. Click Submit.

---

FTP Collector

FTP Collector collects logs from the files uploaded by users using FTP clients. You can add multiple FTP collectors for a single device. Use an FTP client to forward logs to the FTP collector. We recommend FTP Rush or Filezilla.

Configuring FTP Collector in Logpoint

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the " Add collectors/fetchers icon under Actions.

3. Click FTP Collector to see a list of all the FTP Collectors configured for the device.

4. Click Add.

5. Enter a Username and a Password. These credentials are required to configure settings on the client’s side.

6. Enter a Source Name, which is a unique identifier for the collector.

7. Select a Parser, a Processing Policy, and a Charset to apply to the logs.

8. Click Submit.

Configuring FTP Rush or Filezilla

1. Enter the address of Logpoint in Host.

2. Enter the Username and the Password of the FTP Collector.

3. Use Port 21.

4. Click Enter or Quick Connect to connect to the Logpoint.

5. Drag the log files from the Local site and drop it in the Remote site. The files are transferred to Logpoint.

---

SNMP Trap Collector

SNMP Trap Collector collects logs from SNMP-enabled devices. SNMP traps are alert messages devices use to notify the SNMP manager about significant events.

Configuring SNMP Trap Collector in Logpoint

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click SNMP Trap Collector.

4. Select an SNMP Version.

   1. For v_12, enter the Community String.

   2. For v_3, enter a Username, Authorization Key, Security Engine ID, and Private Key. The Authorization Key must contain at least 8 characters.

5. Select a Processing Policy to apply to the logs.

6. Click Submit.

Configuring SNMP for Windows

1. Install Simple Network Management Protocol (SNMP) from Turn Windows feature on or off in the Control Panel.

2. Run services.msc command.

3. Search for the SNMP Service. Right-click on it and select Properties.

4. Select the TRAPS tab.

5. Add the Community name and Trap destinations.

6. Click OK.

7. To manually forward different SNMP traps:

   1. Run the evntwin command and select custom.

   2. Click Edit and add the event sources.

   3. Click OK.

---

sFlow Collector

sFlow monitors networks, wireless, and host devices. The sFlow Collector forwards counter and flow samples using UDP or ARP. Ensure the sFlow Package is installed. The default port for sFlow is 6343.

Configuring sFlow Collector in Logpoint

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click the sFlow Collector.

4. Select a Processing Policy to apply to the logs.

5. Click Submit.

Logpoint Taxonomy for sFlow Collector

sFlow Field	Logpoint Field
agent_ip_address	host_address
cs_ethernet_dot3_stats_AlignmentErrors	alignment_error
cs_ethernet_dot3_stats_CarrierSenseErrors	carrier_sense_error
cs_ethernet_dot3_stats_DeferredTransmissions	deferred_transmission
cs_ethernet_dot3_stats_ExcessiveCollisions	excessive_collision
cs_ethernet_dot3_stats_FCSErrors	fcs_error
cs_ethernet_dot3_stats_FrameTooLongs	frame_too_long
cs_ethernet_dot3_stats_InternalMacReceiveErrors	mac_receive_error
cs_ethernet_dot3_stats_InternalMacTransmitErrors	mac_transmit_error
cs_ethernet_dot3_stats_LateCollisions	late_collision
cs_ethernet_dot3_stats_MultipleCollisionFrames	multiple_collision_frame
cs_ethernet_dot3_stats_SingleCollisionFrames	single_collision_frame
cs_ethernet_dot3_stats_SQETestErrors	sqe_test_error
cs_ethernet_dot3_stats_SymbolErrors	symbol_error
cs_generic_if_direction	direction
cs_generic_if_if_status	status_code
cs_generic_if_in_bcast_pkts	in_broadcast_packet
cs_generic_if_in_discards	in_discard
cs_generic_if_in_errors	in_error
cs_generic_if_in_mcast_pkts	in_multicast_packet
cs_generic_if_in_octets	in_octet
cs_generic_if_in_ucast_pkts	in_unicast_packet
cs_generic_if_in_unknown_proto	in_unknown_protocol
cs_generic_if_index	if_index
cs_generic_if_out_bcast_pkts	out_broadcast_packet
cs_generic_if_out_discards	out_discard
cs_generic_if_out_errors	out_error
cs_generic_if_out_mcast_pkts	out_multicast_packet
cs_generic_if_out_octets	out_octet
cs_generic_if_out_ucast_pkts	out_unicast_packet
cs_generic_if_promisc	if_promiscuous
cs_generic_if_speed	if_speed
cs_generic_if_type	if_type
fs_input_if_format	input_if_format
fs_input_if_value	input_if_value
fs_output_if_format	output_if_format
fs_output_if_value	output_if_value
fs_rph_frame_length	frame_length
fs_rph_header_protocol	header_protocol
fs_rph_header_size	header_size
fs_rph_sample_dst_ip	destination_address
fs_rph_sample_dst_mac	destination_hardware_address
fs_rph_sample_dst_port	destination_port
fs_rph_sample_eth_type	ethernet_type
fs_rph_sample_ip4_flags	ip4_flag
fs_rph_sample_ip_version	ip_version
fs_rph_sample_protocol	protocol
fs_rph_sample_sender_ip_address	source_address
fs_rph_sample_sender_mac_address	sender_hardware_address
fs_rph_sample_src_ip	source_address
fs_rph_sample_src_mac	source_hardware_address
fs_rph_sample_src_port	source_port
fs_rph_sample_target_ip_address	destination_address
fs_rph_sample_target_mac_address	target_hardware_address
fs_rph_sample_tcp_flags	tcp_flag
fs_rph_sample_vlan_id	network_id
fs_rph_stripped	rph_stripped
fs_sample_pool	sample_pool
fs_sampling_rate	sampling_rate
fs_sequence_number	sequence_number
fs_source_id_index	source_id_index
fs_source_id_type	source_id_type
switch_uptime	duration

---

File System Collector

File System Collector collects logs from Logpoint file systems, allowing you to monitor file access, changes, and other activities. It is only applied to the localhost device. The File System collector processes all internal, generated logs captured from collectors, web servers, mergers, normalizers, services and integrations.

File System Collector can only access log files in /var/log/ and /opt/immune/var/log. To add other file paths, create a support ticket.

Configuring File System Collector in Logpoint

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click File System Collector.

4. Click ADD.

5. Enter the File Path and Exclude Paths.

6. Select a Parser, a Processing Policy, and a Charset.

7. Click Submit.

---

Fetchers

WMI Fetcher

The Windows Management Instrumentation (WMI) Fetcher retrieves Windows logs.

Configuring WMI Fetcher in Logpoint

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click WMI Fetcher.

4. Enter the Username and Password of Windows configured for the WMI Fetcher.

If the Windows user is in a domain, the username must be in the format domain\username (domain\username is invalid).

1. Enter the frequency at which data is retrieved in Fetch Interval (minutes).

2. Select Parser.

3. In Facility, select the code used to specify the system that has the log.

4. Select Severity.

5. Select a Processing Policy.

6. Select an encoding format from Charset.

7. Click Submit.

Configuration of Windows for WMI

1. Go to Control Panel >> Administrative Tools >> Components Services in the Windows device.

2. On Components Services, expand Component Services >> Computers.

3. Right-click My Computer and select Properties.

4. Select COM Security.

5. Click Edit Limits in Launch and Activation Permissions.

6. Click Add.

7. Click Advanced.

8. Click Find Now.

9. Select a user and click OK.

10. The username is displayed in the name field in Select Users or Groups.

11. Click OK.

12. In Launch Permission, select Remote Launch and Remote Activation.

13. Click OK.

Configuration of Windows for WMI with Non-admin Rights

1. Create a new user from Administrative Tools >> Active Directory Users and Computers in the Windows device.

2. Add users to the following groups.

   • Distributed COM users

   • Performance monitor users

   • Event log readers

3. Open WMI Control console.

   1. Click Start.

   2. Click Run.

   3. Type wmimgmt.msc.

   4. Click OK.

4. Right-click on WMI control and click Properties.

5. Click Security and click Add.

6. In Select Users, Computers, or Groups, enter the name of Performance monitor users.

7. In Security, under Permissions, select Permissions. Add Remote Enable and Read Security.

8. Assign the user to use Component Services.

9. Go to Component Services under Administrative Tools.

10. On Components Services, expand Component Services/Computers.

11. Right-click My Computer and select Properties.

12. Select COM Security.

13. Grant Access Permissions (Remote Access) and Launch and Activation Permissions (Remote Launch and Remote Activation) to the newly created user.

---

FTP Fetcher

FTP Fetcher fetches logs from the relative file path of the FTP server. You can configure FTP Fetcher from Log Source or Devices. We recommend configuring it from the log source, as it provides a centralized User Interface for all configurations.

Configuration from Log Source

FTP Fetcher consists of the log source template, FTP Fetcher, which has pre-defined settings and configurations to fetch logs. However, some fields in the template must be configured manually.

To configure FTP Fetcher from Log Source:

1. Go to Settings >> Log Sources in the navigation bar and click + Add Log Source.

2. Click + Create New and select FTP Fetcher.

Source

Add details about the FTP server from where the FTP Fetcher fetches logs

1. Click Source.

2. Enter the Log Source’s Name.

3. Enter the IP addresses of the device whose logs are to be monitored in Device Addresses.

4. Select Device Groups.

5. Select the Time Zone.

6. Select the risk values for Confidentiality, Integrity, and Availability.

Connector

Configure how the FTP Fetcher and the FTP server communicate.

1. Click Connector.

2. Enter the FTP server Username and Password. When editing FTP Fetcher, you must re-enter the password.

3. Enter the Port on which the FTP server is running. The default port is 21.

4. Enter the frequency at which data is retrieved in Fetch Interval (min). If logs are not received for two consecutive intervals, the log source is marked as inactive in Last Log Received under Settings >> Log Source.

5. If you are using a Distributed Logpoint, select Distributed Collector from the dropdown.

Fetchers

Configure the file path from where logs are fetched.

1. Click Fetchers.

2. Enter the Relative FilePath.

3. Enter a valid Python regular expression (regex) in Filename Pattern that matches the files to fetch from the FTP server. Example: ^.\.txt$* matches all files ending in .txt.

Glob-style patterns are not supported (e.g. .txt, .log, etc). You must replace the glob-style pattern with an equivalent Python regex, for example:

• Instead of .txt, use .*\.txt$

• Instead of .log, use ^.*\\.log$

• Instead of logs-.json, use ^logs--.json%$

1. Select Forward Old Logs to fetch logs from the file before configuring the log source.

2. Select a Parser to parse the logs.

3. Select the Charset.

Routing

Create repos and routing criteria for FTP Fetcher. Repos are locations where incoming logs are stored, and routing criteria are created to determine the conditions under which these logs are sent to repos.

To create a repo:

1. Click Routing and + Create Repo.

2. Enter a Repo name.

3. In Path, enter the location to store incoming logs.

4. In Retention (Days), enter the number of days logs are kept in a repository before they are automatically deleted.

5. In Availability, select the Remote logpoint and Retention (Days).

6. Click Create Repo.

7. In Repo, select the created repo to store logs.

To create Routing Criteria:

1. Click + Add row.

2. Enter a Key and Value. The routing criteria are only applied to those logs that have this key-value pair.

3. Select an Operation for logs that have this key-value pair.

   1. Select Store raw message to store both the incoming and the normalized logs in the selected repo.

   2. Select Discard raw message to discard the incoming logs and store the normalized ones.

   3. Select Discard entire event to discard both the incoming and the normalized logs.

4. In Repository, select a repo to store logs.

Click the Delete icon under Action to delete the created routing criteria.

Normalization

Select normalizers for the incoming logs. Normalizers translate a raw log message into the

1. Click Normalization.

2. You can either select a previously created normalization policy from the Select Normalization Policy dropdown or select a Normalizer from the list and click the swap icon.

Enrichment

Select an enrichment policy for the incoming logs. Enrichment Policies are used to add additional information to a log, such as user information, device type or geolocation.

1. Click Enrichment.

2. Select an Enrichment Policy.

1. Click Create Log Source to save the configurations of Source, Connector, Fetchers, Routing, Normalization, and Enrichment.

Configuration from Devices

Configuring FTP Fetcher from Devices

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click FTP Fetcher.

4. Click ADD.

5. Enter Username and Password. When editing FTP Fetcher, you must re-enter the password.

6. Enter Port on which the FTP is running. The default port is 21.

7. Enter the Relative FilePath

8. Enter a valid Python regular expression (regex) in Filename Pattern that matches the files to fetch from the FTP server. Example: ^.\.txt$* matches all files ending in .txt.

   Glob-style patterns are not supported (e.g. .txt, .log, etc). You must replace the glob-style pattern with an equivalent Python regex, for example:

   • Instead of .txt, use .*\.txt$

   • Instead of .log, use ^.*\\.log$

   • Instead of logs-.json, use ^logs--.json%$

9. Select Forward Old Logs to fetch logs from the file before configuring the log source.

10. Enter the frequency at which data is retrieved in Fetch Interval (min).

11. Select a Parser to parse the logs.

12. Select a Processing Policy, and a Charset.

13. Click Submit.

---

SCP Fetcher

SCP Fetcher fetches logs from the log files in a remote host using the SSH connection. You must enable SFTP (SSH File Transfer Protocol) in the remote server to fetch logs using the SCP Fetcher.

Configuring SCP Fetcher

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click SCP Fetcher.

4. Click Add.

5. Enter a Username for the fetcher.

6. Enter the Relative FilePath and the Filename Pattern to specify from which files to fetch logs.

7. Select Forward Old Logs to fetch logs from the file before configuring the log source.

8. Enter the Port number the SCP uses to listen for the remote server. The default SCP port is 22.

9. Enter the frequency at which data is retrieved in Fetch Interval (min).

10. Select a mode of Authentication: Password or SSH Certificate.

    1. If you choose Password, enter or type the password.

    2. If you choose SSH Certificate, Logpoint automatically generates a certificate key for you.

    Copy the password or the SSH certificate key, as it is required later for the user validation.

11. Choose a Parser, a Processing Policy, and a Charset.

12. Click Submit.

---

SNMP Fetcher

SNMP Fetcher allows you to send SNMP queries to network devices and receive responses in Logpoint. You can then use these responses as event logs for further analysis.

SNMP Fetcher needs an SNMP Policy to make the SNMP Walk query. SNMP Policy is a set of OIDs and their query time intervals.

SNMP Fetcher makes an SNMPWALK query. The query uses SNMP GETNEXT requests to get the logs from a network entity. An object identifier (OID) is used while making this query. The OID specifies all the branches of the OID tree for fetching. All variables in the sub-tree below the given OID are queried, and their values are presented to the user.

Before configuring the SNMP Fetcher, create an SNMP Policy.

Creating an SNMP Policy

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click SNMP FETCHER.

4. Click Policy and click ADD.

5. Enter the Name of the policy.

6. Enter a list of OIDs and their respective Fetch time (in minutes).

7. Click Submit.

Configuring SNMP Fetcher

1. Go to Settings >> Configuration in the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click SNMP Fetcher.

4. Select the SNMP Version of the server.

   1. For Version1/Version2 (v_12) enter Community String.

   2. For Version3 (v_3) enter Username, Authorization Key, and Private Key. Enter the Authentication Protocol and Privacy Protocol of the SNMP server from which Logpoint fetches logs.

5. Enter the Port number.

6. Select the previously created SNMP Policy. You can also apply a policy from plugins.

7. Select a Processing Policy to apply to the logs.

8. Select an encoding format from the Charset dropdown.

9. Click Submit.

---

SDEE Fetcher

Security Device Event Exchange (SDEE) is a network protocol used by security devices to communicate. You can forward network statistics from SDEE devices to Logpoint via the SDEE Fetcher.

Configuring SDEE Fetcher

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under Actions.

3. Click SDEE Fetcher and configure its essential parameters.

4. Enter Username and Password.

5. Select a Processing Policy, and a Charset.

6. Select Certificate Verification to ensure Logpoint enables connection by verifying SSL/TLS certificates.

7. Browse to upload a server certificate. If no certificate is uploaded, Logpoint uses a default certificate.

8. Click Submit.