Built-in Collectors¶

Syslog Collector¶

The Syslog Collector is used to collect data from the sources following the Syslog protocol. Once you add a device, it can be utilized either as a proxy or as a device depending on its configuration mode.

Configuring Syslog Collector in LogPoint

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under the Actions column of the device.

3. Click Syslog Collector.

4. In the Syslog Collector section, select a Parser, a Processing Policy, and a Charset from the drop-downs.

   Note

   • The ProofPointEmailProtectionLogParser joins the logs having fields with the same session_ids.

   • The EmailParser joins the logs having the same queue_id or message_id.

   A system configured with these parsers display the search results of the collected logs within a time range of 10 seconds.

5. In the Proxy Server section, choose either Use as Proxy, Uses proxy or None.

   1. If you select None, the device works as a Syslog Collector.

   2. If you select Use as Proxy, the device is used as a proxy.

   

   Syslog Collector (Use as Proxy)¶

   Note

   The Processing Policy box disappears once you select Use as Proxy. This is because the logs coming from a proxy device do not need to be normalized and stored.

   • If you select Uses Proxy, the device uses a proxy device to collect the logs. Select an appropriate Proxy IP from the drop-down menu. The IP address must be of a device used as a proxy. Now, provide the device’s HostName.

   

   Syslog Collector (Uses Proxy)¶

   Note

   • The hostname of a proxy device is case-sensitive.

   • The Parser and Charset options disappear once you select the Uses Proxy option. This is because the parser and charset values added for the proxy device are used for all the devices using that proxy.

   For each proxy, a device can have multiple hostnames. The hostnames for all the devices using the same syslog proxy must also be unique. LogPoint takes the first word after the log timestamp as HostName. Hence, the syslog proxy sending the logs in the following formats is supported:

   • Standard RFC 3164, with format:

     <PRI> MTH DD HH:MM:SS Hostname LogContent

     For example:
     
     <12> Aug 12 13:14:59 HostName1 This is test1
     

   • Slightly modified syslog format with year mentioned:

     <PRI> YYYY MTH DD HH:MM:SS Hostname LogContent

     For example:
     
      <13> 2015 Apr 8 05:28:13 HostNameA [warning] 1: #1192 Invalid IP resolution
      2015 Apr 8 05:28:13 HostNameA [warning] 1: #1192 Invalid IP resolution
     

   • If a device not configured as proxy sends logs via a configured syslog proxy device, then it is assumed to be malicious, and hence, its events are rejected.

6. Click Submit.

   Note

   LogPoint also supports the following RFC5424 log format:

   Standard RFC5424 format:

   <PRI> [PRIVAL] [FULL-DATE]T[FULL-TIME] Log Content

   Here,

   FULL-DATE = DATE-FULLYEAR “-” DATE-MONTH “-” DATE-MDAY

   FULL-TIME = PARTIAL-TIME TIME-OFFSET

   PARTIAL-TIME = TIME-HOUR “:” TIME-MINUTE “:” TIME-SECOND [TIME-SECFRAC]

   TIME-SECFRAC = “.” 1*6DIGIT

   IME-OFFSET = Z / (“+” / “-“) TIME-HOUR “:” TIME-MINUTE

   For example:
   
   <165>1 2018-09-14T09:15:15.000003-07:00 Off set value -7
   <165>1 2018-09-14T09:15:15.000003-11:00 Off set value -11
   

   Additionally, the assignment mechanism of log_ts in the syslog parser also supports the format. The offset value of timezone is prioritized over the device timezone while assigning a value to the log_ts. This means that, if a log contains an offset timezone value and also has a device timezone, the log_ts is equal to the offset timezone.

Before you start receiving logs, you need to configure the settings on the log source as well. Refer below to configure your Linux or Windows machines to forward logs to LogPoint.

Configuration for Linux devices

1. Open the syslog configuration.

   /etc/syslog.conf
   

2. Add your remote server in this format.

   *.*     @logserv.example.com:port
   
   
   Example:-
   
   *.*     @192.168.2.205:514
   

3. Restart Syslog facility.

Configuring Linux devices for forwarding SSL encrypted log message

1. Install stunnel, and type the following command in the terminal.

   sudo apt-get install stunnel
   

2. Copy /usr/share/doc/stunnel4/examples/stunnel.conf-sample to /etc/stunnel/stunnel.conf.

3. Go to /etc/stunnel/stunnel.conf and modify as follows (create it if doesn’t exist).

   client=yes
   

   Now, on the same config file add:

   [sslsyslog]
   accept  = 127.0.0.1:60515
   connect = 192.168.2.200:515
   

4. Restart the config file with:

   stunnel4/etc/stunnel/stunnel.conf
   

5. Open /etc/syslog.conf with the administrative privilege.

6. Forward the logs to LogPoint server.

   *.*      @@127.0.0.1:60515
   

7. Restart syslogd facility with:

   service rsyslog restart
   

8. Go to LogPoint server.

9. Now, select the Linux Device from Settings >> Configuration >> Devices and click Add Collectors/Fetchers.

Configuration for Windows devices

You can forward logs from Windows devices to LogPoint using third-party agents that forward the logs in the syslog format. One recommended tool is the InterSect alliance’s Snare for Windows.

1. Go to Network Configuration tab in the Snare application.

2. Provide the IP address of your LogPoint in the Destination Snare Server Address field.

3. Provide port number 514 in the Destination Port.

4. Click Change Configuration.

5. Go to Apply the Latest Audit Configuration tab and press Reload Settings.

Sequence Numbering in logs collected from Syslog Collector

A sequence number is assigned per device per protocol to each log collected from the Syslog Collector. This helps you identify the order of the logs received from a particular device.

The log collected from a device with the device ip 192.168.0.135 and communicating via the TCP protocol is as follows:

The sequence number for the above log is 41, which is shown as the field value for seq_num_tcp. It means that this log is the 41^st TCP log message received from the device with the device ip 192.168.0.135.

Similarly, the log collected from a device with the device ip 192.168.0.135 and communicating via the UDP protocol is as follows:

The sequence number for the above log is 83, which is shown as the field value for seq_num_udp. It means that this log is the 83^rd UDP log message received from the device with the device ip 192.168.0.135.

And, the log collected from a device with the device ip 192.168.0.135 and communicating via the SSL protocol is as follows:

The sequence number for the above log is 68, which is shown as the field value for seq_num_ssl. It means that this log is the 68^th SSL log message received from the device with the device ip 192.168.0.135.

Configuring Sequence Numbering

1. Go to Settings >> System Settings from the navigation bar and click System Settings.

2. Go to General tab, scroll the page until you find the Sequence Numbering section.

3. Select the Add sequence numbers on log received from syslog collector checkbox.

4. Click Save.

When completing the configuration of sequence numbering, every log is provided with a sequence number on an incremental basis starting from 1.

Snare Collector¶

The Snare Collector collects and analyzes logs from the Windows Snare agent.

Configuring Snare Collector in LogPoint

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under the Actions column of the device.

3. Click Snare Collector.

4. Select a Parser, a Processing Policy, and a Charset from the drop-downs.

5. Click Submit.

Configuration for Windows devices

You can forward logs from Windows devices to LogPoint using third-party agents that forward the logs in syslog format. One recommended tool is the InterSect alliance’s SNARE for Windows.

Follow the steps below to configure SNARE for windows.

1. Go to the Network Configuration tab in the Snare Application.

2. Provide the IP address of your LogPoint in the Destination Snare Server Address field.

3. Provide port number 6161 in the Destination Port.

4. Click Change Configuration.

5. Click the Apply the Latest Audit Configuration tab and then Reload Settings.

FTP Collector¶

The FTP Collector collects logs from the files uploaded by users using FTP clients.

Configuring FTP Collector in LogPoint

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under the Actions column of the device.

3. Click FTP Collector to see a list of all the FTP Collectors configured for the device.

4. Click Add.

4. Enter a Username and a Password. These credentials are needed to configure settings on the client’s side.

5. Provide a Source Name. This is the unique identifier for the collector.

6. Select a Parser, a Processing Policy, and a Charset to apply over the logs.

7. Click Submit.

Configuring FTP Collector in FTP Clients

Follow the steps below to configure FTP Rush or Filezilla.

1. Provide the Host address, i.e., the address of LogPoint.

2. Provide the Username and the Password of the FTP Collector.

3. Use Port 21.

4. Press Enter or Quick Connect to connect to the LogPoint.

5. Now drop the log files in the remote tab.

6. The files are now transferred to the LogPoint.

SNMP Trap Collector¶

The SNMP Trap Collector collects logs from SNMP enabled devices. SNMP traps are alert messages that devices use to notify the SNMP manager about the occurrence of significant events.

Configuring SNMP Trap Collector in LogPoint

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under the Actions column of the related device.

3. Click SNMP Trap Collector.

4. Choose an SNMP Version.

   • For v_12, provide the Community String.

   • For v_3, provide a Username, Authorization Key, Security Engine ID, and Private Key.

   Note

   The Authorization Key must contain at least 8 characters.

5. Select a Processing Policy to apply over the logs.

6. Click Submit.

Configuring SNMP for Windows

1. Install Simple Network Management Protocol (SNMP) from Turn Windows feature on or off in the Control Panel.

2. Run services.msc command.

3. Search for the SNMP Service. Right click on it and select Properties.

4. Select TRAPS tab.

5. Add Community name and Trap destinations.

6. Click OK.

7. To manually forward different SNMP traps:

   7.1. Run evntwin command and select custom option.

   7.2. Click Edit and add the event sources.

   7.3. Click OK.

sFlow Collector¶

sFlow is a sampling technology used to monitor networks, wireless and host devices. The sampled packets are called flow packets. You can forward the flow packets into LogPoint via the sFlow Collector.

Configuring sFlow Collector in LogPoint

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under the Actions column of the device.

3. Click sFlow Collector.

4. Select a Processing Policy to apply over the logs.

5. Click Submit.

Starting from LogPoint v6.12.0, the following fields have been renamed according to the LogPoint taxonomy for the sFlow Collector:

sFlow Fields¶

Previously Used Field Name	New Field Name
agent_ip_address	host_address
cs_ethernet_dot3_stats_AlignmentErrors	alignment_error
cs_ethernet_dot3_stats_CarrierSenseErrors	carrier_sense_error
cs_ethernet_dot3_stats_DeferredTransmissions	deferred_transmission
cs_ethernet_dot3_stats_ExcessiveCollisions	excessive_collision
cs_ethernet_dot3_stats_FCSErrors	fcs_error
cs_ethernet_dot3_stats_FrameTooLongs	frame_too_long
cs_ethernet_dot3_stats_InternalMacReceiveErrors	mac_receive_error
cs_ethernet_dot3_stats_InternalMacTransmitErrors	mac_transmit_error
cs_ethernet_dot3_stats_LateCollisions	late_collision
cs_ethernet_dot3_stats_MultipleCollisionFrames	multiple_collision_frame
cs_ethernet_dot3_stats_SingleCollisionFrames	single_collision_frame
cs_ethernet_dot3_stats_SQETestErrors	sqe_test_error
cs_ethernet_dot3_stats_SymbolErrors	symbol_error
cs_generic_if_direction	direction
cs_generic_if_if_status	status_code
cs_generic_if_in_bcast_pkts	in_broadcast_packet
cs_generic_if_in_discards	in_discard
cs_generic_if_in_errors	in_error
cs_generic_if_in_mcast_pkts	in_multicast_packet
cs_generic_if_in_octets	in_octet
cs_generic_if_in_ucast_pkts	in_unicast_packet
cs_generic_if_in_unknown_proto	in_unknown_protocol
cs_generic_if_index	if_index
cs_generic_if_out_bcast_pkts	out_broadcast_packet
cs_generic_if_out_discards	out_discard
cs_generic_if_out_errors	out_error
cs_generic_if_out_mcast_pkts	out_multicast_packet
cs_generic_if_out_octets	out_octet
cs_generic_if_out_ucast_pkts	out_unicast_packet
cs_generic_if_promisc	if_promiscuous
cs_generic_if_speed	if_speed
cs_generic_if_type	if_type
fs_input_if_format	input_if_format
fs_input_if_value	input_if_value
fs_output_if_format	output_if_format
fs_output_if_value	output_if_value
fs_rph_frame_length	frame_length
fs_rph_header_protocol	header_protocol
fs_rph_header_size	header_size
fs_rph_sample_dst_ip	destination_address
fs_rph_sample_dst_mac	destination_hardware_address
fs_rph_sample_dst_port	destination_port
fs_rph_sample_eth_type	ethernet_type
fs_rph_sample_ip4_flags	ip4_flag
fs_rph_sample_ip_version	ip_version
fs_rph_sample_protocol	protocol
fs_rph_sample_sender_ip_address	source_address
fs_rph_sample_sender_mac_address	sender_hardware_address
fs_rph_sample_src_ip	source_address
fs_rph_sample_src_mac	source_hardware_address
fs_rph_sample_src_port	source_port
fs_rph_sample_target_ip_address	destination_address
fs_rph_sample_target_mac_address	target_hardware_address
fs_rph_sample_tcp_flags	tcp_flag
fs_rph_sample_vlan_id	network_id
fs_rph_stripped	rph_stripped
fs_sample_pool	sample_pool
fs_sampling_rate	sampling_rate
fs_sequence_number	sequence_number
fs_source_id_index	source_id_index
fs_source_id_type	source_id_type
switch_uptime	duration

File System Collector¶

The File System Collector reads the contents of the configured files instead of listening on a port. It is only applied to the localhost device to monitor the log files. The File System collector processes all the internal logs generated in LogPoint. It captures all the logs from collectors, web servers, mergers, normalizers, and all the other applications in LogPoint.

Configuring File System Collector in LogPoint

1. Go to Settings >> Configuration from the navigation bar and click Devices.

2. Click the Add collectors/fetchers icon under the Actions column of the device.

3. Click File System Collector.

4. Click Add.

5. Provide File Path and Exclude Paths.

6. Select a Parser, a Processing Policy, and a Charset.

7. Click Submit.