Troubleshooting CloudWatch

Common Issues and Solutions

Installation Issues

Issue: Integration fails to install

  • Solution: Verify Logpoint version compatibility (v7.8.0 or later)

  • Solution: Check available disk space and system resources

  • Solution: Ensure proper administrative privileges

  • Solution: Review Logpoint application logs for specific error messages

Issue: Integration not visible after installation

  • Solution: Refresh the browser and check under Settings >> System Settings >> Plugins

  • Solution: Clear browser cache

  • Solution: Restart Logpoint services if necessary

Pre-Configuration Issues in AWS

Issue: Cannot create access keys in AWS

  • Solution: Verify you have IAM user permissions or administrative access

  • Solution: Check if maximum number of access keys has been reached (limit: 2 per user)

  • Solution: Delete unused access keys before creating new ones

  • Solution: Ensure you're logged in with the correct AWS account

Issue: Access keys created but cannot retrieve them

  • Solution: Access keys can only be viewed once during creation

  • Solution: Download the key file immediately after creation

  • Solution: If keys are lost, delete them and create new ones

  • Solution: Store keys securely in a password manager

Issue: IAM permissions insufficient

  • Solution: Verify IAM policy includes required CloudWatch Logs permissions

  • Solution: Attach CloudWatchLogsReadOnlyAccess managed policy

  • Solution: Create custom IAM policy with minimal required permissions

  • Solution: Wait a few minutes for IAM policy changes to propagate

Issue: Cannot find Log Groups in CloudWatch

  • Solution: Verify CloudWatch Logs service is enabled in your region

  • Solution: Check that applications are actually writing logs to CloudWatch

  • Solution: Ensure you're viewing the correct AWS region

  • Solution: Verify IAM permissions allow listing Log Groups

Issue: Log Streams are empty or not visible

  • Solution: Verify applications are actively writing to Log Streams

  • Solution: Check Log Stream retention settings

  • Solution: Ensure correct time range is selected

  • Solution: Verify IAM permissions allow reading Log Streams

Issue: Specific AWS service logs not appearing in CloudWatch

  • Solution: Enable CloudWatch logging for the specific service (EC2, RDS, Lambda, etc.)

  • Solution: Verify service is configured to send logs to CloudWatch

  • Solution: Check service-specific log configuration settings

  • Solution: Ensure proper IAM role is attached to the service

Configuration Issues in Logpoint

Issue: Cannot configure CloudWatch Fetcher

  • Solution: Verify CloudWatch integration is properly installed

  • Solution: Check that CloudWatchFetcher appears in available fetchers

  • Solution: Ensure proper permissions in Logpoint to create fetchers

  • Solution: Review Logpoint system logs for configuration errors

Issue: Access Key ID or Secret Access Key rejected

  • Solution: Verify keys are copied correctly without extra spaces

  • Solution: Check keys are from the correct AWS account

  • Solution: Ensure keys haven't been deleted or rotated in AWS

  • Solution: Verify keys have proper IAM permissions

  • Solution: Regenerate keys if necessary

Issue: AWS Region selection incorrect

  • Solution: Select the region where your Log Groups are located

  • Solution: Verify Log Groups exist in the selected region

  • Solution: Check endpoint URL matches the selected region

  • Solution: Note that Log Groups are region-specific

Issue: Log Group or Log Stream name incorrect

  • Solution: Copy Log Group name exactly from AWS Console (case-sensitive)

  • Solution: Include full path including forward slashes (e.g., /aws/lambda/function-name)

  • Solution: Verify Log Group exists in the selected region

  • Solution: For multiple Log Streams, separate with commas without spaces

Issue: Start Date configuration causing issues

  • Solution: Select a date when logs actually exist

  • Solution: Don't select dates too far in the past if logs have been deleted

  • Solution: Consider Log Stream retention policies

  • Solution: Start with recent date and adjust as needed

Issue: Fetch Interval too frequent or infrequent

  • Solution: Balance between data freshness and API call costs

  • Solution: For high-volume logs, increase interval to reduce API calls

  • Solution: For critical monitoring, decrease interval for faster detection

  • Solution: Monitor AWS CloudWatch API throttling limits

Issue: Processing Policy configuration errors

  • Solution: Ensure normalization policy is created before processing policy

  • Solution: Verify CloudWatchCompiledNormalizer is selected

  • Solution: Check that routing and enrichment policies are properly configured

  • Solution: Test processing policy with sample logs

Data Ingestion Issues

Issue: No logs being ingested

  • Solution: Verify CloudWatch Fetcher is running in Logpoint

  • Solution: Check AWS access keys are valid and have proper permissions

  • Solution: Confirm Log Groups contain logs in the specified time range

  • Solution: Test network connectivity from Logpoint to AWS endpoints

  • Solution: Review Logpoint fetcher logs for error messages

  • Solution: Verify Log Streams are actively receiving logs

Issue: Incomplete log ingestion

  • Solution: Check fetch interval - may be too infrequent for log volume

  • Solution: Verify AWS API rate limits aren't being exceeded

  • Solution: Monitor CloudWatch Fetcher status for errors

  • Solution: Check if specific Log Streams are being skipped

  • Solution: Verify sufficient storage in Logpoint repository

Issue: Logs not normalized correctly

  • Solution: Verify CloudWatchCompiledNormalizer is selected

  • Solution: Check log format matches expected CloudWatch JSON structure

  • Solution: Ensure logs are valid JSON (not truncated or malformed)

  • Solution: Review normalization policy configuration

  • Solution: Test with known-good log samples

Issue: Duplicate logs appearing

  • Solution: Check if multiple fetchers are configured for same Log Group

  • Solution: Verify routing criteria doesn't cause duplicate storage

  • Solution: Review Start Date and fetch interval settings

  • Solution: Ensure only one Logpoint instance is fetching from the Log Group

Issue: Logs delayed or arriving late

  • Solution: Check fetch interval - increase frequency if needed

  • Solution: Verify network latency between Logpoint and AWS

  • Solution: Monitor Logpoint system performance and resource usage

  • Solution: Check AWS CloudWatch Logs are being written in real-time

  • Solution: Review CloudWatch Logs retention and delivery settings

Issue: Timestamp parsing errors

  • Solution: Verify logs contain valid timestamp fields

  • Solution: Check Logpoint timezone configuration

  • Solution: Ensure CloudWatch eventTime format is supported

  • Solution: Review normalizer timestamp parsing logic

Issue: Missing fields in normalized logs

  • Solution: Verify CloudWatch logs include all expected fields

  • Solution: Check normalizer field mapping for CloudWatch format

  • Solution: Ensure log source configuration includes all required fields

  • Solution: Review AWS CloudWatch documentation for log structure

Issue: Cannot retrieve logs from specific Log Stream

  • Solution: Verify Log Stream name is spelled correctly

  • Solution: Check Log Stream exists and contains logs

  • Solution: Ensure IAM permissions include access to specific Log Stream

  • Solution: Verify Log Stream hasn't been deleted or archived

Performance Issues

Issue: Slow query performance

  • Solution: Optimize queries by adding time range constraints

  • Solution: Use indexed fields in search queries where possible

  • Solution: Consider data retention policies to manage repository size

  • Solution: Filter by specific Log Groups or event types

Issue: High resource usage by fetcher

  • Solution: Monitor CloudWatch Fetcher resource consumption

  • Solution: Adjust fetch interval to reduce API call frequency

  • Solution: Implement log filtering using routing criteria

  • Solution: Consider fetching from fewer Log Streams simultaneously

Issue: AWS API throttling

  • Solution: Reduce fetch frequency to stay within API limits

  • Solution: Request AWS to increase CloudWatch Logs API quota

  • Solution: Distribute fetching across multiple time periods

  • Solution: Use more specific Log Stream filters

  • Solution: Monitor AWS CloudWatch API usage metrics

Issue: High log volume impacting Logpoint

  • Solution: Implement routing criteria to filter unnecessary logs

  • Solution: Adjust CloudWatch Logs retention in AWS to reduce volume

  • Solution: Use Log Stream filtering to ingest only critical logs

  • Solution: Scale Logpoint infrastructure to handle volume

  • Solution: Aggregate or sample logs before sending to Logpoint

AWS Service-Specific Issues

Issue: EC2 instance logs not appearing

  • Solution: Verify CloudWatch Logs agent is installed on EC2 instances

  • Solution: Check EC2 IAM role has CloudWatch Logs write permissions

  • Solution: Ensure agent configuration points to correct Log Group

  • Solution: Verify EC2 security group allows outbound HTTPS

Issue: RDS database logs missing

  • Solution: Enable CloudWatch Logs export in RDS instance settings

  • Solution: Verify log types are enabled (error, slow query, general, audit)

  • Solution: Check RDS parameter group includes log settings

  • Solution: Ensure sufficient storage for RDS logs

Issue: Lambda function logs not captured

  • Solution: Verify Lambda execution role has CloudWatch Logs permissions

  • Solution: Check function is actually being invoked

  • Solution: Ensure function logs to console/stdout (which goes to CloudWatch)

  • Solution: Review Lambda Log Group naming convention (/aws/lambda/function-name)

Issue: ELB access logs not available

  • Solution: Enable access logging on the Load Balancer

  • Solution: Verify S3 bucket for logs is accessible

  • Solution: Check Load Balancer log configuration

  • Solution: Note: ELB logs typically go to S3, not directly to CloudWatch Logs

Issue: VPC Flow Logs not appearing

  • Solution: Enable VPC Flow Logs for the VPC or subnet

  • Solution: Verify Flow Logs are configured to publish to CloudWatch Logs

  • Solution: Check IAM role for Flow Logs has proper permissions

  • Solution: Ensure correct Log Group is specified in Flow Logs configuration

Issue: CloudTrail logs not in CloudWatch

  • Solution: Configure CloudTrail to send logs to CloudWatch Logs

  • Solution: Verify CloudTrail trail is enabled and logging

  • Solution: Check CloudTrail IAM role has CloudWatch Logs permissions

  • Solution: Ensure CloudTrail Log Group exists and is accessible

Authentication and Permission Issues

Issue: Access denied errors

  • Solution: Verify IAM policy includes all required CloudWatch Logs permissions

  • Solution: Check access keys belong to user/role with proper permissions

  • Solution: Ensure IAM policy is attached to the correct user/role

  • Solution: Wait for IAM policy changes to propagate (up to 5 minutes)

Issue: Cannot list Log Groups

  • Solution: Grant logs:DescribeLogGroups permission

  • Solution: Verify user has permissions for the specific region

  • Solution: Check for service control policies (SCPs) blocking access

  • Solution: Ensure account is not in a suspended state

Issue: Cannot read Log Streams

  • Solution: Grant logs:DescribeLogStreams and logs:GetLogEvents permissions

  • Solution: Verify Log Group ARN in IAM policy is correct

  • Solution: Check for resource-based policies restricting access

  • Solution: Ensure Log Group hasn't been encrypted with KMS requiring additional permissions

Issue: Cross-account access not working

  • Solution: Configure cross-account IAM role with proper trust relationship

  • Solution: Verify assume role permissions in source account

  • Solution: Check CloudWatch Logs resource policies allow cross-account access

  • Solution: Use temporary security credentials (STS) for cross-account access

Network and Connectivity Issues

Issue: Cannot connect to AWS CloudWatch API

  • Solution: Verify Logpoint server has internet connectivity

  • Solution: Check firewall rules allow outbound HTTPS (port 443)

  • Solution: Ensure DNS resolution for AWS endpoints works

  • Solution: Test connectivity: curl https://logs.{region}.amazonaws.com

  • Solution: Verify proxy settings if Logpoint uses a proxy

Issue: Intermittent connection failures

  • Solution: Check network stability between Logpoint and AWS

  • Solution: Monitor AWS service health dashboard

  • Solution: Implement retry logic in fetcher configuration

  • Solution: Review Logpoint network logs for patterns

Issue: SSL/TLS certificate errors

  • Solution: Verify Logpoint system time is accurate

  • Solution: Update system CA certificates

  • Solution: Check for SSL inspection/proxy interference

  • Solution: Ensure TLS 1.2 or higher is supported

Multi-Region and Multi-Account Issues

Issue: Logs from multiple regions not appearing

  • Solution: Create separate fetchers for each region

  • Solution: Each region requires separate endpoint URL

  • Solution: Verify

Last updated

Was this helpful?