Top five tips to ensure logs are forwarding correctly to New Relic.
New Relic offers robust log management capabilities that connect your log data with the rest of your application and infrastructure data. This enables you to gain a holistic view of your system and reduces mean time to resolution (MTTR). However, given the complex nature of systems involving multiple log sources, configurations and tools, ensuring accurate log forwarding to New Relic can be challenging.
In this blog, we will provide you with the top five tips to ensure your logs are correctly being forwarded to New Relic. By following these tips, you'll:
- Confirm your network access to New Relic domains
- Verify you have selected a supported log forwarder
- Send troubleshooting logs from the log forwarder for further debugging
- Access designated log files and services to confirm correct permission levels
- Check that you are correctly parsing your logs to avoid errors and truncation
Tip #1: Ensure network access to New Relic domains
To ensure proper functionality of New Relic, it is essential to establish network connectivity between your environment and New Relic domains. Since New Relic is a Software as a Service (SaaS) based offering, your environment must be able to communicate with multiple domains over the Internet. To guarantee this connectivity, add the domains listed in this document to your ‘allow’ list. These domains are essential for New Relic to perform optimally, and blocking them can lead to issues with product features or prevent pages from loading entirely.
To verify your connection to New Relic domains, you can perform the following ping test.
ping collector.newrelic.comExecuting the above command will help confirm the connectivity between your system and the New Relic collector domain.
Please note that if you are using a proxy, it is necessary to include your proxy variables as part of your infrastructure agent configuration settings. This will ensure that your logs are properly forwarded to New Relic.
Important! If you encounter any issues while trying to connect to New Relic, it is most likely due to network connectivity problems. Consult with your network team for potential workarounds, or seek additional whitelisting approvals.
Tip #2: Use supported log forwarders
While using the New Relic infrastructure agent is one way to manage logs, it's not the only option available to you. If you don't need the host metrics that infrastructure log forwarding provides or if you already have a log forwarder in place, you can choose from other supported methods to forward your logs to New Relic. New Relic provides a comprehensive list of log forwarders, including those for popular cloud platforms such as AWS, Azure, and GCP.
Before you explore alternative log forwarding options, let's first understand how log forwarding works.
How log forwarding works
Logstash plugin for log forwarding
If your log data is already being monitored by Logstash, you can use our Logstash plugin to forward and enrich your log data in New Relic. By forwarding your Logstash logs to New Relic, you can take advantage of enhanced log management capabilities. This includes the ability to collect, process, explore, query, and set alerts on your log data more efficiently. Install and configure the Logstash plugin for sending logs to New Relic.
Logs API
If the existing New Relic log forwarding solutions don’t fulfill your requirements, you can use the Log API to directly send log data to New Relic via an HTTP endpoint. The platform is built around the four fundamental telemetry data types: metrics, events, logs, and traces, which are necessary for effective system monitoring. These data types are exposed via an API, often called "MELT" in the observability industry.
Here is a simple Python example for sending logs to the New Relic Logs API.
import json
import requests
# Endpoint for New Relic Log API
url = "https://log-api.newrelic.com/log/v1"
# Headers to include the content-type and Api-Key
headers = {
'Content-type': 'application/json',
'Api-Key': 'INGEST_KEY'
}
# Loading the JSON file
with open('./sampleLogs.json') as file:
sampleLogs = json.load(file)
# Post request to the url with the sampleLogs data and headers
response = requests.post(url, json=sampleLogs, headers=headersYou can also use curl to send logs to New Relic Logs API. Below is a simple example of forwarding logs to New Relic Logs API using curl.
curl -X POST https://log-api.newrelic.com/log/v1 \
-H "Content-Type: application/json" \
-H "Api-Key: <YOUR_LICENSE_KEY>" \
-H "Accept: */*" \
-d '{
"message": "Logs Example",
"logtype": "accesslogs",
"service": "production-service",
"hostname": "production.example.com"
}'Follow our documentation to learn more about how to use the New Relic Log API to forward your logs.
Important! If you encounter any issues when sending logs to New Relic using supported log forwarders, it's likely that you are using an unsupported approach. In such cases, it's recommended to review the log forwarding solutions provided by New Relic and choose a supported method that meets your requirements.
Tip #3: Send troubleshooting logs
To help troubleshoot issues with log forwarding to New Relic, you can configure the agent to send its own logs to New Relic. This provides valuable insights into the log forwarding process and helps identify potential issues. To enable this feature, you need to edit your newrelic-infra.yml configuration file and add a configuration snippet to enable log forwarding to New Relic.
To send troubleshooting logs to New Relic via infrastructure agent, follow these steps:
- Edit your
newrelic-infra.ymlconfiguration file. - Add the configuration snippet below to enable log forwarding to New Relic, as seen here.
log:
level: trace # Recommended: Helps with troubleshooting
forward: true # Enables sending logs to New Relic
format: json # Recommended: Enable agent logging in JSON format
stdout: false # On Windows and systems that don't use `systemd` or where `journald` is inaccessibleBy following these steps, you configure the infrastructure agent to send its logs to New Relic. The above configuration snippet enables log forwarding to New Relic, sets the log level to trace for troubleshooting purposes, specifies the log format as `JSON`, and disables the stdout output in certain system environments.
If you can send troubleshooting logs successfully, it indicates that the Infrastructure agent is communicating with New Relic without any issues. However, keep in mind that trace logging generates an enormous amount of data quickly. To avoid high disk usage and data ingestion, you should set the log level to `info` or lower once you have generated logs.
If you are unable to see troubleshooting logs in New Relic, you need to investigate the underlying causes of the issue. This may include networking problems, firewalls or permission issues.
Verify infrastructure agent connectivity
Linux: On Linux, you can use the following command to check if the New Relic Infrastructure agent is running successfully.
# list and check the td-agent process
ps -ef | grep -i td-agentCopy
Windows: For Microsoft Windows, you can check whether the New Relic Infrastructure service is running in Windows Services.
Address permission issues
If you are experiencing issues with the New Relic agent, it is possible that you have permission issues preventing it from running successfully. The following resources provide additional information to help you resolve any permission-related issues.
Linux: By default, the agent runs and installs as root. You can also select privileged or unprivileged run modes.
Windows: The agent must be installed from an Administrator account and requires Administrator privileges to run.
macOS: The agent can be installed from any user account.
In addition, you can leverage the NrIntegrationError event provided by New Relic to gain more visibility into potential issues related to data ingestion, limits, and feature configuration. This event is specifically designed to capture issues arising from exceeding data ingest and query limits, configuration errors, and malformed data.
Troubleshoot with NrIntegrationError Event
New Relic provides the NrIntegrationError event, which offers visibility into potential issues related to data ingestion, limits, and feature configuration. Use the following NRQL query to examine the structure of the NrIntegrationError event and see its attached attributes.
FROM NrIntegrationError SELECT keyset()The image below shows an example of issues captured by the NrIntegrationError event.
Important! As mentioned earlier, if you encounter any issues with sending troubleshooting logs to New Relic, it is likely that you have networking issues preventing the Logs from being sent, or you do not have the necessary system permissions to run the New Relic Infrastructure agent.
Tip #4: Access the designated log file and service
When working with log files, it's important to ensure that the log file you're attempting to use is accessible and has the necessary permissions. In some cases, the log file may have been created when the agent was running as root, resulting in the nri-agent user to lose write permissions.
To check the necessary permissions and ensure that you have access to the log file, you can use the ls -l command in Linux.
ls -l /path/to/fileThe above command provides detailed information about the permissions and ownership of the file.
Once you have identified any issues with file permissions, you can consider implementing the following options: :
Option 1: Change the owner of the log file: By changing the owner of the log file, you can ensure that the appropriate user has the necessary permissions to access and modify the file.
Option 2: In the /etc/newrelic-infra.yml configuration file, modify the log_file entry to specify a log file location that allows the nri-agent user to write to it. Our installation scripts create the /var/log/newrelic-infra/ folder for this purpose, and we recommend using the same. For example:
log_file: /var/log/newrelic-infra/newrelic-infra.logWhen encountering issues with the agent not running or logs being inaccessible on Microsoft Windows, it is often caused by antivirus or security products installed on the system. These products can sometimes interfere with the proper functioning of the agent and prevent access to important log files. It is important to check the settings of these products and ensure that proper exceptions have been added to allow the agent to run and logs to be accessed. Additionally, it may be helpful to review any recent updates or changes made to the security software to determine if they may be causing the issue.
Important! If you encounter any issues when accessing the designated log file and service, it’s likely that you do not have the right permissions and access. Check with the team to ensure you have all the right configurations and approvals.
Tip #5: Implement appropriate log parsing
In New Relic, logs are often presented in JSON format which is recommended over plain text. JSON log messages are automatically parsed into key-value pairs, thereby reducing the likelihood of unexpected issues.
When logging in plain text, the lack of structure can pose challenges when parsing log data. Extracting helpful information from plain-text logs becomes difficult and may even result in parsing errors. On the other hand, JSON is a structured data format that provides a consistent structure for log data. By using JSON as an output format, log messages are automatically converted into key-value pairs, making it easier to extract useful information from logs.
When logs don’t appear correctly in New Relic, usually there are two reasons.
Reason one: invalid JSON format
One possible reason for logs not appearing correctly in New Relic is an invalid JSON format. To ensure proper parsing and prevent truncation, the log data must be in valid JSON format. Otherwise, it will be stored as a string and truncated if it exceeds the character limit. Therefore, verifying that the log data is in valid JSON format before forwarding it to New Relic is crucial.
Workaround
If you encounter issues with log parsing, it is helpful to know that a wealth of information is available in the technical community regarding parsing logs. You can find many examples provided by FluentBit, New Relic, or other resources commonly available to help. By leveraging this knowledge, you can ensure your logs are correctly parsed into New Relic.
Reason two: “stringified” JSON content
Another potential reason for logs not appearing correctly in New Relic is the presence of "stringified" JSON content. This occurs when valid JSON content is converted into a string format using escape characters, resulting in it being evaluated as a string rather than JSON. This means the content will be truncated to 4,096 characters before being assessed as JSON. This truncation will produce an invalid JSON, storing the data as a string. Additionally, if the JSON contains arrays, they will be flattened and stored as unparsed strings making it difficult to extract and analyze the data within the arrays.
Workaround
To avoid issues related to "stringified" JSON content and the flattening of arrays, make sure to send the data in their original JSON format without converting them to a string. This allows the content to be parsed correctly even if it exceeds the character limit.
Important! If you encounter any issues related to log data formatting and parsing, it is likely that you do not have the correct log format, or that the logs are not being correctly parsed into New Relic. Explore the data structure of your logs and use the appropriate log parsers.
Conclusion
To summarize, we have covered five tips to ensure your logs are correctly forwarded to New Relic, allowing you to gain valuable insights into your application behavior and quickly identify potential issues. The information provided in this post is designed to help you understand the basics of log forwarding and guide you in troubleshooting and verifying your log forwarding configurations. By utilizing these tips, you can easily ensure that your logs are working correctly and being sent to New Relic.
To further explore the topics discussed in this blog post, you can refer to the following documentation for more information:
Forward your logs to New Relic
