Log forwarding from pm2 to new relic

When using pm2, it automatically logs the server logs (both console logs and errors) into local logs files. We can ssh into our servers and check them if necessary. But in reality, we configure some sort of logging framework or platform. By integrating with the logging service, we can see the logs without needing to go into our server. Some platform provides capabilities to notify people by email or slack message in case of unexpected errors or failures. Here I am going to setup log forwarding from pm2 server to new relic.

In order to follow this, I assume you have already setup new relic account and installed new relic infrastructure agent on your server. Also, you should have your pm2 running and serving your web service.

First, I will go into the pm2 logs folder and check the logging files I want to forward to new relic.

root@seajobies-prod-droplet:~# cd .pm2/logs/
root@seajobies-prod-droplet:~/.pm2/logs# ls -al
total 40
drwxr-xr-x 2 root root  4096 Jul 25 09:05 .
drwxr-xr-x 5 root root  4096 Jul 25 08:04 ..
-rw-r--r-- 1 root root     0 Jul 25 08:05 pm2-logrotate-error.log
-rw-r--r-- 1 root root     0 Jul 25 08:05 pm2-logrotate-out.log
-rw-r--r-- 1 root root     0 Jul 25 09:05 seajobies-production-error.log
-rw-r--r-- 1 root root 29109 Jul 25 09:29 seajobies-production-out.log
root@seajobies-prod-droplet:~/.pm2/logs# pwd
/root/.pm2/logs

pm2 logs are inside .pm2/logs folder by default. Here you will see a few files according to the instances running by your pm2 server. I am interested in forwarding logs related to seajobies. I will configure to forward logs from seajobies-production-out.log (normal console logs) and seajobies-production-error.log (error logs). Before going out of this folder, I will also check the path of this logs folder by using pwd command. We will need this later.

Then, I will go to the folder where I will add those folder for log forwarding

cd /etc/newrelic-infra/logging.d/

If you check in this folder, you will see a few example files you can reference.

root@seajobies-prod-droplet:/etc/newrelic-infra/logging.d# ls -al
total 32
drwxr-xr-x 2 root root 4096 Jul 25 09:25 .
drwxr-xr-x 4 root root 4096 Jul 25 09:12 ..
-rw-r--r-- 1 root root 1638 Jul 18 11:28 file.yml.example
-rw-r--r-- 1 root root 1174 Jul 18 11:28 fluentbit.yml.example
-rw-r--r-- 1 root root 2763 Jul 18 11:28 syslog.yml.example
-rw-r--r-- 1 root root  931 Jul 18 11:28 systemd.yml.example
-rw-r--r-- 1 root root 1098 Jul 18 11:28 tcp.yml.example

We will create a new file call pm2_logging.yml.

nano pm2_logging.yml

Inside the file, I will paste the following. Please note that file need to the full path to the file you want to forward. We already got that path before by using pwd command.

logs:
  - name: pm2-prod-error
    file: /root/.pm2/logs/seajobies-production-error.log

  - name: pm2-prod-log
    file: /root/.pm2/logs/seajobies-production-out.log

The new relic infra agent will automatically pick up the changes and the logs will appear in the dashboard. That's it. Then, trigger some logs by calling some apis from your web client, postman, or anything. After a few seconds, the logs will start to show up in the new relic console.