PBS MoM Log Analysis in ELK Stack

Users/Site Administrators
1

Hi,

I am looking forward to visualize mom_logs on Kibana in Elk Stack. As a prerequisite we need to design a Grok Filter. As a part of the setup I do have setup for ELK Stack ready. I am able to send PBS mom_logs to kibana as well.

Example of Log and its filter in logstash
Log: localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0

Grok Filter: %{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}

For e.g as this mom log has multiple factors like date, time stamp, job id. My query is how do I get to know about the parameters that are to be defined in the Grok Filter

Log: 06/04/2019 18:12:05;0008;pbs_mom;Job;224.primaryserver1;Started, pid = 2352

Thanks,
Rakhen

1 Like
2

The log record format is the same as used by other PBS daemons, with the addition of the thread number and the daemon name in the log record. The log record format is as follows:

date-time;event_code;daemon_name(thread number);object_type;object_name;message

event_code : Please refer the Table 2-38: tracejob Filters in the RG-229 ( https://www.pbsworks.com/pdfs/PBSReferenceGuide19.2.1.pdf)
daemon_name: pbs_mom
object_type : type of object the message is about Job, Que, Svr, Req, Node,Hook
object_name: name of the specific object
message_text : text of the log message

It is good to see your project with mom_logs and Kibana, please share your experiences on this with the community once you have some results. This might spawn some interests and discussions.

1 Like
3

Thanks Adarsh for the details. Sure I will share my experience.

1 Like
4

Hi All,

Project: PBS MoM log Analysis in ELK Stack

Below are my leanings from the project:

  1. Just to give an overview ELK Stack consists of four main components i.e: Elasticsearch, Logstash, Kibana and Filebeat(to be installed on client machine).

  2. Elasticsearch, Logstash, Kibana were installed on Centos7 instance and Filebeat was installed on PBS Pro Cluster(Has shared file system).

  3. Filebeat can be installed on multiple clients in-case the log data is coming from different sources.

  4. Grok filter in logstash helps us reading pbs_mom logs. It needs to be designed as per your requirements.

  5. Analyzed PBS mom_logs through Kibana and have created a sample dashboard which shows visualizations of the data. Below are the snapshots:

    image
    image.png786×356 22.2 KB

    image
    image.png866×366 20.9 KB

Any further ideas are highly welcome!!

Thanks,
Rakhen

1 Like
5

Excellent ! . Thank you Rakhen. It is indeed very useful. The same can be extended to

  1. Server logs : To trace job acceptance throughput, common error charting
  2. Scheduler logs : say number of jobs considered at each scheduling cycle, common error charting
  3. Comm logs : mom registration
    If you have the plan to share it, then push it to github and so that community can use, contribute and extend.
  4. Accounting logs : $PBS_HOME/server_priv/accounting

Well done !

6

Thank you so much Adarsh…

1 Like
7

Looks awesome @rgarg

Is it possible to get some walk through document on how to get all of these configured. I had been meaning to play with either the ELK or EFK stack for a long time now, but never found the time.

Thanks!

8

Hi Adarsh,

Please refer to the below link:

Thanks,
Rakhen

1 Like
9

Likewise, I’d love to see your logstash conf for mom_logs. There’s a git repo mentioned that I’ll take a look at too.

10

Just FYI - I recently did a grafana visualization of the PBS logs based on loki and promtail and it was very straightforward.

11

Hi subhasisb,

Can You Please share Entire Configuration Step.It will be helpfull.

Thanks