{"id":190,"date":"2011-03-28T14:27:15","date_gmt":"2011-03-28T13:27:15","guid":{"rendered":"http:\/\/laur.ie\/blog\/?p=190"},"modified":"2011-03-28T14:27:15","modified_gmt":"2011-03-28T13:27:15","slug":"hadoop-and-ganglia-3-1","status":"publish","type":"post","link":"https:\/\/laur.ie\/blog\/2011\/03\/hadoop-and-ganglia-3-1\/","title":{"rendered":"Hadoop and Ganglia 3.1"},"content":{"rendered":"<p>A quick note to anyone setting up a new <a href=\"http:\/\/hadoop.apache.org\/\">Hadoop<\/a> cluster and hoping to quickly use the built in <a href=\"http:\/\/ganglia.sourceforge.net\/\">Ganglia<\/a> metrics collection (which you should! If it moves, graph it!): This works out of the box with Ganglia 3.0, but the protocol changed with Ganglia 3.1.<\/p>\n<p>The official\u00a0<a href=\"http:\/\/wiki.apache.org\/hadoop\/GangliaMetrics\">GangliaMetrics<\/a> pages talks about this, and talks about patching (which is already available if you use the Cloudera releases) but doesn&#8217;t go into more detail than that. I recently set up a new cluster, and remembered there was something I had to change in the default config to make it work out of the box&#8230; After inquiring (and finding the comment I left in my old config file!) I remembered, you must change the default class to have &#8220;31&#8221; (e.g. Ganglia 3.1) on the end.<\/p>\n<p>For example, the default config file:\u00a0(Replacing @GANGLIA@ with your multicast address)<\/p>\n<pre>dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext\r\ndfs.period=10\r\ndfs.servers=@GANGLIA@:8649\r\n\r\nmapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext\r\nmapred.period=10\r\nmapred.servers=@GANGLIA@:8649\r\n\r\njvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext\r\njvm.period=10\r\njvm.servers=@GANGLIA@:8649\r\n\r\nrpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext\r\nrpc.period=10\r\nrpc.servers=@GANGLIA@:8649<\/pre>\n<p>Is changed to this:<\/p>\n<pre>dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31\r\ndfs.period=10\r\ndfs.servers=@GANGLIA@:8649\r\n\r\nmapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31\r\nmapred.period=10\r\nmapred.servers=@GANGLIA@:8649\r\n\r\njvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31\r\njvm.period=10\r\njvm.servers=@GANGLIA@:8649\r\n\r\nrpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31\r\nrpc.period=10\r\nrpc.servers=@GANGLIA@:8649<\/pre>\n<p>Restart the cluster, and the graphs will appear under each host in the Ganglia interface.<\/p>\n<p>There is a LOT of detail in these graphs, with metrics ranging <a href=\"http:\/\/laur.ie\/grb\/49-q7dt023noksgk.png\">from DFS <\/a>(things like bytes written, and how many operations were transferred from other nodes) to <a href=\"http:\/\/laur.ie\/grb\/39-ggblwxyoe8ksw.png\">the JVM<\/a> (monitor those heap memory sizes!)<\/p>\n<p>This is probably old news to most people I&#8217;m sure, but I have a rule that if I didn&#8217;t find it within 30 minutes, maybe this will help someone in the same boat as me \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A quick note to anyone setting up a new Hadoop cluster and hoping to quickly use the built in Ganglia metrics collection (which you should! If it moves, graph it!): This works out of the box with Ganglia 3.0, but the protocol changed with Ganglia 3.1. The official\u00a0GangliaMetrics pages talks about this, and talks about &hellip; <a href=\"https:\/\/laur.ie\/blog\/2011\/03\/hadoop-and-ganglia-3-1\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Hadoop and Ganglia 3.1<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[3,5],"tags":[],"class_list":["post-190","post","type-post","status-publish","format-standard","hentry","category-technology","category-work"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pioRW-34","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/posts\/190","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/comments?post=190"}],"version-history":[{"count":2,"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/posts\/190\/revisions"}],"predecessor-version":[{"id":192,"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/posts\/190\/revisions\/192"}],"wp:attachment":[{"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/media?parent=190"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/categories?post=190"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/laur.ie\/blog\/wp-json\/wp\/v2\/tags?post=190"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}