SlideShare a Scribd company logo
1 of 59
Download to read offline
Improving	
  Hadoop	
  
Cluster	
  Performance	
  via	
  
Linux	
  Configura:on	
  
DevIgni:on	
  2014	
  –	
  Dulles,	
  Virginia	
  
Alex	
  Moundalexis	
  //	
  @technmsg	
  
2	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Tips	
  
from	
  a	
  former	
  system	
  administrator	
  	
  
3	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Click	
  to	
  edit	
  Master	
  :tle	
  style	
  
CC	
  BY	
  2.0	
  /	
  Richard	
  Bumgardner	
  
Been	
  there,	
  done	
  that.	
  
4	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Tips	
  
from	
  a	
  former	
  system	
  administrator	
  field	
  guy	
  	
  
5	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Click	
  to	
  edit	
  Master	
  :tle	
  style	
  
CC	
  BY	
  2.0	
  /	
  Alex	
  Moundalexis	
  
Home	
  sweet	
  home.	
  
6	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Tips	
  
Easy	
  steps	
  to	
  take…	
  
7	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Tips	
  
Easy	
  steps	
  to	
  take…	
  that	
  most	
  people	
  don’t.	
  
8	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
What	
  this	
  talk	
  isn’t	
  about	
  
•  Deploying	
  
• Puppet,	
  Chef,	
  Ansible,	
  homegrown	
  scripts,	
  intern	
  labor	
  
•  Sizing	
  &	
  Tuning	
  
• Depends	
  heavily	
  on	
  data	
  and	
  workload	
  
•  Coding	
  
• Unless	
  you	
  count	
  STDOUT	
  redirec:on	
  
•  Algorithms	
  
• I	
  suck	
  at	
  math,	
  but	
  we’ll	
  try	
  some	
  mul:plica:on	
  later	
  
9	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
“The	
  answer	
  to	
  most	
  Hadoop	
  
ques:ons	
  is…	
  
	
  
10	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
“The	
  answer	
  to	
  most	
  Hadoop	
  
ques:ons	
  is…	
  	
  
it	
  depends.”	
  
11	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
“The	
  answer	
  to	
  most	
  Hadoop	
  
ques:ons	
  is…	
  	
  
it	
  depends.”	
  
(helpful,	
  right?)	
  
12	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
So	
  what	
  ARE	
  we	
  talking	
  about?	
  
•  Seven	
  simple	
  things	
  
• Quick	
  
• Safe	
  
• Viable	
  for	
  most	
  environments	
  and	
  use	
  cases	
  
•  Iden:fy	
  issue,	
  then	
  offer	
  solu:on	
  
•  Note:	
  Commands	
  run	
  as	
  root	
  or	
  sudo	
  
13	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
1.	
  Swapping	
  
Bad	
  news,	
  best	
  not	
  to.	
  
14	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Swapping	
  
•  A	
  form	
  of	
  memory	
  management	
  
•  When	
  OS	
  runs	
  low	
  on	
  memory…	
  
• write	
  blocks	
  to	
  disk	
  
• use	
  now-­‐free	
  memory	
  for	
  other	
  things	
  
• read	
  blocks	
  back	
  into	
  memory	
  from	
  disk	
  when	
  needed	
  
•  Also	
  known	
  as	
  paging	
  
15	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Swapping	
  
•  Problem:	
  Disks	
  are	
  slow,	
  especially	
  to	
  seek	
  
•  Hadoop	
  is	
  about	
  maximizing	
  IO	
  
• spend	
  less	
  :me	
  acquiring	
  data	
  
• operate	
  on	
  data	
  in	
  place	
  
• large	
  streaming	
  reads/writes	
  from	
  disk	
  
•  Memory	
  usage	
  is	
  somewhat	
  limited	
  within	
  JVM	
  
• we	
  should	
  be	
  able	
  to	
  manage	
  our	
  memory	
  
• account	
  for	
  JVM	
  overhead	
  
16	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Limit	
  swapping	
  in	
  kernel	
  
•  Well,	
  as	
  much	
  as	
  possible.	
  
•  Immediate:	
  
	
  #	
  echo	
  1	
  >	
  /proc/sys/vm/swappiness	
  
•  Persist	
  amer	
  reboot:	
  
	
  #	
  echo	
  "vm.swappiness	
  =	
  1"	
  >>	
  /etc/sysctl.conf	
  
17	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Swapping	
  peculiari:es	
  
•  Behavior	
  varies	
  based	
  on	
  Linux	
  kernel	
  
• CentOS	
  6.4+	
  /	
  Ubuntu	
  10.10+	
  
• For	
  you	
  kernel	
  gurus,	
  that’s	
  Linux	
  2.6.32-­‐303+	
  
•  Prior	
  
• We	
  don’t	
  swap,	
  except	
  to	
  avoid	
  OOM	
  condi:on.	
  
•  Amer	
  
• We	
  don’t	
  swap,	
  ever.	
  
•  Details:	
  hpp://:ny.cloudera.com/noswap	
  
18	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
2.	
  File	
  Access	
  Time	
  
Disable	
  this	
  too.	
  
19	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
File	
  access	
  :me	
  
•  Linux	
  tracks	
  access	
  :me	
  
• writes	
  to	
  disk	
  even	
  if	
  all	
  you	
  did	
  was	
  read	
  
•  Problem	
  
• more	
  disk	
  seeks	
  
• HDFS	
  is	
  write-­‐once,	
  read-­‐many	
  
• NameNode	
  tracks	
  access	
  informa:on	
  for	
  HDFS	
  
20	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Don’t	
  track	
  access	
  :me	
  
•  Mount	
  volumes	
  with	
  noatime	
  op:on	
  
• In	
  /etc/fstab:	
  
	
  
/dev/sdc	
  /data01	
  ext3	
  defaults,noatime	
  0	
  	
  
• Note:	
  noatime	
  assumes	
  nodirtime	
  as	
  well	
  
•  What	
  about	
  relatime?	
  
• Faster	
  than	
  atime	
  but	
  slower	
  than	
  noatime	
  
•  No	
  reboot	
  required	
  
• #	
  mount	
  -­‐o	
  remount	
  /data01	
  
21	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
3.	
  Root	
  Reserved	
  Space	
  
Reclaim	
  it,	
  impress	
  your	
  bosses!	
  
22	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Root	
  reserved	
  space	
  
•  EXT3/4	
  reserve	
  5%	
  of	
  disk	
  for	
  root-­‐owned	
  files	
  
• On	
  an	
  OS	
  disk,	
  sure	
  
• System	
  logs,	
  kernel	
  panics,	
  etc	
  
23	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Click	
  to	
  edit	
  Master	
  :tle	
  style	
  
CC	
  BY	
  2.0	
  /	
  Alex	
  Moundalexis	
  
Disks	
  used	
  to	
  be	
  much	
  smaller,	
  right?	
  
24	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Do	
  the	
  math	
  
•  Conserva:ve	
  
• 5%	
  of	
  1	
  TB	
  disk	
  =	
  46	
  GB	
  
• 5	
  data	
  disks	
  per	
  server	
  =	
  230	
  GB	
  
• 5	
  servers	
  per	
  rack	
  =	
  1.15	
  TB	
  
•  Quasi-­‐Aggressive	
  
• 5%	
  of	
  4	
  TB	
  disk	
  =	
  186	
  GB	
  
• 12	
  data	
  disks	
  per	
  server	
  =	
  2.23	
  TB	
  
• 18	
  servers	
  per	
  rack	
  =	
  40.1	
  TB	
  
•  That’s	
  a	
  LOT	
  of	
  unused	
  storage!	
  
25	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Root	
  reserved	
  space	
  
•  On	
  a	
  Hadoop	
  data	
  disk,	
  no	
  root-­‐owned	
  files	
  
•  When	
  crea:ng	
  a	
  par::on	
  
	
  #	
  mkfs.ext3	
  –m	
  0	
  /dev/sdc	
  
•  On	
  exis:ng	
  par::ons	
  
	
  #	
  tune2fs	
  -­‐m	
  0	
  /dev/sdc	
  
• 0	
  is	
  safe,	
  1	
  is	
  for	
  the	
  ultra-­‐paranoid	
  
26	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
4.	
  Name	
  Service	
  Cache	
  
Turn	
  it	
  on,	
  already!	
  
27	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Name	
  Service	
  Cache	
  Daemon	
  
•  Daemon	
  that	
  caches	
  name	
  service	
  requests	
  
• Passwords	
  
• Groups	
  
• Hosts	
  
•  Helps	
  weather	
  network	
  hiccups	
  
•  Helps	
  more	
  with	
  high	
  latency	
  LDAP,	
  NIS,	
  NIS+	
  
•  Small	
  footprint	
  
•  Zero	
  configura:on	
  required	
  
28	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Name	
  Service	
  Cache	
  Daemon	
  
•  Hadoop	
  nodes	
  
• largely	
  a	
  network-­‐based	
  applica:on	
  
• on	
  the	
  network	
  constantly	
  
• issue	
  lots	
  of	
  name	
  lookups,	
  especially	
  HBase	
  &	
  distcp	
  
• can	
  thrash	
  name	
  servers	
  
•  Reducing	
  latency	
  of	
  service	
  requests?	
  Smart.	
  
•  Reducing	
  impact	
  on	
  shared	
  infrastructure?	
  Smart.	
  
29	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Name	
  Service	
  Cache	
  Daemon	
  
•  Turn	
  it	
  on,	
  let	
  it	
  work,	
  leave	
  it	
  alone:	
  
#	
  chkconfig	
  -­‐-­‐level	
  345	
  nscd	
  on	
  
#	
  service	
  nscd	
  start	
  	
  
•  Check	
  on	
  it	
  later:	
  
#	
  nscd	
  -­‐g	
  
•  Unless	
  using	
  Red	
  Hat	
  SSSD;	
  modify	
  nscd	
  config	
  first!	
  
• Don’t	
  use	
  nscd	
  to	
  cache	
  passwd,	
  group,	
  or	
  netgroup	
  
• Red	
  Hat,	
  Using	
  NSCD	
  with	
  SSSD.	
  hpp://goo.gl/68HTMQ	
  
30	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
5.	
  File	
  Handle	
  Limits	
  
Not	
  a	
  problem,	
  un:l	
  they	
  are.	
  
31	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
File	
  handle	
  limits	
  
•  Kernel	
  refers	
  to	
  files	
  via	
  a	
  handle	
  
• Also	
  called	
  descriptors	
  
•  Linux	
  is	
  a	
  mul:-­‐user	
  system	
  
•  File	
  handles	
  protect	
  the	
  system	
  from	
  
• Poor	
  coding	
  
• Malicious	
  users	
  
• Poor	
  coding	
  of	
  malicious	
  users	
  
• Pictures	
  of	
  cats	
  on	
  the	
  Internet	
  
32	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  32	
  
Microsom	
  Office	
  EULA.	
  Really.	
  
java.io.FileNotFoundExcep:on:	
  (Too	
  many	
  open	
  files)	
  
33	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
File	
  handle	
  limits	
  
•  Linux	
  defaults	
  usually	
  not	
  enough	
  
•  Increase	
  maximum	
  open	
  files	
  (default	
  1024)	
  
#	
  echo	
  hdfs	
  –	
  nofile	
  32768	
  >>	
  /etc/security/limits.conf	
  
#	
  echo	
  mapred	
  –	
  nofile	
  32768	
  >>	
  /etc/security/limits.conf	
  
#	
  echo	
  hbase	
  –	
  nofile	
  32768	
  >>	
  /etc/security/limits.conf	
  
•  Bonus:	
  Increase	
  maximum	
  processes	
  too	
  
#	
  echo	
  hdfs	
  –	
  nproc	
  32768	
  >>	
  /etc/security/limits.conf	
  
#	
  echo	
  mapred	
  –	
  nproc	
  32768	
  >>	
  /etc/security/limits.conf	
  
#	
  echo	
  hbase	
  –	
  nproc	
  32768	
  >>	
  /etc/security/limits.conf	
  
•  Note:	
  Cloudera	
  Manager	
  will	
  do	
  this	
  for	
  you.	
  
34	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
6.	
  Dedicated	
  Disks	
  
Don’t	
  be	
  tempted	
  to	
  share,	
  even	
  with	
  monster	
  disks.	
  
35	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
The	
  Situa:on	
  
1.  Your	
  new	
  server	
  has	
  a	
  dozen	
  1	
  TB	
  disks	
  
2.  Eleven	
  disks	
  are	
  used	
  to	
  store	
  data	
  
3.  One	
  disk	
  is	
  used	
  for	
  the	
  OS	
  
• 20	
  GB	
  for	
  the	
  OS	
  
• 980	
  GB	
  sits	
  unused	
  	
  
4.  Someone	
  asks	
  “can	
  we	
  store	
  data	
  there	
  too?”	
  
5.  Seems	
  reasonable,	
  lots	
  of	
  space…	
  “OK,	
  why	
  not.”	
  
Sound	
  familiar?	
  
36	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Microsom	
  Office	
  EULA.	
  Really.	
  
“I	
  don’t	
  understand	
  it,	
  there’s	
  	
  
no	
  consistency	
  to	
  these	
  run	
  >mes!”	
  
37	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
No	
  love	
  for	
  shared	
  disk	
  
•  Our	
  quest	
  for	
  data	
  gets	
  interrupted	
  a	
  lot:	
  
• OS	
  opera:ons	
  
• OS	
  logs	
  
• Hadoop	
  logging,	
  quite	
  chapy	
  
• Hadoop	
  execu:on	
  
• userspace	
  execu:on	
  
•  Disk	
  seeks	
  are	
  slow,	
  remember?	
  
38	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Dedicated	
  disk	
  for	
  OS	
  and	
  logs	
  
•  At	
  install	
  :me	
  	
  	
  
• Disk	
  0,	
  OS	
  &	
  logs	
  
• Disk	
  1-­‐n,	
  Hadoop	
  data	
  
•  Amer	
  install,	
  more	
  complicated	
  effort,	
  requires	
  manual	
  HDFS	
  block	
  rebalancing:	
  
1.  Take	
  down	
  HDFS	
  
•  If	
  you	
  can	
  do	
  it	
  in	
  under	
  10	
  minutes,	
  just	
  the	
  DataNode	
  
2.  Move	
  or	
  distribute	
  blocks	
  from	
  disk0/dir	
  to	
  disk[1-­‐n]/dir	
  
3.  Remove	
  dir	
  from	
  HDFS	
  config	
  (dfs.data.dir)	
  
4.  Start	
  HDFS	
  
39	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
7.	
  Name	
  Resolu:on	
  
Sane,	
  both	
  forward	
  and	
  reverse.	
  
40	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Name	
  resolu:on	
  op:ons	
  
1.  Hosts	
  file,	
  if	
  you	
  must	
  
2.  DNS,	
  much	
  preferred	
  
	
  
	
  
41	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Name	
  resolu:on	
  with	
  hosts	
  file	
  
•  Set	
  canonical	
  names	
  properly	
  	
  
•  Right	
  
	
  10.1.1.1	
   	
  r01m01.cluster.org	
  r01m01 	
  master1	
  
	
  10.1.1.2	
   	
  r01w01.cluster.org	
  	
  r01w01 	
  worker1	
  
•  Wrong	
  
	
  10.1.1.1	
   	
  r01m01 	
   	
   	
   	
   	
  r01m01.cluster.org	
  master1	
  
	
  10.1.1.2	
   	
  r01w01 	
   	
   	
   	
   	
  r01w01.cluster.org	
  worker1	
  
42	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Name	
  resolu:on	
  with	
  hosts	
  file	
  
•  Set	
  loopback	
  address	
  properly	
  
• Ensure	
  127.0.0.1	
  resolves	
  to	
  “localhost,”	
  NOT	
  hostname	
  
•  Right	
  
	
  127.0.0.1 	
  localhost	
  
•  Wrong	
  
	
  127.0.0.1 	
  r01m01	
  
43	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Name	
  resolu:on	
  with	
  DNS	
  
•  Forward	
  
•  Reverse	
  
•  Hostname	
  should	
  match	
  the	
  FQDN	
  in	
  DNS	
  
44	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
This	
  is	
  what	
  you	
  ought	
  to	
  see	
  
45	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Name	
  resolu:on	
  errata	
  
•  Mismatches?	
  Expect	
  odd	
  results.	
  
• Problems	
  star:ng	
  DataNodes	
  
• Non-­‐FQDN	
  in	
  Web	
  UI	
  links	
  
• Security	
  features	
  are	
  extra	
  sensi:ve	
  to	
  FQDN	
  
•  Errors	
  so	
  common	
  that	
  link	
  to	
  FAQ	
  is	
  included	
  in	
  logs!	
  
• hpp://wiki.apache.org/hadoop/UnknownHost	
  
•  Get	
  name	
  resolu:on	
  working	
  BEFORE	
  enabling	
  nscd!	
  
46	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Summary	
  
Now	
  is	
  the	
  appropriate	
  :me	
  to	
  take	
  out	
  your	
  camera	
  
phone.	
  
47	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
A	
  white	
  background	
  is	
  supposedly	
  
beper	
  for	
  prin:ng.	
  
(who	
  prints	
  things	
  anymore?)	
  
48	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
A	
  white	
  background	
  is	
  supposedly	
  
beper	
  for	
  prin:ng.	
  
(but	
  makes	
  for	
  very	
  pale	
  slides)	
  
49	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Summary	
  
1.  disable	
  vm.swappiness	
  
2.  data	
  disks:	
  mount	
  with	
  noatime	
  op:on	
  
3.  data	
  disks:	
  disable	
  root	
  reserve	
  space	
  
4.  enable	
  nscd	
  
5.  increase	
  file	
  handle	
  limits	
  
6.  use	
  dedicated	
  OS/logging	
  disk	
  
7.  sane	
  name	
  resolu:on	
  
hpp://:ny.cloudera.com/7steps	
  
50	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Recommended	
  reading	
  
•  Hadoop	
  Opera:ons	
  
hpp://amzn.to/1ydMrLf	
  
51	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Ques:ons?	
  
Preferably	
  related	
  to	
  the	
  talk…	
  
52	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Thanks!	
  
Alex	
  Moundalexis|	
  @technmsg	
  
53	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
8.	
  Bonus	
  Round	
  
Because	
  we	
  have	
  enough	
  :me	
  (or	
  I	
  talked	
  really	
  fast)…	
  
54	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Other	
  things	
  to	
  check	
  
•  Disk	
  IO	
  
• hdparm	
  
•  #	
  hdparm	
  -­‐Tt	
  /dev/sdc	
  
•  Looking	
  for	
  at	
  least	
  70	
  MB/s	
  from	
  7200	
  RPM	
  disks	
  
•  Slower	
  could	
  indicate	
  a	
  failing	
  drive,	
  disk	
  controller,	
  array,	
  etc.	
  
• dd	
  
•  hpp://romanrm.ru/en/dd-­‐benchmark	
  
55	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Other	
  things	
  to	
  check	
  
•  Disable	
  Red	
  Hat	
  Transparent	
  Huge	
  Pages	
  (RH6+	
  un:l	
  6.5)	
  
• Can	
  reduce	
  elevated	
  CPU	
  usage	
  
• In	
  rc.local:	
  
echo	
  never	
  >	
  /sys/kernel/mm/redhat_transparent_hugepage/defrag	
  
echo	
  never	
  >	
  /sys/kernel/mm/redhat_transparent_hugepage/enabled	
  
• Reference:	
  Linux	
  6	
  Transparent	
  Huge	
  Pages	
  and	
  Hadoop	
  Workloads,	
  hpp://
goo.gl/WSF2qC	
  
56	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Other	
  things	
  to	
  check	
  
•  Enable	
  Jumbo	
  Frames	
  
• Only	
  if	
  your	
  network	
  infrastructure	
  supports	
  it!	
  
• Can	
  easily	
  (and	
  arguably)	
  boost	
  throughput	
  by	
  10-­‐20%	
  
57	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Other	
  things	
  to	
  check	
  
•  Enable	
  Jumbo	
  Frames	
  
• Only	
  if	
  your	
  network	
  infrastructure	
  supports	
  it!	
  
• Can	
  easily	
  (and	
  arguably)	
  boost	
  throughput	
  by	
  10-­‐20%	
  
•  Monitor	
  and	
  Chart	
  Everything	
  
• How	
  else	
  will	
  you	
  know	
  what’s	
  happening?	
  
• Nagios	
  
• Ganglia	
  
58	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Ques:ons?	
  
Preferably	
  related	
  to	
  the	
  talk…	
  
59	
  ©	
  Cloudera,	
  Inc.	
  All	
  rights	
  reserved.	
  
Thanks!	
  
Alex	
  Moundalexis|	
  @technmsg	
  

More Related Content

What's hot

RNUG - HCL Notes 11.0.1 FP2 — Performance Boost Re-Reloaded
RNUG - HCL Notes 11.0.1 FP2 — Performance Boost Re-ReloadedRNUG - HCL Notes 11.0.1 FP2 — Performance Boost Re-Reloaded
RNUG - HCL Notes 11.0.1 FP2 — Performance Boost Re-ReloadedChristoph Adler
 
Kvm performance optimization for ubuntu
Kvm performance optimization for ubuntuKvm performance optimization for ubuntu
Kvm performance optimization for ubuntuSim Janghoon
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceArgus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceHBaseCon
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Brian Brazil
 
Hardware Assisted Latency Investigations
Hardware Assisted Latency InvestigationsHardware Assisted Latency Investigations
Hardware Assisted Latency InvestigationsScyllaDB
 
BPF Hardware Offload Deep Dive
BPF Hardware Offload Deep DiveBPF Hardware Offload Deep Dive
BPF Hardware Offload Deep DiveNetronome
 
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...DataWorks Summit/Hadoop Summit
 
AIXpert - AIX Security expert
AIXpert - AIX Security expertAIXpert - AIX Security expert
AIXpert - AIX Security expertdlfrench
 
Linux Kernel Memory Model
Linux Kernel Memory ModelLinux Kernel Memory Model
Linux Kernel Memory ModelSeongJae Park
 
Linux kernel architecture
Linux kernel architectureLinux kernel architecture
Linux kernel architectureSHAJANA BASHEER
 
Spectrum Scale Best Practices by Olaf Weiser
Spectrum Scale Best Practices by Olaf WeiserSpectrum Scale Best Practices by Olaf Weiser
Spectrum Scale Best Practices by Olaf WeiserSandeep Patil
 
Windows Privilege Escalation
Windows Privilege EscalationWindows Privilege Escalation
Windows Privilege EscalationRiyaz Walikar
 
NGINX: High Performance Load Balancing
NGINX: High Performance Load BalancingNGINX: High Performance Load Balancing
NGINX: High Performance Load BalancingNGINX, Inc.
 
Tips on High Performance Server Programming
Tips on High Performance Server ProgrammingTips on High Performance Server Programming
Tips on High Performance Server ProgrammingJoshua Zhu
 
Engage2022 - Domino Admin Tips
Engage2022 - Domino Admin TipsEngage2022 - Domino Admin Tips
Engage2022 - Domino Admin TipsGabriella Davis
 
IBM Traveler Management, Security and Performance
IBM Traveler Management, Security and PerformanceIBM Traveler Management, Security and Performance
IBM Traveler Management, Security and PerformanceGabriella Davis
 
Moving to PCI Express based SSD with NVM Express
Moving to PCI Express based SSD with NVM ExpressMoving to PCI Express based SSD with NVM Express
Moving to PCI Express based SSD with NVM ExpressOdinot Stanislas
 

What's hot (20)

RNUG - HCL Notes 11.0.1 FP2 — Performance Boost Re-Reloaded
RNUG - HCL Notes 11.0.1 FP2 — Performance Boost Re-ReloadedRNUG - HCL Notes 11.0.1 FP2 — Performance Boost Re-Reloaded
RNUG - HCL Notes 11.0.1 FP2 — Performance Boost Re-Reloaded
 
Kvm performance optimization for ubuntu
Kvm performance optimization for ubuntuKvm performance optimization for ubuntu
Kvm performance optimization for ubuntu
 
storage device
storage device storage device
storage device
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceArgus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)
 
Hardware Assisted Latency Investigations
Hardware Assisted Latency InvestigationsHardware Assisted Latency Investigations
Hardware Assisted Latency Investigations
 
BPF Hardware Offload Deep Dive
BPF Hardware Offload Deep DiveBPF Hardware Offload Deep Dive
BPF Hardware Offload Deep Dive
 
60 Admin Tips
60 Admin Tips60 Admin Tips
60 Admin Tips
 
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
 
NetApp against ransomware
NetApp against ransomwareNetApp against ransomware
NetApp against ransomware
 
AIXpert - AIX Security expert
AIXpert - AIX Security expertAIXpert - AIX Security expert
AIXpert - AIX Security expert
 
Linux Kernel Memory Model
Linux Kernel Memory ModelLinux Kernel Memory Model
Linux Kernel Memory Model
 
Linux kernel architecture
Linux kernel architectureLinux kernel architecture
Linux kernel architecture
 
Spectrum Scale Best Practices by Olaf Weiser
Spectrum Scale Best Practices by Olaf WeiserSpectrum Scale Best Practices by Olaf Weiser
Spectrum Scale Best Practices by Olaf Weiser
 
Windows Privilege Escalation
Windows Privilege EscalationWindows Privilege Escalation
Windows Privilege Escalation
 
NGINX: High Performance Load Balancing
NGINX: High Performance Load BalancingNGINX: High Performance Load Balancing
NGINX: High Performance Load Balancing
 
Tips on High Performance Server Programming
Tips on High Performance Server ProgrammingTips on High Performance Server Programming
Tips on High Performance Server Programming
 
Engage2022 - Domino Admin Tips
Engage2022 - Domino Admin TipsEngage2022 - Domino Admin Tips
Engage2022 - Domino Admin Tips
 
IBM Traveler Management, Security and Performance
IBM Traveler Management, Security and PerformanceIBM Traveler Management, Security and Performance
IBM Traveler Management, Security and Performance
 
Moving to PCI Express based SSD with NVM Express
Moving to PCI Express based SSD with NVM ExpressMoving to PCI Express based SSD with NVM Express
Moving to PCI Express based SSD with NVM Express
 

Viewers also liked

Improving Hadoop Performance via Linux
Improving Hadoop Performance via LinuxImproving Hadoop Performance via Linux
Improving Hadoop Performance via LinuxAlex Moundalexis
 
Hadoop configuration & performance tuning
Hadoop configuration & performance tuningHadoop configuration & performance tuning
Hadoop configuration & performance tuningVitthal Gogate
 
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your ApplicationHadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your ApplicationYahoo Developer Network
 
Hortonworks.Cluster Config Guide
Hortonworks.Cluster Config GuideHortonworks.Cluster Config Guide
Hortonworks.Cluster Config GuideDouglas Bernardini
 
Best Practices for Deploying Hadoop (BigInsights) in the Cloud
Best Practices for Deploying Hadoop (BigInsights) in the CloudBest Practices for Deploying Hadoop (BigInsights) in the Cloud
Best Practices for Deploying Hadoop (BigInsights) in the CloudLeons Petražickis
 
Hortonworks Technical Workshop - Operational Best Practices Workshop
Hortonworks Technical Workshop - Operational Best Practices WorkshopHortonworks Technical Workshop - Operational Best Practices Workshop
Hortonworks Technical Workshop - Operational Best Practices WorkshopHortonworks
 
Hadoop & Big Data benchmarking
Hadoop & Big Data benchmarkingHadoop & Big Data benchmarking
Hadoop & Big Data benchmarkingBart Vandewoestyne
 
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Advanced Hadoop Tuning and Optimization - Hadoop ConsultingAdvanced Hadoop Tuning and Optimization - Hadoop Consulting
Advanced Hadoop Tuning and Optimization - Hadoop ConsultingImpetus Technologies
 
Optimizing MapReduce Job performance
Optimizing MapReduce Job performanceOptimizing MapReduce Job performance
Optimizing MapReduce Job performanceDataWorks Summit
 
TestDFSIO
TestDFSIOTestDFSIO
TestDFSIOhhyin
 
DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"
DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"
DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"Dr. Mirko Kämpf
 
Implementação_cluster alto_desempenho_fernando_eduardo_20090726-Imetro_2013
Implementação_cluster alto_desempenho_fernando_eduardo_20090726-Imetro_2013Implementação_cluster alto_desempenho_fernando_eduardo_20090726-Imetro_2013
Implementação_cluster alto_desempenho_fernando_eduardo_20090726-Imetro_2013Kanda Kassobo
 
Performance Tuning para o mercado financeiro
Performance Tuning para o mercado financeiroPerformance Tuning para o mercado financeiro
Performance Tuning para o mercado financeiroRodrigo Missiaggia
 
Admin e suas gambiarras
Admin e suas gambiarrasAdmin e suas gambiarras
Admin e suas gambiarrasDaniel Lara
 
Application Architectures with Hadoop | Data Day Texas 2015
Application Architectures with Hadoop | Data Day Texas 2015Application Architectures with Hadoop | Data Day Texas 2015
Application Architectures with Hadoop | Data Day Texas 2015Cloudera, Inc.
 
Taming Operations in the Hadoop Ecosystem
Taming Operations in the Hadoop EcosystemTaming Operations in the Hadoop Ecosystem
Taming Operations in the Hadoop EcosystemCloudera, Inc.
 

Viewers also liked (20)

Improving Hadoop Performance via Linux
Improving Hadoop Performance via LinuxImproving Hadoop Performance via Linux
Improving Hadoop Performance via Linux
 
Hadoop configuration & performance tuning
Hadoop configuration & performance tuningHadoop configuration & performance tuning
Hadoop configuration & performance tuning
 
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your ApplicationHadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
 
Hortonworks.Cluster Config Guide
Hortonworks.Cluster Config GuideHortonworks.Cluster Config Guide
Hortonworks.Cluster Config Guide
 
Best Practices for Deploying Hadoop (BigInsights) in the Cloud
Best Practices for Deploying Hadoop (BigInsights) in the CloudBest Practices for Deploying Hadoop (BigInsights) in the Cloud
Best Practices for Deploying Hadoop (BigInsights) in the Cloud
 
Hortonworks Technical Workshop - Operational Best Practices Workshop
Hortonworks Technical Workshop - Operational Best Practices WorkshopHortonworks Technical Workshop - Operational Best Practices Workshop
Hortonworks Technical Workshop - Operational Best Practices Workshop
 
Hadoop & Big Data benchmarking
Hadoop & Big Data benchmarkingHadoop & Big Data benchmarking
Hadoop & Big Data benchmarking
 
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Advanced Hadoop Tuning and Optimization - Hadoop ConsultingAdvanced Hadoop Tuning and Optimization - Hadoop Consulting
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
 
Optimizing MapReduce Job performance
Optimizing MapReduce Job performanceOptimizing MapReduce Job performance
Optimizing MapReduce Job performance
 
Cloudera Impala
Cloudera ImpalaCloudera Impala
Cloudera Impala
 
YARN
YARNYARN
YARN
 
TestDFSIO
TestDFSIOTestDFSIO
TestDFSIO
 
Powered by the Sun
Powered by the SunPowered by the Sun
Powered by the Sun
 
DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"
DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"
DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"
 
Hadoop Puzzlers
Hadoop PuzzlersHadoop Puzzlers
Hadoop Puzzlers
 
Implementação_cluster alto_desempenho_fernando_eduardo_20090726-Imetro_2013
Implementação_cluster alto_desempenho_fernando_eduardo_20090726-Imetro_2013Implementação_cluster alto_desempenho_fernando_eduardo_20090726-Imetro_2013
Implementação_cluster alto_desempenho_fernando_eduardo_20090726-Imetro_2013
 
Performance Tuning para o mercado financeiro
Performance Tuning para o mercado financeiroPerformance Tuning para o mercado financeiro
Performance Tuning para o mercado financeiro
 
Admin e suas gambiarras
Admin e suas gambiarrasAdmin e suas gambiarras
Admin e suas gambiarras
 
Application Architectures with Hadoop | Data Day Texas 2015
Application Architectures with Hadoop | Data Day Texas 2015Application Architectures with Hadoop | Data Day Texas 2015
Application Architectures with Hadoop | Data Day Texas 2015
 
Taming Operations in the Hadoop Ecosystem
Taming Operations in the Hadoop EcosystemTaming Operations in the Hadoop Ecosystem
Taming Operations in the Hadoop Ecosystem
 

Similar to Improving Hadoop Cluster Performance via Linux Configuration

Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)Kathleen Ting
 
Hadoop 3 (2017 hadoop taiwan workshop)
Hadoop 3 (2017 hadoop taiwan workshop)Hadoop 3 (2017 hadoop taiwan workshop)
Hadoop 3 (2017 hadoop taiwan workshop)Wei-Chiu Chuang
 
Troubleshooting Hadoop: Distributed Debugging
Troubleshooting Hadoop: Distributed DebuggingTroubleshooting Hadoop: Distributed Debugging
Troubleshooting Hadoop: Distributed DebuggingGreat Wide Open
 
Risk Management for Data: Secured and Governed
Risk Management for Data: Secured and GovernedRisk Management for Data: Secured and Governed
Risk Management for Data: Secured and GovernedCloudera, Inc.
 
The Ultimate IBM and Lotus on Linux Workshop for Windows Admins
The Ultimate IBM and Lotus on Linux Workshop for Windows AdminsThe Ultimate IBM and Lotus on Linux Workshop for Windows Admins
The Ultimate IBM and Lotus on Linux Workshop for Windows AdminsBill Malchisky Jr.
 
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in HadoopKudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoopjdcryans
 
5 Apache Spark Tips in 5 Minutes
5 Apache Spark Tips in 5 Minutes5 Apache Spark Tips in 5 Minutes
5 Apache Spark Tips in 5 MinutesCloudera, Inc.
 
Deployment of WebObjects applications on CentOS Linux
Deployment of WebObjects applications on CentOS LinuxDeployment of WebObjects applications on CentOS Linux
Deployment of WebObjects applications on CentOS LinuxWO Community
 
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataDatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataHakka Labs
 
Microservices with Terraform, Docker and the Cloud. Chicago Coders Conference...
Microservices with Terraform, Docker and the Cloud. Chicago Coders Conference...Microservices with Terraform, Docker and the Cloud. Chicago Coders Conference...
Microservices with Terraform, Docker and the Cloud. Chicago Coders Conference...Derek Ashmore
 
Microservices with Terraform, Docker and the Cloud. JavaOne 2017 2017-10-02
Microservices with Terraform, Docker and the Cloud. JavaOne 2017 2017-10-02Microservices with Terraform, Docker and the Cloud. JavaOne 2017 2017-10-02
Microservices with Terraform, Docker and the Cloud. JavaOne 2017 2017-10-02Derek Ashmore
 
Strata London 2019 Scaling Impala
Strata London 2019 Scaling ImpalaStrata London 2019 Scaling Impala
Strata London 2019 Scaling ImpalaManish Maheshwari
 
Strata London 2019 Scaling Impala.pptx
Strata London 2019 Scaling Impala.pptxStrata London 2019 Scaling Impala.pptx
Strata London 2019 Scaling Impala.pptxManish Maheshwari
 
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...Data Con LA
 
Deployment Strategy
Deployment StrategyDeployment Strategy
Deployment StrategyMongoDB
 
Infrastructure Around Hadoop
Infrastructure Around HadoopInfrastructure Around Hadoop
Infrastructure Around HadoopDataWorks Summit
 

Similar to Improving Hadoop Cluster Performance via Linux Configuration (20)

Hadoop Operations
Hadoop OperationsHadoop Operations
Hadoop Operations
 
Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)
 
Hadoop 3 (2017 hadoop taiwan workshop)
Hadoop 3 (2017 hadoop taiwan workshop)Hadoop 3 (2017 hadoop taiwan workshop)
Hadoop 3 (2017 hadoop taiwan workshop)
 
Troubleshooting Hadoop: Distributed Debugging
Troubleshooting Hadoop: Distributed DebuggingTroubleshooting Hadoop: Distributed Debugging
Troubleshooting Hadoop: Distributed Debugging
 
Risk Management for Data: Secured and Governed
Risk Management for Data: Secured and GovernedRisk Management for Data: Secured and Governed
Risk Management for Data: Secured and Governed
 
The Ultimate IBM and Lotus on Linux Workshop for Windows Admins
The Ultimate IBM and Lotus on Linux Workshop for Windows AdminsThe Ultimate IBM and Lotus on Linux Workshop for Windows Admins
The Ultimate IBM and Lotus on Linux Workshop for Windows Admins
 
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in HadoopKudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
 
Chicago spark meetup-april2017-public
Chicago spark meetup-april2017-publicChicago spark meetup-april2017-public
Chicago spark meetup-april2017-public
 
Kudu austin oct 2015.pptx
Kudu austin oct 2015.pptxKudu austin oct 2015.pptx
Kudu austin oct 2015.pptx
 
5 Apache Spark Tips in 5 Minutes
5 Apache Spark Tips in 5 Minutes5 Apache Spark Tips in 5 Minutes
5 Apache Spark Tips in 5 Minutes
 
Deployment of WebObjects applications on CentOS Linux
Deployment of WebObjects applications on CentOS LinuxDeployment of WebObjects applications on CentOS Linux
Deployment of WebObjects applications on CentOS Linux
 
Tuning Linux for MongoDB
Tuning Linux for MongoDBTuning Linux for MongoDB
Tuning Linux for MongoDB
 
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataDatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
 
Microservices with Terraform, Docker and the Cloud. Chicago Coders Conference...
Microservices with Terraform, Docker and the Cloud. Chicago Coders Conference...Microservices with Terraform, Docker and the Cloud. Chicago Coders Conference...
Microservices with Terraform, Docker and the Cloud. Chicago Coders Conference...
 
Microservices with Terraform, Docker and the Cloud. JavaOne 2017 2017-10-02
Microservices with Terraform, Docker and the Cloud. JavaOne 2017 2017-10-02Microservices with Terraform, Docker and the Cloud. JavaOne 2017 2017-10-02
Microservices with Terraform, Docker and the Cloud. JavaOne 2017 2017-10-02
 
Strata London 2019 Scaling Impala
Strata London 2019 Scaling ImpalaStrata London 2019 Scaling Impala
Strata London 2019 Scaling Impala
 
Strata London 2019 Scaling Impala.pptx
Strata London 2019 Scaling Impala.pptxStrata London 2019 Scaling Impala.pptx
Strata London 2019 Scaling Impala.pptx
 
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
 
Deployment Strategy
Deployment StrategyDeployment Strategy
Deployment Strategy
 
Infrastructure Around Hadoop
Infrastructure Around HadoopInfrastructure Around Hadoop
Infrastructure Around Hadoop
 

Recently uploaded

OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 

Recently uploaded (20)

OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 

Improving Hadoop Cluster Performance via Linux Configuration

  • 1. Improving  Hadoop   Cluster  Performance  via   Linux  Configura:on   DevIgni:on  2014  –  Dulles,  Virginia   Alex  Moundalexis  //  @technmsg  
  • 2. 2  ©  Cloudera,  Inc.  All  rights  reserved.   Tips   from  a  former  system  administrator    
  • 3. 3  ©  Cloudera,  Inc.  All  rights  reserved.   Click  to  edit  Master  :tle  style   CC  BY  2.0  /  Richard  Bumgardner   Been  there,  done  that.  
  • 4. 4  ©  Cloudera,  Inc.  All  rights  reserved.   Tips   from  a  former  system  administrator  field  guy    
  • 5. 5  ©  Cloudera,  Inc.  All  rights  reserved.   Click  to  edit  Master  :tle  style   CC  BY  2.0  /  Alex  Moundalexis   Home  sweet  home.  
  • 6. 6  ©  Cloudera,  Inc.  All  rights  reserved.   Tips   Easy  steps  to  take…  
  • 7. 7  ©  Cloudera,  Inc.  All  rights  reserved.   Tips   Easy  steps  to  take…  that  most  people  don’t.  
  • 8. 8  ©  Cloudera,  Inc.  All  rights  reserved.   What  this  talk  isn’t  about   •  Deploying   • Puppet,  Chef,  Ansible,  homegrown  scripts,  intern  labor   •  Sizing  &  Tuning   • Depends  heavily  on  data  and  workload   •  Coding   • Unless  you  count  STDOUT  redirec:on   •  Algorithms   • I  suck  at  math,  but  we’ll  try  some  mul:plica:on  later  
  • 9. 9  ©  Cloudera,  Inc.  All  rights  reserved.   “The  answer  to  most  Hadoop   ques:ons  is…    
  • 10. 10  ©  Cloudera,  Inc.  All  rights  reserved.   “The  answer  to  most  Hadoop   ques:ons  is…     it  depends.”  
  • 11. 11  ©  Cloudera,  Inc.  All  rights  reserved.   “The  answer  to  most  Hadoop   ques:ons  is…     it  depends.”   (helpful,  right?)  
  • 12. 12  ©  Cloudera,  Inc.  All  rights  reserved.   So  what  ARE  we  talking  about?   •  Seven  simple  things   • Quick   • Safe   • Viable  for  most  environments  and  use  cases   •  Iden:fy  issue,  then  offer  solu:on   •  Note:  Commands  run  as  root  or  sudo  
  • 13. 13  ©  Cloudera,  Inc.  All  rights  reserved.   1.  Swapping   Bad  news,  best  not  to.  
  • 14. 14  ©  Cloudera,  Inc.  All  rights  reserved.   Swapping   •  A  form  of  memory  management   •  When  OS  runs  low  on  memory…   • write  blocks  to  disk   • use  now-­‐free  memory  for  other  things   • read  blocks  back  into  memory  from  disk  when  needed   •  Also  known  as  paging  
  • 15. 15  ©  Cloudera,  Inc.  All  rights  reserved.   Swapping   •  Problem:  Disks  are  slow,  especially  to  seek   •  Hadoop  is  about  maximizing  IO   • spend  less  :me  acquiring  data   • operate  on  data  in  place   • large  streaming  reads/writes  from  disk   •  Memory  usage  is  somewhat  limited  within  JVM   • we  should  be  able  to  manage  our  memory   • account  for  JVM  overhead  
  • 16. 16  ©  Cloudera,  Inc.  All  rights  reserved.   Limit  swapping  in  kernel   •  Well,  as  much  as  possible.   •  Immediate:    #  echo  1  >  /proc/sys/vm/swappiness   •  Persist  amer  reboot:    #  echo  "vm.swappiness  =  1"  >>  /etc/sysctl.conf  
  • 17. 17  ©  Cloudera,  Inc.  All  rights  reserved.   Swapping  peculiari:es   •  Behavior  varies  based  on  Linux  kernel   • CentOS  6.4+  /  Ubuntu  10.10+   • For  you  kernel  gurus,  that’s  Linux  2.6.32-­‐303+   •  Prior   • We  don’t  swap,  except  to  avoid  OOM  condi:on.   •  Amer   • We  don’t  swap,  ever.   •  Details:  hpp://:ny.cloudera.com/noswap  
  • 18. 18  ©  Cloudera,  Inc.  All  rights  reserved.   2.  File  Access  Time   Disable  this  too.  
  • 19. 19  ©  Cloudera,  Inc.  All  rights  reserved.   File  access  :me   •  Linux  tracks  access  :me   • writes  to  disk  even  if  all  you  did  was  read   •  Problem   • more  disk  seeks   • HDFS  is  write-­‐once,  read-­‐many   • NameNode  tracks  access  informa:on  for  HDFS  
  • 20. 20  ©  Cloudera,  Inc.  All  rights  reserved.   Don’t  track  access  :me   •  Mount  volumes  with  noatime  op:on   • In  /etc/fstab:     /dev/sdc  /data01  ext3  defaults,noatime  0     • Note:  noatime  assumes  nodirtime  as  well   •  What  about  relatime?   • Faster  than  atime  but  slower  than  noatime   •  No  reboot  required   • #  mount  -­‐o  remount  /data01  
  • 21. 21  ©  Cloudera,  Inc.  All  rights  reserved.   3.  Root  Reserved  Space   Reclaim  it,  impress  your  bosses!  
  • 22. 22  ©  Cloudera,  Inc.  All  rights  reserved.   Root  reserved  space   •  EXT3/4  reserve  5%  of  disk  for  root-­‐owned  files   • On  an  OS  disk,  sure   • System  logs,  kernel  panics,  etc  
  • 23. 23  ©  Cloudera,  Inc.  All  rights  reserved.   Click  to  edit  Master  :tle  style   CC  BY  2.0  /  Alex  Moundalexis   Disks  used  to  be  much  smaller,  right?  
  • 24. 24  ©  Cloudera,  Inc.  All  rights  reserved.   Do  the  math   •  Conserva:ve   • 5%  of  1  TB  disk  =  46  GB   • 5  data  disks  per  server  =  230  GB   • 5  servers  per  rack  =  1.15  TB   •  Quasi-­‐Aggressive   • 5%  of  4  TB  disk  =  186  GB   • 12  data  disks  per  server  =  2.23  TB   • 18  servers  per  rack  =  40.1  TB   •  That’s  a  LOT  of  unused  storage!  
  • 25. 25  ©  Cloudera,  Inc.  All  rights  reserved.   Root  reserved  space   •  On  a  Hadoop  data  disk,  no  root-­‐owned  files   •  When  crea:ng  a  par::on    #  mkfs.ext3  –m  0  /dev/sdc   •  On  exis:ng  par::ons    #  tune2fs  -­‐m  0  /dev/sdc   • 0  is  safe,  1  is  for  the  ultra-­‐paranoid  
  • 26. 26  ©  Cloudera,  Inc.  All  rights  reserved.   4.  Name  Service  Cache   Turn  it  on,  already!  
  • 27. 27  ©  Cloudera,  Inc.  All  rights  reserved.   Name  Service  Cache  Daemon   •  Daemon  that  caches  name  service  requests   • Passwords   • Groups   • Hosts   •  Helps  weather  network  hiccups   •  Helps  more  with  high  latency  LDAP,  NIS,  NIS+   •  Small  footprint   •  Zero  configura:on  required  
  • 28. 28  ©  Cloudera,  Inc.  All  rights  reserved.   Name  Service  Cache  Daemon   •  Hadoop  nodes   • largely  a  network-­‐based  applica:on   • on  the  network  constantly   • issue  lots  of  name  lookups,  especially  HBase  &  distcp   • can  thrash  name  servers   •  Reducing  latency  of  service  requests?  Smart.   •  Reducing  impact  on  shared  infrastructure?  Smart.  
  • 29. 29  ©  Cloudera,  Inc.  All  rights  reserved.   Name  Service  Cache  Daemon   •  Turn  it  on,  let  it  work,  leave  it  alone:   #  chkconfig  -­‐-­‐level  345  nscd  on   #  service  nscd  start     •  Check  on  it  later:   #  nscd  -­‐g   •  Unless  using  Red  Hat  SSSD;  modify  nscd  config  first!   • Don’t  use  nscd  to  cache  passwd,  group,  or  netgroup   • Red  Hat,  Using  NSCD  with  SSSD.  hpp://goo.gl/68HTMQ  
  • 30. 30  ©  Cloudera,  Inc.  All  rights  reserved.   5.  File  Handle  Limits   Not  a  problem,  un:l  they  are.  
  • 31. 31  ©  Cloudera,  Inc.  All  rights  reserved.   File  handle  limits   •  Kernel  refers  to  files  via  a  handle   • Also  called  descriptors   •  Linux  is  a  mul:-­‐user  system   •  File  handles  protect  the  system  from   • Poor  coding   • Malicious  users   • Poor  coding  of  malicious  users   • Pictures  of  cats  on  the  Internet  
  • 32. 32  ©  Cloudera,  Inc.  All  rights  reserved.  32   Microsom  Office  EULA.  Really.   java.io.FileNotFoundExcep:on:  (Too  many  open  files)  
  • 33. 33  ©  Cloudera,  Inc.  All  rights  reserved.   File  handle  limits   •  Linux  defaults  usually  not  enough   •  Increase  maximum  open  files  (default  1024)   #  echo  hdfs  –  nofile  32768  >>  /etc/security/limits.conf   #  echo  mapred  –  nofile  32768  >>  /etc/security/limits.conf   #  echo  hbase  –  nofile  32768  >>  /etc/security/limits.conf   •  Bonus:  Increase  maximum  processes  too   #  echo  hdfs  –  nproc  32768  >>  /etc/security/limits.conf   #  echo  mapred  –  nproc  32768  >>  /etc/security/limits.conf   #  echo  hbase  –  nproc  32768  >>  /etc/security/limits.conf   •  Note:  Cloudera  Manager  will  do  this  for  you.  
  • 34. 34  ©  Cloudera,  Inc.  All  rights  reserved.   6.  Dedicated  Disks   Don’t  be  tempted  to  share,  even  with  monster  disks.  
  • 35. 35  ©  Cloudera,  Inc.  All  rights  reserved.   The  Situa:on   1.  Your  new  server  has  a  dozen  1  TB  disks   2.  Eleven  disks  are  used  to  store  data   3.  One  disk  is  used  for  the  OS   • 20  GB  for  the  OS   • 980  GB  sits  unused     4.  Someone  asks  “can  we  store  data  there  too?”   5.  Seems  reasonable,  lots  of  space…  “OK,  why  not.”   Sound  familiar?  
  • 36. 36  ©  Cloudera,  Inc.  All  rights  reserved.   Microsom  Office  EULA.  Really.   “I  don’t  understand  it,  there’s     no  consistency  to  these  run  >mes!”  
  • 37. 37  ©  Cloudera,  Inc.  All  rights  reserved.   No  love  for  shared  disk   •  Our  quest  for  data  gets  interrupted  a  lot:   • OS  opera:ons   • OS  logs   • Hadoop  logging,  quite  chapy   • Hadoop  execu:on   • userspace  execu:on   •  Disk  seeks  are  slow,  remember?  
  • 38. 38  ©  Cloudera,  Inc.  All  rights  reserved.   Dedicated  disk  for  OS  and  logs   •  At  install  :me       • Disk  0,  OS  &  logs   • Disk  1-­‐n,  Hadoop  data   •  Amer  install,  more  complicated  effort,  requires  manual  HDFS  block  rebalancing:   1.  Take  down  HDFS   •  If  you  can  do  it  in  under  10  minutes,  just  the  DataNode   2.  Move  or  distribute  blocks  from  disk0/dir  to  disk[1-­‐n]/dir   3.  Remove  dir  from  HDFS  config  (dfs.data.dir)   4.  Start  HDFS  
  • 39. 39  ©  Cloudera,  Inc.  All  rights  reserved.   7.  Name  Resolu:on   Sane,  both  forward  and  reverse.  
  • 40. 40  ©  Cloudera,  Inc.  All  rights  reserved.   Name  resolu:on  op:ons   1.  Hosts  file,  if  you  must   2.  DNS,  much  preferred      
  • 41. 41  ©  Cloudera,  Inc.  All  rights  reserved.   Name  resolu:on  with  hosts  file   •  Set  canonical  names  properly     •  Right    10.1.1.1    r01m01.cluster.org  r01m01  master1    10.1.1.2    r01w01.cluster.org    r01w01  worker1   •  Wrong    10.1.1.1    r01m01          r01m01.cluster.org  master1    10.1.1.2    r01w01          r01w01.cluster.org  worker1  
  • 42. 42  ©  Cloudera,  Inc.  All  rights  reserved.   Name  resolu:on  with  hosts  file   •  Set  loopback  address  properly   • Ensure  127.0.0.1  resolves  to  “localhost,”  NOT  hostname   •  Right    127.0.0.1  localhost   •  Wrong    127.0.0.1  r01m01  
  • 43. 43  ©  Cloudera,  Inc.  All  rights  reserved.   Name  resolu:on  with  DNS   •  Forward   •  Reverse   •  Hostname  should  match  the  FQDN  in  DNS  
  • 44. 44  ©  Cloudera,  Inc.  All  rights  reserved.   This  is  what  you  ought  to  see  
  • 45. 45  ©  Cloudera,  Inc.  All  rights  reserved.   Name  resolu:on  errata   •  Mismatches?  Expect  odd  results.   • Problems  star:ng  DataNodes   • Non-­‐FQDN  in  Web  UI  links   • Security  features  are  extra  sensi:ve  to  FQDN   •  Errors  so  common  that  link  to  FAQ  is  included  in  logs!   • hpp://wiki.apache.org/hadoop/UnknownHost   •  Get  name  resolu:on  working  BEFORE  enabling  nscd!  
  • 46. 46  ©  Cloudera,  Inc.  All  rights  reserved.   Summary   Now  is  the  appropriate  :me  to  take  out  your  camera   phone.  
  • 47. 47  ©  Cloudera,  Inc.  All  rights  reserved.   A  white  background  is  supposedly   beper  for  prin:ng.   (who  prints  things  anymore?)  
  • 48. 48  ©  Cloudera,  Inc.  All  rights  reserved.   A  white  background  is  supposedly   beper  for  prin:ng.   (but  makes  for  very  pale  slides)  
  • 49. 49  ©  Cloudera,  Inc.  All  rights  reserved.   Summary   1.  disable  vm.swappiness   2.  data  disks:  mount  with  noatime  op:on   3.  data  disks:  disable  root  reserve  space   4.  enable  nscd   5.  increase  file  handle  limits   6.  use  dedicated  OS/logging  disk   7.  sane  name  resolu:on   hpp://:ny.cloudera.com/7steps  
  • 50. 50  ©  Cloudera,  Inc.  All  rights  reserved.   Recommended  reading   •  Hadoop  Opera:ons   hpp://amzn.to/1ydMrLf  
  • 51. 51  ©  Cloudera,  Inc.  All  rights  reserved.   Ques:ons?   Preferably  related  to  the  talk…  
  • 52. 52  ©  Cloudera,  Inc.  All  rights  reserved.   Thanks!   Alex  Moundalexis|  @technmsg  
  • 53. 53  ©  Cloudera,  Inc.  All  rights  reserved.   8.  Bonus  Round   Because  we  have  enough  :me  (or  I  talked  really  fast)…  
  • 54. 54  ©  Cloudera,  Inc.  All  rights  reserved.   Other  things  to  check   •  Disk  IO   • hdparm   •  #  hdparm  -­‐Tt  /dev/sdc   •  Looking  for  at  least  70  MB/s  from  7200  RPM  disks   •  Slower  could  indicate  a  failing  drive,  disk  controller,  array,  etc.   • dd   •  hpp://romanrm.ru/en/dd-­‐benchmark  
  • 55. 55  ©  Cloudera,  Inc.  All  rights  reserved.   Other  things  to  check   •  Disable  Red  Hat  Transparent  Huge  Pages  (RH6+  un:l  6.5)   • Can  reduce  elevated  CPU  usage   • In  rc.local:   echo  never  >  /sys/kernel/mm/redhat_transparent_hugepage/defrag   echo  never  >  /sys/kernel/mm/redhat_transparent_hugepage/enabled   • Reference:  Linux  6  Transparent  Huge  Pages  and  Hadoop  Workloads,  hpp:// goo.gl/WSF2qC  
  • 56. 56  ©  Cloudera,  Inc.  All  rights  reserved.   Other  things  to  check   •  Enable  Jumbo  Frames   • Only  if  your  network  infrastructure  supports  it!   • Can  easily  (and  arguably)  boost  throughput  by  10-­‐20%  
  • 57. 57  ©  Cloudera,  Inc.  All  rights  reserved.   Other  things  to  check   •  Enable  Jumbo  Frames   • Only  if  your  network  infrastructure  supports  it!   • Can  easily  (and  arguably)  boost  throughput  by  10-­‐20%   •  Monitor  and  Chart  Everything   • How  else  will  you  know  what’s  happening?   • Nagios   • Ganglia  
  • 58. 58  ©  Cloudera,  Inc.  All  rights  reserved.   Ques:ons?   Preferably  related  to  the  talk…  
  • 59. 59  ©  Cloudera,  Inc.  All  rights  reserved.   Thanks!   Alex  Moundalexis|  @technmsg