cdh3u0的jetty导致Error Reading IndexFile

?

在36个机器上面跑一个大作业，8千多个map，2w多个reduce，跑reduce的时候经常会出现如下问题：

?

Map output lost, rescheduling: getMapOutput(attempt_201204130934_0012_m_000022_0,5416) failed :java.io.IOException: Error Reading IndexFile   at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113)   at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66)   at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3651)   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)   at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:824)   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)   at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)   at org.mortbay.jetty.Server.handle(Server.java:326)   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)   at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)   at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)   at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)Caused by: java.io.EOFException   at java.io.DataInputStream.readFully(DataInputStream.java:180)   at java.io.DataInputStream.readLong(DataInputStream.java:399)   at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:75)   at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:55)   at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109)   ... 23 more

?reduce读取不到map的输出结果，然后map重新跑，reduce重新读取数据，整个job运行时间就变长了。。在官方issue?MAPREDUCE-2389 Spurious EOFExceptions reading SpillRecord index files中有介绍。https://issues.apache.org/jira/browse/MAPREDUCE-2389https://issues.apache.org/jira/browse/MAPREDUCE-2980
解决方案：（1）升级到成3u3（没有测试，但是应该能解决）（2）把3u0的lib目录下面jetty相关包替换成3u3使用的就行（经过测试没有问题）jetty-6.1.26.cloudera.1.jar 、jetty-servlet-tester-6.1.26.cloudera.1.jar 、jetty-util-6.1.26.cloudera.1.jar（3）官方issue中有提到换成jetty-6.1.14（没有测试） 1 楼 tsbian 2012-06-08 我也遇到了类似的问题。

这个错误好像是由于Web服务器Jetty造成的。


参考：https://issues.apache.org/jira/browse/MAPREDUCE-2389

Map output lost, rescheduling: getMapOutput(attempt_201205161733_6589_m_000313_0,40) failed :
java.io.IOException: Illegal seek
at java.io.FileInputStream.skip(Native Method)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3841)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)