简介

继续上一篇博客远程调度（三）来介绍hadoop远程调度过程。上一篇主要写rpc 服务端启动，和接受到客户端请求，处理请求，及把请求封装成一个call对象。接下来介绍怎么执行客户端请求，和结果返回客户端。

Hander 中处理Call对象

hander对象数组初始化在Server.start()方法中，这个方法中启动了一系列hander线程，这些线程在run方法中从Server.callQueue队列中获取call对象，并且处理请求，获取返回结果，并且把结果返回个对应的客户端。

   @Override
    public void run() {
      while (running) {
        TraceScope traceScope = null;
        try {
          final Call call = callQueue.take(); // 排队获取需要处理的call
          try {
            // Make the call as the user via Subject.doAs, thus associating
            // the call with the Subject 调用call方法，处理请求，底层调用  
            //Object value = method.invoke(protocolImpl.protocolImpl, call.getParameters());方法，返回处理结果。
            if (call.connection.user == null) {
              value = call(call.rpcKind, call.connection.protocolName, call.rpcRequest, 
                           call.timestamp);
            } else {
              value = 
                call.connection.user.doAs
                  (new PrivilegedExceptionAction<Writable>() {
                     @Override
                     public Writable run() throws Exception {
                       // make the call
                       return call(call.rpcKind, call.connection.protocolName, 
                                   call.rpcRequest, call.timestamp);

                     }
                   }
                  );
            }
          } catch (Throwable e) {
          
          }
          CurCall.set(null);
          synchronized (call.connection.responseQueue) {
            setupResponse(buf, call, returnStatus, detailedErr,
                value, errorClass, error);
            call.sendResponse();  //处理结果
          }
        } catch (InterruptedException e) {
        } finally {
          IOUtils.cleanup(LOG, traceScope);
        }
      }
      LOG.debug(Thread.currentThread().getName() + ": exiting");
    }

  }

#setupResponse() 方法

整理返回结果header
根据不同请求方式序列化返回结果数据到缓冲区中

Server.Call.sendResponse()方法

调用connection的sendResponse（）方法
connection.sendResponse()调用了responder.doRespond(call)方法，这时候请求被推送到responder线程中，这个线程在server.start()时被启动。用来等待处理返回结果

Server.Responder.doRespond() 方法

    void doRespond(Call call) throws IOException {  
    //给responseQueue添加同步锁，确保同时只有一个响应被处理
      synchronized (call.connection.responseQueue) {  
        call.connection.responseQueue.addLast(call);
        if (call.connection.responseQueue.size() == 1) {
        //调用processResponse方法处理返回结果
          processResponse(call.connection.responseQueue, true);
        }
      }
    }

Server.Responder.processResponse()

作用：处理一个响应，如果channel没有数据需要处理则返回true

 private boolean processResponse(LinkedList<Call> responseQueue,
                                    boolean inHandler) throws IOException {
      boolean error = true;
      boolean done = false;       // there is more data for this channel.
      int numElements = 0;
      Call call = null;
      try {
        synchronized (responseQueue) {
          numElements = responseQueue.size();
          if (numElements == 0) {
            error = false;
            return true;              // no more data for this channel.
          }
          //提取需要处理的call
          call = responseQueue.removeFirst();
          SocketChannel channel = call.connection.channel;
          // Send as much data as we can in the non-blocking fashion
           ////尽可能多的写数据到channel中 这个是最后想客户端发送数据的的地方，是不是很奇怪，因为在Responder线程中run方法，最终会循环调用这个方法，持续写数据到通道中，如果说没有更多数据，那么到这里其实整个请求已经完成，但是为了处理不能一次把数据都发送过去的情况，才有后续的处理。
          int numBytes = channelWrite(channel, call.rpcResponse);
          if (numBytes < 0) {
            return true;
          }
           //如果没有更多数据需要处理，清理rpcResponse
          if (!call.rpcResponse.hasRemaining()) {
            call.rpcResponse = null;
            call.connection.decRpcCount();
            if (numElements == 1) {    // last call fully processes.
              done = true;             // no more data for this channel.
            } else {
              done = false;            // more calls pending to be sent.
            }
          } else {
            //如果还有数据需要处理，把call重新加入responseQueue中
            call.connection.responseQueue.addFirst(call);
            
            if (inHandler) {
              // set the serve time when the response has to be sent later
              call.timestamp = Time.now();
              incPending();
              try {
                // Wakeup the thread blocked on select, only then can the call 
                // to channel.register() complete.
                writeSelector.wakeup();
                channel.register(writeSelector, SelectionKey.OP_WRITE, call); //注册写时间到writeSelector中，在run方法中会后续处理
              } catch (ClosedChannelException e) {
                //Its ok. channel might be closed else where.
                done = true;
              } finally {
                decPending();
              }
            }
            if (LOG.isDebugEnabled()) {
              LOG.debug(Thread.currentThread().getName() + ": responding to " + call
                  + " Wrote partial " + numBytes + " bytes.");
            }
          }
          error = false;              // everything went off well
        }
      } finally {
        if (error && call != null) {
          LOG.warn(Thread.currentThread().getName()+", call " + call + ": output error");
          done = true;               // error. no more data for this channel.
          closeConnection(call.connection);
        }
      }
      return done;
    }

Server.Responder.run()方法

 @Override
    public void run() {
      LOG.info(Thread.currentThread().getName() + ": starting");
      SERVER.set(Server.this);
        doRunLoop(); //调用方法
    }
 
    private void doRunLoop() {
      long lastPurgeTime = 0;   // last check for old calls.
      while (running) {
        try {
          waitPending();     // If a channel is being registered, wait.
          writeSelector.select(PURGE_INTERVAL);
          Iterator<SelectionKey> iter = writeSelector.selectedKeys().iterator();
          while (iter.hasNext()) {
            SelectionKey key = iter.next();
            iter.remove();
            try {
              if (key.isValid() && key.isWritable()) {
                  doAsyncWrite(key); //底层调用Server.Responder.processResponse()方法。对数据进行输出。
              }
            } 
          }
          long now = Time.now();
          if (now < lastPurgeTime + PURGE_INTERVAL) {
            continue;
          }
          lastPurgeTime = now;
          //
          // If there were some calls that have not been sent out for a
          // long time, discard them.
          //
          if(LOG.isDebugEnabled()) {
            LOG.debug("Checking for old call responses.");
          }
          ArrayList<Call> calls;
          
          // get the list of channels from list of keys.
          synchronized (writeSelector.keys()) {
            calls = new ArrayList<Call>(writeSelector.keys().size());
            iter = writeSelector.keys().iterator();
            while (iter.hasNext()) {
              SelectionKey key = iter.next();
              Call call = (Call)key.attachment();
              if (call != null && key.channel() == call.connection.channel) { 
                calls.add(call);
              }
            }
          }
          
          for(Call call : calls) {
            doPurge(call, now);
          }
        } catch (OutOfMemoryError e) {
        }
      }

对结果返回的后续操作参考了参考链接博客

Scott-Blog

hadoop 远程调度（四）

简介

Hander 中处理Call对象

Server.Call.sendResponse()方法

Server.Responder.doRespond() 方法

Server.Responder.processResponse()

Server.Responder.run()方法