NVMe over PCIe最新的NVMe协议是1.3。 在7.2.1讲了Command Processing流程。有图有真相。

This section describes command submission and completion processing. Figure 251 shows the steps that are followed to submit and complete a command. The steps are:
本节描述了命令提交和命令完成处理。图251列出了提交和完成一个命令的步骤。

• 1. The host places one or more commands for execution in the next free Submission Queue slot(s) in memory. 主机将一个或多个命令放置在位于内存中的提交队列(SQ)的下一个可用的槽位中执行。
• 2. The host updates the Submission Queue Tail Doorbell register with the new value of the Submission Queue Tail entry pointer. This indicates to the controller that a new command(s) is submitted for processing. Host用SQ尾部指针的新值去更新SQ的TailDB寄存器。这告诉了SSD控制器有一个新的命令被提交需要被处理。
• 3. The controller transfers the command(s) from in the Submission Queue slot(s) into the controller for future execution. Arbitration is the method used to determine the Submission Queue from which the controller starts processing the next candidate command(s), refer to section 4.11. SSD控制器将命令从SQ中转移到控制器中以供下一步执行。(从哪一个SQ中取出下一条候选命令去执行的仲裁方法，请参见4.11一节。)
• 4. The controller then proceeds with execution of the next command(s). Commands may complete out of order (the order submitted or started execution). 控制器接下来执行下一条命令。命令的执行完成可能是乱序的(与提交或开始执行的时间点无关)。
• 5. After a command has completed execution, the controller places a completion queue entry in the next free slot in the associated Completion Queue. As part of the completion queue entry, the controller indicates the most recent Submission Queue entry that has been consumed by advancing the Submission Queue Head pointer in the completion entry. Each new completion queue entry has a Phase Tag inverted from the previous entry to indicate to the host that this completion queue entry is a new entry. 在命令完成执行之后，SSD控制器将一个完成队列条目(CQE)放在相关的完成队列(CQ)的下一个空闲槽位中。作为CQE的一部分，SSD控制器通过修改完成条目的SQ头指针指示最新的SQE已经被消费了。每一个新的CQE都有一个从前一个条目中反转的相位标记(Phase Tag), 以向Host表明这个CQE是一个新条目。
• 6. The controller optionally generates an interrupt to the host to indicate that there is a new completion queue entry to consume and process. In the figure, this is shown as an MSI-X interrupt, however, it could also be a pin-based or MSI interrupt. Note that based on interrupt coalescing settings, an interrupt may or may not be generated for each new completion queue entry. 可选地，SSD控制器给Host产生一个中断，以表明有一个新的CQE已经产生，可以被消费和处理了。在图中演示的是MSI-X中断，然而，中断也可以是基于PIN或者MSI的中断。注意：基于中断联合设置，可能或不能为每一个新的CQE产生一个中断。
• 7. The host consumes and then processes the new completion queue entries in the Completion Queue. This includes taking any actions based on error conditions indicated. The host continues consuming and processing completion queue entries until it encounters a previously consumed entry with a Phase Tag inverted from the value of the current completion queue entries. Host消费和处理在CQ中放置的新的CQE。包括基于错误情况采取的任何操作。Host继续消耗和处理CQE，直到它遇到以前消费的一个条目的相位标签(Phase Tag)从当前完成队列条目(CQEs)的值中反转。
• 8. The host writes the Completion Queue Head Doorbell register to indicate that the completion queue entry has been consumed. The host may consume many entries before updating the associated Completion Queue Head Doorbell register. Host更新CQ的HeadDB寄存器，表明CQE已经被消费了。在更新相关联的CQ的HeadDB寄存器之前，Host可能消费了多个CQE。

以上8步，写得比较具体，稍微优点罗嗦，那么，用更通俗易懂的话总结一下就是：

• 1. Host写命令到SQ
• 2. Host更新SQ的TailDB, 通知SSD取命令
• 3. SSD收到命令，于是从SQ中取出命令
• 4. SSD执行命令
• 5. 命令执行完成后，SSD往CQ中写入命令执行结果，同时修改CQ的TailDB
• 6. SSD发短信通知Host命令已经执行完成
• 7. Host收到命令后，到CQ中查看命令完成状态
• 8. Host处理完CQ中的命令执行结果，更新CQ中的HeadDB, 回复SSD, "命令执行结果已经处理完毕，辛苦啦"

P.S. NVMe over PCIe和我们前面提到的RDMA本质上都是“玩队列”。 NVMe over PCIe有两条队列，一条提交队列(SQ)和一条完成队列(CQ)；而RDMA有三条队列，一条发送队列(SQ)，一条接收队列(RQ)和一条完成队列(CQ)，而一个SQ和一个RQ被称之为一个QP(队列对)。

If you don't aim high you will never hit high.