Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
NVME(4)		       FreeBSD Kernel Interfaces Manual		       NVME(4)

     nvme -- NVM Express core driver

     To	compile	this driver into your kernel, place the	following line in your
     kernel configuration file:

	   device nvme

     Or, to load the driver as a module	at boot, place the following line in


     Most users	will also want to enable nvd(4)	or nda(4) to expose NVM	Ex-
     press namespaces as disk devices which can	be partitioned.	 Note that in
     NVM Express terms,	a namespace is roughly equivalent to a SCSI LUN.

     The nvme driver provides support for NVM Express (NVMe) controllers, such

     o	 Hardware initialization

     o	 Per-CPU IO queue pairs

     o	 API for registering NVMe namespace consumers such as nvd(4) or	nda(4)

     o	 API for submitting NVM	commands to namespaces

     o	 Ioctls	for controller and namespace configuration and management

     The nvme driver creates controller	device nodes in	the format /dev/nvmeX
     and namespace device nodes	in the format /dev/nvmeXnsY.  Note that	the
     NVM Express specification starts numbering	namespaces at 1, not 0,	and
     this driver follows that convention.

     By	default, nvme will create an I/O queue pair for	each CPU, provided
     enough MSI-X vectors and NVMe queue pairs can be allocated.  If not
     enough vectors or queue pairs are available, nvme(4) will use a smaller
     number of queue pairs and assign multiple CPUs per	queue pair.

     To	force a	single I/O queue pair shared by	all CPUs, set the following
     tunable value in loader.conf(5):


     To	assign more than one CPU per I/O queue pair, thereby reducing the num-
     ber of MSI-X vectors consumed by the device, set the following tunable
     value in loader.conf(5):


     To	force legacy interrupts	for all	nvme driver instances, set the follow-
     ing tunable value in loader.conf(5):


     Note that use of INTx implies disabling of	per-CPU	I/O queue pairs.

     To	control	maximum	amount of system RAM in	bytes to use as	Host Memory
     Buffer for	capable	devices, set the following tunable:


     The default value is 5% of	physical memory	size per device.

     The nvd(4)	driver is used to provide a disk driver	to the system by de-
     fault.  The nda(4)	driver can also	be used	instead.  The nvd(4) driver
     performs better with smaller transactions and few TRIM commands.  It
     sends all commands	directly to the	drive immediately.  The	nda(4) driver
     performs better with larger transactions and also collapses TRIM commands
     giving better performance.	 It can	queue commands to the drive; combine
     BIO_DELETE	commands into a	single trip; and use the CAM I/O scheduler to
     bias one type of operation	over another.  To select the nda(4) driver,
     set the following tunable value in	loader.conf(5):


     This value	may also be set	in the kernel config file with

	   options NVME_USE_NVD=0

     When there	is an error, nvme prints only the most relevant	information
     about the command by default.  To enable dumping of all information about
     the command, set the following tunable value in loader.conf(5):


     Prior versions of the driver reset	the card twice on boot.	 This proved
     to	be unnecessary and inefficient,	so the driver now resets drive con-
     troller only once.	 The old behavior may be restored in the kernel	config
     file with

	   options NVME_2X_RESET

     The following controller-level sysctls are	currently implemented:

	     (R) Number	of CPUs	associated with	each I/O queue pair.

	     (R/W) Interrupt coalescing	timer period in	microseconds.  Set to
	     0 to disable.

	     (R/W) Interrupt coalescing	threshold in number of command comple-
	     tions.  Set to 0 to disable.

     The following queue pair-level sysctls are	currently implemented.	Admin
     queue sysctls take	the format of dev.nvme.0.adminq	and I/O	queue sysctls
     take the format of	dev.nvme.0.ioq0.

	     (R) Number	of entries in this queue pair's	command	and completion

	     (R) Number	of nvme_tracker	structures currently allocated for
	     this queue	pair.

	     (R) Number	of nvme_prp_list structures currently allocated	for
	     this queue	pair.

	     (R) Current location of the submission queue head pointer as ob-
	     served by the driver.  The	head pointer is	incremented by the
	     controller	as it takes commands off of the	submission queue.

	     (R) Current location of the submission queue tail pointer as ob-
	     served by the driver.  The	driver increments the tail pointer af-
	     ter writing a command into	the submission queue to	signal that a
	     new command is ready to be	processed.

	     (R) Current location of the completion queue head pointer as ob-
	     served by the driver.  The	driver increments the head pointer af-
	     ter finishing with	a completion entry that	was posted by the con-

	     (R) Number	of commands that have been submitted on	this queue

	     (W) Writing 1 to this sysctl will dump the	full contents of the
	     submission	and completion queues to the console.

     In	addition to the	typical	pci attachment,	the nvme driver	supports at-
     taching to	a ahci(4) device.  Intel's Rapid Storage Technology (RST)
     hides the nvme device behind the AHCI device due to limitations in	Win-
     dows.  However, this effectively hides it from the	FreeBSD	kernel.	 To
     work around this limitation, FreeBSD detects that the AHCI	device sup-
     ports RST and when	it is enabled.	See ahci(4) for	more details.

     nda(4), nvd(4), pci(4), nvmecontrol(8), disk(9)

     The nvme driver first appeared in FreeBSD 9.2.

     The nvme driver was developed by Intel and	originally written by Jim
     Harris <>, with contributions	from Joe Golio at EMC.

     This man page was written by Jim Harris <>.

FreeBSD	13.0			 June 6, 2020			  FreeBSD 13.0


Want to link to this manual page? Use this URL:

home | help