Direct I/O and Cached I/O

[sechs]

I think that I've confused the heck out of myself on this one...

My MegaRAID controller offers these two options for each array. My impression was that this was an overriding option, basically turning all caching on and off.

The manual says:

3.5.2 Cache Policy
The cache policy applies to reads on a specific logical drive. It does not
affect the read ahead cache. The options are Cached I/O or Direct I/O.
Cached I/O buffers all reads in cache memory and is the default setting.
Direct I/O does not buffer reads in cache memory. Direct I/O does not
override the cache policy settings. Direct I/O transfers data to cache and
the host concurrently. If the same data block is read again, the host reads
it from cache memory.

This is, of course, different from what I thought. And doesn't make sense to me.

What the heck is the difference between these two settings, and what bearing do they have on the read-ahead and write back policies?

 

[mubs]

My interpretation of that marvelous piece of technical writing is:

No matter what the cache policy, reads are always cached (read ahead)

On writes, "cached" results in write-back; i.e., data is written only to cache, and is written to the drive by the controller at a later time when the drive is not busy.

Direct I/O results in write-thru (writes simultaneously to cache as well as the drive). While safer, direct I/O could significantly impact performance. Realistically, one would use write-thru only is there's no UPS; there is less chance of data in cache but not yet written to the drive.

No matter what the cache policy, if data that is required is in the cache, it is pulled off the cache rather than by a read from the drive.

My apologies in advance if I'm stating the obvious.

 

[CityK]

Their explaination certainly doesn't shed any light on the matter.

I started to take a stab at it, but then the "Direct I/O does not overide the cache policy settings" threw my theory out the window and then I paused to think: Just what the heck is the difference then between these two settings?

 

[CityK]

Perhaps Direct I/O just doesn't engage in speculative caching for reads?

 

[sechs]

There's a separate option for write-back versus write-through.

That said, I found another section on the manual with a slightly different description of the option:

This option enables read buffering in cache memory. The default setting is Direct I/O.
Cached I/O specifies that the controller buffers all reads in cache memory. Direct I/O
specifies that the controller does not buffer reads in cache memory. This parameter
applies to reads on a specific logical drive. It does not affect the read-ahead cache.

Now it sounds like this only has to do with reads. But it doesn't affect the read caching policy?

 

[Fushigi]

The default setting is Direct I/O.

and

Cached I/O buffers all reads in cache memory and is the default setting.

Nice.

Direct: All reads are pass-through and are cached so subsequent reads of the same data create a cache hit. Maybe. It could mean reads aren't cached at all depending on how you interpret it; i.e. the buffer is merely used as a pass-through channel so the electronics don't have to support non-buffered transfers.

Cached: Not sure, but I'd imagine it either does a little look-ahead caching or also keeps write data in the cache.

Whatever the true meanings, it seems to me that Cached I/O would be the better performer in most cases.

 

[sechs]

That sounds backwards. Shouldn't direct be unbuffered?

Still doesn't explain the relationship that it *doesn't* have with the read-ahead policy.

 

[sechs]

For reference

I think that I got an answer on this.

The I/O policy refers to caching of reads, while the read-ahead policy refers to the speculative read-ahead cache.

In direct I/O, a read always goes to the media.

In cached I/O, the first time a read is made, it comes from the media, but is also cached. Subsequent read requests of the same data come from cache.

I presume that data that is in the read-ahead cache is always read from that cache, and never from disk, irrelevant of the I/O policy.

Despite its name, the I/O policy seems to have nothing to do with writes; just reads.