Input/Output, generally referred to as I/O, is a term that covers the ways that a computer interacts with the world. Screens, keyboards, files, and networks are all forms of I/O. Data from these devices is sent to and from programs as a stream of characters/bytes.
Unix-like systems treat all external devices as files. We can see these under
the /dev directory.
Read this
list
for a quick description of all the devices we might find under /dev for OS X.
For example (truncated for brevity):
$ tree /dev
/dev
├── disk0
├── fd
│ ├── 0
│ ├── 1
│ ├── 2
│ └── 3 [error opening dir]
├── null
├── stderr -> fd/2
├── stdin -> fd/0
├── stdout -> fd/1
├── tty
└── zero
I/O streams are located under the /dev/fd directory. Files there are given a
number, known as a file descriptor. The operating system provides three streams
by default. They are:
- Standard input (
/dev/fd/0) - Standard output (
/dev/fd/1) - Standard error (
/dev/fd/2)
They are often abbreviated to stdin, stdout, and stderr respectively. Standard
input will default to reading from the keyboard while standard output and
standard error both default to writing to the terminal. As can be seen above,
/dev/stdout, /dev/stdin, and /dev/stderr are just symlinks to the
appropriate file descriptor.
The IO class
Ruby IO objects wrap Input/Output streams. The constants STDIN, STDOUT, and
STDERR point to IO objects wrapping the standard streams. By default the
global variables $stdin, $stdout, and $stderr point to their respective
constants. While the constants should always point to the default streams, the
globals can be overwritten to point to another I/O stream such as a file. IO
objects can be written to via puts and print.
$stdout.puts 'Hello World'
We’ve all written the shorthand version of this program:
puts 'Hello World'
The bare puts method is provided by ruby’s Kernel module that is just an
alias to $stdout.puts. Similarly, IO objects can be read from via gets.
The bare gets provided by Kernel is an alias to $stdin.gets
$stdin is read-only while $stdout and $stderr are write-only.
[1] pry(main)> $stdin.puts 'foo'
IOError: not opened for writing
[2] pry(main)> $stdout.gets
IOError: not opened for reading
[3] pry(main)> $stderr.gets
IOError: not opened for reading
To create a new IO object, we need a file descriptor.
In this case, 1 (stdout).
[1] pry(main)> io = IO.new(1)
=> #<IO:fd 1>
[2] pry(main)> io.puts 'hello world'
hello world
=> nil
What about creating IOs to other streams? They don’t have constant file
descriptors so we first need to get that via IO.sysopen.
[1] pry(main)> fd = IO.sysopen('/dev/null', 'w+')
=> 8
[2] pry(main)> dev_null = IO.new(fd)
=> #<IO:fd 8>
[3] pry(main)> dev_null.puts 'hello'
=> nil
[4] pry(main)> dev_null.gets
=> nil
[5] pry(main)> dev_null.close
=> nil
/dev/null (sometimes referred to as the “bit bucket” or “black hole”) is the
null device on Unix-like systems. Writing to it does nothing and attempting to
read from it returns nothing (nil in Ruby)
First, we get a file descriptor for a stream that that is read/write to the
dev/null device. Then we create an IO object for this stream so we can
interact with it in Ruby. When writing to dev_null, the text no longer appears
on the screen. When reading from dev_null, we get nil.
Since everything on a Unix-like system is a file, we can open an IO stream to
a text file in the same way we would open a device. We just create a file
descriptor with the path to our file and then create an IO object for that
file descriptor. When we are done with it, we close the stream to flush Ruby’s
buffer and release the file descriptor back to the operating system. Attempting
read or write from a closed stream will raise an IOError.
Position
When working with an IO, we have to keep position in mind. Given that we’ve
opened a stream to the following file:
Lorem ipsum
dolor
sit amet...
and we call gets on it:
[1] pry(main)> IO.sysopen '/Users/joelquenneville/Desktop/lorem.txt'
=> 8
[2] pry(main)> lorem = IO.new(8)
=> #<IO:fd 8>
[3] pry(main)> lorem.gets
=> "Lorem ipsum\n"
it returns the first line of the file and moves the cursor to the next line. If we check the position of the cursor:
[4] pry(main)> lorem.pos
=> 12
If we call gets a few more times:
[5] pry(main)> lorem.gets
=> "dolor\n"
[6] pry(main)> lorem.gets
=> "sit amet...\n"
[7] pry(main)> lorem.pos
=> 30
we can see ruby’s “cursor” has moved. Now that we have read the whole file, what
happens if we try to call gets?
[8] pry(main)> lorem.gets
=> nil
[9] pry(main)> lorem.eof?
=> true
We see that it returns nil. We can ask a stream if we have reached “end of
file” via eof?. To return to the beginning of the stream, we can call
rewind.
[10] pry(main)> lorem.rewind
=> 0
[11] pry(main)> lorem.pos
=> 0
This can lead to surprises when writing to a stream.
[1] pry(main)> fd = IO.sysopen '/Users/joelquenneville/Desktop/test.txt', 'w+'
=> 8
[2] pry(main)> io = IO.new(fd)
=> #<IO:fd 8>
[3] pry(main)> io.puts 'hello world'
=> nil
[4] pry(main)> io.puts 'goodbye world'
=> nil
This stream has the lines “hello world” and “goodbye world”. If we were to attempt to read:
[5] pry(main)> io.gets
=> nil
[6] pry(main)> io.eof?
=> true
Our cursor is currently at the end of the file. In order to read we would need to first rewind.
[7] pry(main)> io.rewind
=> 0
[8] pry(main)> io.gets
=> "hello world\n"
Any write operations in the middle of a stream will overwrite the existing data:
[9] pry(main)> io.pos
=> 12
[10] pry(main)> io.puts "middle"
=> nil
[11] pry(main)> io.rewind
=> 0
[12] pry(main)> io.read
=> "hello world\nmiddle\n world\n"
This kind of behavior is necessary because streams do not get loaded into
memory. Instead, only the lines being operated on are loaded. This is very
useful because some streams can point to very large files that would be
expensive to load in memory all at once. Streams can also be infinite. For
example, $stdin has no end. We can always read more data from it (when it
receive the message gets, it waits for the user to type something).
Sub-classes and Duck-types
Ruby gives us a couple subclasses of IO that are more specialized for a
particular type of IO:
File
Probably the most well known IO subclass. File allows us to read/write files
without messing around with file descriptors. It also adds file-specific
convenience methods such as File#size, File#chmod, and File.path.
The Sockets
Socket docs:
Ruby’s various socket classes inherit all ultimately inherit from IO.
For example, I have a server running on localhost:3000
[1] pry(main)> require 'socket'
=> true
[2] pry(main)> socket = TCPSocket.new 'localhost', 3000
=> #<TCPSocket:fd 10>
[3] pry(main)> socket.puts 'GET "/"'
=> nil
[4] pry(main)> socket.gets
=> "HTTP/1.1 400 Bad Request \r\n"
StringIO
StringIO allows strings to behave like IOs. This is useful when we want to
pass strings into systems that consume streams. This is common in tests where
we might inject a StringIO instead of reading an actual file from disk.
Unlike previous classes showcased, StringIO does not inherit from IO.
[1] pry(main)> string_io = StringIO.new('hello world')
=> #<StringIO:0x007feacb0cd4e8>
[2] pry(main)> string_io.gets
=> "hello world"
[3] pry(main)> string_io.puts 'goodby world'
=> nil
[4] pry(main)> string_io.rewind
=> 0
[5] pry(main)> string_io.read
=> "hello worldgoodby world\n"
Tempfile
Tempfile is another class that doesn’t inherit from IO. Instead, it
implements File‘s interface and deals with temporary files. As such, it can be
passed to any object that consumes IO-like objects.
Putting it all together
Say we have the following class for some command-line program:
class SystemTask
def execute
puts "preparing to execute"
puts "starting first task"
first_task
puts "starting second task"
second_task
puts "execution complete"
end
end
Testing this class causes all these messages to be output, cluttering our
results. One approach to solving this problem would be to inject IO objects
instead of calling Kernel#puts and to pass in a null object in tests.
class SystemTask
def initialize(io=$stdout)
@io = io
end
def execute
@io.puts "preparing to execute"
@io.puts "starting first task"
first_task
@io.puts "starting second task"
second_task
@io.puts "execution complete"
end
end
In production, we can still call SystemTask.new.execute as before.
Now we can pass in our own IO in tests. This could be a test double, a
StringIO, or a stream to /dev/null
describe SystemTask do
# test double
it "executes tasks" do
io = double("io", puts: nil)
system_task = SystemTask.new(io)
system_task.execute
# expect things to have happened
# if we care about the messages, we can also expect on the double
expect(io).to have_received(:puts).with("preparing to execute")
end
# StringIO
it "executes tasks" do
io = StringIO.new
system_task = SystemTask.new(io)
system_task.execute
# expect things to have happened
# if we care about the messages read from the string io
io.rewind
expect(io.read).to eq "preparing to execute\nstarting first task\nstarting
second task\nexecution complete\n"
end
# /dev/null
it "executes tasks" do
io = File.open(File::NULL, 'w')
system_task = SystemTask.new(io)
system_task.execute
# expect things to have happened
# only use /dev/null if we don't care about the messages
end
end
Working with disparate APIs
While working on a recent project that pulled reports from several APIs, we noticed some responses were strings, others were CSV documents, and others generate the report and then we had to make a request to another endpoint to download it
The solution was to create an adapter for each API that would get the data and return in a
standard format wrapped in some type of IO-like object. A persistor object could
then process and persist any of the reports as long as they were formatted the
same way and were IO-like. For example:
class API1Report
def fetch
# fetch report (comes down as a CSV doc)
# process it to get it in a standard format
# return standardized report as a Tempfile object
end
end
class API2Report
def fetch
# fetch report
# returns it as a File object
end
end
class Persistor
def initialize(report)
@report = report
end
def persist
# process and persist the report
end
end
What’s next
Read an overview of 4.4 BSD’s I/O to develop a deeper understanding of Unix I/O, file descriptors, and devices.
Read the TTY system to understand the relationship between Unix jobs, processes, and I/O with the TTY device.
Practice Ruby I/O by cloning this repo.
Finally, go deeper into Ruby’s I/O in this chapter from Read Ruby.
