Background
In this post, I will show you the usage and implementation of two Golang standard packages’ : bytes
(especially bytes.Buffer
) and bufio
.
These two packages are widely used in the Golang ecosystem especially works related to networking, files and other IO tasks.
Demo application
One good way to learn new programming knowledge is checking how to use it in real-world applications. The following great demo application is from the open source book Network Programming with Go by Jan Newmarch
.
For your convenience, I paste the code here. This demo consists of two parts: client side and server side, which together form a simple directory browsing protocol. The client would be at the user end, talking to a server somewhere else. The client sends commands to the server side that allows you to list files in a directory and print the directory on the server.
First is the client side program:
1 | package main |
client.go
Then is server side code:
1 | package main |
server.go
Bytes.Buffer
Based on the above demo, let’s review how Bytes.Buffer
is used.
According to Go official document:
Package bytes implements functions for the manipulation of byte slices.
A Buffer is a variable-sized buffer of bytes with Read and Write methods.
The bytes
package itself is easy to understand, which provides functionalities to manipulate byte slice. The concern is bytes.Buffer
, what benefits can we get by using it? Let’s review the demo code where it is used.
1 | func dirRequest(conn net.Conn) { |
The above code block is from client.go
part. And the scenario is: the client send DIR
command to server side, server run this DIR
command which will return contents of current directory. Client and server use conn.Read
and conn.Write
to communicate with each other. The client keeps reading data in a for
loop until all the data is consumed which is marked by two continuous \r\n
strings.
In this case, a new bytes.Buffer
object is created by calling NewBuffer
method and three other member methods are called: Write
, Len
and Bytes
. Let’s review their source code:
1 | type Buffer struct { |
The implementation is easy to understand and no need to add more explanation. One interesting point is inside the Write
function. It will first check whether the buffer has enough room for new bytes, if no then it will call internal grow
method to add more space.
In fact, this is the biggest benefit you can get from Buffer
. You don’t need to manage the dynamic change of buffer length manually, bytes.Buffer
will help you to do that. In this way you won’t waste memory by setting the possible maximum length just for providing enough space. To some extend, it is similar to the vector in C++ language.
Bufio
Next, let’s review how Bufio
pacakge works. In our demo, it is used as following:
1 | reader := bufio.NewReader(os.Stdin) |
Before we dive into the details about the demo code, let’s first understand what is the purpose of bufio
package.
First we need to understand that when applications run IO operations like read or write data from or to files, network and database. It will trigger system call
in the bottom level, which is heavy in the performance point of view.
Buffer IO is a technique used to temporarily accumulate the results for an IO operation before transmitting it forward. This technique can increase the speed of a program by reducing the number of system calls. For example, in case you want to read data from disk byte by byte. Instead of directly reading each byte from the disk every time, with buffer IO technique, we can read a block of data into buffer once, then consumers can read data from the buffer in whatever way you want. Performance will be improved by reducing heavy system calls.
Concretely, let’s review how bufio
package do this. The Go official document goes like this:
Package bufio implements buffered I/O. It wraps an io.Reader or io.Writer object, creating another object (Reader or Writer) that also implements the interface but provides buffering and some help for textual I/O.
Let’s understand the definition by reading the source code:
1 | // NewReader and NewReaderSize in bufio.go |
In our demo, we use NewReader
which then calls NewReaderSize
to create a new Reader
instance. One thing need to notice is that the parameter is io.Reader
type, which is an important interface implements only one method Read
.
1 | // the Reader interface in io.go file |
In our case, we use os.Stdin
as the function argument, which will read data from standard input.
Then let’s reivew declaration of bufio.Reader
which wraps io.Reader
:
1 | // Reader implements buffering for an io.Reader object. |
bufio.Reader
has many methods defined, in our case we use ReadString
, which will call another low-level method ReadSlice
.
1 | func (b *Reader) ReadSlice(delim byte) (line []byte, err error) { |
When buf
byte slice contains data, it will search the target value inside it. But initially buf
is empty, it need firstly load some data, right? That is the most interesting part. The b.fill()
is just for that.
1 | func (b *Reader) fill() { |
The data is loaded into buf
by calling the underlying Reader,
1 | n, err := b.rd.Read(b.buf[b.w:]) |
in our case is os.Stdin
.
Customized Reader
To have a better understand about the buffering IO technique, we can define our own customized Reader
and pass it bufio.NewReader
as follows:
1 | package main |
Please run the demo code above, observe the output and think about why it generates such result.
Summary
In this post, I only talked about Reader
part of bufio, if you understand the behavior explained above clearly, it’s easy to understand Writer
quickly as well.