Using checksum effectively

You have seen how server can use checksum to verify the integrity of the file. However, on the previous lesson, I intentionally read the content of r.Body to a buffer and create two io.ReadCloser from the buffer to calculate the checksum and to write the content to a file.

As you might have realized, this is not the most effective way to handle the checksum or other cases where you need to read the content of the body multiple times.

Using io.TeeReader

Let’s discuss a more effective way to handle this. Instead of reading the content of the body to a buffer and create multiple io.ReadCloser from the buffer, you can use io.TeeReader to read the content of the body and write it to multiple io.Writer at the same time.

io.TeeReader is a utility in Go’s io package that allows you to read from an io.Reader while simultaneously writing the read data to an io.Writer. This is particularly useful when you need to process the data as it is being read, without storing it entirely in memory.

TeeReader returns a Reader that writes to w what it reads from r. All reads from r performed through it are matched with corresponding writes to w. There is no internal buffering - the write must complete before the read completes. Any error encountered while writing is reported as a read error. source

When you use io.TeeReader, it returns a new Reader that reads from the original Reader and writes the data to the specified Writer. The key advantage is that it allows you to perform operations like checksum calculation and file writing concurrently, without duplicating the data in memory.

io.Reader -> io.TeeReader -> io.Writer
                |
                v
                io.Writer

Here’s a simple example of how io.TeeReader can be used:

checksum := r.Header.Get("X-Checksum")

hash := md5.New()
tee := io.TeeReader(r.Body, hash)

// ... open a file named f
n, err = io.Copy(f, tee)
// handle error

calculatedHash := hex.EncodeToString(hash.Sum(nil))

In the example above, we create a new hash object to calculate the checksum. We then create a new TeeReader that reads from r.Body and writes the data to the hash object.

We then copy the data from the TeeReader to a file. Remember that TeeReader also implements the io.Reader interface. Finally, we calculate the checksum by calling hash.Sum(nil) and convert it to a hexadecimal string using hex.EncodeToString.

Hence, by using io.TeeReader you can avoid allocating additional memory used to store the content of the body temporarily. The memory usage is contant and does not depend on the size of the file being uploaded. This is particularly useful when you are dealing with large files.