Uploading the content

We have done few initial checks to the request. Now, let’s start the actual upload process.

Opening the file

Since we have store the path of the file in the metadata, we can now open the file for writing. We will use the os.OpenFile function to open the file.

func (c *Controller) ResumeUpload() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // ... the rest of the code
        f, err := os.OpenFile(fm.Path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
        if err != nil {            
            writeError(w, http.StatusBadRequest, errors.New("error opening the file"))
            return
        }
        defer f.Close()
        // ... the rest of the code
    }
}

We open the file with O_CREATE flag to create the file if it doesn’t exist, O_WRONLY to open the file for writing, and O_APPEND to append the content to the end of the file. Thus, using O_APPEND is important here to ensure that we only append the content to the end of the file, not overwrite the file content.

Writing the content

Let’s process this requirement:

The Server SHOULD accept PATCH requests against any upload URL and apply the bytes contained in the message at the given offset specified by the Upload-Offset header.

In the previous lesson, we have checked the Upload-Offset header should be equal to the current offset of the resource, otherwise return error. In addition to that we also use O_APPEND when opening the file to ensure we always start writing from the end.

So at this point, we can just use the io.Copy function to copy the content from the request body to the file.

func (c *Controller) ResumeUpload() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // ... the rest of the code
        n, err = io.Copy(f, r.Body)
        if err != nil {
            writeError(w, http.StatusInternalServerError, fmt.Errorf("error writing the file: %w", err))
            return
        }
        fm.UploadedSize += uint64(n)
        c.store.Save(fm.ID, fm)
        // ... the rest of the code
    }
}

Above, we know that when the writing is done, we get n as the number of bytes written. We then add the number of bytes written to the fm.UploadedSize and save the metadata to the storage. By doing so, the file metadata now is updated with the new offset. Later, client can use this offset to continue the upload.

Responding to the client

The implementation you have seen earlier should satisfy the requirement for normal cases where there is no error or any interruption. Let’s stick with it for now.

Before we handle the negative scenarios, let’s reponse to the client with the correct status code and headers as required by the protocol.

The Server MUST acknowledge successful PATCH requests with the 204 No Content status. It MUST include the Upload-Offset header containing the new offset. The new offset MUST be the sum of the offset before the PATCH request and the number of bytes received and processed or stored during the current PATCH request.

func (c *Controller) ResumeUpload() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // ... the rest of the code
        w.Header().Add(UploadOffsetHeader, fmt.Sprint(fm.UploadedSize))
        w.WriteHeader(http.StatusNoContent)
    }
}

Since we have stored the uploaded size in the metadata, we can just simply return the fm.UploadedSize as the new offset.

Storing as much as possible

Now, Let’s take a look at the requirement:

The Server SHOULD always attempt to store as much of the received data as possible.

As you might have expect, the implementation above doesn’t really satisfy the requirement when there is any issue during the writing process. The issue could be anything from network error, disk full, network congestion, timeout, etc. In the current implementation, when io.Copy fails, the function will just exit and return 5xx error.

However, when there is any issue during the write, there is still possibility that some data has been written to the file. This is why the number of bytes written, n, plays an important role. We can use this information to determine how much data has been written to the file. In case of error, we can save the number of bytes written to the metadata so that on the next request to get the offset, the client can continue from the correct offset.

The fix to this is trivial:

func (c *Controller) ResumeUpload() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // ... the rest of the code
        n, err = io.Copy(f, r.Body)
        if err != nil {

            fm.UploadedSize += uint64(n)
            c.store.Save(fm.ID, fm)

            writeError(w, http.StatusInternalServerError, fmt.Errorf("error writing the file: %w", err))
            return
        }
        fm.UploadedSize += uint64(n)
        c.store.Save(fm.ID, fm)
        // ... the rest of the code
    }
}

The fix is simple, we will just update the fm.UploadedSize with the number of bytes written before returning the error. This way, the client can continue from the correct offset. Later, we will play around with other error scenarios such as timeout and hacking attempt.

Notes: Probably some of you might see code smell where we repeat the same code to update the fm.UploadedSize and save the metadata. We can refactor this later.