Handling the upload size

Knowing the file size is important when we are building an resumable upload system. It helps us to validate the file size before we start processing the file further.

As stated on thus documentation here, there are at least two ways to inform the server about the file size:

Upload length

Let’s start with the simplest way to capture the file size. The client must inform the server about the file size by sending the Upload-Length header. From tus.io:

The Upload-Length request and response header indicates the size of the entire upload in bytes. The value MUST be a non-negative integer.

const (
    UploadLengthHeader      = "Upload-Length"
)
func (c *Controller) CreateUpload() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // ... the rest of the code
        totalLength := r.Header.Get(UploadLengthHeader)
        totalSize, err := strconv.ParseUint(totalLength, 10, 64)
        if err != nil {
            w.WriteHeader(http.StatusBadRequest)
            w.Write([]byte("Invalid Upload-Length header"))
            return
        }
        fm.TotalSize = totalSize
        // ... the rest of the code for handling metadata        
    }
}

Based on tus specification, the file size may be set to 0. This will indicate that the client wants to upload an empty file. This might not be necessary for all use cases, so we can decide to ignore this case for now and not return an error if the file size is 0.

Deferring upload length

However, there could be a case where client doesn’t know the file size ahead of time when initiating the upload. In this case, the client can choose to defer the file size by sending the Upload-Defer-Length header. From tus.io:

The Upload-Defer-Length request and response header indicates that the size of the upload is not known currently and will be transferred later. Its value MUST be 1. If the length of an upload is not deferred, this header MUST be omitted.

If client decides to defer, when should client set the Upload-Length header? From tus.io:

If the length was deferred using Upload-Defer-Length: 1, the Client MUST set the Upload-Length header in the next PATCH request, once the length is known. Once set the length MUST NOT be changed. As long as the length of the upload is not known, the Server MUST set Upload-Defer-Length: 1 in all responses to HEAD requests.

We know that client should set the Upload-Length header in the next PATCH request. But what if the client doesn’t set the Upload-Length header in the next PATCH request? In this case, since we are using uint64 to store the TotalSize and we might allow 0 as the value, we need to differentiate between the case where the client doesn’t know the value and the case where the client intentionally set it to 0. In this case, I will just add IsDeferLength bool to the File struct to indicate whether the client has deferred the length or not.

type File struct {
    // ... the rest of the fields
    IsDeferLength bool
}

Then I want to update the constructor of the File struct to set the IsDeferLength to true. This is assuming that the client will always defer the length when creating the file.

func NewFile() File {
    f := File{
        // ... the rest of the initialization
        IsDeferLength: true,		
    }
    return f
}

By doing so, as required by the protocol, we later can set Upload-Defer-Length to 1 when returning the response to HEAD requests. We will revisit this later once we implement the HEAD request.

Then, let’s update the CreateUpload handler to handle the Upload-Defer-Length header:

const (
    UploadDeferLengthHeader = "Upload-Defer-Length"
)

func (c *Controller) CreateUpload() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // ... the rest of the code
        uploadDeferLength := r.Header.Get(UploadDeferLengthHeader)
        if uploadDeferLength != "" && uploadDeferLength != "1" {
            writeError(w, http.StatusBadRequest, errors.New("invalid Upload-Defer-Length header"))
            return
        }

        isDeferLength := uploadDeferLength == "1"				
        if !isDeferLength {
            totalLength := r.Header.Get(UploadLengthHeader)
            totalSize, err := strconv.ParseUint(totalLength, 10, 64)
            if err != nil {
                w.WriteHeader(http.StatusBadRequest)
                w.Write([]byte("Invalid Upload-Length header"))
                return
            }
            fm.IsDeferLength = false
            fm.TotalSize = totalSize
        }
        // ... the rest of the code for handling metadata
    }
}

On the code above, we check if the Upload-Defer-Length header is not set to 1, then we check if the Upload-Length header is set. If it is set, we parse the value and set it to the TotalSize field of the File struct. We also set the IsDeferLength to false to indicate that the client has set the file size.

Easy, right? At this point, we have stored the given file size. Next, let’s see how we can handle the maximum file size.

Max file size

When initiating the upload, we can also check the value against the maximum file size that we allow. So, instead of waiting until the client uploads the entire file, server can reject the request if the file size exceeds the maximum size.

If the length of the upload exceeds the maximum, which MAY be specified using the Tus-Max-Size header, the Server MUST respond with the 413 Request Entity Too Large status.

Now, assuming that we have a max limit of 100 MB (MegaBytes), let’s update the CreateUpload handler check the file size against the maximum size:

const (
    defaultMaxSize = 100 * 1024 * 1024 // 100MB
)
func (c *Controller) CreateUpload() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // ... the rest of the code
        if fm.TotalSize > defaultMaxSize {
            writeError(w, http.StatusRequestEntityTooLarge, errors.New("upload length exceeds the maximum size"))
            return
        }
    }
}

Then, how client can know the maximum file size that the server allows ahead of time? The server can inform the client about the maximum file size by sending the Tus-Max-Size header. Tus.io has OPTIONS method to get the information about the server before starting the upload. This is where the client can get the information about the maximum file size that the server allows. We will implement this later.