Form data upload

Are you a web developer? I know that you have been waiting for this, right? :p

In the previous lesson, we have built a HTTP server and client that can upload and handle binary of stream. However, if you are providing a web interface for the user to upload a file along with other information in the form of form, you might want to use the form data instead of sending the binary data directly.

Moving forward, we will only use octet-stream as the content type for the binary upload. This is because we are not going to handle the form data in the server. We will only focus on the binary upload.

Now, lets assume where you have this requirements:

  1. You want to allow the user to upload a file along with other information in the form. File that you want to upload is stored in form data with key file.
  2. When uploading a big file, you want to make sure that the server’s memory is not overwhelmed by the file upload.

ParseMultiPartForm

To handle the form data, in Go, we can use r.ParseMultipartForm to parse the form data. This function accepts the maximum size of the form data that can be parsed.

Let’s consider the following code;

func FormUpload() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        if err := r.ParseMultipartForm(5 << 20); err != nil {
            log.Error().Err(err).Msg("Error Parsing the Form")
            w.WriteHeader(http.StatusBadRequest)
        return
        }
        // the rest of the code ...
    }
}

In the code above, we parse the form data with a maximum size of memory of 5MB. Note that this is not the maximum size of the file that can be uploaded. This is the maximum size of the form data that can be stored in memory. The whole request body is parsed and up to a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files.

So if you are uploading a file with 20MB, the file will be stored in memory up to 5MB and the rest will be stored in disk as temporary file in the /tmp directory. By doing so, the memory usage of the server is limited to 5MB per file upload.

If you don’t trust me, you can try to upload a file with 20MB and inspect the /tmp directory before the function is exited. You will see a file with a random name in the /tmp/multipart-form-*.

Clean up the temporary files

While ParseMultipartForm will attempt to clean up the temporary files after the request is done, the clean up is not always guaranteed. My experiment shows that the temporary files won’t be cleaned up if we have a HTTP middleware that updates the context of the request.

Hence, it is always a good practice to clean up the temporary files as soon as we are done with it. Here is the trick:

defer r.MultipartForm.RemoveAll()

By adding the line above, we make sure that the temporary files are cleaned up as soon as we are done with the form data.

Handling the file upload

From the requirement, we know that the file is stored in the form data with key file. To get the file, we can use r.FormFile to get the file from the form data.

// ... the rest of the code on the handler
file, handler, err := r.FormFile("file")
if err != nil {
    w.WriteHeader(http.StatusBadRequest)
    return
}
defer file.Close()

The r.FormFile returns multipart.File which is also an io.Reader. This means that we can use it to read the content of the file or copy it to other io.Writer. If you want to copy the content of the file to another file, you can use io.Copy as we have discussed in the previous lesson.

// open the target file
target, err := os.OpenFile(....)
// handle the error
n, err := io.Copy(target, file)

Summary

In this lesson, we learned how to handle file uploads using HTML form data in Go. Here are the key points:

  • We used ParseMultipartForm to parse form data with a memory limit of 5MB
  • The memory limit only affects how much form data is stored in memory - larger files are automatically stored in temporary files on disk
  • It’s important to clean up temporary files using r.MultipartForm.RemoveAll() to prevent disk space leaks
  • Files can be accessed from form data using r.FormFile() which returns an io.Reader
  • We can efficiently copy uploaded files using io.Copy to minimize memory usage

This approach provides a more user-friendly way to handle file uploads compared to raw binary uploads, while still maintaining good performance characteristics through streaming and proper resource cleanup.

In the next lesson, we’ll look at more advanced topics like handling multiple file uploads and providing upload progress feedback to users.