Naive upload server

In this lesson, we will build a simple HTTP server that can accept a file upload from the user and store the file in the server. We will use the net/http package to build the server and the io package to handle the file upload.

You probably have done something similar, but let’s do it again to refresh our memory.

Setting up the server

Let’s start with the initialization of simple HTTP server that can accept a file uploaded by user and store the file in the server.

package main

import (
  "fmt"
  "io"
  "net/http"
  "os"
  "github.com/gorilla/mux"
  "github.com/rs/zerolog/log"
)

func main() {
  ctx := context.Background()
  ctx, cancel := context.WithCancel(ctx)
  ch := make(chan os.Signal, 1)
  signal.Notify(ch, os.Interrupt)
  signal.Notify(ch, syscall.SIGTERM)

  go func() {
    oscall := <-ch
    log.Warn().Msgf("system call:%+v", oscall)
    cancel()
  }()

  httpServer := &http.Server{
    Addr:    ":8080",
    Handler: s.newHTTPHandler(),
  }

  go func() {
    if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
      log.Fatal().Err(err).Msgf("listen:%+s\n", err)
    }
  }()

  <-ctx.Done()

  gracefulShutdownPeriod := 30 * time.Second
  shutdownCtx, cancel := context.WithTimeout(context.Background(), gracefulShutdownPeriod)
  defer cancel()
  if err := httpServer.Shutdown(shutdownCtx); err != nil {
    log.Error().Err(err).Msg("failed to shutdown http server gracefully")
  }
}

Let’s break down the implementation above:

ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
ch := make(chan os.Signal, 1)
signal.Notify(ch, os.Interrupt)
signal.Notify(ch, syscall.SIGTERM)

go func() {
  oscall := <-ch
  log.Warn().Msgf("system call:%+v", oscall)
  cancel()
}()

This part of the code is to handle the graceful shutdown of the server. When the server receives an interrupt signal, it will cancel the context and shutdown the server gracefully. While this looks like an overkill for a simple server, it is a good practice to have this in places so that we can make sure that we can close all of the opened resources properly.

httpServer := &http.Server{
  Addr:    ":8080",
  Handler: newHTTPHandler(),
}

This part of the code is just to create a new HTTP server with the address :8080 and the handler is the newHTTPHandler function which we will defined later.

go func() {
  if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
    log.Fatal().Err(err).Msgf("listen:%+s\n", err)
  }
}()

<-ctx.Done()

This part of the code is to start the HTTP server in a goroutine.

gracefulShutdownPeriod := 30 * time.Second
shutdownCtx, cancel := context.WithTimeout(context.Background(), gracefulShutdownPeriod)
defer cancel()
if err := httpServer.Shutdown(shutdownCtx); err != nil {
  log.Error().Err(err).Msg("failed to shutdown http server gracefully")
}

If you observe, whenever the SIGTERM signal is called, the goroutine will be unblocked and cancel() will cancel the context. This cancelation will close to the ctx.Done() channeland unblock block the execution of the program. At this point, we will start the graceful shutdown of the server. We will give the server 30 seconds to shutdown gracefully. If the server is not shutdown within 30 seconds, we will forcefully shutdown the server.

Handling the simple file upload

When a file is being uploaded or when you make an API call to a HTTP server, data is sent in the form of stream of bytes. Thus, what we need to do is just to read the stream of bytes and write it to a file.

Here is the implementation of the file upload handler named BinaryUpload:

import (
	"io"
	"net/http"
	"os"
	"path/filepath"
	"github.com/google/uuid"
)

func BinaryUpload() http.HandlerFunc {
  return func(w http.ResponseWriter, r *http.Request) {
    defer r.Body.Close()

    f, err := os.OpenFile(filepath.Join("/tmp", uuid.NewString()), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)
    if err != nil {
      log.Error().Err(err).Msg("Error Creating the File")
      w.WriteHeader(http.StatusBadRequest)
      return
    }
    defer f.Close()
    defer os.Remove(f.Name())

    d, err := io.ReadAll(r.Body)
    if err != nil {
      log.Error().Err(err).Msg("Error Copying the File")
    }

    n, err := f.Write(d)
    if err != nil {
      log.Error().Err(err).Msg("Error Copying the File")
    }

    log.Info().
      Int("written_size", n).
      Str("stored_file", f.Name()).
      Msg("File Uploaded")

    w.WriteHeader(http.StatusOK)
  }
}

Let’s break down the implementation above:

defer r.Body.Close()

Since r.Body is an io.ReadCloser, we need to make sure that we close the body after we are done with it. We use defer to make sure that the body is closed after the function is done.

f, err := os.OpenFile(filepath.Join("/tmp", uuid.NewString()), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)
if err != nil {
  log.Error().Err(err).Msg("Error Creating the File")
  w.WriteHeader(http.StatusBadRequest)
  return
}
defer f.Close()
defer os.Remove(f.Name())

Since we want to store the uploaded file to a file on the server, we will create a new file in the /tmp directory with a random UUID as the filename. We use os.OpenFile to create a new file. The os.O_CREATE flag is to create the file if it does not exist, os.O_WRONLY is to open the file for writing, and os.O_TRUNC is to truncate the file if it exists. The 0644 is the permission of the file. We use defer f.Close() to make sure that the file is closed after we are done with it. We also use defer os.Remove() to make sure that the file is removed after we are done with it (just to safe some disk space).

d, err := io.ReadAll(r.Body)
if err != nil {
  log.Error().Err(err).Msg("Error Copying the File")
}

If you ever read a file and then do some processing against the file (e.g. read how many lines in the file), you typically will read the entire content of the file into the computer memory and then do your things, right? This is exactly what io.ReadAll does. It reads the entire content of the r.Body into the memory (stored in var named d) so that you can write the content later to the target file.

:::warn This might not be a good idea if you are dealing with a large file. I will show you how this can be bad for performance and how to fix it in the next lesson. For now, please bare with me. :::

n, err := f.Write(d)
if err != nil {
  log.Error().Err(err).Msg("Error Copying the File")
}
log.Info().Int("written_size", n).Str("stored_file", f.Name()).
  Msg("File Uploaded")

This part of the code where we write the content of the file to the target file. We use f.Write() to write the content of the file to the target file. We also log the size of the file that has been written and the name of the file that has been stored.

To use this upload handler, let’s define the newHTTPHandler function:

func newHTTPHandler() http.Handler {
	mux := mux.NewRouter()
  mux.Handle("/api/v1/binary", http.HandlerFunc(BinaryUpload))
	return mux
}

Now you have a simple HTTP server that can accept a file upload from the user and store the file in the server. Run your server and try to upload a file to the server. You can use curl to upload a file to the server.

On the example below, we are uploading a file named file.pdf to the server:

curl --location 'localhost:8080/api/v1/binary/slow' \
  --header 'Content-Type: application/pdf' \
  --data-binary '@/path/to/file.pdf'

Nice! You might have noticed something off in the implementation above. If yes, that’s good! But if you still don’t see it, don’t worry! On the next lesson, we will discuss how the implementation above can be bad for performance and how to fix it. See ya!