File metadata

Similar to checksum, metadata is another important aspect of file upload. Metadata is data that provides information about other data. In the context of file upload, metadata can include information such as the file name, file size, file type, and other attributes.

When uploading a file, it is essential to include metadata to provide additional context about the file being uploaded. This information can be used to validate the file, enforce file size limits, and ensure that the file is processed correctly on the server-side.

What metadata to include

When designing an upload system, consider including the following metadata:

File name: The name of the file being uploaded.
File size: The size of the file in bytes.
File type: The MIME type of the file.
Additional attributes: Any other relevant information about the file, such as creation date, modification date, or custom attributes.

Including metadata in the upload request allows the server to validate the file and enforce any constraints or business rules related to the file. For example, the server can check the file size against a maximum limit, verify the file type to prevent malicious uploads, or extract additional information from the metadata to process the file correctly.

Sending metadata

There are several ways to send metadata along with the file upload:

HTTP headers

One common approach is to include metadata in the HTTP headers of the upload request. You can add custom headers to the request to send metadata information. You may use custom headers such as X-File-Name, X-File-Size, X-File-Type, or any other header name that suits your needs and send each value as a header field.

Another approach is to send metadata as encoded text in a single header. For example, you can encode the metadata as JSON or XML and send it as a single header field. This approach allows you to send multiple metadata fields in a structured format and is suitable for more complex metadata structures.

Assume you have the following metadata:

{
  "name": "example.txt",
  "size": 1024,
  "type": "text/plain"
}

You can encode the metadata as JSON and send it as a header field while sending the file to the server:

curl -X POST --data-binary @path/to/file.txt \
    -H "X-Metadata: $(base64 path/to/metadata.json)" \ 
    http://localhost:8080/upload

Then, on the server-side, you can decode the metadata from the header and process it accordingly.

Keep in mind that headers have size limitations, so avoid sending large amounts of metadata in the headers. For small metadata fields, headers are a convenient way to include additional information in the request.

Preflight Requests

There are cases where you need to send metadata before the actual file upload. In such cases, you can use preflight requests to send metadata information to the server and receive a response with additional instructions, validation results, or even URL that you can use to upload the actual file. This technique is commonly used in resumable upload systems, where you need to initiate the upload process and send metadata before uploading the file in chunks. If you are not sure about resumable uploads, worry not! We will cover that in a later lesson.

Instead of sending the file directly, you send a preflight request with the metadata information:

curl -X POST --data-binary @path/to/metadata.json \
    -H "Content-Type: application/json" \
    http://localhost:8080/upload/preflight

When the server receives the preflight request, it can validate the metadata, generate a unique identifier for the upload session, and return the necessary information to the client (e.g. URL used to upload the actual fiel). The client can then use this information to upload the file in subsequent requests.

Later in resumable upload lessons, you will see how to use preflight requests to send metadata and initiate the upload process by using the URL provided by the server.

To summarize, metadata is essential for providing context and additional information about the files being uploaded. Including metadata in the upload request allows the server to validate the file, enforce constraints, and process the file correctly. You can send metadata as HTTP headers or use preflight requests to send metadata before the actual file upload. By including metadata in the upload process, you can enhance the reliability and security of your file upload system.