Hooli Drive is my client and server implementation of a personal cloud storage service (sometimes referred to as an online backup service) that can be used for file sharing and collaboration. This project was programmed in C and was my first exposure to memory leaks and other complex C problems as well as the OSI network stack.
About the tool
When the server and client programs are run simultaneously the client’s specified directory will be synced with the same directory on the server. This means any files in the client directory that do not already exist or are outdated on the server, will be replaced. The process starts with the HMDS (Hooli MetaData Server) sending filenames and checksums of those files to the server, then the server compares that data with it’s directory. The server sends back a request specifying the files that need to be replaced and then those files are transmitted in blocks and reconstructed on the server side for maximum transfer speed and reliability.
The filenames and checksums are sent using TCP within HMDS because this data needs reliable data transfer and is not expected to be large. The file data itself is sent using protocol I designed on top of UDP which ensures reliable data transfer while taking advantage of the speed benefit of UDP.
Structure of the Project
Above you can see the structure of the project repository (along with my angry commit messages). It is categorised into four sections; client, common, HFTPD, and HMDS. Client stores the code that runs on the client’s machine and is responsible for creating messages, sending files, and broadcasting which files are in the directory. Common stores code that is used more than once by multiple components, like the message struct, input parsing code, and UDP networking code. HFTPD contains code to actually transfer files, which includes detecting different types of control and acknowledgement messages and creating files/directories. HMDS is responsible for transferring file metadata, comparing server / client files and handling all communication with redis.
This is one of my favourite personal projects because I love learning about networking and low level C programming. I got a chance to experiment with the different layers of networking using hands-on programming.
I began working on this project while reading the textbook ‘Networking. A top-down approach’ to better understand UDP and TCP. I used both of these protocols and learned when they are applicable and how I can leverage their strengths in my projects.
Development of this project was done on a virtual environment generated by Vagrant. This turned out to be a great decision when I had to switch development environments because it allowed me to continue programming without any problems. I plan to continue using Vagrant in my future projects and maybe experiment with Docker as well.
Overall this was a very good learning experience for me and I will continue seeking for more difficult adventures like this one.