Homemade Beowulf Cluster

A commodity Beowulf cluster built from old desktops and laptops, running distributed MPI jobs across nodes

Technology Stack

Linux

Ubuntu Server

MPICH

MPI

NFS

SSH

Demo Video

Project Overview

A commodity Beowulf cluster assembled from old desktops and laptops. I installed Ubuntu Server on each node and configured the networking, shared storage, and process management needed to distribute parallel MPI jobs across the cluster.

Key Features

• Built from commodity hardware running Ubuntu Server, with static IP networking, hostname resolution, and passwordless SSH across nodes
• NFS-shared filesystem and a common MPI user across the cluster
• MPICH deployed with the Hydra process manager to distribute jobs across nodes
• Validated by creating and running MPI test programs across all nodes, verifying process distribution and inter-node communication

Technical Approach

• Cluster networking configured for reliable node-to-node communication and discovery
• Shared NFS storage so all nodes access a common working directory and MPI binaries
• Hydra process manager to launch and coordinate distributed MPI processes
• Test programs to confirm correct process placement and message passing between nodes

Results & Insights

Performance

The Beowulf cluster successfully handled distributed MPI jobs across all nodes, demonstrating the effectiveness of the cluster architecture. The use of MPICH and Hydra allowed for efficient process management and communication.

Development Insights

This was a project for my Master level Parallel Programming course that a group of students and I were able to complete instead of a traditional research paper to gain some hands-on experience with distributed computing. It provided valuable insights into cluster architecture, networking, parallel programming techniques, and especially debugging in the terminal.

4-Node

Distributed Cluster