Metronome

Metronome is a research project that implements an intelligent scheduler for serverless functions with the advanced locality-aware scheduling policy and machine learning-based prediction models. It extends OpenLambda with sophisticated scheduling algorithms and comprehensive experimental frameworks.

Project Structure

.
├── metronome/                    # Metronome Core Components
│   ├── experiments/              # Experimental results and analysis data
│   ├── faas-scripts/            # Workload generation and experiment orchestration
│   ├── my-scheduler/            # Metronome intelligent scheduler
│   └── predict-model/           # ML-based execution time prediction service
└── ol/                          # OpenLambda (Extended)
    ├── src/                     # OpenLambda source code
    ├── my-code/                 # Serverless function registry
    ├── docs/                    # Documentation
    └── scripts/                 # Utility scripts

Component Details

• Metronome Scheduler: Implements the delay-aware locality scheduling
• FaaS Scripts: Generates realistic workloads and orchestrates experiments
• Predict Model: Uses Random Forest and other ML models to predict function execution times
• OpenLambda: Extended version with custom APIs and monitoring capabilities
• Function Registry: serverless functions for various experimental scenarios

Experimental Setup

Experimental Applications

1. Video Processing Application

• Functions: video-split → video-transcode → video-merge
• Data: Video files
• SLA: 7,000ms per stage
• Use case: CPU-intensive multimedia processing

2. Log Analysis Application

• Functions: log-split → log-analyze → log-merge
• Data: Nginx log files
• SLA: 6,000ms for upload/split, 1,000ms for analyze/merge
• Use case: Data analytics and log processing

3. ML Inference Application

• Functions: ml-normalize → ml-process
• SLA: 13,000ms
• Use case: AI/ML model serving

4. Document Processing Application

• Functions: any2md-validate → any2md-process
• Data: PDF documents
• SLA: 17,000ms
• Use case: Document transformation and processing

Configuration

Environment Variables

Scheduler Configuration (metronome/my-scheduler/.env)

# Node topology (required)
NODE_LIST=node1:100.2.3.100,node2:100.2.3.101,node3:100.2.3.102,node4:100.2.3.103

# OpenLambda integration
OL_REGISTRY_PATH=/root/ol/my-code/registry
OL_ZYGOTE_FILENAME=test-zygotes.json

FaaS Scripts Configuration (metronome/faas-scripts/.env)

# Current node IP (required)
NODE_IP=100.2.3.100

# Scheduler endpoint
SCHEDULER_ENDPOINT=http://100.2.3.100:5555

# OpenLambda endpoint
OL_RAW_ENDPOINT=http://100.2.3.100:5050

Setup and Installation

Prerequisites

1. Install Taskfile: https://taskfile.dev/

   # macOS
   brew install go-task/tap/go-task
   
   # Linux
   wget -O task.tar.gz https://github.com/go-task/task/releases/latest/download/task_linux_amd64.tar.gz
   tar -xzf task.tar.gz && sudo mv task /usr/local/bin/

2. Install Docker: https://docs.docker.com/engine/install/

3. Install Go: Version 1.21+ required

   wget -O go.tar.gz https://go.dev/dl/go1.21.5.linux-amd64.tar.gz
   tar -C /usr/local -xzf go.tar.gz
   echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc

4. Install Python: Version 3.8+ with required packages

   pip install pandas scikit-learn grpcio grpcio-tools joblib numpy

Network Setup

Multi-node Configuration

Assumes controller node (node1) with IP 100.2.3.100 and worker nodes (node2, node3, node4, etc.)

1. Configure /etc/hosts on all nodes:

   100.2.3.100 node1
   100.2.3.101 node2
   100.2.3.102 node3
   100.2.3.103 node4
   ...
   

2. Install and configure rclone on all worker nodes:

   # Install rclone
   curl https://rclone.org/install.sh | sudo bash
   
   # Configure SFTP backend for each worker node
   rclone config create node2 sftp host 100.2.3.101 user root
   rclone config create node3 sftp host 100.2.3.102 user root
   rclone config create node4 sftp host 100.2.3.103 user root
   ...

3. SSH Key Setup:

   # Generate SSH key pair on controller
   ssh-keygen -t ed25519
   
   # Copy public key to all worker nodes
   ssh-copy-id root@node2
   ssh-copy-id root@node3
   ssh-copy-id root@node4
   ...

4. Ensure OpenLambda directory exists at /root/ol/ on all nodes.

Build and Deployment

Automated Setup with Taskfile

The project uses Taskfile for automated build and deployment:

# 1. Build OpenLambda binaries and sync to all nodes
task build-go

# 2. Build Docker images locally
task build-docker

# 3. Initialize Lambda instances on all nodes
task init-lambda

# 4. Update function code across all nodes
task update-my-code

# 5. Start worker processes
task up-worker

Please refer to ol/Taskfile.yml for more details.

Running Experiments

1. Start OpenLambda Workers

On each worker node (node2, node3, node4, etc.):

cd /root/ol
./ol worker up

2. Start Metronome Scheduler

On the controller node:

cd metronome/my-scheduler
./my-scheduler

The scheduler will:

• Load node topology from NODE_LIST
• Initialize resource monitoring
• Start web API server on port 5555
• Begin scheduling service

3. Start Prediction Model Server

On the controller node:

cd metronome/predict-model
python server.py

The model server provides:

• gRPC service on port 50051
• ML-based execution time prediction
• Incremental learning capabilities
• Model serving for all function types

4. Run Experimental Workloads

Workflow Experiments

cd metronome/faas-scripts
./faas-scripts -we    # Run workflow experiments

Generate Training Data

If you want to generate training data from scratch:

./faas-scripts -gte   # Generate training data

Experiment Configuration

Workflow Experiment Parameters

• Concurrency levels: 1, 3, 5 (configurable)
• Workflow run count: 100 runs per configuration (configurable)
• Background load: Configurable generation

Results Location

• Scheduler traces: metronome/my-scheduler/traces/
• Expected results: metronome/experiments/

Troubleshooting

Common Issues

1. Connection Issues

• Problem: Cannot connect to worker nodes
• Solution: Check SSH keys, network connectivity, and firewall settings

2. Docker Issues

• Problem: Docker image load failures
• Solution: Ensure Docker daemon is running and images are properly built

3. Model Server Issues

• Problem: gRPC connection failures
• Solution: Check if port 50051 is available and model files exist

4. Resource Exhaustion

• Problem: Out of memory or CPU errors
• Solution: Adjust SlotsPerNode and TotalMemoryPerNode constants

License

This project is licensed under the MIT License.