MLX

Overview

MLX is Apple's machine learning framework optimized for Apple Silicon. Built by Apple's machine learning research team and available on GitHub, MLX provides a NumPy-like API with automatic differentiation, lazy computation, and unified memory. When integrated with Autohand, you get:

• Native Metal acceleration on M1, M2, M3, and M4 chips
• Unified memory architecture for efficient model loading
• Complete privacy since your code never leaves your machine
• No API costs with unlimited local inference
• Low latency responses from your local hardware
• Support for popular open-source models in MLX format

Requirements

Before setting up MLX, make sure you meet these requirements:

Hardware

• Mac with Apple Silicon (M1, M2, M3, or M4 series)
• Minimum 8GB unified memory (16GB+ recommended)
• macOS Sonoma 14.0 or later recommended

Software

• Python 3.9 or later
• pip or uv package manager
• Autohand CLI v0.7.0 or later

Memory recommendations

Installation

Install the MLX LM package to run a local inference server.

Install mlx-lm

The mlx-lm package provides both the inference engine and an OpenAI-compatible server:

# Using pip
pip install mlx-lm

# Using uv (faster)
uv pip install mlx-lm

# Verify installation
python -c "import mlx_lm; print('MLX LM installed successfully')"

Download a model

MLX models are available on Hugging Face. Download one optimized for coding:

# Download a coding-optimized model
mlx_lm.convert --hf-path codellama/CodeLlama-7b-Instruct-hf -q

# Or download a pre-converted MLX model
pip install huggingface_hub
huggingface-cli download mlx-community/CodeLlama-7b-Instruct-mlx

Start the server

Run the OpenAI-compatible API server:

# Start the server with your model
mlx_lm.server --model mlx-community/CodeLlama-7b-Instruct-mlx

# Or specify a local model path
mlx_lm.server --model ./models/codellama-7b-mlx

# The server runs on http://localhost:8080 by default

Verify the server

# Check if the server is running
curl http://localhost:8080/v1/models

# Test a completion
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'

CLI configuration

Configure Autohand to use MLX as your model provider in ~/.autohand/config.json:

{
  "provider": "mlx",
  "mlx": {
    "baseUrl": "http://localhost:8080",
    "model": "codellama-7b-instruct",
    "temperature": 0.7,
    "maxTokens": 4096
  }
}

Or use YAML format in ~/.autohand/config.yaml:

provider: mlx
mlx:
  baseUrl: http://localhost:8080
  model: codellama-7b-instruct
  temperature: 0.7
  maxTokens: 4096

Configuration options

Environment variables

You can also configure MLX using environment variables:

# Set MLX as the default provider
export AUTOHAND_PROVIDER="mlx"
export AUTOHAND_MODEL="codellama-7b-instruct"

Available models

The MLX community maintains optimized versions of popular models on Hugging Face.

Coding models

General-purpose models

Download and run models

# Start server with a Hugging Face model (downloads automatically)
mlx_lm.server --model mlx-community/CodeLlama-7b-Instruct-mlx

# Use a quantized model for lower memory usage
mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit

# Switch models in Autohand during a session
/model mlx/codellama-13b-instruct

Server options

Fine-tune the mlx-lm server for optimal performance:

Basic options

mlx_lm.server \
  --model mlx-community/CodeLlama-7b-Instruct-mlx \
  --host 127.0.0.1 \
  --port 8080

Memory and performance

# Use 4-bit quantization for lower memory
mlx_lm.server \
  --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit

# Specify cache directory for models
mlx_lm.server \
  --model mlx-community/CodeLlama-7b-Instruct-mlx \
  --cache-dir ~/.cache/mlx-models

Server options reference

Commands

Use these commands to interact with MLX through Autohand:

Switch to MLX

# Start Autohand with MLX
autohand --provider mlx --model codellama-7b-instruct

# Or use the /model command during a session
/model mlx/codellama-7b-instruct

Check connection

# Test the MLX server
curl http://localhost:8080/v1/models

# Check server health
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "test"}], "max_tokens": 5}'

Converting models

Convert Hugging Face models to MLX format for optimal performance:

Convert a model

# Convert with default settings
mlx_lm.convert --hf-path codellama/CodeLlama-7b-Instruct-hf

# Convert with 4-bit quantization (recommended for most users)
mlx_lm.convert --hf-path codellama/CodeLlama-7b-Instruct-hf -q

# Specify output directory
mlx_lm.convert \
  --hf-path codellama/CodeLlama-7b-Instruct-hf \
  --mlx-path ./models/codellama-7b-mlx \
  -q

Quantization options

Best practices

• Use quantized models: 4-bit quantized models offer the best balance of quality and memory usage on Apple Silicon.
• Match model to memory: Choose models that fit comfortably in your unified memory with room for the OS and other apps.
• Keep the server running: Start the MLX server once and keep it running to avoid model reload times.
• Use coding models for code: CodeLlama and DeepSeek Coder models perform better on coding tasks than general-purpose models.
• Monitor Activity Monitor: Check memory pressure in Activity Monitor to ensure you're not swapping.

Recommended configurations

8GB unified memory (M1/M2 base)

{
  "provider": "mlx",
  "mlx": {
    "model": "codellama-7b-instruct-4bit",
    "maxTokens": 2048
  }
}

16GB unified memory (M1/M2 Pro)

{
  "provider": "mlx",
  "mlx": {
    "model": "codellama-13b-instruct",
    "maxTokens": 4096
  }
}

32GB+ unified memory (M1/M2/M3 Max)

{
  "provider": "mlx",
  "mlx": {
    "model": "codellama-34b-instruct",
    "maxTokens": 8192
  }
}

Troubleshooting

Common issues

Check platform support

# Verify Apple Silicon
uname -m
# Should output: arm64

# Check macOS version
sw_vers
# Should show macOS 14.0 or later for best performance

Debug mode

# Run Autohand with debug logging
AUTOHAND_DEBUG=true autohand --provider mlx

# Check MLX server logs
mlx_lm.server --model your-model 2>&1 | tee mlx-server.log

Memory pressure

If you see yellow or red memory pressure in Activity Monitor:

• Switch to a smaller model (7B instead of 13B)
• Use 4-bit quantization
• Close other memory-intensive applications
• Restart the MLX server to clear cached data

MLX vs other providers

Choose the right local provider for your setup: