# Download Directory with Subdirectories - Complete Guide

This guide provides multiple methods to download all files from a directory (including subdirectories) from your golf club's web page to your business workspace.

---

## Method 1: Using wget (Recommended for HTTP/HTTPS)

**Best for:** Public web directories accessible via HTTP/HTTPS

### Basic Command:
```bash
wget -r -np -nH --cut-dirs=0 -P /path/to/download/folder https://your-golf-club-website.com/directory/
```

### Options Explained:
- `-r` : Recursive download (downloads subdirectories)
- `-np` : No parent - don't go up to parent directories
- `-nH` : Don't create hostname directories
- `--cut-dirs=N` : Skip N directory levels from the URL
- `-P` : Specify download directory
- `-k` : Convert links for local viewing (optional)
- `--user=username --password=password` : If authentication required

### Example:
```bash
# Download from public directory
wget -r -np -nH -P /var/www/html/wordpress6/wordpress/EP/downloads https://your-golf-club.com/files/

# With authentication
wget -r -np -nH -P ./downloads --user=yourusername --password=yourpassword https://your-golf-club.com/files/

# Convert links for local viewing
wget -r -np -nH -k -P ./downloads https://your-golf-club.com/files/
```

---

## Method 2: Using curl (Alternative HTTP/HTTPS)

**Best for:** When wget is not available or you need more control

### Basic Script:
```bash
#!/bin/bash
# download_directory.sh

URL="https://your-golf-club-website.com/directory/"
OUTPUT_DIR="./downloads"

mkdir -p "$OUTPUT_DIR"

# Download recursively (curl doesn't have built-in recursive, so we use a workaround)
curl -L -o "$OUTPUT_DIR/index.html" "$URL"

# For recursive download, use curl with find or wget is better
```

**Note:** curl doesn't have built-in recursive download. Use wget or the Python script below.

---

## Method 3: Using Python Script (Most Flexible)

**Best for:** When you need custom logic, authentication, or filtering

### Python Script:
```python
#!/usr/bin/env python3
"""
Download all files from a web directory recursively
"""
import os
import sys
import requests
from urllib.parse import urljoin, urlparse
from bs4 import BeautifulSoup
import argparse

def download_file(url, local_path):
    """Download a single file"""
    try:
        response = requests.get(url, stream=True, timeout=30)
        response.raise_for_status()
        
        os.makedirs(os.path.dirname(local_path), exist_ok=True)
        
        with open(local_path, 'wb') as f:
            for chunk in response.iter_content(chunk_size=8192):
                f.write(chunk)
        
        print(f"✓ Downloaded: {local_path}")
        return True
    except Exception as e:
        print(f"✗ Failed to download {url}: {e}")
        return False

def get_directory_listing(url, auth=None):
    """Get list of files and directories from a web directory"""
    try:
        if auth:
            response = requests.get(url, auth=auth, timeout=30)
        else:
            response = requests.get(url, timeout=30)
        response.raise_for_status()
        
        soup = BeautifulSoup(response.text, 'html.parser')
        links = []
        
        # Find all links (adjust selector based on your server's directory listing format)
        for link in soup.find_all('a', href=True):
            href = link['href']
            if href not in ['../', './'] and not href.startswith('?'):
                full_url = urljoin(url, href)
                links.append((full_url, link.text.strip()))
        
        return links
    except Exception as e:
        print(f"✗ Failed to list directory {url}: {e}")
        return []

def download_recursive(base_url, output_dir, auth=None, visited=None):
    """Recursively download all files from a directory"""
    if visited is None:
        visited = set()
    
    if base_url in visited:
        return
    
    visited.add(base_url)
    print(f"\n📁 Processing directory: {base_url}")
    
    links = get_directory_listing(base_url, auth)
    
    for url, name in links:
        if url.endswith('/'):
            # It's a directory, recurse
            download_recursive(url, output_dir, auth, visited)
        else:
            # It's a file, download it
            parsed = urlparse(url)
            relative_path = parsed.path.lstrip('/')
            local_path = os.path.join(output_dir, relative_path)
            download_file(url, local_path)

def main():
    parser = argparse.ArgumentParser(description='Download directory recursively from web')
    parser.add_argument('url', help='URL of the directory to download')
    parser.add_argument('-o', '--output', default='./downloads', help='Output directory')
    parser.add_argument('-u', '--username', help='Username for authentication')
    parser.add_argument('-p', '--password', help='Password for authentication')
    
    args = parser.parse_args()
    
    auth = None
    if args.username and args.password:
        from requests.auth import HTTPBasicAuth
        auth = HTTPBasicAuth(args.username, args.password)
    
    os.makedirs(args.output, exist_ok=True)
    download_recursive(args.url, args.output, auth)
    print(f"\n✅ Download complete! Files saved to: {args.output}")

if __name__ == '__main__':
    main()
```

### Usage:
```bash
# Install required packages
pip3 install requests beautifulsoup4

# Run the script
python3 download_directory.py https://your-golf-club.com/files/ -o ./downloads

# With authentication
python3 download_directory.py https://your-golf-club.com/files/ -o ./downloads -u username -p password
```

---

## Method 4: Using FTP/SFTP (If you have FTP access)

**Best for:** When you have FTP/SFTP credentials

### Using lftp (Recommended):
```bash
# Install lftp
sudo apt-get install lftp

# Download recursively
lftp -c "open -u username,password ftp://your-golf-club.com; mirror -c /remote/directory /local/directory"
```

### Using sftp:
```bash
# Using sftp (interactive)
sftp username@your-golf-club.com
# Then in sftp prompt:
get -r /remote/directory /local/directory
```

### Using rsync over SSH:
```bash
rsync -avz --progress username@your-golf-club.com:/remote/directory/ /local/directory/
```

---

## Method 5: Using Browser Extension (Easiest for Non-Technical Users)

**Best for:** Quick downloads without command line

### Recommended Extensions:
1. **DownThemAll** (Firefox) - Can download all links from a page
2. **Download Master** (Chrome) - Batch download manager
3. **wget for Chrome** - Command-line style downloads

### Steps:
1. Navigate to the directory listing page
2. Right-click and select "Download all links" or use the extension
3. Select the files/directories you want
4. Start download

---

## Method 6: PHP Script (If you have web server access)

**Best for:** When you can run PHP scripts on the server

### PHP Download Script:
```php
<?php
// download_directory.php
// Run this on the server to create a zip of the directory

$directory = '/path/to/directory'; // Change this
$zipFile = 'directory_backup_' . date('Y-m-d') . '.zip';

if (!is_dir($directory)) {
    die("Directory not found: $directory\n");
}

$zip = new ZipArchive();
if ($zip->open($zipFile, ZipArchive::CREATE | ZipArchive::OVERWRITE) !== TRUE) {
    die("Cannot create zip file\n");
}

$iterator = new RecursiveIteratorIterator(
    new RecursiveDirectoryIterator($directory),
    RecursiveIteratorIterator::LEAVES_ONLY
);

foreach ($iterator as $file) {
    if (!$file->isDir()) {
        $filePath = $file->getRealPath();
        $relativePath = substr($filePath, strlen($directory) + 1);
        $zip->addFile($filePath, $relativePath);
        echo "Added: $relativePath\n";
    }
}

$zip->close();
echo "Zip file created: $zipFile\n";
echo "Size: " . filesize($zipFile) . " bytes\n";
?>
```

### Usage:
```bash
php download_directory.php
# Then download the zip file via browser or scp
```

---

## Quick Start - Choose Your Method

### If you have the URL and it's public:
```bash
wget -r -np -nH -P ./downloads https://your-golf-club.com/files/
```

### If you need authentication:
```bash
wget -r -np -nH -P ./downloads --user=username --password=password https://your-golf-club.com/files/
```

### If you have FTP access:
```bash
lftp -c "open -u username,password ftp://your-golf-club.com; mirror -c /remote/directory ./downloads"
```

### If you have SSH access:
```bash
rsync -avz --progress username@your-golf-club.com:/remote/directory/ ./downloads/
```

---

## Troubleshooting

### Issue: "403 Forbidden" or "401 Unauthorized"
**Solution:** You need authentication. Use `--user` and `--password` with wget, or provide credentials in the Python script.

### Issue: "Connection timeout"
**Solution:** 
- Check your internet connection
- Verify the URL is correct
- Try with `--timeout=60` option in wget

### Issue: "Too many files" or "Disk space"
**Solution:**
- Use `--quota` option in wget to limit download size
- Download in smaller batches
- Check available disk space: `df -h`

### Issue: "Directory listing not enabled"
**Solution:**
- You may need to use FTP/SFTP instead
- Or contact the web administrator to enable directory listing
- Or use a sitemap if available

---

## After Download - Upload to Business Workspace

Once you have the files downloaded locally, you can upload them to your business workspace:

### Using Google Drive API (if you have the setup):
```bash
# Use your existing upload scripts
php event-upload.php
```

### Using Google Drive Web Interface:
1. Go to Google Drive
2. Create a folder in your Business Workspace
3. Upload the downloaded files/folders

### Using rclone (Command line):
```bash
# Install rclone
sudo apt-get install rclone

# Configure (one time)
rclone config

# Copy files
rclone copy ./downloads/ gdrive:BusinessWorkspace/FolderName/
```

---

## Need Help?

If you need assistance with a specific method or encounter issues:
1. Check what type of access you have (HTTP, FTP, SSH)
2. Verify your credentials
3. Test with a single file first
4. Check server logs if you have access

---

## Security Notes

⚠️ **Important:**
- Never share credentials in scripts that will be committed to version control
- Use environment variables for passwords
- Consider using SSH keys instead of passwords when possible
- Verify the source before downloading files