From a local folder

Method 1: Using aws-cli with cron

This method involves setting up a cron job to periodically run the aws s3 sync command to synchronize the local directory with the S3 bucket.

1. Install AWS CLI: Make sure you have the AWS CLI installed and configured with your AWS credentials.

   pip install awscli
   aws configure
   

2. Set Up a Cron Job: Create a cron job to run the aws s3 sync command at regular intervals. Edit your cron jobs by running:

   crontab -e
   

   Add the following line to sync every 5 minutes (adjust the timing as needed):

   */5 * * * * aws s3 sync /path/to/local/dir s3://your-bucket-name
   

Method 2: Using inotify-tools for Real-Time Sync

This method uses inotify-tools to monitor file changes in real-time and aws-cli to sync the changes to S3.

1. Install inotify-tools: Install inotify-tools to monitor filesystem events.

   sudo apt-get install inotify-tools
   

2. Create a Script to Sync Changes: Create a shell script that uses inotifywait to detect changes and sync them to S3.

   #!/bin/bash
   
   LOCAL_DIR="/path/to/local/dir"
   S3_BUCKET="s3://your-bucket-name"
   
   inotifywait -m -r -e modify,create,delete,move $LOCAL_DIR |
   while read path action file; do
       echo "The file '$file' at path '$path' was $action"
       aws s3 sync $LOCAL_DIR $S3_BUCKET
   done
   

3. Make the Script Executable: Make your script executable.

   chmod +x /path/to/your/script.sh
   

4. Run the Script: Run the script in the background.

   nohup /path/to/your/script.sh &
   

Example Script to Continuously Sync Changes

#!/bin/bash

LOCAL_DIR="/path/to/local/dir"
S3_BUCKET="s3://your-bucket-name"

# Log file for recording sync operations
LOG_FILE="/path/to/your/logfile.log"

# Monitor the directory for changes
inotifywait -m -r -e modify,create,delete,move $LOCAL_DIR --format '%w%f %e' |
while read file event; do
    echo "$(date) - File $file was $event" >> $LOG_FILE
    aws s3 sync $LOCAL_DIR $S3_BUCKET >> $LOG_FILE 2>&1
done

Conclusion

By using either aws-cli with cron or inotify-tools for real-time synchronization, you can keep your local directory and S3 bucket in sync continuously. Choose the method that best suits your needs based on the required synchronization frequency and system resources.