How to Rsync files and media to your web server
FTP has always been the go to tool for developers looking to push changes to a server. Today we’re going to learn a better, faster and more reliable way to deploy changes using the command line and Rsync.
What is rsync?
Rsync is an abbreviation of “remote sync”. It’s a command line utility that’s used to transfer (or synchronise) files between two directories. For web developers it’s a great tool to push changes to a remote server as only files that have changed will be transferred and there’s no need to manually keep track of changed files to upload.
Sound good? Great, but before we move on there’s a few pre requisites:
Your web host must be Linux based, allow SSH connections and have Rsync enabled
Rsync is a Unix / Linux based tool, as such you need to ensure your hosting environment is not Windows based. You’ll also have to check your host allows SSH connections and Rsync is installed. If you’re unsure a quick call or email to your hosts support team should help.
You’ve setup and uploaded SSH keys to your server
To simplify the process (and avoid password prompts) it’s best to setup SSH keys before you start. You can learn how to create and add SSH keys to your server in this blog post.
Your local machine is running Linux or Mac OS
As mentioned Rsync isn’t natively available to Windows users, there are workarounds including the Windows Unix subsystem. A Mac or Linux based OS gives the best results as Rsync is built into the OS and available to use from the command line.
Finally, don’t sue me
Working in the command line and transferring files via Rsync should only be attempted by those who are confident and fully understand the commands they are executing. A lot can wrong with the smallest of typos! Attempt what you read in this post at your own risk, I accept no responsibility should anything go wrong.
Okay, so we’re ready to start using Rsync, let’s start by going over the basic syntax. A simple Rsync command looks like the following:
rsync -flags source/ destination
Let’s break this down, as you may have guessed
rsync is calling our Rsync utility. Next we pass our
flags, these are options that change the behaviour of the utility. Finally we indicate what we want to sync, this is always written as
destination. That means we will synchronise (or copy) all the contents from our source directory to our destination directory.
An important note:
See the use of the trailing slash
source? This slash means copy the contents of
destination, without this slash the source folder and its contents would be placed inside the destination directory.
Useful Rsync flags
Now you have an understanding of the basic Rsync commands, let’s explore some useful options (flags) available. Flags are case sensitive, so ensure you enter them correctly!
rsync --verbose source destination
-v flag stands for verbose. By default an Rsync transfer will operate silently, however most of the time it’s handy to have some form of feedback of what’s transferred. Verbose will print information on what files have been sent to your destination and include a brief summary at the end of the process. You can establish what files have been skipped by including two
rsync --progress source destination
Fairly self explanatory this one, but handy for those that need it. The
--progress flag will output the progress of each file transfer, it’s best used with the verbose flag
-v to prevent any strange output to your console.
rsync --partial source destination
By default if a connection is terminated mid transfer the partially uploaded file with be deleted and re-uploaded on the next attempt. By using the
--partial flag the part transferred file will be kept and an attempt to resume the transfer will be made on the next connection.
-P flag can be passed, which is equivalent to using both
rsync --dry-run source destination
For beginners and seasoned users the option to perform a dry run on new transfers is highly recommended. Using the flag
-n the results of the transfer will be output to your console without actually moving or deleting any files on your destination.
rsync --exclude 'pattern_file_or_directory_to_exclude' source destination
The default behaviour is to process all files. If you want to exclude a certain file or files matching a pattern you can do so using the
--exclude flag, excluded files will be skipped when Rsync is called.
You can exclude multiple files, directories or patterns by repeating the
--exclude flag, but this can get a little messy. A more elegant solution is to pass a file containing a list of files to exclude using the
Finally, the alternative to
--exclude is the
--include flag. This allows you to include files in the transfer, which is handy for overriding an exclusion pattern.
rsync --compress source destination
-z flag set, all data transmitted from source to destination will be compressed. This option is good for large files and/or slower connections.
rsync --delete source destination
By default, if a file is removed from the source the file will not be removed from the destination. This can cause your destination directory to become cluttered with unnecessary files, not ideal. In order to truly synchronise two directories we have to also ensure removed files are deleted from the destination, to achieve this we use the
--delete flag. It is always advisable to perform a
--dry-run prior to using delete to test what files and / or directories would be removed.
How to Rsync to your web server
Syncing directories and files to a remote server is simple if you have both SSH access and SSH keys setup. Once you have confirmed a successful SSH connection we can run our command to transfer our files with the following command.
rsync -avz --delete --dry-run source/ [email protected]_host:destination_directory
This command will push our source to our remote destination, we start by calling the operation with
rsync and pass in our flags, in this case
-avz. Let’s break these flags down:
-ais for archive, which is a shorthand for
-venables verbose mode
-zturns on compression
--deleteremoves files on the destination not present on the source
--dry-runallows us to check the output prior to committing to a live run
Finally we define our source and destination directories, replace
source with your source directory,
[email protected]_host: with your remote host details and
destination_directory with your chosen destination directory.
Running the process via a shell script
So you’re now up to speed with how to deploy changes from your local machine to your web server using Rsync, great. However, you’ll soon realise that typing the same long command out time after time gets a little tedious. We can automate the process by saving the command to a script and calling this script via the command line. Let’s do that now.
Start by opening a new terminal window, we’re going to kick things off by creating a new shell script. Enter the following into terminal and hit return.
The above command
touch will create a new file called rsync_deploy.sh and save it to your working directory. I recommend that this script is saved in the root of the working directory for your project. We now need to make this file executable so we can run it from the command line, enter the following into terminal and hit return.
sudo chmod +x rsync_deploy.sh
Here we’re using the
chmod which stands for “change mode”, it can be used to change permissions, modes and owners of files and directories. In this example we are passing the
+x flag, this will make the file executable. We finally have to pass the file we want to change, which in this case is
Now to add he Rsync command to our script. Open
rsync_deploy.sh in your text editor and enter the following and save the file.
#!/bin/bash rsync -avz --delete --dry-run source/ [email protected]_host:destination_directory
Great, we now have a shell script that can be executed from the command line. On doing so it will run our Rsync command without us having to enter the full command each time we want to deploy changes. To run the script (assuming your in the same working directory as the script is saved), enter the following into the command line. That’s a lot better isn’t it?
If all goes to plan you should get the same verbose feedback on the status of the Rsync process, just as if you had called the command manually.
Hopefully by now you have managed to make a few deployments over SSH using Rsync and can see the benefits this has over a traditional FTP upload. The technology has so many uses, it’s not just limited to web developers, it can be used to sync files between computers on a local or external network and even as backup solution. Just remember, it’s a powerful tool and one small typo can cause a lot of problems! Always use
--dry-run when establishing new connections and ensure you have a backup of both your local and remote destinations before you deploy.
That’s all folks, thanks for reading.