I am interested in scheduling the copying and sending of large files over the internet from one server to another. The servers are not on the same network.
I do not know about all of my available options for solving this issue, but a couple protocols/specifications with which I am vaguely familiar which may be useful for this task are (1) FTP and (2) bittorrent.
I have used Filezilla before for manually sending large files between groups of servers. There are no scheduling capabilities of that particular FTP application, so it will not suffice for meeting my scheduling requirement. Ideally, I would like to accomplish this task using something more programmatic (non-GUI-based), and not proprietary (like a for-pay application).
I am already using Python scripts which run on a scheduled basis which perform other operations on a group of servers. The servers are running Windows and the scripts are automated via the Windows Task Scheduler. If I could send files via FTP using Python, that would be ideal for me specifically because I could just throw the FTP code into what I have already got working and scheduled. However, I briefly tried toying with Python's “ftplib” and could not make heads or tails of it. So, before I attempt to go down that path, I wanted to post here first.
I am uncertain as to whether I should pursue a FTP-based approach, so I would like some recommendations on that. Also, I do not really care what programming language I use for this task, although I would slightly prefer Python, as it seems like a natural choice since I am dealing with server-side scripting. I would like recommendations on that too.
As this is my first serverfault post, you can probably imagine I am mostly clueless about many of the topics mentioned above. Thanks in advance.
Best Answer
I would start by saying that you would want to avoid FTP and Bit Torrent. Bit Torrent is a great technology when many people have the file that you'd like and the download can be distributed amongst all of them (fast download of a linux ISO for instance). I would not recommend it for a simple site A to site B transfer.
FTP is largely regarded as out-dated because it's insecure. All of the information is transmitted in plain text and easily intercepted by malicious parties who may be listening to your traffic. You will want to do complete this transfer over a secure channel.
For that, you have a couple of options to encrypt your transmission.
I would recommend the following
That is probably the easiest way I could recommend. Your other option would be to establish a VPN with the remote machine and simply write a one line script that would copy the file from your local server to the remote server.
Welcome to SF.