Ssh – Speed up mongodump+mongorestore

mongodbsocketsshssh-tunnel

I have created a script to copy a MongoDB database to my machine. I am creating an SSH tunnel (ssh -L ...) then I connect to the tunnelled port with mongodump then I pipe its output to mongorestore:

mongodump --host=127.0.0.1:##### --db=***** --archive | mongorestore --host={mongo_dest} --drop --archive

I would like to speed up the copying. --gzip doesn't make sense to be used in this case — because the same machine and memory are used by mongodump and mongorestore. The data comes uncompressed via the SSH socket.

Is there a way to run mongodump on the SSHed machine and pipe its output to a process on my machine?

Of course I could dump the database, archive, copy over SSH and restore it. But I don't want taking temporary space.

Best Answer

Don't know why I didn't think of it:

ssh *** ". /etc/profile; mongodump --host=127.0.0.1:27017 --db=**** --archive --gzip" | mongorestore --host=127.0.0.1:27017 --drop --archive --gzip

Was 1603.96 real 45.24 user 43.57 sys.

Now 209.52 real 7.25 user 6.03 sys.

Related Solutions

SSH connection times out

This is either an issue of what the sshd service is binding to on VM, or a networking issue of some sort, perhaps a firewall which is still nagging you.

To be sure, check what the sshd service is binding to. I'm assuming this won't be the problem though since it seems you mirrored the setup on a couple of machines, and one works. Probably best to check to be sure though.

Next thing is to double check that there are no firewalls blocking, or any other type of network control on the VM host software that may cause the packets to not reach destination. I would start right off the bat by running a tcpdump or whatever packet sniffer on the VM, and try to connect again via SSH, just to determine whether or not the packets are even making it that far. If they are not, my bet is the vm host software is blocking this connection.

It would help to have more detail about the network information. You went to good detail on some stuff but didn't include this. Are they all on the same subnet? what is the interconnectivity between all, etc...

I think this will help too

EDIT: So it looks like it's some sort of networking thing perhaps. I would kill the sshd service and try to run it manually, as described in This Answer from a related question. My guess is that you'll see nothing show up at all, which confirms that you are not reaching the VM - which has me thinking that either something on the network is blocking, or despite your efforts to disable any firewalling on the vm-host, it's still not getting beyond there. HTH.

Linux – Using mongodb tools (mongodump, mongorestore) from another machine

I know this is quite an old question, but googling took me there. On ubuntu 14.04 I have a following line in /etc/sources.list.d/mongo.list to have the 3.2 version there:

deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse

The source contains few packages. According to the MongoDB Manual it's like this:

mongodb-org - A metapackage that will automatically install the four component packages listed below.
mongodb-org-server - Contains the mongod daemon and associated configuration and init scripts.
mongodb-org-mongos - Contains the mongos daemon.
mongodb-org-shell - Contains the mongo shell.
mongodb-org-tools - Contains the following MongoDB tools: mongoimport bsondump, mongodump, mongoexport, mongofiles, mongooplog, mongoperf, mongorestore, mongostat, and mongotop.

So in your case the mongodb-org-tools should be enough.

Best Answer

Related Solutions

SSH connection times out

Linux – Using mongodb tools (mongodump, mongorestore) from another machine

Related Topic