Python – Using Selenium with Python and PhantomJS to download file to filesystem

downloadphantomjspythonseleniumselenium-webdriver

I've been grappling with using PhantomJS/Selenium/python-selenium to download a file to the filesystem.
I'm able to easily navigate through the DOM and click, hover etc. Downloading a file is, however, proving to be quite troublesome. I've tried a headless approach with Firefox and pyvirtualdisplay but that wasn't working well either and was unbelievably slow. I know That CasperJS allows for file downloads. Does anyone know how to integrate CasperJS with Python or how to utilize PhantomJS to download files. Much appreciated.

Best Answer

Despite this question is quite old, downloading files through PhantomJS is still a problem. But we can use PhantomJS to get download link and fetch all needed cookies such as csrf tokens and so on. And then we can use requests to download it actually:

import requests
from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get('page_with_download_link')
download_link = driver.find_element_by_id('download_link')
session = requests.Session()
cookies = driver.get_cookies()

for cookie in cookies: 
    session.cookies.set(cookie['name'], cookie['value'])
response = session.get(download_link)

And now in response.content actual file content should appear. We can next write it with open or do whatever we want.

Related Solutions

Python – How to copy a file in Python

shutil has many methods you can use. One of which is:

from shutil import copyfile
copyfile(src, dst)

# 2nd option
copy(src, dst)  # dst can be a folder; use copy2() to preserve timestamp

Copy the contents of the file named src to a file named dst. Both src and dst need to be the entire filename of the files, including path.
The destination location must be writable; otherwise, an IOError exception will be raised.
If dst already exists, it will be replaced.
Special files such as character or block devices and pipes cannot be copied with this function.
With copy, src and dst are path names given as strs.

Another shutil method to look at is shutil.copy2(). It's similar but preserves more metadata (e.g. time stamps).

If you use os.path operations, use copy rather than copyfile. copyfile will only accept strings.

Python – How to upgrade all Python packages with pip

There isn't a built-in flag yet, but you can use

pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U

Note: there are infinite potential variations for this. I'm trying to keep this answer short and simple, but please do suggest variations in the comments!

In older version of pip, you can use this instead:

pip freeze --local | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U

The grep is to skip editable ("-e") package definitions, as suggested by @jawache. (Yes, you could replace grep+cut with sed or awk or perl or...).

The -n1 flag for xargs prevents stopping everything if updating one package fails (thanks @andsens).

Best Answer

Related Solutions

Python – How to copy a file in Python

Python – How to upgrade all Python packages with pip

Related Topic