A few months back I had written a small bash script to bypass the download limit (14MB) imposed by our proxy server. What it essentially does is, downloads files chunk-wise; for instance, in my case, it'd download 14MB chunks at a time and appends it to the file... all on-the-fly! It served me pretty well all these days. Moreover, I also wanted to use it as my Pacman's XferCommand to stay up-to-date.
Though the original version works, it needs a few tweaks. For bigger files, output doesn't look streamlined. Like this...
I removed all unwanted code like i) filename guessing from URLs and ii) output directory validations. And beautified/simplified it's output to look like the below screenshot. To use it with pacman, go to /etc/pacman.conf and add a XferCommand like shown in the top pane of the below screenshot. That's it, now it works well with pacman too...
Here's the new code... It's also on github
This can also be used as a standalone script to download files normally. Just that you need to mention the full filepath of where you want to download. Like this...
Though the original version works, it needs a few tweaks. For bigger files, output doesn't look streamlined. Like this...
I removed all unwanted code like i) filename guessing from URLs and ii) output directory validations. And beautified/simplified it's output to look like the below screenshot. To use it with pacman, go to /etc/pacman.conf and add a XferCommand like shown in the top pane of the below screenshot. That's it, now it works well with pacman too...
Here's the new code... It's also on github
#!/bin/bash # # Vikas Reddy @ # http://vikas-reddy.blogspot.in/2012/04/bypass-proxy-servers-file-size-download.html # # # Erase the current line in stdout erase_line() { echo -ne '\r\033[K' } # Asynchronously (as a different process) display the filesize continously # (once per second). # * Contains an infinite loop; runs as long as this script is active # * Takes one argument, the total filepath async_display_size() { PARENT_PID=$BASHPID { # Run until this script ends until [[ -z "$(ps x | grep -E "^\s*$PARENT_PID")" ]]; do # Redraw the `du` line every second erase_line echo -n "$(du -sh "$1") " sleep 1 done }& updater_pid="$!" } # Defaults fsize_limit=$((14*1024*1024)) # 14MB user_agent="Firefox/20.0" # Command-line options while getopts 'f:d:u:y' opt "$@"; do case "$opt" in f) filepath="$OPTARG" ;; u) user_agent="$OPTARG" ;; esac done shift $((OPTIND - 1)) # Exit if no URL or filepath argument is provided if [[ $# -eq 0 ]] || [[ -z "$filepath" ]]; then exit fi # Only one argument, please! url="$1" # Create/truncate the output file truncate --size 0 "$filepath" # Asynchronously (as a different process) start displaying the filesize # even before the download is started! async_display_size "$filepath" # infinite loop, until the file is fully downloaded for (( i=1; 1; i++ )); do # setting the range [ $i -eq 1 ] && start=0 || start=$(( $fsize_limit * ($i - 1) + 1)) stop=$(( $fsize_limit * i )) # downloading curl --fail \ --location \ --user-agent "$user_agent" \ --range "$start"-"$stop" \ "$url" >> "$filepath" 2> /dev/null; # No progress bars and error msgs, please! # catching the exit status exit_status="$?" if [[ $exit_status -eq 22 ]] || [[ $exit_status -eq 36 ]]; then # Download finished erase_line echo "$(du -sh "$filepath")... done!" break elif [[ $exit_status -gt 0 ]]; then # Unknown exit status! Something has gone wrong erase_line echo "Unknown exit status: $exit_status. Aborting..." break fi done
This can also be used as a standalone script to download files normally. Just that you need to mention the full filepath of where you want to download. Like this...
./pacman-curl.sh -f ~/downloads/video.mp4 'http://the-whole-url/'