A few months back I had written a small bash script to bypass the download limit (14MB) imposed by our proxy server. What it essentially does is, downloads files chunk-wise; for instance, in my case, it'd download 14MB chunks at a time and appends it to the file... all on-the-fly! It served me pretty well all these days. Moreover, I also wanted to use it as my Pacman's XferCommand to stay up-to-date.
Though the original version works, it needs a few tweaks. For bigger files, output doesn't look streamlined. Like this...
I removed all unwanted code like i) filename guessing from URLs and ii) output directory validations. And beautified/simplified it's output to look like the below screenshot. To use it with pacman, go to /etc/pacman.conf and add a XferCommand like shown in the top pane of the below screenshot. That's it, now it works well with pacman too...
Here's the new code... It's also on github
This can also be used as a standalone script to download files normally. Just that you need to mention the full filepath of where you want to download. Like this...
Though the original version works, it needs a few tweaks. For bigger files, output doesn't look streamlined. Like this...
I removed all unwanted code like i) filename guessing from URLs and ii) output directory validations. And beautified/simplified it's output to look like the below screenshot. To use it with pacman, go to /etc/pacman.conf and add a XferCommand like shown in the top pane of the below screenshot. That's it, now it works well with pacman too...
Here's the new code... It's also on github
#!/bin/bash
#
# Vikas Reddy @
# http://vikas-reddy.blogspot.in/2012/04/bypass-proxy-servers-file-size-download.html
#
#
# Erase the current line in stdout
erase_line() {
echo -ne '\r\033[K'
}
# Asynchronously (as a different process) display the filesize continously
# (once per second).
# * Contains an infinite loop; runs as long as this script is active
# * Takes one argument, the total filepath
async_display_size() {
PARENT_PID=$BASHPID
{
# Run until this script ends
until [[ -z "$(ps x | grep -E "^\s*$PARENT_PID")" ]]; do
# Redraw the `du` line every second
erase_line
echo -n "$(du -sh "$1") "
sleep 1
done
}&
updater_pid="$!"
}
# Defaults
fsize_limit=$((14*1024*1024)) # 14MB
user_agent="Firefox/20.0"
# Command-line options
while getopts 'f:d:u:y' opt "$@"; do
case "$opt" in
f) filepath="$OPTARG" ;;
u) user_agent="$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
# Exit if no URL or filepath argument is provided
if [[ $# -eq 0 ]] || [[ -z "$filepath" ]]; then
exit
fi
# Only one argument, please!
url="$1"
# Create/truncate the output file
truncate --size 0 "$filepath"
# Asynchronously (as a different process) start displaying the filesize
# even before the download is started!
async_display_size "$filepath"
# infinite loop, until the file is fully downloaded
for (( i=1; 1; i++ )); do
# setting the range
[ $i -eq 1 ] && start=0 || start=$(( $fsize_limit * ($i - 1) + 1))
stop=$(( $fsize_limit * i ))
# downloading
curl --fail \
--location \
--user-agent "$user_agent" \
--range "$start"-"$stop" \
"$url" >> "$filepath" 2> /dev/null; # No progress bars and error msgs, please!
# catching the exit status
exit_status="$?"
if [[ $exit_status -eq 22 ]] || [[ $exit_status -eq 36 ]]; then
# Download finished
erase_line
echo "$(du -sh "$filepath")... done!"
break
elif [[ $exit_status -gt 0 ]]; then
# Unknown exit status! Something has gone wrong
erase_line
echo "Unknown exit status: $exit_status. Aborting..."
break
fi
done
This can also be used as a standalone script to download files normally. Just that you need to mention the full filepath of where you want to download. Like this...
./pacman-curl.sh -f ~/downloads/video.mp4 'http://the-whole-url/'
