A few months back I had written a small
bash script to bypass the download limit (14MB) imposed by our proxy server. What it essentially does is, downloads files chunk-wise; for instance, in my case, it'd download 14MB chunks at a time and appends it to the file... all on-the-fly! It served me pretty well all these days. Moreover, I also wanted to use it as my
Pacman's XferCommand to stay up-to-date.
Though the original version works, it needs a few tweaks. For bigger files, output doesn't look streamlined. Like this...
I removed all unwanted code like i) filename guessing from URLs and ii) output directory validations. And beautified/simplified it's output to look like the below screenshot. To use it with pacman, go to /etc/pacman.conf and add a XferCommand like shown in the top pane of the below screenshot. That's it, now it works well with pacman too...
Here's the new code... It's also on
github
#!/bin/bash
#
# Vikas Reddy @
# http://vikas-reddy.blogspot.in/2012/04/bypass-proxy-servers-file-size-download.html
#
#
# Erase the current line in stdout
erase_line() {
echo -ne '\r\033[K'
}
# Asynchronously (as a different process) display the filesize continously
# (once per second).
# * Contains an infinite loop; runs as long as this script is active
# * Takes one argument, the total filepath
async_display_size() {
PARENT_PID=$BASHPID
{
# Run until this script ends
until [[ -z "$(ps x | grep -E "^\s*$PARENT_PID")" ]]; do
# Redraw the `du` line every second
erase_line
echo -n "$(du -sh "$1") "
sleep 1
done
}&
updater_pid="$!"
}
# Defaults
fsize_limit=$((14*1024*1024)) # 14MB
user_agent="Firefox/20.0"
# Command-line options
while getopts 'f:d:u:y' opt "$@"; do
case "$opt" in
f) filepath="$OPTARG" ;;
u) user_agent="$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
# Exit if no URL or filepath argument is provided
if [[ $# -eq 0 ]] || [[ -z "$filepath" ]]; then
exit
fi
# Only one argument, please!
url="$1"
# Create/truncate the output file
truncate --size 0 "$filepath"
# Asynchronously (as a different process) start displaying the filesize
# even before the download is started!
async_display_size "$filepath"
# infinite loop, until the file is fully downloaded
for (( i=1; 1; i++ )); do
# setting the range
[ $i -eq 1 ] && start=0 || start=$(( $fsize_limit * ($i - 1) + 1))
stop=$(( $fsize_limit * i ))
# downloading
curl --fail \
--location \
--user-agent "$user_agent" \
--range "$start"-"$stop" \
"$url" >> "$filepath" 2> /dev/null; # No progress bars and error msgs, please!
# catching the exit status
exit_status="$?"
if [[ $exit_status -eq 22 ]] || [[ $exit_status -eq 36 ]]; then
# Download finished
erase_line
echo "$(du -sh "$filepath")... done!"
break
elif [[ $exit_status -gt 0 ]]; then
# Unknown exit status! Something has gone wrong
erase_line
echo "Unknown exit status: $exit_status. Aborting..."
break
fi
done
This can also be used as a standalone script to download files normally. Just that you need to mention the full filepath of where you want to download. Like this...
./pacman-curl.sh -f ~/downloads/video.mp4 'http://the-whole-url/'
