Showing posts with label automation. Show all posts
Showing posts with label automation. Show all posts

Tuesday, June 26, 2012

Convert videos using ffmpeg to watch in Nokia 5230 / 5233 / 5800 / 5530 / X6 / N97

Nokia 5230/5800/.../X6 are (err... were) great smartphones. At least before the advent of the now-ubiquitous Android/Windows ones. They have got excellent, high resolution (640x360) screens which are great for watching videos on the go.

ffmpeg too has become the de facto encoding suite for converting videos, at least on linux. Here are the ffmpeg commands used to convert videos to a format which the aforementioned devices play without a hiccup. My 5230 plays MPEG4 videos encoded with lavc/xvid upto resolution 640x360. However, it'd play videos encoded with the advanced H264 codec only upto 320x240.

# ffmpeg libxvid
 ffmpeg -i Input-Filename.avi -f mp4 -y \
   -vcodec libxvid -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
   -r 25 -s 640x272 -aspect 640:360 -vf pad=640:360:0:44 \
   -threads 2 -async 1 -pass 1 /dev/null
 ffmpeg -i Input-Filename.avi -f mp4 \
   -y -vcodec libxvid -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
   -r 25 -s 640x272 -aspect 640:360 -vf pad=640:360:0:44 \
   -threads 2 -async 1 -pass 2 ./Input-Filename-ffmpeg.mp4

# ffmpeg libx264
 ffmpeg -i Input-Filename.avi -f mp4 -y \
   -vcodec libx264 -b:v 600k -vpre ipod320 -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
   -r 25 -s 320x196 -aspect 320:240 -vf pad=320:240:0:22 \
   -threads 2 -async 1 -pass 1 /dev/null
 ffmpeg -i Input-Filename.avi -f mp4 -y \
   -vcodec libx264 -b:v 600k -vpre ipod320 -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
   -r 25 -s 320x196 -aspect 320:240 -vf pad=320:240:0:22 \
   -threads 2 -async 1 -pass 2 ./Input-Filename-ffmpeg.mp4

However, for batch processing and automation, a handy Bash shell script would be great. This is a small script I use to convert my videos. (Note: I'm continually working on it. The latest code will be on my github account)
#!/bin/bash
#
#    Vikas Reddy @ http://vikas-reddy.blogspot.com/
#
# ffmpeg libxvid
# --------------
# ffmpeg -i Input-Filename.avi -f mp4 -y \
#   -vcodec libxvid -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
#   -r 25 -s 640x272 -aspect 640:360 -vf pad=640:360:0:44 \
#   -threads 2 -async 1 -pass 1 /dev/null
# ffmpeg -i Input-Filename.avi -f mp4 \
#   -y -vcodec libxvid -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
#   -r 25 -s 640x272 -aspect 640:360 -vf pad=640:360:0:44 \
#   -threads 2 -async 1 -pass 2 ./Input-Filename-ffmpeg.mp4
#
# ffmpeg libx264
# --------------
# ffmpeg -i Input-Filename.avi -f mp4 -y \
#   -vcodec libx264 -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
#   -r 25 -s 320x196 -aspect 320:240 -vf pad=320:240:0:22 \
#   -threads 2 -async 1 -pass 1 /dev/null
# ffmpeg -i Input-Filename.avi -f mp4 -y \
#   -vcodec libx264 -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
#   -r 25 -s 320x196 -aspect 320:240 -vf pad=320:240:0:22 \
#   -threads 2 -async 1 -pass 2 ./Input-Filename-ffmpeg.mp4
#  
#  Usage
#  -----
#  Command-line options: 
#  -a : Video aspect ratio. Could be either 1.66 or 2.35 (default)
#  -b : Video bitrate. Should be in the form of 600k (default)
#  -c : Video codec. Should be either libx264 or libxvid (default)
#  -d : Output directory. Current directory (.) is the default
#  -y : Whether to ask confirmation before overwriting any file.
#       Should be either "yes" (default) or "no"
#  
#  Examples
#  --------
#  1) ./ffmpeg-encode.sh The.Movie.Filename.avi 
#     would output the xvid-encoded video to The.Movie.Filename-ffmpeg.mp4 in the current directory
#  2) ./ffmpeg-encode.sh -a 1.66 -b 650k -c libx264 -d /home/vikas/downloads/ -y The.Movie.Filename.avi 
#  


# Command-line options
while getopts 'a:b:c:d:o:p:y' opt "$@"; do
    case "$opt" in
        a) video_aspect="$OPTARG" ;;
        b) vbitrate="$OPTARG" ;;
        c) video_codec="$OPTARG" ;;
        d) output_dir="$OPTARG" ;;
        o) addl_options="$OPTARG" ;;
        p) passes="$OPTARG" ;;
        y) ask_confirmation="no" ;;
    esac
done
shift $((OPTIND - 1))


# Defaults
video_aspect="${video_aspect:-2.35}"
video_codec="${video_codec:-libxvid}" # or libx264
vbitrate="${vbitrate:-600k}"
passes="${passes:-2}"
output_dir="${output_dir:-.}"


vpre_pass1=""
vpre_pass2=""

if [[ "$video_codec" == "libx264" ]]; then
    #vpre_pass1="-vpre fastfirstpass -vpre baseline"
    #vpre_pass2="-vpre hq -vpre baseline"
    aspect="320:240"

    if [[ "$video_aspect" == "2.35" ]]; then
        resolution="320x196"
        pad="pad=320:240:0:22"
    elif [[ "$video_aspect" == "1.66" ]]; then
        resolution="320x240"
        pad="pad=320:240:0:0"
    fi;

elif [[ "$video_codec" == "libxvid" ]]; then
    aspect="640:360"

    if [[ "$video_aspect" == "2.35" ]]; then
        resolution="640x272"
        pad="pad=640:360:0:44"
    elif [[ "$video_aspect" == "1.66" ]]; then
        resolution="640x360"
        pad="pad=640:360:0:0"
    fi;
fi;


echo "Encoding '${#@}' video(s)";

for in_file in "$@"; do

    # If the filename has no extension
    if [[ -z "$(echo "$in_file" | grep -Ei "\.[a-z]+$")" ]]; then
        fname="$(basename "${in_file}")-ffmpeg.mp4"
    else
        fname="$(basename "$in_file" | sed -sr 's/^(.*)(\.[^.]+)$/\1-ffmpeg.mp4/')"
    fi
    out_file="${output_dir%/}/${fname}"

    # Avoid overwriting files
    if [[ "$ask_confirmation" != "no" ]] && [[ -f "$out_file" ]]; then
        echo -n "'$out_file' already exists. Do you want to overwrite it? [y/n] "; read response
        [[ -z "$(echo "$response" | grep -i "^y")" ]] && continue
    fi

    # 1st pass
    ffmpeg -i "$in_file" \
           -f mp4 -y $addl_options \
           -vcodec "$video_codec" -b:v "$vbitrate" \
           -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
           -r 25 -s "$resolution" -aspect "$aspect" -vf "$pad" \
           -threads 2 -async 1 -pass 1  \
           "/dev/null"; # $out_file;

    # 2nd pass
    ffmpeg -i "$in_file" \
           -f mp4 -y $addl_options \
           -vcodec "$video_codec" -b:v "$vbitrate" \
           -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
           -r 25 -s "$resolution" -aspect "$aspect" -vf "$pad" \
           -threads 2 -async 1 -pass 2  \
           "$out_file";
done

Its usage is simple too. Without having to remember, edit and type in the lengthy ffmpeg command, this one makes my life a lot easier.
To start with, without any command-line options, the given file is assumed to be of 2.35 (cinema scope) aspect ratio, and consequently encoded using libxvid library to produce a nice 640x360 mp4 video. See below...

A small documentation with available command-line options and a few examples is bundled in the script itself.

NOTE: Because of licensing issues, ffmpeg binaries that are available on the repositories of most of the linux distros are not compiled with "non-free" codec support. This is especially true in the case of libx264 and libfaac. You may have to abandon them and compile the software from sources. Google it!

Do post your views in the comments section below...


Monday, April 30, 2012

Bypass proxy server's file size download limit restriction


   Many organizations and colleges restrict their employees and students respectively from downloading files from the Internet which are larger than a prescribed limit. It is way too low at 14MB where I work. Fret not! There are ways to bypass this. And here is a simple bash script I wrote to download much larger files at my workplace.

Note: This script works only with direct links and with servers which support resume-download functionality.

  I'm continually working on it. So, the latest version will be available on my github account.

How to run it?
  1. Download the following code to a text file named curldownload.sh
  2. Give executable permissions to it chmod +x curldownload.sh
  3. File size limit, fsize_limit variable, is set to 14MB. You may change it to your liking.
  4. The script takes two arguments; the first one being the url of the file to be downloaded; the second one which is optional (defaults to "./") is the output directory.
  5. For ex:- ./curldownload.sh http://ftp.jaist.ac.jp/pub/mozilla.org/firefox/releases/12.0/linux-i686/en-US/firefox-12.0.tar.bz2 "$HOME/Downloads"
  6. A little more complex example of it using multiple urls, and two command-line arguments (-d for output directory, and -u for user-agent http header) is:  ./curl-multi-url.sh -d ~/downloads/ -u "Chromium/18.0" http://ftp.jaist.ac.jp/pub/mozilla.org/firefox/releases/11.0/linux-x86_64/en-US/firefox-11.0.tar.bz2 http://ftp.jaist.ac.jp/pub/mozilla.org/firefox/releases/12.0/linux-i686/en-US/firefox-12.0.tar.bz2
#!/bin/bash
#
# Vikas Reddy @
#   http://vikas-reddy.blogspot.in/2012/04/bypass-proxy-servers-file-size-download.html
#
# 
# Usage:
#     ./curl-multi-url.sh -d OUTPUT_DIRECTORY -u USER_AGENT http://url-1/ http://url-2/;
#     Arguments -d and -u are optional
#
#

# Defaults
fsize_limit=$((14*1024*1024))
user_agent="Firefox/10.0"
output_dir="."


# Command-line options
while getopts 'd:u:' opt "$@"; do
    case "$opt" in
        d) output_dir="$OPTARG";;
        u) user_agent="$OPTARG";;
    esac
done
shift $((OPTIND - 1))


# output directory check
if [ -d "$output_dir" ]; then
    echo "Downloading all files to '$output_dir'"
else
    echo "Target directory '$output_dir' doesn't exist. Aborting..."
    exit 1
fi;


for url in "$@"; do
    filename="$(echo "$url" | sed -r 's|^.*/([^/]+)$|\1|')"
    filepath="$output_dir/$filename"

    # Avoid overwriting the file
    if [[ -f "$filepath" ]]; then
        echo -n "'$filepath' already exists. Do you want to overwrite it? [y/n] "; read response
        [ -z "$(echo "$response" | grep -i "^y")" ] && continue
    else
        cat /dev/null > "$filepath"
    fi

    echo -e "\nDownload of $url started..."
    i=1
    while true; do   # infinite loop, until the file is fully downloaded

        # setting the range
        [ $i -eq 1 ] && start=0 || start=$(( $fsize_limit * ($i - 1) + 1))
        stop=$(( $fsize_limit * i ))

        # downloading
        curl --fail --location --user-agent "$user_agent" --range "$start"-"$stop" "$url" >> "$filepath"

        exit_status="$?"

        # download finished
        [ $exit_status -eq 22 ] && echo -e "Saved $filepath\n" && break

        # other exceptions
        [ $exit_status -gt 0 ] && echo -e "Unknown exit status: $exit_status. Aborting...\n" && break

        i=$(($i + 1))
    done
]]>

Friday, January 13, 2012

A simple command-line Megaupload download manager

Have a bunch of megaupload links and ever felt the need for a simple, lightweight, text-based/command-line and eminently extensible script which does all the downloading for you?

Your search ends here!

Requirements

  1. wget: which is present by default in almost linux distros
  2. ruby and nokogiri: These can be installed as shown here.
Steps
  1. Save the script as megaupload_dm.rb
  2. Make it executable using chmod +x megaupload_dm.rb
  3. Install the dependencies
  4. Dump all your megaupload links to a file named megaupload_list.txt in the current directory. You can also insert any content or comments in this file
  5. Tweak the LinkFile and OutputDirectory variables to your liking.
  6. Limit the download rate by adding --limit-rate=100k to the wget command at line #43 (optional)
  7. Run it using ./megaupload_dm.rb
  8. Your files are downloaded to ~/Downloads directory!


#!/usr/bin/env ruby


# Is the url a valid one?
def valid?(url)
  url =~ %r|^http://www.megaupload.com/\?d=[a-zA-Z0-9]+$|
end

require 'rubygems'
require 'nokogiri'
require 'open-uri'

LinkFile = "./megaupload_list.txt"
OutputDirectory = "/home/vikas/Downloads"
TempFile = "/tmp/megaupload-tmp.html"


File.open(LinkFile, 'r') do |f|
  f.each_line do |link|

    link.strip!

    next unless valid?(link)

    begin

      puts "Fetching information for #{link} ..."
      `wget "#{link}" -U "Firefox/4.0.0" --quiet --output-document #{TempFile}`

      doc = Nokogiri::HTML(File.read(TempFile))

      # Filename and Direct Link
      fname = doc.css("div.downl_main div.download_file_name").first.content
      direct_link = doc.css("div.downl_main div.download_member_bl a").last.attributes['href'].value

      # Actual download
      puts "Downloading #{fname} ... \n"
      `wget -c -P "#{OutputDirectory}" -U "Firefox/3.6.3" "#{direct_link}"`
      puts "*********************************** DONE *******************************************"
      puts "\n\n"

      # Sleep 5 secs. Take some load off megaupload ;)
      sleep 5

    rescue
      puts "ERROR! FileNotFound or otherwise; Skipping this file...."
      puts "\n\n"
      next
    end

  end
end

Tuesday, January 3, 2012

A script to download wallpapers from wallpaperswide.com


Going through wallpapers websites like http://www.wallpaperswide.com/ and ever wondered how cool would it be if you have a magic wand with which you could download all the wallpapers without having to manually download the ten images in all of the 2500-odd pages?

Your search ends here!

Dependencies
  1. This script is written in Ruby. Most Linux distros ship a basic version of this scripting language. If not, install it along with rubygems using the following commands.
  2. This script needs wget utility to download. wget is shipped with almost all linux distros. If not, install it using your distro's package manager.
# If you don't use a Debian-based distro like Ubuntu or Linux Mint, replace 'apt-get' with your distro's package manager

# Install 'ruby' and 'rubygems' if not already installed
sudo apt-get install ruby rubygems

# Install 'wget' if not installed
sudo apt-get install wget

# Install 'nokogiri' gem
gem install --no-r{doc,i} nokogiri
  1. Save the following ruby code into a file named wallpaperswide_dot_com.rb
  2. Choose the resolution of the wallpapers you want to download. The list of resolutions supported are given in the beginning of the script. Tweak Resolution variable accordingly.
  3. Adjust the Output_Directory variable.
  4. Run the script by ruby wallpaperswide_dot_com.rb
  5. It'll be a 2+GB download depending on the resolution. If you desire to limit your download speed to say 100kbps, you could add an additional argument to the wget command (the third line from the end) like wget -c --limit-rate=100k -U "Firefox/4.5.6" ...;
#!/usr/bin/env ruby
#
#  Vikas Reddy
#
#  A little script to download ALL the wallpapers of a given
#  resolution from http://www.wallpaperswide.com/
#
#  Requirements
#  ============
#  Ruby Version: 1.9.2
#  Gems: nokogiri, open-uri
#  Other programs: wget
#
#
#  Available Resolutions
#  =====================
#
#  Wide
#  
#  * 16:10 960x600
#  * 16:10 1152x720
#  * 16:10 1280x800
#  * 16:10 1440x900
#  * 16:10 1680x1050
#  * 16:10 1920x1200
#  * 16:10 2560x1600
#  * 16:10 3840x2400
#  * 16:10 5120x3200
#  * 5:3 800x480
#  * 5:3 1280x768
#  
#  HD
#  
#  * 16:9 960x540
#  * 16:9 1024x576
#  * 16:9 1280x720
#  * 16:9 1366x768
#  * 16:9 1600x900
#  * 16:9 1920x1080
#  * 16:9 2048x1152
#  * 16:9 2400x1350
#  * 16:9 2560x1440
#  * 16:9 3554x1999
#  * 16:9 3840x2160
#  
#  Standard
#  
#  * 4:3 800x600
#  * 4:3 1024x768
#  * 4:3 1152x864
#  * 4:3 1280x960
#  * 4:3 1400x1050
#  * 4:3 1440x1080
#  * 4:3 1600x1200
#  * 4:3 1680x1260
#  * 4:3 1920x1440
#  * 4:3 2048x1536
#  * 4:3 2560x1920
#  * 4:3 2800x2100
#  * 4:3 3200x2400
#  * 4:3 4096x3072
#  * 5:4 1280x1024
#  * 5:4 2560x2048
#  * 5:4 3750x3000
#  
#  Mobile Ratio
#  
#  * VGA 240x320
#  * VGA 480x640
#  * VGA 320x240
#  * VGA 640x480
#  * WVGA 240x400
#  * WVGA 480x800
#  * WVGA 400x240
#  * WVGA 800x480
#  * HVGA 320x480
#  * HVGA 480x320
#  * HVGA 640x960
#  * HVGA 960x640
#  * iPad 1024x768
#  * iPad 768x1024
#  * HD 16:9 480x272
#  * HD 16:9 272x480
#  * Phone 176x220
#  * Phone 220x176
#  
#  Dual
#  
#  * 4:3 1600x600
#  * 4:3 2048x768
#  * 4:3 2304x864
#  * 4:3 2560x960
#  * 4:3 2800x1050
#  * 4:3 2880x1080
#  * 4:3 3200x1200
#  * 4:3 3360x1260
#  * 4:3 3840x1440
#  * 4:3 4096x1536
#  * 4:3 5120x1920
#  * 4:3 5600x2100
#  * 4:3 6400x2400
#  * 4:3 8192x3072
#  * 5:4 2560x1024
#  * 5:4 5120x2048
#  * 5:4 7500x3000
#  * 5:4 10240x4096
#  * 16:10 1920x600
#  * 16:10 2304x720
#  * 16:10 2560x800
#  * 16:10 2880x900
#  * 16:10 3360x1050
#  * 16:10 3840x1200
#  * 16:10 5120x1600
#  * 16:10 7680x2400
#  * 16:10 10240x3200
#  * 5:3 1600x480
#  * 5:3 2560x768
#  * 16:9 1920x540
#  * 16:9 2048x576
#  * 16:9 2560x720
#  * 16:9 3200x900
#  * 16:9 3840x1080
#  * 16:9 4096x1152
#  * 16:9 4800x1350
#  * 16:9 5120x1440
#  * 16:9 7108x2000
#  * 16:9 7680x2160
#  * 3:2 2880x960
#  * 3:2 4000x1333
#  * 3:2 2304x768
#  
#  Other
#  
#  * 3:2 1152x768
#  * 3:2 1440x960
#  * 3:2 2000x1333


require 'open-uri'
require 'nokogiri'

Resolution = "1600x900"
Base_URL   = "http://wallpaperswide.com/#{Resolution}-wallpapers-r/page/"
Output_Directory = "/home/vikas/Wallpapers/"

# Create the Output_Directory if needed
`mkdir -p "#{Output_Directory}"`

(1..2492).each do |page_num|

  # Go page by page
  url = Base_URL + page_num.to_s

  # Parse html
  f = open(url)
  doc = Nokogiri::HTML(f)

  # Loop over image-boxes
  doc.css("div.thumb").each do |wallp|

    # Extract wallpaper subpage url
    wallp.css("div[onclick]").attr("onclick").value =~ /prevframe_show\('(.*)'\)/
    subpage_url = $1
    subpage_url =~ %r|http://wallpaperswide\.com/[^/]+/([\w\d]+)\.html|

    # Generate url of the required wallpaper
    wallp_url = %|http://wallpaperswide.com/download/#{$1}-#{Resolution}.jpg|
   
    # Download... with a user-agent parameter just in case...
    # use '--limit-rate=100k' to limit download speed
    `wget -c -U "Firefox/4.5.6" -P "#{Output_Directory}" "#{wallp_url}"`
  end
end

Monday, January 2, 2012

A script to compile a latex-beamer presentation into all themes

I always wondered how great and handy would it be had I had a nifty script to compile my latex presentation using all the themes present in my current tex installation so that I can have a look at all the output files and choose the one which looks best.

And, this is the little bash shell script I've written to fulfill my need.
  1. Save this script into your Latex project directory
  2. Adjust the variables listed in the script to your liking ($themes_path and $output_directory)
  3. Give executable permission using "chmod +x $filename" or otherwise
  4. Execute it! It takes your ".tex" as the command-line argument and outputs the pdfs into a directory named "change_themes". There will be as many themes as your Beamer installation could possibly output.
PS: It'll not disturb/write to your original .tex file though. It just reads it.

#!/bin/bash

themes_path="/usr/share/texmf/tex/latex/beamer/base/themes/theme"
filename="$1"
output_directory="./change_themes"

# Needs the .tex file as the command-line argument
if [[ -z "$filename" ]]; then
    echo "Needs a command-line argument";
    exit;
fi

# Create the output directory if needed
if ! [[ -d "$output_directory" ]]; then
    mkdir -p "$output_directory";
fi

for theme in $themes_path/*.sty; do
    theme_name="$(echo "$(basename "$theme")" | sed -re 's/beamertheme(.*)\.sty/\1/')"
    sed -r "
s/usetheme\{(.*)\}/usetheme{$theme_name}/" "$filename" > "$output_directory/$filename"
    pdflatex -output-directory="
$output_directory" "$output_directory/$filename"
    pdf_fname="
$(basename "$filename" ".tex").pdf"
    pdf_fname_new="
${theme_name}_${pdf_fname}"
    mv "
$output_directory/$pdf_fname" "$output_directory/$pdf_fname_new"
done