Archlinux: Compile ffmpeg with nonfree codecs

The ffmpeg binaries that are present in the official repository of Archlinux are not compiled with nonfree codecs. Some excellent ones like libfaac, libx264 and lib-nonfree are not included by default. However, with AUR and its package building automation tool yaourt, one can easily accomplish this. Here are the steps...

1. Install yaourt using pacman -S yaourt, if you haven't already.
2. Key in yaourt -Sb ffmpeg, and answer 'Yes' when asked for confirmation. Remember, you'd have to edit PKGBUILD when asked for. Add --enable-nonfree and --enable-libfaac and other needed configuration options to the build() function, exit the editor, and proceed.
3. That's it, get yourself a cup of tea while the compilation process to finishes, and you're good to go.

PKGBUILD in vim

For a list of available configuration options, see http://git.videolan.org/?p=ffmpeg.git;a=blob;f=configure;h=f30998b37c80ff83a81f4949ea8ab0cfb8043376;hb=HEAD

Wallpaperswide download script in Windows

A few of the visitors of my blog found it difficult to run my wallpapers download script (http://vikas-reddy.blogspot.in/2012/01/script-to-download-wallpapers-from.html) in their Windows OSes. So, here's what I did to run it in my Win XP Pro SP2 (32-bit).

1. Install Ruby 1.9.3-p194 from  http://rubyinstaller.org/downloads/. Normally the defaults are fine. I installed it to my "C:\" folder
2. Install nokogiri gem using "C:\Ruby193\bin\gem" install nokogiri
3. Install wget from  http://gnuwin32.sourceforge.net/packages/wget.htm. Keep the defaults, or else, change the "wget" line in the script to reflect the correct executable path.
4. Download the following code into a file named wallpaperswide-script.rb, and adjust Resolution and OutputDirectory variables. It's 1600x900 and "C:\Wallpapers" respectively by default. Check out the original post for available resolutions.
5. Run it using "C:\Ruby193\bin\ruby.exe" "C:\wallpaperswide-script.rb"
6. That's it, all your wallpapers will get downloaded into the given folder.

require 'open-uri'
require 'nokogiri'

Resolution = "1600x900"
Base_URL   = "http://wallpaperswide.com/#{Resolution}-wallpapers-r/page/"
Output_Directory = "C:\\Wallpapers"


(1..2805).each do |page_num|

  # Go page by page
  url = Base_URL + page_num.to_s

  # Parse html
  f = open(url)
  doc = Nokogiri::HTML(f)

  # Loop over image-boxes
  doc.css("div.thumb").each do |wallp|

    # Extract wallpaper subpage url
    wallp.css("div[onclick]").attr("onclick").value =~ /prevframe_show\('(.*)'\)/
    subpage_url = $1
    subpage_url =~ %r|http://wallpaperswide\.com/[^/]+/([\w\d]+)\.html|

    # Generate url of the required wallpaper
    wallp_url = %|http://wallpaperswide.com/download/#{$1}-#{Resolution}.jpg|
    
    # Download... with a user-agent parameter just in case...
    # use '--limit-rate=100k' to limit download speed
    system(%|"C:\\Program Files\\GnuWin32\\bin\\wget.exe" -c -U "Firefox/4.5.6" -P "#{Output_Directory}" "#{wallp_url}"|)

  end
end

I'm continually working on it. You can find the latest version of the code on my github account.


The below listed screenshots will guide you through the process...

Convert videos using ffmpeg to watch in Nokia 5230 / 5233 / 5800 / 5530 / X6 / N97

Nokia 5230/5800/.../X6 are (err... were) great smartphones. At least before the advent of the now-ubiquitous Android/Windows ones. They have got excellent, high resolution (640x360) screens which are great for watching videos on the go.

ffmpeg too has become the de facto encoding suite for converting videos, at least on linux. Here are the ffmpeg commands used to convert videos to a format which the aforementioned devices play without a hiccup. My 5230 plays MPEG4 videos encoded with lavc/xvid upto resolution 640x360. However, it'd play videos encoded with the advanced H264 codec only upto 320x240.

# ffmpeg libxvid
 ffmpeg -i Input-Filename.avi -f mp4 -y \
   -vcodec libxvid -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
   -r 25 -s 640x272 -aspect 640:360 -vf pad=640:360:0:44 \
   -threads 2 -async 1 -pass 1 /dev/null
 ffmpeg -i Input-Filename.avi -f mp4 \
   -y -vcodec libxvid -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
   -r 25 -s 640x272 -aspect 640:360 -vf pad=640:360:0:44 \
   -threads 2 -async 1 -pass 2 ./Input-Filename-ffmpeg.mp4

# ffmpeg libx264
 ffmpeg -i Input-Filename.avi -f mp4 -y \
   -vcodec libx264 -b:v 600k -vpre ipod320 -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
   -r 25 -s 320x196 -aspect 320:240 -vf pad=320:240:0:22 \
   -threads 2 -async 1 -pass 1 /dev/null
 ffmpeg -i Input-Filename.avi -f mp4 -y \
   -vcodec libx264 -b:v 600k -vpre ipod320 -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
   -r 25 -s 320x196 -aspect 320:240 -vf pad=320:240:0:22 \
   -threads 2 -async 1 -pass 2 ./Input-Filename-ffmpeg.mp4

However, for batch processing and automation, a handy Bash shell script would be great. This is a small script I use to convert my videos. (Note: I'm continually working on it. The latest code will be on my github account)

#!/bin/bash
#
#    Vikas Reddy @ http://vikas-reddy.blogspot.com/
#
# ffmpeg libxvid
# --------------
# ffmpeg -i Input-Filename.avi -f mp4 -y \
#   -vcodec libxvid -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
#   -r 25 -s 640x272 -aspect 640:360 -vf pad=640:360:0:44 \
#   -threads 2 -async 1 -pass 1 /dev/null
# ffmpeg -i Input-Filename.avi -f mp4 \
#   -y -vcodec libxvid -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
#   -r 25 -s 640x272 -aspect 640:360 -vf pad=640:360:0:44 \
#   -threads 2 -async 1 -pass 2 ./Input-Filename-ffmpeg.mp4
#
# ffmpeg libx264
# --------------
# ffmpeg -i Input-Filename.avi -f mp4 -y \
#   -vcodec libx264 -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
#   -r 25 -s 320x196 -aspect 320:240 -vf pad=320:240:0:22 \
#   -threads 2 -async 1 -pass 1 /dev/null
# ffmpeg -i Input-Filename.avi -f mp4 -y \
#   -vcodec libx264 -b:v 600k -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
#   -r 25 -s 320x196 -aspect 320:240 -vf pad=320:240:0:22 \
#   -threads 2 -async 1 -pass 2 ./Input-Filename-ffmpeg.mp4
#  
#  Usage
#  -----
#  Command-line options: 
#  -a : Video aspect ratio. Could be either 1.66 or 2.35 (default)
#  -b : Video bitrate. Should be in the form of 600k (default)
#  -c : Video codec. Should be either libx264 or libxvid (default)
#  -d : Output directory. Current directory (.) is the default
#  -y : Whether to ask confirmation before overwriting any file.
#       Should be either "yes" (default) or "no"
#  
#  Examples
#  --------
#  1) ./ffmpeg-encode.sh The.Movie.Filename.avi 
#     would output the xvid-encoded video to The.Movie.Filename-ffmpeg.mp4 in the current directory
#  2) ./ffmpeg-encode.sh -a 1.66 -b 650k -c libx264 -d /home/vikas/downloads/ -y The.Movie.Filename.avi 
#  


# Command-line options
while getopts 'a:b:c:d:o:p:y' opt "$@"; do
    case "$opt" in
        a) video_aspect="$OPTARG" ;;
        b) vbitrate="$OPTARG" ;;
        c) video_codec="$OPTARG" ;;
        d) output_dir="$OPTARG" ;;
        o) addl_options="$OPTARG" ;;
        p) passes="$OPTARG" ;;
        y) ask_confirmation="no" ;;
    esac
done
shift $((OPTIND - 1))


# Defaults
video_aspect="${video_aspect:-2.35}"
video_codec="${video_codec:-libxvid}" # or libx264
vbitrate="${vbitrate:-600k}"
passes="${passes:-2}"
output_dir="${output_dir:-.}"


vpre_pass1=""
vpre_pass2=""

if [[ "$video_codec" == "libx264" ]]; then
    #vpre_pass1="-vpre fastfirstpass -vpre baseline"
    #vpre_pass2="-vpre hq -vpre baseline"
    aspect="320:240"

    if [[ "$video_aspect" == "2.35" ]]; then
        resolution="320x196"
        pad="pad=320:240:0:22"
    elif [[ "$video_aspect" == "1.66" ]]; then
        resolution="320x240"
        pad="pad=320:240:0:0"
    fi;

elif [[ "$video_codec" == "libxvid" ]]; then
    aspect="640:360"

    if [[ "$video_aspect" == "2.35" ]]; then
        resolution="640x272"
        pad="pad=640:360:0:44"
    elif [[ "$video_aspect" == "1.66" ]]; then
        resolution="640x360"
        pad="pad=640:360:0:0"
    fi;
fi;


echo "Encoding '${#@}' video(s)";

for in_file in "$@"; do

    # If the filename has no extension
    if [[ -z "$(echo "$in_file" | grep -Ei "\.[a-z]+$")" ]]; then
        fname="$(basename "${in_file}")-ffmpeg.mp4"
    else
        fname="$(basename "$in_file" | sed -sr 's/^(.*)(\.[^.]+)$/\1-ffmpeg.mp4/')"
    fi
    out_file="${output_dir%/}/${fname}"

    # Avoid overwriting files
    if [[ "$ask_confirmation" != "no" ]] && [[ -f "$out_file" ]]; then
        echo -n "'$out_file' already exists. Do you want to overwrite it? [y/n] "; read response
        [[ -z "$(echo "$response" | grep -i "^y")" ]] && continue
    fi

    # 1st pass
    ffmpeg -i "$in_file" \
           -f mp4 -y $addl_options \
           -vcodec "$video_codec" -b:v "$vbitrate" \
           -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
           -r 25 -s "$resolution" -aspect "$aspect" -vf "$pad" \
           -threads 2 -async 1 -pass 1  \
           "/dev/null"; # $out_file;

    # 2nd pass
    ffmpeg -i "$in_file" \
           -f mp4 -y $addl_options \
           -vcodec "$video_codec" -b:v "$vbitrate" \
           -acodec libfaac -b:a 96k -ac 2 -ar 44100 \
           -r 25 -s "$resolution" -aspect "$aspect" -vf "$pad" \
           -threads 2 -async 1 -pass 2  \
           "$out_file";
done

Its usage is simple too. Without having to remember, edit and type in the lengthy ffmpeg command, this one makes my life a lot easier.

To start with, without any command-line options, the given file is assumed to be of 2.35 (cinema scope) aspect ratio, and consequently encoded using libxvid library to produce a nice 640x360 mp4 video. See below...

A small documentation with available command-line options and a few examples is bundled in the script itself.

NOTE: Because of licensing issues, ffmpeg binaries that are available on the repositories of most of the linux distros are not compiled with "non-free" codec support. This is especially true in the case of libx264 and libfaac. You may have to abandon them and compile the software from sources. Google it!

Do post your views in the comments section below...

My Archlinux screenshot

This is a screenshot of my Archlinux installation. I have been using it since 2007. Since it's a rolling distribution, I'd never ever need to re-install it again. And I've ported/copied this installation to many other machines with all the packages/settings/configuration intact.

The desktop consists of:

1. Xfce 4.10
2. Conky 1.9.0
3. XfPanel, for which no screenspace is reserved. This is great for conserving desktop realestate!
4. Iconset: Faience-Claire
5. A 1280x800 wallpaper from wallpaperswide.com

Bypass proxy server's file size download limit restriction

   Many organizations and colleges restrict their employees and students respectively from downloading files from the Internet which are larger than a prescribed limit. It is way too low at 14MB where I work. Fret not! There are ways to bypass this. And here is a simple bash script I wrote to download much larger files at my workplace.

Note: This script works only with direct links and with servers which support resume-download functionality.

  I'm continually working on it. So, the latest version will be available on my github account.

How to run it?

1. Download the following code to a text file named curldownload.sh
2. Give executable permissions to it chmod +x curldownload.sh
3. File size limit, fsize_limit variable, is set to 14MB. You may change it to your liking.
4. The script takes two arguments; the first one being the url of the file to be downloaded; the second one which is optional (defaults to "./") is the output directory.
5. For ex:- ./curldownload.sh http://ftp.jaist.ac.jp/pub/mozilla.org/firefox/releases/12.0/linux-i686/en-US/firefox-12.0.tar.bz2 "$HOME/Downloads"
6. A little more complex example of it using multiple urls, and two command-line arguments (-d for output directory, and -u for user-agent http header) is:  ./curl-multi-url.sh -d ~/downloads/ -u "Chromium/18.0" http://ftp.jaist.ac.jp/pub/mozilla.org/firefox/releases/11.0/linux-x86_64/en-US/firefox-11.0.tar.bz2 http://ftp.jaist.ac.jp/pub/mozilla.org/firefox/releases/12.0/linux-i686/en-US/firefox-12.0.tar.bz2

#!/bin/bash
#
# Vikas Reddy @
#   http://vikas-reddy.blogspot.in/2012/04/bypass-proxy-servers-file-size-download.html
#
# 
# Usage:
#     ./curl-multi-url.sh -d OUTPUT_DIRECTORY -u USER_AGENT http://url-1/ http://url-2/;
#     Arguments -d and -u are optional
#
#

# Defaults
fsize_limit=$((14*1024*1024))
user_agent="Firefox/10.0"
output_dir="."


# Command-line options
while getopts 'd:u:' opt "$@"; do
    case "$opt" in
        d) output_dir="$OPTARG";;
        u) user_agent="$OPTARG";;
    esac
done
shift $((OPTIND - 1))


# output directory check
if [ -d "$output_dir" ]; then
    echo "Downloading all files to '$output_dir'"
else
    echo "Target directory '$output_dir' doesn't exist. Aborting..."
    exit 1
fi;


for url in "$@"; do
    filename="$(echo "$url" | sed -r 's|^.*/([^/]+)$|\1|')"
    filepath="$output_dir/$filename"

    # Avoid overwriting the file
    if [[ -f "$filepath" ]]; then
        echo -n "'$filepath' already exists. Do you want to overwrite it? [y/n] "; read response
        [ -z "$(echo "$response" | grep -i "^y")" ] && continue
    else
        cat /dev/null > "$filepath"
    fi

    echo -e "\nDownload of $url started..."
    i=1
    while true; do   # infinite loop, until the file is fully downloaded

        # setting the range
        [ $i -eq 1 ] && start=0 || start=$(( $fsize_limit * ($i - 1) + 1))
        stop=$(( $fsize_limit * i ))

        # downloading
        curl --fail --location --user-agent "$user_agent" --range "$start"-"$stop" "$url" >> "$filepath"

        exit_status="$?"

        # download finished
        [ $exit_status -eq 22 ] && echo -e "Saved $filepath\n" && break

        # other exceptions
        [ $exit_status -gt 0 ] && echo -e "Unknown exit status: $exit_status. Aborting...\n" && break

        i=$(($i + 1))
    done
]]>

Port your GTalk contacts to another account

Porting your GTalk contacts to another GTalk account is not as easy as the export-import feature of Gmail Contacts would have you believe. For, your GTalk accounts are only a subset of your Gmail ones. So, essentially, if you'd export all your contacts to a csv file from your Account #1 and import them to your Account #2, they would get imported but do not appear in your Gtalk.

I've been using Pidgin since many years now and I never knew it'd store all the contacts it imported in an xml file locally, with groups they belong to, their buddy icons, last seen time, etc.,  ~/.purple/blist.xml is where all these are present.

Steps


1. Install Pidgin, create accounts for both your gmail ids.
2. Open your Ruby console/irb and run the following code

# Parse the xml document...
doc = REXML::Document.new(File.read("/home/YOUR_USERNAME/.purple/blist.xml"))

# Initialize empty arrays
contacts_1 = []
contacts_2 = []

# Get all your contacts in your account number 1, assume it as account_1@gmail.com
doc.elements.each('//buddy[starts_with(@account,"account_1@gmail.com")]/name') {|e| contacts_1 << e.text}

# Get all your contacts in your account number 2, assume it as account_2@gmail.com
doc.elements.each('//buddy[starts_with(@account,"account_2@gmail.com")]/name') {|e| contacts_2 << e.text}

# Get the email ids of all contacts which are present in account#2 but not in account#2
# i.e., the contacts who need to be invited
(contacts_2 - contacts_1).join(", ")

3. You'll get some thing like this as output of the last line of code...

4. multiple contacts can be invited at once using Windows GTalk. So, open it and paste this output into the Invite Contacts wizard as shown below

That's it! You have the requests sent to all your invitees from your new account.

cp: An amazing swiss knife of GNU

  The other day I was banging my head wondering how to infuse life (read, extra space) into my ageing laptop and into the Arch Linux installation, which, by the way, is turning five this year. The rolling release model and its light-weight packaging has bitten me way back in 2007. I had used my old 80GB, Turion 64x2, 1.5GB DDR2 Presario V3029AU to the maximum, and it's showing.

   Last week, I tried to install Oracle 10g XE on Fedora 16 and it asked for a minimum of 1024MB of swap, a whisker short of what my harddisk actually had. So, I was only left with one dreaded option: to re-format my internal drive and to make space for three Linux distros and a Windows 7 x64 primary partition with sufficient swap space. For that, I had to retain the contents of my Arch. A few options I found after scouting the Internet were:

1. Use dd to take an as-is backup of the partition to an image file
2. Use clonezilla to take backup to an image file
3. Use partimage
4. Use cp to simple copy all the files onto a separate "linux-compatible" filesystem (read, one which retains the permissions and ownership of the files)

  Obviously, as my main intention was to expand in-place the partition or move files onto a bigger partition, the first option was ruled out. Clonezilla had to be used by burning the live iso  onto a CD or a pen drive. The only sensible option I was left with was to use cp.

cp -ax /source-path/* /destination-path/

   Mind you, I didn't know what all cp could do even after using it for the past seven years on a daily basis. This command simply copies all the files onto the new partition, of course, with all the necessary bells and whistles such as permissions, file ownership and other necessary attributes. Note the two command-line options "-a" and "-x". The latter is to indicate to cp that only one file system needs to be copied. Ignoring this option will end you up with copies of files present on mounted partitions too on your new partition. These two options are absolutely essential for your new partition to work as intended.

  This left me with sufficient space (13GB each) for Arch , LMDE and Fedora 16.

  This scenario reinforces the simplicity of design and everything-is-transparent-to-user philosophy of GNU/Linux. And, cp truly is an amazing swiss knife of it!