The aim of this tutorial is to help you create a web fetching script that can extract content from a password protected area using the necessary login credentials. I will use the well know cURL command line tool to connect to the protected web area. PHP supports libcurl which is required in order to use cURL functions in PHP.

The following script signs in to YouTube and fetches the latest favorite videos.

First, let’s create the file functions.php which should contain a practical cURL function & another function that extracts content between 2 delimiters (click here to view details about it):

<?php
function LoadCURLPage($url, $agent='', $cookie='', $referer='', $post_fields='', $ssl='')
{
$ch = curl_init(); 

curl_setopt($ch, CURLOPT_URL, $url);

if($ssl) curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,  2);

curl_setopt ($ch, CURLOPT_HEADER, 0);

if($agent) curl_setopt($ch, CURLOPT_USERAGENT, $agent);

if($post_fields)
{
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
}

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

if($referer) curl_setopt($ch, CURLOPT_REFERER, $referer);

if($cookie)
{
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
}

$result = curl_exec ($ch);

curl_close ($ch);

return $result;
}

function extract_unit($string, $start, $end)
{
$pos = stripos($string, $start);  

$str = substr($string, $pos);  

$str_two = substr($str, strlen($start));  

$second_pos = stripos($str_two, $end);  

$str_three = substr($str_two, 0, $second_pos);  

$unit = trim($str_three); // remove whitespaces  

return $unit;
} 

?>

Let’s start creating the index.php file that will login in to YouTube members area. First, we will include ‘functions.php’ and create some configuration variables:

/*
Credits: Bit Repository
URL: http://www.bitrepository.com/web-programming/php/curl-login-to-members-area.html
*/

include 'functions.php';

// YouTube: Username & Password
$username = 'user_here';
$password = 'pass_here'; 

$cookie_file_path = 'tmp/yt-cookie.txt'; // Please set your Cookie File path
$login_url = 'http://www.youtube.com/login?next=/index';

$referer = $login_url;

$agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4)
 Gecko/20030624 Netscape/7.1 (ax)';

$username, $password – YouTube Login

$cookie_file_path – This is the file containing the cookie data, which can be in Netscape format, or regular HTTP-style headers dumped into a file.

NOTE: Make sure that the folder (where the cookie file is located), is writable (chmod 0777).

$login_url – This is the URL where the POST data will be sent
$referer – It is used to set the content of “Referer: ” header that is used in a HTTP Request. In this case it will be the same as the $login_url;
$agent – Sets the content of “User-Agent: ” header that is sent in a HTTP Request

In many cases, it is not enough to send only the login info to the parse page. Many forms have hidden inputs which should be taken into consideration. This is the case with the YouTube login form too. There are 3 additional parameters that are sent to the ‘process login’ page:

current_form – It has the value ‘loginForm’
next – It has the value ‘/my_favorites’. If the user successfully logs in, he/she will be redirected to “My Videos” page.
action_login – It has the value ‘Log In’

/*
--------------------
2. SEND LOGIN INFO
--------------------
*/

// Data to send
$post_info = array('current_form' => 'loginForm',
                   'next' => '/my_favorites',
				   'username' => $username,
                   'password'   => $password,
				   'action_login'  => 'Log In');

$post_data = '';

foreach($post_info as $name => $value)
{
$post_data .= $name.'='.$value.'&';
}

$post_data = trim(substr($post_data, 0, -1));

$result = LoadCURLPage($login_url, $agent, $cookie_file_path, $referer, $post_data);

Eventually, extract the latest videos from $result and output them:

/*
-----------------------------------------
3. EXTRACT THE LATEST FAVORITE VIDEOS
-----------------------------------------
*/

preg_match_all('/<div class="clipper">(.*)<\/div>/i', $result, $out);

echo "<strong>".$username."'s Latest YouTube Favorite Videos</strong><br>";

foreach($out[1] as $key => $r)
{
$video_id = extract_unit($r, "['", "']");
$video_title = strip_tags($r);

echo ($key + 1).'. <a target="_blank" href="http://www.youtube.com/watch?v='.$video_id.'">'.$video_title.'</a>'."\n<br>";
}

IMPORTANT – Please note that in some cases, you need to open the login page first in order to set the cookie & continue sending the data to ‘process login’ page. Some unique info is registered for the user that accessed the login form and it is necessary to be passed to the parse page too.

I hope that this tutorial gave you an idea about how you can create a bot that authenticates in to a password protected area. If you have any comments regarding this article, please post them.

Happy coding!

Do you wish to receive the latest updates as soon as they are posted? Get our RSS Feed or Subscribe to the Newsletter!