Parsing Webpage Content Into KeywordsRequirements:- PHP - version 5.3.0 used. Should be set up on same machine as the MySQL server, unless you modify the code in ParseSQL.
- simple_html_dom - version 1.11 used. This class enables PHP to extract information from HTML pages.
- ParseSQL - Class made in tutorial two of this series. Used for adding parsed data to the MySQL database.
- page_search - The database defined in tutorial one of this series should be created and accessible on your MySQL server.
Analyzing Requirements:
At the end of the "Filling the Database" tutorial, we had created a database for inserting pages and search keywords, and a class for performing the insertions. The next step is to retrieve data for insertion. For this purpose, we will create a static class named HTMLParser. HTMLParser will need to be able to perform the following tasks:
- Use an instance of ParseSQL to perform data insertion.
- Open a HTML file on the local server, and optionally on a remote server. (if PHP settings support it)
- Extract the title, description, and any textual content from the HTML file.
- Convert local file paths to URLs so they can be used to create links.
- Scan a file system at a certain url or path in order to list and parse files (This way each url which should support searching won't have to be individually entered by hand)
- Check the file extension each file in a directory to determine if it potentially contains HTML.
All of these requirements will be met using the four static functions of HTMLParser:
- parsePage - Requirements: 1, 2, 3
- fileToURL - Requirement: 4
- parseDirectory - Requirement: 5
- hasValidExt - Requirement: 6
Creating the HTMLParser Class:
require_once(dirname(__FILE__) . '/simple_html_dom.php');
require_once(dirname(__FILE__) . '/ParseSQL.php');
abstract class HTMLParser
{
}
Code Info:This is just a simple abstract class definition with two required classes. The required pathnames are set up for ParseSQL and simple_html_dom to be placed in the same directory as HTMLParser, you can adjust them to your need. Because the class is abstract, it cannot be instantiated (new HTMLParser() is invalid), so there is no need to define a constructor.
Defining the parsePage() Function:
This function will take an initialized ParseSQL instance, a path string that can either be a url or a local file path, and a boolean 'islocal' which is should be true if the path string is a file path. It will use these parameters to parse the file specified by the path string, and add it to the search database used by the ParseSQL instance. The code is as follows:
public static function parsePage($parse_sql, $path_name, $islocal)
{
// Create an html dom from the provided pathname
$html = file_get_html($path_name);
// Verify the file contains an html tag
if($html->find('html',0))
{
// if pathname is a local file, convert to url
$url = ($islocal ? HTMLParser::fileToURL($path_name) : $path_name);
// Get the title of this page, if no title tag is set, use the url
$title = $html->find('title',0);
$title = ($title == null ? $url : $title->innertext);
// Get the description meta tag of this page, or null if there isn't one
$desc = $html->find('description',0);
$desc = ($desc == null ? $desc : $desc->innertext);
// Extract an array of keywords from this page.
$terms = ParseSQL::extractKeywords($html->plaintext);
// Add Page to database and store the page id
$page_id = $parse_sql->addPage($url, $title, $desc);
// Add terms to database using the page id
$parse_sql->addTerms($terms, $page_id);
}
}
Code Info:Using a URL for $path_name: If a url is specified in $path_name, PHP needs to have fopen() set up to be able to open remote files. Info about setting up php to allow_url_fopen can be found
here.
Defining the fileToURL() Function:I slightly modified a function submitted by someone in the PHP doc comments to create this function. It takes a file path, and uses the PHP home directory and server url to determine the url that represents that file path. It will only work on file paths that are sub-paths of the home directory. The code is as follows:
private static function fileToURL($file_path) {
// Get the resolved path of the provided file path
$full_path = realpath($file_path);
// Get just the directory part of the provided file path
$dir_path = realpath(dirname($file_path));
// Get the root path of the web server
$root_path = realpath($_SERVER['DOCUMENT_ROOT']);
if (strlen($root_path) > strlen($dir_path)) {
// File is lower than root
return false;
}
// Create a url from the provided path
return 'http://' . $_SERVER['HTTP_HOST'] . str_replace('\\', '/', substr($full_path, strlen($root_path)));
}
Code Info:Line 15: This line essentially replaces the home path of the server with the url of the server. substr() eliminates the root file path from the provided file path. str_replace() replaces '\' characters with '/' characters, because windows file paths use '\' slashes for file paths. Altogether this function will create a valid url that can be used to link to the file provided.
Creating the parseDirectory() Function:
This function is used to scan the files in a directory, or at an url that allows indexing. Same as the parsePage() function, PHP needs to have fopen() set up to be able to open remote files. Info about setting up php to allow_url_fopen can be found
here. The code is as follows:
public static function parseDirectory($parse_sql, $path, $deep_scan = false, $valid_types = array('html', 'htm', 'php'))
{
// Create a DirectoryIterator for single directory search,
// or a RecursiveDirectoryIterator to also scan subdirectories
$dir = ($deep_scan == true ? new RecursiveDirectoryIterator($path) : new DirectoryIterator($path));
// Scan through each file
foreach($dir as $page)
{
if($page->isDot() !HTMLParser::hasValidExt($page->getFilename(), $valid_types))
{
// If page is a '.' or '..' directory, or has an extension not in $valid_types
// continue to next file
continue;
}
else
{
// If page has a valid extension attempt to parse it
HTMLParser::parsePage($parse_sql, $page->getPathName(), true);
}
}
}
Code Info:Parameters: $parse_sql = an initialized instance of ParseSQL, $path - a string representing the directory containing the files to be parsed, $deep_scan - If set to true, subdirectories of the provided path will also be parsed, $valid_types - An array containing the lower case extensions of filetypes that should be scanned, defaults to php, htm, and html files.
Line 8: DirectoryIterator can be used in a foreach loop to scan through each value of the DirectoryIterator object. More info on DirectoryIterator can be found at the php documentation
here.
Line 10: hasValidExtension() is defined next, it will verify that the file currently being scanned has an extension included that is contained in $valid_types.
Creating the hasValidExtension() Function:This function returns a boolean value indicating if the extension of a provided file string is contained in a provided array. The code is as follows:
private static function hasValidExt($filename, $valid_types)
{
return in_array(strtolower(pathinfo($filename, PATHINFO_EXTENSION)), $valid_types);
}
Code Info:Code Broken Down: pathinfo($filename, PATHINFO_EXTENSION) gets the file extension part of a filename as a string. strtolower() converts the extension to lowercase so that case won't affect the comparison with array values. in_array() determines if the file extension is present in $valid_types.
Example Usage of HTMLParser:This code will parse the pages in the path it is contained in:
<?php
require_once(dirname(__FILE__) . '/HTMLParser.php');
require_once(dirname(__FILE__) . '/ParseSQL.php');
$connection = new ParseSQL('user','password','page_search');
HTMLParser::parseDirectory($connection, dirname(__FILE__) . '/');
$connection->close();
?>
The source code for this tutorial can be found
here.
The source code for all 5 tutorials in this series can be found
here.
Tutorials in this Series:1.
Defining a MySQL Database to be Populated / Searched Using PHP2.
Filling the MySQL Search Keyword Database3.
Parsing Webpage Content Into Keywords4.
Creating the Search Query Class5.
Creating the User Interface