TECKpert » Coding Tips

DOM manipulation with PHP, the ultimate page scraper?

Andres — Tue, 12 Jan 2010 13:21:10 +0000

These days we hear a good deal about DOM manipulation with JavaScript but some little known technologies ( for now – they’re quickly gaining ground ) are xPath, xQuery & XSLT.

Fellow developers will know that historically we’ve had to rely on a number of regular expressions to scrape a page and while this can most often be fast, it’s sometimes horrendous to read and edit as a TINY mis-write can effectively render the Regular Expression useless. That’s not to say it’s not useful when the hierarchy is small and simple but in today’s world of web 2.0 designs they’re often not.

What is page scraping?

Page scraping is a method that allows you to pull information from a web page, so that the data can be manipulated inside your own script. In your script, you can connect to another URL and request a page, just like a browser would do it. Once you make the request, the web server will send back the page you asked for and your script can manipulate the data and extract specific information.

What exactly does the DomDocument Object do?

If you’re not familiar with the DOM model than the following explanation probably isn’t going to make much sense as there really isn’t too much to say besides, DomDocument transform’s an HTML page into a tree-model of elements. JavaScript does this upon load and is the entire point to the language: Dom Manipulation. I find a visual representation often helps to understand the tree model so here is a simplified version of the aforementioned model present in DOM:

The above is a representation of the following HTML:

Sidenote: If your development background is based on the more traditional languages this will look familiar to you as resembles the Binary Tree, how ever, DOM supports a limitless number of children per node.

Through the DomDocument object we are given to the ability to traverse the nodes, create them, and remove them as we see fit. However sometimes traversing the levels of DOM solely through the methods provided to us by the object is cumbersome and altogether impractical given the depth of the information that we sometimes need. Enter xPath; it is to XML compliant mark up languages what SQL is to databases, a query language. The entire breadth of xPath is outside the scope of this particular post but is covered in depth here. If you’re familiar with jQuery or any other JavaScript framework which supports CSS style selectors this’ll be easy for you.

How to use xPath with DomDocument

We’ll start off with something basic as an introduction, we’ll scrape the “Why us” section of TECKpert’’s homepage.

loadHTMLFile($url);

// Alternatively this works too
$html = file_get_contents($url);
$doc->loadHTML($html);

// Now that we've created our dom object proper
// call the xPath object
$xPath = new DOMXPath( $doc );

// Query TECKpert's dom for the 'why us' section
$results = $xPath->query('//div[@class="why_us"]');

echo $results->item(0)->textContent;

Simple to use right? There will be more to come on the onset of technologies associates with XML traversing and it’s related query languages.

Note on this article
There exists the possibility of violating copyright laws using techniques outlined in this article if you misuse data you scrape. Please scrape responsibly.

jQuery Development: Your own Plugins Part II

Andres — Tue, 08 Dec 2009 23:26:16 +0000

Extending on my previous post jQuery development: Your own plugins , we now have to take a look into a more complex plugin as well as making greater use of both JavaScript’s built in Regular Expression objects and jQuery provided functionality.

For the purposes of this post, I’ll be extending on the previous plugin to readily handle more video services: Dailymotion, Megavideo & Vimeo as well as the already supported YouTube. Also, we’ll be adding an option to leave the link but make the video pop up in a modal box using ColorBox.

Okay enough intro…let’s code!

First, we’re going to start off with the basic structure of every jQuery plugin:

(function($){
    $.fn.videa = function(){
    }
})(jQuery)

Note: We use this format for the purpose of ease and using the familiar $ variable, however we can just do this.

jQuery.fn.videa = function(){
   //How ever, within here we can only use the jQuery object,
   // $ is undefined.
}

First we’re going to need the embed code for each service. This is readily
available on their websites as well as the format for their actual flv’s.
I’ve already done this below.

Note: I’ve made use of JSON. If you don’t know about this, read up here.

(function($){

    $.fn.videa= function(){

        var supportedSites = {
            'youtube' : '',
            'dailymotion': '',
            'megavideo': '',
            'vimeo'    : ''
        }

        var handle = {
            'youtube'    : ['watch?v=', 'v/'],
            'dailymotion': [/.*dailymotion.*\/video\/(.*?)_.*/, 'http://dailymotion.com/swf/$1'],
            'megavideo'  : ['?v=', 'v/'],
            'vimeo'      : [/.*vimeo.*\/(.*[0-9])/, 'http://vimeo.com/moogaloop.swf?clip_id=$1&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=00adef&fullscreen=1']
        }

Next, we’ll need to make use of each utility; this is something that will be used in nearly ALL your plugins. The reason why is because you want the plugin to affect more than one HTML element.

(function($){
    $.fn.videa= function(){
        // We're of course assuming the aforementioned json is here
        // ommited for the purposes of readability
      this.each(function(){
      })
})(jQuery)

Now, for our purposes we don’t actually need to instantiate a jQuery object per element – yet. All we need is the href and class attributes which are easily available via JavaScript’s native getAttribute function.

(function($){
    $.fn.videa= function(){
        // We're of course assuming the aforementioned json is here
        // ommited for the purposes of readability
      this.each(function(){
           var href    = this.getAttribute('href');
           var aClass  = this.getAttribute('class');
           var key     = href.match(/(http:\/\/www\.|http:\/\/|www\.|)(.*?)\.(\.com|\.*)/)[2];

          // we now have the base such as 'youtube' and 'vimeo' in key
          // check if we support it
          if( supportedSites[key] )
          {
          }
      })
})(jQuery)

Now I’ll explain the regex

The first bit:

This matches the http protocol with or without the www or just the www

‘(http:\/\/www\.|http:\/\/|www\.|)’

The second bit:
Simply grabs everything AFTER the http://www.

‘(.*?)’

The third bit:

This matches against any .com or .* extension

‘\.(\.com|\.*)’

The rest is rather straight forward and can be explained in code comments.

// Check if the site is supported
if( supportedSites[key] )
            {
               // Construct properly url to the flv
               if( href.indexOf('http://www.') == -1 )
                        href = 'http://www.' + href.replace('www.', '');

                // Check if we want a modal box
                if( aClass.indexOf('modal') > -1 )
                {
                    // Bind to the click and fire off colorbox
                    $(this).click(function(){
                        $.fn.colorbox({
                            html: supportedSites[key].replace(/\[URL\]/ig, href.replace(handle[key][0], handle[key][1])) ,
                            open:true
                        });
                                return false;
                     })
                }else{
                    // We don't want a modal box so just do the html
                    $(this).before(supportedSites[key].replace(/\[URL\]/ig, href.replace(handle[key][0], handle[key][1]))).remove();
                }

Note: jQuery is a jackhammer, sometimes all we have is nails.

That’s the plugin broken down into the important sections. The end result is as expected. Everything which has a videa class is converted. The modal class is optional (it creates a link that opens a modal box with your video).

As for plugin development, once you understand DOM and the entire jQuery API, you can pretty much make anything you can come up with. JavaScript support is strong and growing.

The Entire plugin below:

(function($){

    $.fn.embeddedables = function(){

        var supportedSites = {
            'youtube' : '',
            'dailymotion': '',
            'megavideo': '',
            'vimeo'    : ''
        }

        var handle = {
            'youtube'    : ['watch?v=', 'v/'],
            'dailymotion': [/.*dailymotion.*\/video\/(.*?)_.*/, 'http://dailymotion.com/swf/$1'],
            'megavideo'  : ['?v=', 'v/'],
            'vimeo'      : [/.*vimeo.*\/(.*[0-9])/, 'http://vimeo.com/moogaloop.swf?clip_id=$1&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=00adef&fullscreen=1']
        }

        // Looad in this element for all a tags
        this.each(function(){

            var href   = this.getAttribute('href');
            var aClass = this.getAttribute('class');
            var key = href.match(/(http:\/\/www\.|http:\/\/|www\.|)(.*?)\.(\.com|\.*)/)[2];

            if( supportedSites[key] )
            {
               if( href.indexOf('http://www.') == -1 )
                        href = 'http://www.' + href.replace('www.', '');

                if( aClass.indexOf('modal') > -1 )
                {
                    $(this).click(function(){
                        $.fn.colorbox({
                            html: supportedSites[key].replace(/\[URL\]/ig, href.replace(handle[key][0], handle[key][1])) ,
                            open:true
                        });
                                return false;
                     })
                }else{
                    $(this).before(supportedSites[key].replace(/\[URL\]/ig, href.replace(handle[key][0], handle[key][1]))).remove();
                }
            }

        });

    }

})(jQuery)

TECKpert’s Tips: Using Google Base with .NET for Real Estate

Sean — Fri, 27 Nov 2009 15:41:03 +0000

Adding listings to Google Base Real Estate is simple and completely free. Today I’ll show how to, using the Google Base .NET library, add listings to Google Base and then search for those listings. The library can be downloaded at http://code.google.com/apis/gdata/client-cs.html.

First import the following classes from the Google Base Client library.

using Google.GData.Client;

using Google.GData.GoogleBase;

To insert items, you need to be authenticated. Specify your Google Base account name and password to GBaseService as follows. You can use your Gmail account but you have to first sign into Google Base to activate the account.

GBaseService service = new GBaseService("Application Name", developerKey);

service.setUserCredentials("username", "password");

The next step is to create the GBaseEntry you would like to insert.

GBaseEntry entry = new GBaseEntry();

entry.Title.Text = "My Title";

entry.Content.Content = description;

It is important to set the ItemType to “Housing”.

entry.GBaseAttributes.ItemType = "Housing";

Now set the other attributes. To see a list of all the Google Base attributes go to http://base.google.com/support/bin/answer.py?hl=en&answer=78170.

FloatUnit unit = new FloatUnit(price, "usd");

entry.GBaseAttributes.Price = unit;

entry.GBaseAttributes.AddTextAttribute("feature", feature);

entry.GBaseAttributes.AddImageLink(imgLink);

entry.GBaseAttributes.AddTextAttribute("property_type", propertyType);

entry.GBaseAttributes.AddTextAttribute("listing_type", listType);

entry.GBaseAttributes.AddTextAttribute("listing_status", listStatus);

FloatUnit sunit = new FloatUnit(sqft, "square ft.");

entry.GBaseAttributes.AddFloatUnitAttribute("square_feet", sunit);

entry.GBaseAttributes.AddFloatAttribute("bathrooms", baths);

entry.GBaseAttributes.AddIntAttribute("bedrooms", beds);

entry.GBaseAttributes.Location = "Address, City, State xxxxx USA ";

entry.GBaseAttributes.AddTextAttribute("mls_listing_id", listingID);

entry.GBaseAttributes.AddTextAttribute("broker", listBroker);

entry.GBaseAttributes.AddTextAttribute("agent", listAgent);

Finally, insert this GBaseEntry object into the Items feed.

 GBaseEntry ent = service.Insert(GBaseUriFactory.Default.ItemsFeedUri, entry);

The method service.Insert returns the item you just inserted. The Google Base server assigns pre-computed attributes for your entry such as the creation date and time, the author, and most importantly, the identifier (or URL) of the new entry. The entry may take a few hours to publish so it may not be searchable immediately.

Now you can search for your listings on Google Base by connecting to a snippets feed URL and then interpreting the resulting atom feed. This can be done programmatically by creating a GBaseService object and executing a query on it.

GBaseService service = new GBaseService("Application Name", developerKey);

GBaseQuery query = new GBaseQuery(GBaseUriFactory.Default.SnippetsFeedUri);

The following query will get single family homes for sale within a 5 mile radius of San Francisco CA.

query.GoogleBaseQuery = @"[item type:housing] [listing type:for sale] [property_type:single] [location:@""San Francisco, CA, USA"" + 5mi]";

You can also set the sort by and order, number of results, and search start index.

query.NumberToRetrieve = 20;

query.StartIndex = 0;

query.AscendingOrder = true;

query.OrderBy = "price(float USD)";

Lastly, execute the query and save the atom feed into a readable xml format. This can be parsed and displayed on a web page.

XmlTextReader reader = new XmlTextReader(query.Uri.AbsoluteUri);

XmlDocument doc = new XmlDocument();

doc.Load(reader);

Pretty simple right? Now you can have your listings searchable on Google along with your website.

jQuery development: Your own plugins

Andres — Mon, 16 Nov 2009 17:32:19 +0000

Often there is a need for functionality that goes above jQuery’s core set of methods, there are a host of plugins which can be used to attain this functionality. Such as ColorBox when a modal box is needed or jFlow for a basic gallery. However, sometimes these plugins cannot perform exactly the way we need them to without attacking the plugin’s source and in the end causing a fair bit of frustration, confusion, and a plugin that not only doesn’t do what you want but doesn’t do anything.

Solution? Develop your own plugin

The benefits of this course are:

Extreme familiarization with the code base ( You wrote it! )
Makes for identifying the area where bugs are coming from a fair bit easier
You don’t have to worry about licencing, you wrote it, it’s yours.

The above being said, I suppose it’s about time to get on to exactly how one goes about developing their own plugin and it’s assuming familiarization with their core and some basic JavaScript it can be done rather easily. For the purpose of this blog post we’ll be doing a plugin that transforms every YouTube link into a nice embedded video.

First, we start off with a basic structure which is necessary to eliminate the possibility of Namespace conflicts ( libraries aside from jQuery make use of the dollar sign $ )

(function($){

})(jQuery)

Now we must define our function, with jQuery however we’ll be making use of closures or anonymous functions and than assign it to a variable. This is how:

(function($){
    $.fn.youtubualize = function(){

    }
})(jQuery)

Now we get to the fun bits! jQuery feeds us a copy of itself ( available though the this supervariable ) ready to be manipulated, seeing as we’ll most likely have more than 1 link we’ll make use of jQuery’s each utility to cycle through all the links.

(function($){
     $.fn.youtubualize = function(){

        this.each(function(){

        });
    }
})(jQuery)

Now we must be away that within the callback we’re providing to the each utility this is only a string representing the YouTube link so we need to use jQuery’s CSS selectors to single out the element and modify it accordingly, this will be the most verbose part

(function($){
     $.fn.youtubualize = function(){

        this.each(function(){
            var url    = String(this);
            var flvURL = url.replace('watch?v=','v/');
            $('a[href="'+ url +'"]')
                .before('\
 \
 \
 \
 \
 \
 ')
                .remove();
        });
    }
})(jQuery)