How quickly your web pages are delivered is becoming an important factor in running any site. Not only does Google take speed into account but slow sites frustrate users and send them away.

While pages may load quickly when you look at them what about other times?  Do pages always get delivered quickly or only under some conditions? What about during your peak times? What about quiet times when caching is less effective?

Idea

There is plenty of discussion in forums about measuring page speeds but it's always a one-off fudge. I decided that I wanted a consistent way of measuring speeds that I could drop into any LAMP site I worked on. One class that does everything.

There are 2 main types of measurement that need to be measured: overall page delivery and targeted areas of the page (eg. processing a form before even starting to deliver any html).

The class has to take these into account: it has an overall page time as well as being able to meausre time getting to "milestones" within the code.

The class uses the database for storage (allows it to be easily centralised) and is completely self contained. There is no need for any other database abstractions - it takes care of things it's self using PDO.

Time for class... or class for time

The basic tool is one PHP class that can be included in the page. The class specifically avoids doing anything that may introduce delays such as database activity until the page is completed.

Download: LAMP (PHP) Pagetimer Class

Using the class is easy. "require" it:

require_once ( 'pagetimer.php' );

... if it's in your PHP path or

require_once ( '/path/to/pagetimer.php' );

Then at the top of a page kick it off:

$pagetimer = new pagetimer ();

By default the class will look for /etc/pagetimer/dbconfig.php where it eexpects to look something like:

<?php
        $dsn = 'mysql:host=localhost;dbname=pagetimer';
        $username = 'pagetimer';
        $password = 'pagetimerpassword';
        $options = array ();
?>

These map onto the arguments of PDO which it uses for database access. Make sure the file has appropriate permissions to allow it to be accesed by the PHP scripts as well as snmpd. You can also specify these same arguments in your script:

$pagetimer = new pagetimer ( 'mysql:host=localhost;dbname=pagetimer', 'pagetimer', 'pagetimerpassword', array () );

At this point the clock is ticking. Along the way you can also collect "miletone" times:

$pagetimer->miletone ( 'name of miletone' );

And then finally at the end of the page we stop the clock:

$pagetimer->stop ();

At this point the database connection is opened and the page time and milestone times are recorded. Simple as that.

One important thing to note is that if you don't have buffering turned on that is big enough to hold the whole page then you will not be measuring the speed the page was generated, but rather the speed it was delivered to the client which is unlikely to give much useful information about the performance of your site / code. You need something like this in your php.ini with a buffer size suited to your site:

output_buffering = 8192

Database setup

Obviously we need a database to store this data in. The basic setup (logged into MySQL as root) follows.

Create a database and user:

CREATE DATABASE IF NOT EXISTS pagetimer CHARACTER SET = utf8;
CREATE USER 'pagetimer'@'localhost' IDENTIFIED BY 'pagetimerpass';
GRANT ALL PRIVILEGES ON pagetimer.* TO 'pagetimer'@'localhost';
FLUSH PRIVILEGES;

Create the neccessary tables:

CREATE TABLE IF NOT EXISTS `pagetimer`.`Pages` (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT,
    Start FLOAT NOT NULL,
    Stop FLOAT NOT NULL,
    Time FLOAT NOT NULL,
    Method CHAR(10) NOT NULL,
    Host CHAR(20) NOT NULL,
    Request CHAR(100) NOT NULL,
    Name CHAR(50),
    INDEX (Time),
    INDEX (Method),
    INDEX (Host),
    INDEX (Request),
    INDEX (Name),
    PRIMARY KEY (id)
);
CREATE TABLE IF NOT EXISTS `pagetimer`.`Milestones` (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT,
    Start FLOAT NOT NULL,
    Milestone FLOAT NOT NULL,
    Time FLOAT NOT NULL,
    Page INT UNSIGNED NOT NULL REFERENCES Pages(id),
    MilestoneName CHAR(50),
    Name CHAR(50),
    INDEX (Time),
    INDEX (MilestoneName),
    INDEX (Name),
    PRIMARY KEY (id)
);
CREATE TABLE IF NOT EXISTS `pagetimer`.`HistogramCache` (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT,
    Time INT UNSIGNED NOT NULL,
    Band INT UNSIGNED NOT NULL,
    Value FLOAT NOT NULL,
    Filter CHAR(40) NOT NULL,
    INDEX (Filter),
    UNIQUE INDEX (Band,Filter),
    PRIMARY KEY (id)
);
CREATE TABLE IF NOT EXISTS `pagetimer`.`PagetimeCache` (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT,
    Time INT UNSIGNED NOT NULL,
    Filter CHAR(40) NOT NULL UNIQUE,
    Value FLOAT,
    INDEX (Time),
    INDEX (Filter),
    PRIMARY KEY (id)
);

Getting the data out

There is a second PHP script for extracting the data and pruning the database. This is intended to be run from snmpd so that Cacti can easily collect the data remotely. You will need PHP CLI installed for this. The script assumes it's in /usr/bin/php so you will need to modify the script accordingly.

Download: snmpd extension LAMP (PHP) Timer Statistics

Make this executable and add it to your snmpd.conf with something like (assuming you put the script in /etc/snmp) this:

extend pagetimerfilters /etc/snmp/getpagetimer.php filters
extend pagetimerhistlevels /etc/snmp/getpagetimer.php histogramlog 1 levels
extend pagetimerhist0 /etc/snmp/getpagetimer.php histogramlog 1 0
extend pagetimerhist1 /etc/snmp/getpagetimer.php histogramlog 1 1
extend pagetimerhist2 /etc/snmp/getpagetimer.php histogramlog 1 2
extend pagetimerhist3 /etc/snmp/getpagetimer.php histogramlog 1 3
extend pagetimerhist4 /etc/snmp/getpagetimer.php histogramlog 1 4
extend pagetimerhist5 /etc/snmp/getpagetimer.php histogramlog 1 5
extend pagetimerhist6 /etc/snmp/getpagetimer.php histogramlog 1 6
extend pagetimerhist7 /etc/snmp/getpagetimer.php histogramlog 1 7
extend pagetimerhist8 /etc/snmp/getpagetimer.php histogramlog 1 8
extend pagetimerhist9 /etc/snmp/getpagetimer.php histogramlog 1 9
extend pagetimerhist10 /etc/snmp/getpagetimer.php histogramlog 1 10
extend pagetimerrawhist0 /etc/snmp/getpagetimer.php histogramrawlog 1 0
extend pagetimerrawhist1 /etc/snmp/getpagetimer.php histogramrawlog 1 1
extend pagetimerrawhist2 /etc/snmp/getpagetimer.php histogramrawlog 1 2
extend pagetimerrawhist3 /etc/snmp/getpagetimer.php histogramrawlog 1 3
extend pagetimerrawhist4 /etc/snmp/getpagetimer.php histogramrawlog 1 4
extend pagetimerrawhist5 /etc/snmp/getpagetimer.php histogramrawlog 1 5
extend pagetimerrawhist6 /etc/snmp/getpagetimer.php histogramrawlog 1 6
extend pagetimerrawhist7 /etc/snmp/getpagetimer.php histogramrawlog 1 7
extend pagetimerrawhist8 /etc/snmp/getpagetimer.php histogramrawlog 1 8
extend pagetimerrawhist9 /etc/snmp/getpagetimer.php histogramrawlog 1 9
extend pagetimerrawhist10 /etc/snmp/getpagetimer.php histogramrawlog 1 10
extend pagetimertimes0 /etc/snmp/getpagetimer.php pagetimes 80
extend pagetimertimes1 /etc/snmp/getpagetimer.php pagetimes 95
extend pagetimertimes2 /etc/snmp/getpagetimer.php pagetimes 100
extend pagetimertimes /etc/snmp/getpagetimer.php pagetimes 80 95 100

Note that this uses the log histograms which are normally more useful, but you can also use linear timing.

There are three modes:

filters

This mode lists the filter names (more about them in a mement) common to all. Filters are simply configurations that allow you to filter and isolate specific data. The filters are used as the index for the Cacti graphs so you can view any set of data from a filter with any of the graph types.

This does not take any furhter arugments.

histograms

This mode produces a banded graph showing what proportion of pages are being served at what speed. There are four histogram types:

• histogram - basic linear histogram by proportion (%) of pages
• histogramraw - as above but unscaled so shows the number of pages
• histogramlog - histogram with log scaling to give a wider range of times by proportion (%) of pages
• histogramrawlog - as above but unscaled so shows the number of pages

These all take a further argument of the reference (maximum) time for the histogram, and a last argument which is the band number (0-9 and 10 for everything above) or levels which lists the time bands.

Generally the log histograms will be used which is what the snmp.conf entries do.

pagetimes

This is a different view of the problem and allows you to specify the proportions of the pages you want times for.

eg. 80 means best time that 80% of the pages are delivered, 100 means the time that all (100%) pages are delivered in.

If more than one argument is given then it assumes it's listing the bands for graph labels.

Caching & Pruning

The script will automatically cache data and prune the database entries. By default it leaves 30 minutes (1800 seconds) of data and caches for 2 minutes (120 seconds) which can be changed to suit your traffic levels and polling intervals within the script.

Over to Cacti

There is a data query .xml file that is intended to go in /usr/local/share/cacti/resources/data_queries or wherever is appropriate for unpackaged files on your system (modify the template accordingly).

Download: LAMP Timer Cacti Data Query

Then add the Cacti Template.

Download: LAMP Timer Cacti Template

You can then add the data query to the host running snmpd and it should pick up the filters (by default only one called allPages) and allow you to add the different graph types.

More on Filters

The snmp etension will look not only for /etc/pagetimer/dbconfig.php to get the database credentials, but also for /etc/pagetimer/filters.php which contains an array for looking at specific times.

This file may look something like this:

<?php
$filters = array (
        'Home' => array (
                'table'=>'Pages',
                'condition' => array (
                                'pagesql' => 'Request = :request',
                                'params' => array ( ':request'=>'/' )
                        )
                ),
        'somearea' => array (
                'table'=>'Pages',
                'condition' => array (
                                'pagesql' => '( Request = :request OR Request LIKE :belowreq ) AND NOT Request = :nrequest',
                                'params' => array ( ':request'=>'/questions/', 'belowreq'=>'/somearea/%', ':nrequest'=>'/somearea/not_here/' )
                        )
                ),

        'Milestone Home' => array (
                'table'=>'Milestones',
                'condition' => array (
                                'pagesql' => 'Request = :request',
                                'milestonesql' => 'MilestoneName = :milestonename',
                                'params' => array ( ':request'=>'/', ':milestonename'=>'formprocessed' )
                        )
                ),
        'Milestone somearea' => array (
                'table'=>'Milestones',
                'condition' => array (
                                'pagesql' => '( Request = :request OR Request LIKE :question ) AND NOT Request = :nrequest',
                                'milestonesql' => 'MilestoneName = :milestonename',
                                'params' => array ( ':request'=>'/questions/', 'belowreq'=>'/somearea/%', ':nrequest'=>'/somearea/not_here/', ':milestonename'=>'thinkingdone' )
                        )
                ),
?>

Essentially it's an array where the filter names (no spaces) reference another array with the filter in it. The array has a 'table' (Pages or Milestones) and the filter 'condition' which may contain:

• pagesql - SQL after "WHERE" to filter to a specific entry in the Pages table. This is needed both for Pages and Milestone filters.
• milestonesql - SQL after "WHERE" to filter to a specific entry in the Milestones table. This is only used for Milestone filters.
• params - is the parameters used in the SQL above. As quotes are not allowed (for security) in the SQL all parameters have to be done this way to ensure they are properly escaped.

The allPages filter is automatically added by the script so if no filters are specified then all the pages that are timed will be graphed by this.

The Graphs

The basic (proportional) histogram looks like this:

That may look bad in places - significan proportions of pages being delivered >1second, but the corrisponding raw histogram looks like this:

That looks much more healthy - really only a significan proportion of pages are being delivered really slow at minimal traffic times which probably means that caches are expired etc. so not a big deal. There is however scope for some improvement at peak times.

The other way to look at it is by the time taken to serve a proportion of pages:

Again we see that 80% of monitored pages are delivered <250ms and 95% of monitored pages <400ms with occasional peaks up to just over 1 second. The past 5% of pages can peak up to 1 second and occasionally up to or over 2 seconds.

Each of these can be put against any filter to make the graph specific to a page or a milestone.