Story of Memcached

One MemcacheClass to rule them all, One MemcacheClass to find them,
One MemcacheClass to bring them all and in the RAMs bind them.

Below is a story I found on memcached. It is an amazing story how memcache work.

Two plucky adventurers, Programmer and Sysadmin, set out on a journey. Together they make websites. Websites with webservers and databases. Users from all over the Internet talk to the webservers and ask them to make pages for them. The webservers ask the databases for junk they need to make the pages. Programmer codes, Sysadmin adds webservers and database servers.

One day the Sysadmin realizes that their database is sick! It’s spewing bile and red stuff all over! Sysadmin declares it has a fever, a load average of 20! Programmer asks Sysadmin, “well, what can we do?” Sysadmin says, “I heard about this great thing called memcached. It really helped livejournal!” “Okay, let’s try it!” says the Programmer.

Our plucky Sysadmin eyes his webservers, of which he has six. He decides to use three of them to run the ‘memcached’ server. Sysadmin adds a gigabyte of ram to each webserver, and starts up memcached with a limit of 1 gigabyte each. So he has three memcached instances, each can hold up to 1 gigabyte of data. So the Programmer and the Sysadmin step back and behold their glorious memcached!

“So now what?” they say, “it’s not DOING anything!” The memcacheds aren’t talking to anything and they certainly don’t have any data. And NOW their database has a load of 25!

Our adventurous Programmer grabs the pecl/memcache client library manual, which the plucky Sysadmin has helpfully installed on all SIX webservers. “Never fear!” he says. “I’ve got an idea!” He takes the IP addresses and port numbers of the THREE memcacheds and adds them to an array in php.

$MEMCACHE_SERVERS = array(
"10.1.1.1", //web1
"10.1.1.2", //web2
"10.1.1.3", //web3
);

Then he makes an object, which he cleverly calls ‘$memcache’.

$memcache = new Memcache();
foreach($MEMCACHE_SERVERS as $server){
$memcache->addServer ( $server );
}

Now Programmer thinks. He thinks and thinks and thinks. “I know!” he says. “There’s this thing on the front page that runs SELECT * FROM hugetable WHERE timestamp > lastweek ORDER BY timestamp ASC LIMIT 50000; and it takes five seconds!” “Let’s put it in memcached,” he says. So he wraps his code for the SELECT and uses his $memcache object. His code asks:

Are the results of this select in memcache? If not, run the query, take the results, and PUT it in memcache! Like so:

$huge_data_for_front_page = $memcache->get("huge_data_for_front_page");
if($huge_data_for_front_page === false){
$huge_data_for_front_page = array();
$sql = "SELECT * FROM hugetable WHERE timestamp > lastweek ORDER BY timestamp ASC LIMIT 50000";
$res = mysql_query($sql, $mysql_connection);
while($rec = mysql_fetch_assoc($res)){
$huge_data_for_frong_page[] = $rec;
}
// cache for 10 minutes
$memcache->set("huge_data_for_front_page", $huge_data_for_front_page, 0, 600);
}

// use $huge_data_for_front_page how you please

Programmer pushes code. Sysadmin sweats. BAM! DB load is down to 10! The website is pretty fast now. So now, the Sysadmin puzzles, “What the HELL just happened!?” “I put graphs on my memcacheds! I used cacti, and this is what I see! I see traffic to one memcached, but I made three :(.” So, the Sysadmin quickly learns the ascii protocol and telnets to port 11211 on each memcached and asks it:

Hey, ‘get huge_data_for_front_page’ are you there?

The first memcached does not answer…

The second memcached does not answer…

The third memcached, however, spits back a huge glob of crap into his telnet session! There’s the data! Only once memcached has the key that the Programmer cached!

Puzzled, he asks on the mailing list. They all respond in unison, “It’s a distributed cache! That’s what it does!” But what does that mean? Still confused, and a little scared for his life, the Sysadmin asks the Programmer to cache a few more things. “Let’s see what happens. We’re curious folk. We can figure this one out,” says the Sysadmin.

“Well, there is another query that is not slow, but is run 100 times per second. Maybe that would help,” says the Programmer. So he wraps that up like he did before. Sure enough, the server loads drops to 8!

So the Programmer codes more and more things get cached. He uses new techniques. “I found them on the list and the faq! What nice blokes,” he says. The DB load drops; 7, 5, 3, 2, 1!

“Okay,” says the Sysadmin, “let’s try again.” Now he looks at the graphs. ALL of the memcacheds are running! All of them are getting requests! This is great! They’re all used!

So again, he takes keys that the Programmer uses and looks for them on his memcached servers. ‘get this_key’ ‘get that_key’ But each time he does this, he only finds each key on one memcached! Now WHY would you do this, he thinks? And he puzzles all night. That’s silly! Don’t you want the keys to be on all memcacheds?

“But wait”, he thinks “I gave each memcached 1 gigabyte of memory, and that means, in total, I can cache three gigabytes of my database, instead of just ONE! Oh man, this is great,” he thinks. “This’ll save me a ton of cash. Brad Fitzpatrick, I love your ass!”

“But hmm, the next problem, and this one’s a puzzler, this webserver right here, this one runing memcached it’s old, it’s sick and needs to be upgraded. But in order to do that I have to take it offline! What will happen to my poor memcache cluster? Eh, let’s find out,” he says, and he shuts down the box. Now he looks at his graphs. “Oh noes, the DB load, it’s gone up in stride! The load isn’t one, it’s now two. Hmm, but still tolerable. All of the other memcacheds are still getting traffic. This ain’t so bad. Just a few cache misses, and I’m almost done with my work. So he turns the machine back on, and puts memcached back to work. After a few minutes, the DB load drops again back down to 1, where it should always be.

“The cache restored itself! I get it now. If it’s not available it just means a few of my requests get missed. But it’s not enough to kill me. That’s pretty sweet.”

So, the Programmer and Sysadmin continue to build websites. They continue to cache. When they have questions, they ask the mailing list or read the faq again. They watch their graphs. And all live happily ever after.

Author: Dormando via IRC. Edited by Brian Moon for fun. Further fun editing by Emufarmers.

This story has been illustrated by the online comic TOBlender.com.

Chinese translation by Wei Liu.

Source: http://code.google.com/p/memcached/wiki/TutorialCachingStory

Advertisements

PHP Data Object: How to

With reference to previous post on PDO. Here is a small tutorial and explanation for connecting to database using PDO object. I recommend using PDO as it is more structured way to connect to database rather than doing it by self plus the general functions like mysql_connect are to be deprecated in upcoming versions of PHP. There is another class called mysqli, if you are using mysql you can use it but again if in case you have to change your database, PDO is best.

Required: PHP 5.0+

Using:

<?php
$user = 'user';
$pass = 'mypass';

$color = 'red';

/** PDO is a class for php data object in php. First argument takes the 
  "typeofdatabase:host=localhost;dbname=db"
* typeofdatabase here is mysql, which can postgre, mssql etc.
* host is in general localhost or you use user with which you connect to 
  database.
* dbname is the name of database you want to connect with.
* return the object of the database connection.
**/
$dbh = new PDO('mysql:host=localhost;dbname=test', $user, $pass);

/** Begin transaction, which is very general term in database. Actually 
   any of CRUD is called transaction.
**/
$dbh->beginTransaction();

$query = "SELECT name FROM fruit WHERE color = ?";    // Query

/** Prepares the query/statement for execution and return the 
   object of statement.
**/
$sth = $dbh->prepare($query);

/** Binds the variables to the prepared query
* first argument takes the position of question mark to which 
  the variable has to be bound.
* second argument is the variable which is to be bound.
* type of variable in the query. Like here the color is string, 
  it can be int etc. and for it there are predefined constants. 
  Its optional.
**/
$sth->bindParam(1, $color, PDO::PARAM_STR);

$sth->execute();     //Query execution.

/** Fetch method is used to fetch each row. There are other methods 
  also like fetchAll
* it takes predefined constant in PDO as argument, FETCH_BOTH returns 
  the row with number and name indexes. Like $result['name'] and $result[0].
* it returns the row array.
**/
while($result = $sth->fetch(PDO::FETCH_BOTH)){
     echo $result['name'];
}

?>

Explanation:

Most of the things which are used in general are explained in the code snippet. But I would still recommend PHP: PDO Manual. Look at the other methods and predefined constants, specially understand the difference between bindParam method and bindValue method. For the record bindParam is better way because it binds the variable at the time of execution of query.

Look at mysqli also for a method called bind_result. Which is quite good, you can extend this PDO class and include this method in it.

Hope you will find this post useful.

PDO: PHP Data Object

There are different ways of connecting to database in PHP. The most general way in use is MySQL extension, which we have discussed here and even everywhere. Well there are two other ways of connecting to database. First one is MySQLi and other is PDO.

Now, question is what is the difference? So, the difference is MySQLi is MySQL improved. As name states, it is improved version of the MySQL extension. It uses both direct function which we use in MySQL extension and the object oriented way of implementing the same. So if I would be using MySQLi, I would choose OOP way. But MySQLi extension is only for MySQL. It is more secured if we use it in OOP way like it eliminates the threat of sql injection.

PDO is more generic because through it you can connect to different type of database by changing just the name of the database you want to connect. You can connect to MySQL, MS SQL, PostgreSQL, SQLite and many more. So it is more generalized, as you will not be needed to change the function call and other things. All you have to do is change the connector name, this is the term which is used for software that connects application to database. It is also based on the OOPs the concepts, which makes it easy to use.

PDO and MySQLi have very much the same way of implementation. Moreover MySQL extension is being maintained and can be deprecated in near future. While the other two are in active development.

Next time we’ll be learning implementation using PDO. Its very easy, meanwhile if you want to look it by yourself, here are the links. PDO and MySQLi.

JSON – Javascript Object Notation

There were days when we used to create and use XML for transporting and store data and it was good at it. But I consider there was a problem with it getting data from it, I feel nausea. I used ‘was’ here because I also consider it as the way our daddies used to transfer data. Now is the day of JSON.

JSON is cool, easy, requires less memory(though we are not bothered). You can use it with any language and I love it. Frankly, I gave up XML before learning it because of JSON.

JSON looks like array we make in C\C++ or any other language. As we are familiar with the array, it makes understanding of JSON easy. It looks like :

{'data' : ['J','S','O','N'], 'this' : { 'is' : 'json' } };

Let’s use it with PHP and Javascript.

PHP:

$array = new array('data'=>array('J','S','O','N'), 'this'=>'json');
$json = json_encode($array);
$json = json_decode($json);
echo $json->data[0];
//output: J
echo $json->this;
//output: json

Javascript:

var json = {'data' : ['J','S','O','N'], 'this' : { 'is' : 'json' } };
alert(json['data'][0]);
//output: J
alert(json->data[0])
//output: J
alert(json->this);
//output: json
alert(json['this'])
//output: json

We have studying here for PHP and Javscript as it is used mostly for web development. Go to json.org for better understanding and using with other languages.

Configuring SSL on Apache

Our local server run on HTTP protocol. Then how will you work with facebook APIs where you need secured protocol now to work with APIs. Well its very easy to configure WAMP for ssl. Follow the steps below and you are done with it.

  1.  Left click on wamp icon in taskbar in windows and go to apache->apache modules and select ssl_module.
  2. Left click on wamp icon again. php->php extensions. Select php_openssl.
  3. Most important restart your wamp and you are done.
For those who want to configure by editing file do the following.
  1. Open httpd.conf in apache->conf folder.
  2. Uncomment the line saying or write ‘LoadModule ssl_module modules/mod_ssl.so’ minus the quotes(‘).
  3. Uncomment the line php.ini file in apache folder ‘extension=php_openssl.dll’ minus quotes(‘).
  4. Save  the doument and restart apache.
Configuring php for ssl is done by editing php.ini.
  1. Open the php.ini in PHP folder.
  2. Uncomment the line ‘extension=extension=php_openssl.dll’.
  3. Save it restart the server.

Fetching from database using PHP

If you have database, then of course you will need to fetch data out of it. Four methods are described below to fetch from database and whats the difference between them.

Required: PHP, MySQL.
Using:

SQL table

table1(
name varchar(20)
)

PHP query:

$query = "SELECT name FROM table1";
$rows = mysql_query($query);

PHP fetch functions:

$row = mysql_fetch_array($rows, $result_type);
$row = mysql_fetch_assoc($rows);
$row = mysql_fetch_row($rows);
$row = mysql_fetch_object($rows, 'class', $params);
Note: All parameters are not defined here but are explained below.
Explanation:

Methods to fetch are described below:

  1. mysql_fetch_array($rows, $result_type): First argument, $rows is same as we have described earlier. Second argument is the result type which has three values MYSQL_ASSOC, MYSQL_NUM, MYSQL_BOTH, which is default and this argument is optional.
    • MYSQL_ASSOC is used when we want to use associative key or table name for fetching. So, if we want to echo result it would be like
      echo $row[‘name’];
    • MYSQL_NUM is used if we are going to use numbers as associative keys also called index. It would go like
      echo $row[0].
    • MYSQL_BOTH is used if you don’t know what you are going to use. With this way you can use both key and index.
  2. This is just the same as mysql_fetch_array() with result type argument as MYSQL_ASSOC. Syntax will be
    $row[‘name’];
  3. It is identical to mysql_fetch_array() with result type as MYSQL_NUM. Syntax will be
    echo $row[0];
  4. mysql_fetch_object($rows, ‘class’, $params): This is same as mysql_fetch_assoc() but instead of array this returns object of class ‘class’. That is it can be accessed by only with column name in table. Syntax would be:
    echo class->name;
    The last argument, $params, is used when parameters are needed to be passed in constructor of ‘class’, it has to be array. Also $params is optional.

Note: All the fetch function are almost equivalent in terms of performance.

         Also see PHP manual on these function.