What is the difference between controller and extension in visualforce pages in salesforce ?

In the context of visualforce, to take advantage of the built-in features of Visualforce to access your org's data, you must use a controller. All controllers are Apex classes, either standard Apex classes (ApexPages.StandardController or ApexPages.StandardSetController) or your own custom Apex class. You can also extend functionality of any of these with an Extension class.

I tend to be a bit picky with my terminology here. Controllers are controllers. Extensions are not controllers. And for me, the difference is that extensions cannot live by themselves on a page. In other words, I'm not permitted to use extensions, unless my page has a controller in the first place.

Another distinguishing feature is that a page is allowed only one controller ever. But I can add as many extensions as I need.

So let me begin (what has become a dissertation) with controllers:

Controllers are attached to your page using one of three mutually exclusive formats, standard controller, standard list/set controller, custom controller.

For a page with a single record, accessing standard data and behavior of your object you would use standard controller:

<apex:page standardController="Account"...> ... </apex:page>
If you want a page with many records, using standard list view, and list actions you could use standard list controller:

<apex:page standardController="Account" recordsetVar="accountsList"...> ... </apex:page>
If you want to use completely custom functionality and don't really need most of the standard object behaviors, custom controller is a good option, but bear in mind, any standard actions you want to use will have to be implemented in your custom class. A page with a custom controller would use the following attribute:

<apex:page controller="MyApexClass"...> ... </apex:page>
But bear in mind, you yourself can instantiate ApexPages.StandardController to use its features and functionality. I often use the pattern below to instantiate standard controller to get access to its methods:

public with sharing class MyApexClass {
private ApexPages.StandardController ctrl;
public Account acct {get;set;}
public MyApexClass(){
try{
acct = [select Name from Account
where Id = : ApexPages.currentPage().getParameters().get('id')];
} catch (QueryException e) {
acct = new Account();
}
ctrl = new ApexPages.StandardController(acct);
}
public PageReference save(){
//here I can put additional functionality or just leave it as-is.
return ctrl.save();
}
}
The above instantiates Standard Controller privately, but then surfaces a public save method that is just a pass-through of the controller's save method. The constructor does the housekeeping of creating the actual record I'm working with.

If I want to add to the functionality of any controller, I can then use an extension. Extensions work with either standard or custom controllers. Standard controller with custom extension, for me, is the most common construct I use for Visualforce that will operate inside my org. I get all the power of the standard controller, but all the flexibility of Apex. The syntax for an extension with a standard controller page would be as follows:

<apex:page standardController="account" extensions="MyExtClass"...> ... </apex:page>
The extension class is then required to implement a constructor that accepts the type of the controller. You will then typically want to get a handle to the standard controller instance that is floating around, and the sObject instance that it represents. So for a standard controller page:

public with sharing class MyExtClass {
private ApexPages.StandardController ctrl;
public Account acct {get;set;}
public MyExtClass(ApexPages.StandardController controllerParam){
ctrl = controllerParam;
acct = (Account) ctrl.getRecord();
}
public PageReference customsave(){
//here I override my standard save functionality before I return the save call.
return ctrl.save();
}
}
With a custom controller and extension, you can probably extrapolate from the above, but just to close the loop.

<apex:page controller="MyApexClass" extensions="MyExtClass"...> ... </apex:page>
Again, the extension class implements a constructor that accepts the type of the controller, only now it is my custom class. From there, apart from getting a local handle to your custom controller instance, there isn't a pattern, per-se, but the local handle allows you to interact directly with your custom controller, in whatever way necessary, of course. :

public with sharing class MyExtClass {
private MyApexClass ctrl;
public MyExtClass(MyApexClass controllerParam){
ctrl = controllerParam;
}
}