My 2 penny worth:

What gauge of wire are you using in your setup? I've faced a similar problem when I had to use 18 gauge wires and everything was heating up. Once I switched to a thicker 14 gauge, things were 'cool'. Of course, this was on brushed set up and I remember raising a similar query on the forum on my Mermaid build.

@kewal ... From what I know a ESC of an higher rating is always better ... As it does not cause any  damage .. But a lower rated ESC will definitely burn if you load it excessively ..Seniors ... Correct me if am wrong

@VC Sir ... There are no extensions used all the wires are the one that came with the stock motor and ESC ...
By any chance does ventilation play a major part ?

@K K Iyer ... The motor is Emax GF2215/20 prop is 10*6 ....
Does higher pitch also overload the motor ?

yes higher pitch overload the motor. 10x6 is not recommended for this motor, that is why your setup is getting hot. Recommended props are, 10x4.7
and 10x5

sory my frnd u r wrong. Do u know y kV, resistance ohms and  watts of motor is mentioned in there specification? Bro it is not useless information. With that info we can make good electronic combination so there is no need to see on chart for combination. Bro for ur kind info I strated motor winding experiment wen I was 8 year old.

Have u program ur ESE for ur motor or u kept default?

@kewal .. I have high regards for your knowledge... No doubts about that ...
Its just that the KV .. Resistance and wattage are defiantly mentioned for a reason ... But here I guess u have some confusion coz the current drawn by a motor on a specific load is mentioned that is for a particular prop the current drawn by a motor is fixed you can use any ESC greater then that max current drawn ...
Like if the maximum current drawn by a motor is 20 amps you can use an ESC with 25 amps or 30 amps or 35 amps with out any problems in the electric configuration ...
( except it will increase the  AUW )

@kewal just for your reference
....
http://www.modelairplanenews.com/blog/2011/05/29/choosing-a-motor-speed-control-battery-pack-taking-the-mystery-out/

bro for simple example i m giving u how to check esc amps for motor which can run on 11.1 volt. jut devide watts given.

Motor: Avionic M2028/12 KV2300 MICRO brushless motor
KV (rpm/v): 2300
Power: 65W
volt 7.4

now ese will be = 65/7.4 = 8.5
so 10  amps ese is much suitable than any other ese.
 

in other example


rc bazzar has listed some motor 1of them has as following specification

Motor: Avionic M1818/17 KV4500 MICRO brushless motor
KV (rpm/v): 4500
Power: 30W
Winds: 17
Resistance: 263 mOhm
Idle current: 0.8 A
Weight: 9gms

Combination of usage:
PROP - 6x4 E
● Get  300 gms Thrust
Lipo - 2 cell 7.4V
ESC - 20 amp



now see the are using 20 amps ese

if u calculate 20*7.4 is 148 watts
which is greater than the watts given in detail.
it can burn ur motor or batt.


i have same motor cobination with 6 amps ese. it works great with no heating stuff.

bro this easy tricks for u bignners to select ur combination and save ur electronics.

Can you explain why you feel docnayeem is wrong and using a higher amps ESC is a problem ?  You have said nothing in way of "why" except to point to specs and talk about your experience.  What is the technical reason why a higher amps ESC would be a problem ?

PS: Please take extra 10 to 15 seconds while posting to write proper English and DO NOT USE SMS language !

@kewal yes you are right .... But confused ... What you are calculating is the maximum wattage the ESC can support ...
The actual max wattage ur  motor can give is 30W and you are using a 6 amp ESC  so your ESC can support a max wattage of 42 watts that's quite above the actual wattage of your motor that is 30 ...
Even in this example a higher rated ESC is used with out any harm ...
I hope you get the point ...

@docnayeem  battery and ese combination should not exceed to the watts given for ur motor.

sir i have already given simple example for selecting combination. if u want perfect technical reason than it has  big explanation and it should need to take lecture.

battery and ESC combination  should exceed than the watts given for ur motor...... Motor pulls the desired current from esc and battery.

if its not exceeding you may have low performance from motor. There is no harm in having power reservoir for motor. in case if excess current drawn ESC will autocut (choose Esc with such feature).

@kewal ... You seem to be confused again ....
Let me explain
The motor draws a current depending on a load ( the prop ) your ESC  should be able to deliver that current ... Say if your motor draws 30 amps your ESC should be able to give 30 amps of current ... So a 30 amp ESC or a 35 amp ESC or a 40 amp ESC will be able to deliver the required current .... A 40 amp ESC used does not mean its pushing 40 amps current to the motor ... The motor will draw only the required current according to the load .
The battery and ESC combination will decide what max motor it can support ...
I hope your confusion is cleared

I have all the time in the world for you !

You have got your basics (and the simple example) entirely mixed up !

See this post, there are tons like this on this forum.  In fact you should start by reading this thread :

http://www.rcindia.org/electric-power/understanding-electric-power-systems/msg44026/#msg44026

The question is quite simple. Why should I not use a 30A ESC in a place where even a 20A ESC may work just fine ? Please think about the reason.

Kewal your 3D planes will have much more power and punch if you have a bit higher capacity esc and motor.

we all know that the mentioned stats are with +- variations so having higher side is better.

In fact, just brush up the stickied posts at the top of the "Electric Power" board.

http://www.rcindia.org/electric-power/

The normal recommendation is that you should always use an ESC that is slightly higher than what is needed for the motor.

http://www.rcindia.org/electric-power/amp-rating-and-motor-size/msg1566/#msg1566  (see the reference to "Sizing of the ESC: The ESC has to be sized such that it exceeds the maximum rating of the motor so that it can operate all load conditions of the motor.")

http://www.rcindia.org/electric-power/lipo-selection-for-motor/msg34361/#msg34361 (see this post also, and read this whole thread).

sir plz read all post care fully before giving sugetion on my 3d model. Have seen my all modes? I have models from 10 gram to 4000 gm  and 100 km to 2000mm size and all are electric. I have 100 of models design by me.

The proper way to choose an ESC has been established.  The OP was about troubleshooting and the causes for a hot motor.  

What is hot to me might be different to others.  The general guideline is if you can't touch it for more than a second then it's too hot.  A better more accurate method is to measure the temp and use something like this for Rs 600

http://www.hobbyking.com/hobbyking/store/__8297__Turnigy_Infrared_Thermometer_33_180Celsius_.html

For all we know, the temps of your motor and ESC are within specs.  But what we don't know is if you are using too big a prop.

Invest is a watt meter, fairly cheap considering you don't want to burn out an ESC in mid-flight, and you'll see before your eyes the difference in how many watts and amps a 9" prop and a 10" prop will draw.  Once you have the numbers, you will save lots of time from writing and reading these posts. No more guessing.

It can't be any more simple than that.  

kewal kalsaria and rastsaurabh, when you guys hijack the thread and go off in a tangent, it's distracting and not helpful to the OP. 

Hi everyone thanks for all the inputs .... Ventilation was the problem ... Increased the size of ventilation inlets and outlet .... Things are cool now ....

I guess ..Ayub had pointed out the probable cause thanks ...