Factorize $5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc$.

Factorize $5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc$.

My solution:

The given expression is written so neatly and beautifully at first glance, it's like it couldn't be factored. And we feel so reluctant to expand the second product, but we have to, if we want to factor the expression correctly, we have to get a clearer picture of what this expression is all about by expanding and then rearranging terms in decreasing order:

$5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc$

$=5a^3+5b^3+5c^3-3(a^3+a^2b+a^2c+ab^2+b^3+b^2c+ac^2+bc^2+c^3)+12abc$

$=2a^3+2b^3+2c^3-3(ab(a+b)+c(a^2+b^2)+c^2(a+b))+12abc$

Tough nut like this is hard to crack, that is for certain. But, we got to have a plan, we got to plan for a strategy in order to proceed, and our strategy is to use the substitution.

First, let's substitute $a+b$ by $x$.

We work separately to get the relation between $a^3+b^3$ and $a^2+b^2$ with $x$:

$a+b=x$

$(a+b)^2=x^2$

$a^2+2ab+b^2=x^2$(*)

and

$(a+b)^3=x^3$

$a^3+3ab(a+b)+b^3=x^3$

$a^3+3abx+b^3=x^3$(**)

We now replace (*) and (**) into 

$5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc$

and obtain:

$5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc$

$=2(a^3+b^3)+2c^3-3(ab(a+b)+c(a^2+b^2)+c^2(a+b))+12abc$

$=2(x^3-3abx)+2c^3-3(abx+c(x^2-2ab)+c^2x)+12abc$

$=2x^3-6abx+2c^3-3abx-3cx^2+6abc-3c^2x+12abc$

$=2x^3-3cx^2-3c^2x+2c^3-9abx+18abc$

$=2x^3-3cx^2-3c^2x+2c^3+9ab(2c-x)$

Now, if we can prove set $(2c-x)$ is a factor for $2x^3-3cx^2-3c^2x+2c^3=0$, that would suggest
$5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc=2x^3-3cx^2-3c^2x+2c^3+9ab(2c-x)$ has a factor of $a+b-2c$.

Let's now determine if $2c-x$ is a factor for $2x^3-3cx^2-3c^2x+2c^3$:

$\begin{align*}2x^3-3cx^2-3c^2x+2c^3&=2(2c)^3-3c(2c)^2-3c^2(2c)+2c^3\\&=16c^3-12c^3-6c^3+2c^3\\&=0\end{align*}$

So we've found one factor for $5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc$, that is  $a+b-2c$.

It must be true that the other two $a+c-2b$ and $2a-b-c$ are the remaining factors for $5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc$, let's verify it:

$(a+b-2c)(a+c-2b)(2a-b-c)$

$=(a^2+ac-2ab+ab+bc-2b^2-2ac-2c^2+4bc)(2a-b-c)$

$=(a^2-2b^2-2c^2-ac-ab+5bc)(2a-b-c)$

$=2a^3-a^2b-a^2c-4ab^2++2b^3+2b^2c-4ac^2+2bc^2+2c^3$

$\,\,\,\,\,\,\,\,-2a^2c+abc+ac^2-2a^2b+ab^2+abc+10abc-5b^2c-5bc^2$

$=2a^3+2b^3+2c^3-3a^2b-3ab^2-3b^2c-3bc^2-3a^2c-3ac^2+12abc$

$=2a^3+2b^3+2c^3-3a^2b-3ab^2-3b^2c-3bc^2-3a^2c-3ac^2+12abc$

$=5a^3+5b^3+5c^3-3(a^3+a^2b+a^2c+ab^2+b^3+b^2c+ac^2+bc^2+c^3)+12abc$

$=5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc$

Therefore, $5(a^3+b^3+c^3)-3(a^2+b^2+c^2)(a+b+c)+12abc=(a+b-2c)(a+c-2b)(2a-b-c)$.