Suppose we have a surface $S$ and that ${\bf n}$ is a unit vector normal to $S$ at a point ${\bf x} \in S$. There are many common transformations we can perform on $S$: rotation, scaling, shear etc. After we transform $S$ into a new surface $S'$, the point ${\bf x}$ from $S$ will end up at a new position ${\bf x}'$ in $S'$. This post answers the following question: when we apply an invertible linear transformation to all points of $S$, what is the relation between ${\bf n}$ and the corresponding ${\bf n}'$ which is a unit vector normal to $S'$ at ${\bf x}'$?

We will illustrate our points in this post using a one-dimensional surface on a two-dimensional plane, but everything here applies to any surface on an $n$-dimensional space which undergoes an invertible linear transformation $M$. A natural but incorrect assumption would be that ${\bf n}$ is converted to ${\bf n}' = M{\bf n}$ since every point ${\bf x}$ in $S$ is converted to ${\bf x}' = M{\bf x}$ in $S'$. In fact, as we will show, ${\bf n}'$ is actually parallel to $(M^{-1})^T{\bf n}$, which is in general not parallel to $M{\bf n}$.

Consider first what happens when we apply a shear transformation to a surface $S$ in a two-dimensional space, with the transformation matrix $M$ having the following form (here we assume $a \gt 0$): $$ M = \left(\begin{matrix} 1 & a \\ 0 & 1 \end{matrix}\right) \label{post_a2386030d8d82e457a8c7dc124d00564_shear_matrix} $$ A point ${\bf x}$ in $S$ is transformed into ${\bf x}' = M{\bf x}$: $$ {\bf x} \;\longrightarrow\; {\bf x}' = M{\bf x} = \left(\begin{matrix} x + ay \\ y \end{matrix}\right) $$ The transformation $M$ preserves the $y$ coordinates of all points in $S$, but moves the $x$ coordinates by $ay$. Figure 1 shows an example of how a unit square is transformed by $M$.

A normal vector ${\bf n}_{\textrm{t}}$ on the top edge of $S$ has coordinates $(0,1)$, and $M{\bf n}_{\textrm{t}} = (a,1)$, which is not parallel to ${\bf n}_{\textrm{t}}'$ since ${\bf n}_{\textrm{t}}' = {\bf n}_{\textrm{t}}$ and $a \neq 0$. Also, ${\bf n}_{\textrm{r}}$ on the right edge has coordinates $(1,0)$, $M{\bf n}_{\textrm{r}} = (1,0) = {\bf n}_{\textrm{r}}$, which is not parallel to ${\bf n}_{\textrm{r}}'$ because ${\bf n}_{\textrm{r}}' \neq {\bf n}_{\textrm{r}}$. In other words, surface normals do not in general transform like points from $S$, so we cannot just take a normal vector ${\bf n}$ and expect that $M{\bf n}$ has the same direction as the corresponding normal ${\bf n}'$ in $S'$.

Fortunately, finding an expression for ${\bf n}'$ in terms of ${\bf n}$ and $M$ is easy. All we need to do is use the fact that tangent vectors in $S$ are transformed into tangent vectors in $S'$. In fact, consider a tangent vector ${\bf t}$ connecting ${\bf x}$ and an infinitesimally close point $\tilde{\bf x} = {\bf x} + \Delta{\bf x}$ from $S$, i.e., ${\bf t} = \tilde{\bf x} - {\bf x} = \Delta{\bf x}$ (${\bf t}$ is not a unit vector, but we only care about its direction here). Since ${\bf x}$ and $\tilde{\bf x}$ are converted into ${\bf x}' = M{\bf x}$ and $\tilde{\bf x}' = M({\bf x} + \Delta{\bf x}) = {\bf x}' + M\Delta{\bf x}$ respectively, the infinitesimal tangent vector ${\bf t}'$ connecting ${\bf x}'$ and $\tilde{\bf x}'$ is then: $$ {\bf t}' = \tilde{\bf x}' - {\bf x}' = M{\Delta{\bf x}} = M{\bf t} $$ In other words, an infinitesimal vector ${\bf t}$ tangent to ${\bf x} \in S$ is transformed into an infinitesimal vector ${\bf t}' = M{\bf t}$ which is tangent to ${\bf x}' \in S'$. As a matter of fact, since $S$ undergoes a linear transformation, the argument applies even if ${\bf t}$ is not an infinitesimal vector, in which case ${\bf t}' = M{\bf t}$ is also not infinitesimal. Representing vectors as columns, we have that: $$ {\bf t}\cdot{\bf n} = {\bf t}^T{\bf n} = 0 \label{post_a2386030d8d82e457a8c7dc124d00564_dotp1} $$ because ${\bf t}$ and ${\bf n}$ are orthogonal. Equivalently, we also have that: $$ {\bf t}' \cdot {\bf n}' = (M{\bf t})^T {\bf n}' = 0 \label{post_a2386030d8d82e457a8c7dc124d00564_dotp2} $$ Equation \eqref{post_a2386030d8d82e457a8c7dc124d00564_dotp2} is satisfied if we take ${\bf n}' = (M^T)^{-1}{\bf n}$. Indeed: $$ (M{\bf t})^T {\bf n}' = (M{\bf t})^T (M^T)^{-1}{\bf n} = {\bf t}^T M^T (M^T)^{-1}{\bf n} = {\bf t}^T{\bf n} = 0 $$ Therefore, we have that $(M^T)^{-1}{\bf n}$ is normal to the surface $S'$ at ${\bf x}'$. All we need to do now is normalize it: $$ \boxed{ {\bf n}' = (M^T)^{-1}{\bf n} \; \big/ \; \big\| (M^T)^{-1}{\bf n} \big\| } $$ It is important to say, however, that if $M$ is an orthogonal matrix (e.g. a rotation), then $(M^T)^{-1} = M$ and $\|M{\bf n}\| = \|{\bf n}\| = 1$, so in this case surface normal vectors transform just like regular points from $S$.