Concave means to "bulge inwards".

In mathematics this adjective is usually used when referring to a polygon (i.e. a concave polygon).