If you have come through the village on the B3443 then you will take a left hand turn to reveal The Counting House, an unmistakable landmark, and should take the next left turn up the hill.
Below you can see it, approaching from the other direction, with The Counting House on the left of the picture, and a clear view of the little lane up the hill that you need to take.
This is also what you would see if approaching on the B3443 from the Ysbyty Ystwyth direction, just after you pass the Pont-Rhyd-Y-Groes village sign.