Preheat the oven to 350 degrees F.
Line a 9-inch round cake pan with parchment. Brush the bottom and sides with olive oil.
Place sugar, olive oil, eggs, creme fraiche, and lemon zest in a large bowl and whisk until well combined.
Place a fine mesh sieve directly over the bowl, and sift in dry ingredients (flours, baking powder, baking soda, cardamom, and salt).
Whisk in dry ingredients until well combined.
Pour the cake batter into the prepared cake pan. Smooth the top with an offset spatula.
Place the cake pan in the oven and baked for 15 minutes.
Carefully remove the hot pan from the oven. Top the batter with plum halves. Do not press down.
Sprinkle demerara sugar over the top and almond flakes at the edges.
Return the cake to the oven and bake for 30 minutes until golden brown or a toothpick inserted in the middle comes out clean. Let the cake cool in the pan.
When ready to serve, transfer the cake to a serving plate. Garnish with a generous sprinkle of confectioners' sugar before slicing.