I'm a geek about this sort of thing so happy to try and help. First of all I wouldn't think of it just as memory, think of it as brain health and performance, with memory as a subset of that.
As Stu touched on sleep, and nutrition are part of the fundamentals and you can add movement to that.
People think of sleep as about physical rest, but it's so much more than that - while you're asleep the brain disposes of metabolic waste and consolidates memory - think of it like a filing system. If your sleep is poor in terms of quality and/or volume, that system gets disrupted.
Neuroplasticity is really important (and reassuring) to understand - it's the brain's ability to learn and adapt. What you need to do is stimulate it through doing new things that will forge new neural pathways - work on a new skill, learn a new language etc.
What Controversial has said is right - far too many people 'narrow' as they get older and just stick to old routines - this is not good for the brain as it requires stimulus.
You can check out the likes of Nicole Vignola (Rewire), Anne-Laure Le Cunff (Tiny Experiments) and Dr Hannah Critchlow (21st Century Brain) and you'll find a ton of good stuff that is evidence-based.
Hope this is useful!
