Preheat the oven to 325°F and position the rack to the center. Spray a 3 quart (9x13 inch) casserole dish with cooking spray or softened butter and set aside.
Melt the butter in a large stock pot or dutch oven over medium heat then add the thinly sliced onion and cook for 10 minutes to soften. Turn the heat down to medium-low and add the wine and cook for another 5 more, stirring frequently, until the onions are caramelized and dark amber color. Add more wine as needed as they cook down. Add the minced garlic and cook for another minute, until fragrant, then remove the caramelized onions to a clean plate or cutting board.
While the onions are cooking, prep the potatoes to save time. Peel the potatoes first if you’re using russet potatoes. You can leave the skin on Yukon gold potatoes for a more rustic look if you prefer. Slice the potatoes thinly, into 1/4 inch - 1/8 inch slices using a mandolin slicer. You can use a sharp knife instead if you don’t have a mandoline, just be sure to get them very thin and similar size.
When the onions are done and you’ve removed them from the pot, add the milk and cream over medium heat and use a spoon or spatula to scrape any extra brown bits off the bottom of the pan from the onions. Add the sliced raw potatoes with the salt, pepper, thyme, and nutmeg and turn the heat up to medium-high to bring them to a boil, then reduce to medium heat and simmer for 10 minutes. Remove them from the heat and let them cool slightly so they’re easier to handle. Boiling the milk and potatoes slightly helps the starches to release from the potatoes for a creamier sauce and it also partially cooks the potatoes so they’re more tender in less amount of time.
Layer 1: Pour a bit of the milk mixture into the bottom of the greased dish then layer in one-third of the potatoes. Spoon a third of the boiled milk over top then sprinkle with 3/4 cup of gruyere cheese and half of the caramelized onions.
Layer 2: repeat layer one with another third of the potatoes and milk mixture then another 3/4 cup of gruyere cheese and caramelized onion.
Layer 3: finish with the remaining potatoes and milk, saving the cheddar cheese for later.
Cover with foil and bake for an hour and a half then remove foil and sprinkle with 1/2 cup of cheddar cheese and the remaining gruyere cheese and bake for another 30 mins, until the cheese is golden and melted. When it’s done, the potatoes should be tender, not mushy and liquid should be mostly absorbed. Allow the potatoes to cool for at least 30 minutes before serving.