How can I load a page template from a plugin?

I want to add page templates to a theme directly from the plugin. The idea is that the template would show up in the dropdown under Page Attributes, and all the code would need to be in the plugin.

Any tips on how to achieve that ?

You can use the theme_page_templates filter to add templates to the dropdown list of page templates like this:

function wpse255804_add_page_template ($templates) {
    $templates['my-custom-template.php'] = 'My Template';
    return $templates;
    }
add_filter ('theme_page_templates', 'wpse255804_add_page_template');

Now WP will be searching for my-custom-template.php in the theme directory, so you will have to redirect that to your plugin directory by using the page_template filter like this:

function wpse255804_redirect_page_template ($template) {
    if ('my-custom-template.php' == basename ($template))
        $template = WP_PLUGIN_DIR . '/mypluginname/my-custom-template.php';
    return $template;
    }
add_filter ('page_template', 'wpse255804_redirect_page_template');

Read more about this here: Add custom template page programmatically