As long as you can get mono running, it should be possible. Most of the libraries associated with monogame can be ported to Raspbian. (ARM/Linux)
I do have some suggestions, that might help you in this endeavor. For starters, don't rely on the de-facto installation of SDL that comes with Raspbian, or gets installed with the Raspbian package manager. Download the latest version of SDL from the official site, and build it from source yourself. The process is relatively easy, and the latest source version includes much better hardware support for the Raspberry Pi. (in all of its various flavors)
Once you've got that covered, you will probably also want to cook up basic make-file to compile your project using a C#-compatible compiler. Rigging up a make-file will allow you to automate the compilation process, with flags for include folders and library linking and such. Platforms like the Raspberry Pi typically don't have full-featured IDEs like Visual Studio, and even the few that they do aren't often designed to work properly on the Pi's file-system. But make-files are designed to work with it, and will be one of the more head-ache free ways to get a more complex project compiling quickly and painlessly.
Once you have your project compiling properly, you will also want to cook up a quickie shell script for actually running your compiled project. This will allow you to run the project with a single command, instead of having to type out the full line of calling mono and pointing it at your project's compiled executable. Just a little quality-of-life feature to speed things up.
As far as running just the game, I actually have a suggestion for that as well. You will likely want to do most of your development in the Raspbian GUI, and that's fine. Go ahead and install the version of Raspbian that comes with the GUI. It won't stop you from doing what you're proposing down the line. Libraries like Monogame, as well as other systems fueled by SDL2, require a windowing system to access system resources. (like display devices, drivers, input devices, etc...) For SDL2, you are basically limited to either using DirectFB (direct frame-buffer) or X11. (the standard Raspbian windowing system) DirectFB is not installed by default, and requires a little extra set-up. It is also not fully compatible with all of SDL2's features.
I would recommend sticking with X11. For the kind of game-only implementation you are seeking, you just have to run X11 targeted at your particular game, and NOT run the full desktop environment. This can be done, and fairly easily. All you have to do is exit the desktop environment to the CLI, and then run "xinit" targeted at mono, with your executable as an argument for mono. This will initialize the X11 server, but it will only run it against your single program, instead of firing up the entire desktop environment. Basically, it is the best way to run X11 with minimal resources, so all of the xserver will be devoted to your program, with no other windows or applications.