Download the source code for this post (3k)
Greetings from Seattle! This is the start of a multi-part series about what I’ve learned about Visual Studio debugger addins while writing the FNameAddin.
First, however, we need to take a minute to discuss some basic stuff that most engineers I’ve met are aware of, but few bother to dig into. The autoexp.dat file is a peculiar beast in that you are encouraged to add to it, yet it is squirrelled away deep in the Visual Studio installation folder. It can be customized for your project, but has to be shared among all projects. All in all, it seems poorly thought out. We’ll blast through the really basic stuff first, so anyone who is seeing this for the first time should read up on the supplementary links.
AutoExp.dat
Deep in the Visual Studio install directory, at <VisualStudioInstall>/Common7/Packages/Debugger, there lives the autoexp.dat file. It’s just a text file, so go ahead and open it up and take a look around if you’ve never done that before. This is where you can give the Visual Studio debugger some hints as to how you want your data to be displayed in the watch window. There are three sections: [AutoExpand], [Visualizer], and [hresult]. We’ll only deal with the first one for the time being.
The AutoExpand section is fairly simple and quite well documented at the top of the file, so I won’t bother with it here. Sufficed to say, you can crack lots of simple data types by just adding them to the mix and possibly using one of the type suffixes. Go ahead and play around with this stuff. It’s very safe in that it won’t destabilize Visual Studio if you get it wrong, the debugger reloads the file every time you start a new debugging session, so feel free to edit and play around – just make a copy of the original file before you start. It’s always nice to revert in a pinch.
I’m far more interested in covering the $ADDIN-dlls functionality that is briefly mentioned. The file recommends looking at Microsoft’s EEAddin sample in order to get started, but you should use some caution. It turns out that EEAddin has been broken for as long as I can remember and will crash if you use it as-is – how unfortunate! So let’s take a different route. Let’s make our own sample: MT_Addin.dll. The process looks like this.
- Make a new Win32 DLL project called MT_Addin.
- I ripped out the PCH plumbing to reduce the file count, but you’ll probably want to leave it in.
- Write your handler function using the correct function signature. (see below)
- Add a .def file so our handler function is exported with a nice undecorated name.
- Add an entry in the autoexp.dat file that references our Type, DLL and entry point
- MyType=$ADDIN(<MT_AddinProjectDir>\Debug\MT_Addin.dll,HandleMyType)
You can download my sample project here (3k)
Ok, now let’s see what we have. Starting with the cpp file, we have our handler function with the following signature:
ADDIN_API HRESULT WINAPI HandleMyType( DWORD /*dwAddress*/
, DEBUGHELPER * pHelper
, int nBase
, BOOL /*IsUnicode*/
, char * pResult
, size_t maxResult
, DWORD /*reserved*/ );
There are a few things to note here. First, we declare the entry point as WINAPI which is just a #define for the __stdcall calling convention. This is what’s missing from the EEAddin sample that causes it to crash. Next, notice that we don’t use the dwAddress or bIsUnicode arguments. It might seem a little strange given that the whole point of this exercise is to look up addresses in the target process, but these are really just legacy arguments.
The most important thing here is the DEBUGHELPER. Strangely, you have to define the type for yourself even though the format isn’t under your control. (You can find it in MT_Addin.h) Fortunately, it’s not that complex and contains very little nuance.
| dwVersion | – Structure version – <0x200000 for VS6, >=0x20000 for VS7 and beyond |
| GetDebugeeMemory() | – Useful for VS6 otherwise deprecated. Doesn’t support 64-bit pointers |
| GetRealAddress() | – Gets the 64-bit value of the variable to be inspected. (Replaces dwAddress) |
| GetDebugeeMemoryEx() |
– Query raw memory from the target process. |
| GetProcessorType() | – Its use seems clear enough, but it always returns 0 for me. |
Our example doesn’t do any memory queries at all, but it does create a simple string to tell you it’s working. Finally, it returns S_OK for success – your addin should always return S_OK even when something goes wrong. When data can’t be interpreted correctly, you should return success and set the output string to “Error processing the data!” because returning an error will only display “???” in the watch window. Think of the return code to mean “The addin succeeded/failed” instead of “processing this data type succeeded/failed”
Installing, Testing, and Debugging
In order for Visual Studio to find your DLL, we have two options – specify an absolute path in the autoexp.dat file or copy our DLL into the “<VisualStudioInstall>\Common\IDE” folder. I recommend the former method because it prevents pollution of the Visual Studio folder, and it facilitates debugging the addin.
In order to test the addin, we need a second project that contains a type called “MyType”. At this point, it really doesn’t matter what MyType contains since our addin doesn’t actually query process memory. We just need the match the type name in order to invoke our handler – so go ahead and try it. If you put a MyType variable in the watch window, it should display, “Handled Data Type!!” Don’t worry, we’ll eventually make some memory queries.
Debugging these sorts of addins is fairly straight forward. Edit the addin project’s Debugging properties so that the Command item points to DevStudio IDE binary – it lives at “<VisualStudioInstall>\Common\IDE \devenv.exe”. When you run, your DLL won’t be loaded, so your breakpoints are invalid initially. Don’t worry about it though. The debugger will load your dll just in time and your breakpoints will go active. In fact, it will load and unload your any time it it’s required.
Next Time
Ok, that’s it for now. I think we have the basics pretty well covered. There’s a bit more to do with the standard addins, but hopefully things get a bit more interesting when we get to the Visual Studio extensibility framework.