Let’s Make a Visual Studio Addin – part 4

---

Download the source code for this post (19k)

---

Alright then…Part 4…I think it’s time we tie the first three parts together. We want to use the type of add-in that was introduced in part 3 to fill in the knowledge gaps of the add-ins we played around with in parts 1 & 2. Furthermore, for simplicity, we want to literally combine the two types of add-ins into one DLL.

Filling in the Gaps

In part 2, I spent some time talking about all the things we don’t know when we get a raw pointer from a target process through the AutoExp.Dat file. We don’t know:

1. The target platform’s endianess
2. The size of an int or pointer on the target platform
3. How to access any global symbols because there’s no general way to know where they reside in memory.
4. The data layout for a type in the target process

When our OnEnterBreakMode handler is triggered, we want to quickly gather all the extra information we might need in anticipation of our autoexp.dat handlers being triggered. We don’t know exactly what pointers will be sent to those handlers, but we can answer our nagging questions that are common to all of them.

Endianess

Unfortunately, endianess isn’t something we can just ask the debugger about, but it is almost always something we can infer. The scope of our work here covers Xbox360 debugging as well as Win32/Win64 debugging. The Windows-targets will use the AutoExp.Dat file in <VisualStudioRoot>\Common7\Packages\Debugger while the Xbox360 will use the one in <XDKROOT>\bin\win32. You can use this difference to link our MyType structure to two different handler functions in our add-in.

That’s probably the simplest way, but there are others. If you dig through the EnvDTE::Debugger object, you’ll see that you can get a Process2 object for the current target process and then ask it about the transport being used. That can also tell you if you’re attached to an Xbox.

Sizes of Types

At the end of part 3 this series, I mentioned that the Debugger object has a GetExpression() function. This thing is pure gold. You can feed it strings, and it returns the evaluations of those strings just like you had typed them into the watch window. It’s a programmatic way to evaluate expressions within the context of the target process. So what if our add-in asked it to evaluate this?

CComBSTR const bsSizeOfVoidPtr(L"(int)sizeof(void*),d");
CComPtr<EnvDTE::Expression> pExpr=NULL;
hr = pDebugger->GetExpression(bsSizeOfVoidPtr,  // our string as a BSTR
                              VARIANT_FALSE,    // Don't use autoexp.dat
                              -1,               // no timeout
                              &pExpr);          // result

It would definitively tell us how large our target platform thinks a pointer is, and allow us to query nested pointer types safely.

Since this is the sort of question that will only ever have exactly one answer for the lifetime of a given process, we only want ask it once and then cache the result for later use. Performance isn’t a huge concern inside the debugger, but we can’t ignore it either.

Global Variables

Globals are fairly similar to data sizes in the way we evaluate them. Let’s say our string table example from part 2 was instead held in a global variable. If we knew the name of that global, then we could feed it into GetExpression and get the current base pointer of the table.

Bear in mind that a table pointer might change if the table is reallocated for any reason, but the pointer to the base pointer will reside in a single place throughout the lifetime of the target process. Depending on the nature and expected lifetime of your data, you should cache it appropriately. Unfortunately, even if a value changes only once in a blue moon, you still need to check it ever time.

Data Layouts

Let’s say you know there is a string data member on your object called StrId that is perfect for displaying in the watch window. You won’t ever change it’s name, but you’re less sure about it being shifted around inside of your object.

If you look elsewhere on this blog, you’ll find an interview question that I ask a lot. I posted it before I posted this entry because the contents are important to what we’re doing here. You can use the answer to the struct-offset question along with the GetExpression function to give you the answer you’re after for the target process.

CComBTR const bsOffsetToStrId(L"(int)&(((MyType*)0)->StrId),d");

If we can figure out the critical part of our data layouts at debug-time, we no longer have to mirror our data in our AutoExp.Dat add-in. We don’t have to worry about things getting out of synch because we can ask the right questions against the actual target every time.

A final word about these strings we’re passing into GetExpression. As I said, they’re evaluated as though you typed them into the watch window. That means they’re subject to the platform’s specific expression evaluator nuances as well as the Hex-display button. Because these things are somewhat unpredictable and difficult to account for, I’ve taken to completely specifying what I want to get back. That’s why I’ve explicitly cast my expressions to integers and suffixed my expressions with “,d”  to force the display to decimal. It’s all about predictability.

Tying It All Together

As I’ve already mentioned, we have we have two types of add-ins across the first three installments. The first add-in is a Visual Studio extension that can extract lots of information from the target process. The second can’t extract much information, but has the ability to augment the watch window’s output. It’s no secret that we want the first add-in type to cache some data that’s useful to the second.

It turns out that sharing data is a pretty simple thing to do. Anywhere that you can stash data in Windows can be used to share this information between the two DLLs. The registry, named-pipes, temporary files, swap-backed files, shared memory, etc are all viable ways to make the connection, but they’re not the way I’ve used in the past. When I originally wrote the FNameAddin, I used a simple assumption and a shared secret.

I assumed that the COM add-in was always loaded at debugging time. When the AutoExp.Dat add-in is loaded, it calls LoadLibrary() on the COM DLL. I only had to know what it was called – not where it was installed – because I could assume it was already loaded. Then I could call GetProcAddress on my special GetTheDataCache() function to get do the data transfer. Everything worked fine, but there were a few caveats that needed attention.

Because I had two DLLs, I had to keep their versions in synch or at least make sure they could handle the edge cases where the versions were out of synch with one another. I also had to make sure I was installing the AutoExp.Dat add-in to an appropriate folder so it would be found by the debugger.

A Key Insight, and An Unintended Benefit!

It’s not hard, but it can be a lot easier – we’re making two DLLs here, why can’t they simply be combined into the same DLL? That way, they’d share the same memory space which makes these coupling shenanigans completely unnecessary. In fact, by combining the DLLs, we don’t even have to worry about putting the AutoExp.Dat DLL in the correct place.

Remember all that messiness in part one where we used an absolute path to our add-in DLL? That requirement just went away. Because our DLL will always be loaded by virtue of being a registered Visual Studio add-in, the AutoExp.Dat mechanism won’t ever have a problem finding it. We can put it anywhere we want as long as it’s registered with Visual Studio to be loaded on start-up. Not too shabby!

The Sample

The sample code is really a re-work of the code from part 2, but it uses all of the Visual Studio extension framework we discussed in part 3. It accomplishes the same fundamental task as part 2, but it doesn’t rely on any shared secrets with respect to types or any fragile linkages. Instead, it derives all of its data at runtime using the methods discussed above. It should also be reasonably portable between Win32, Win64, and Xbox targets.

There’s a lot more code there than in the other samples, but take your time to follow what it’s doing. The sample code also includes a trivial target application. Once you compile the add-in project and use Visual Studio as the debugging target as we discussed in the other parts of this series, load the target-app project in the second Visual Studio instance. It should give you all the breakpoints you need to exercise the add-in.

Be sure to add the correct lines to the right instance of AutoExp.Dat:

VS 2008 –	%PROGRAMFILES%/Microsoft Visual Studio 9.0\Common7\Packages\Debugger\autoexp.dat
	MyType=$ADDIN(MT_Addin4.dll,HandleMyType_WinXX);
VS 2010 –	%PROGRAMFILES%/Microsoft Visual Studio 10.0\Common7\Packages\Debugger\autoexp.dat
	MyType=$ADDIN(MT_Addin4.dll,HandleMyType_WinXX);
Xbox360 –	%PROGRAMFILES%/Microsoft Xbox 360 SDK\bin\win32\autoexp.dat
	MyType=$ADDIN(MT_Addin4.dll,HandleMyType_Xbox360);

Conclusion

That should just about wrap it up for this series on Visual Studio add-ins. I’ve covered enough of the techniques used in the FNameAddin that you should be able to adapt them to your projects. I hope you’ve found it useful. Please let me know if you have any questions or problems with the sample code.