Oct 23 2015 (Mole Day)
Today's task is to revive Win7 and resume debugging the NetMon driver using WinDbg on a disposable system.
10:45> I traveled to the Felton Satellite office for this, since it may involve downloading updates. I was able to reconfigure the network and boot up Win7 fairly quickly.
The first big obstacle was disk space on Win7. It currently uses a single 40GB drive and has only 3.5GB available. It is running VS13 and I need to use VS15, which requires at least 7GB. So I need to expand the C: drive. This involves a few time-consuming steps:
11:05> That was easy. Win7 now has 23GB available. Long gone and good riddance are the days of opening cases and replacing hard drives! I can resume installing VS15...
11:24> Well... Win7 BSoD'd during the VS15 installation. Hmmm.
12:09> Setup completed. Rebooting...
12:12> Win7 is back online. I am taking advantage of the fast internet connection to run Windows Update.
12:25> Well, in typical fashion Windows Update spent 15 minutes to accomplish absolutely nothing.
12:37> The Driver menu item is missing from my VS15. I followed links that eventually led me to downloading the WDK for Windows 10. I saved the WDK to \\Pogo\R\Disks\MSDN\.
12:44> Installing WDK10 onto VS12...
12:56> Installation complete. The Driver menu item is back. It even picked up the DbgWin764 configuration from VS13.
13:00> I am attempting to start a remote debugging session using my notes from VS12.
13:07> VS15 failed to connect to the target. I am now trying to install the WDK10 on WinDbg764. In the "Installers" folder there is an MSI named "WDK Test Target Setup" for x64.
13:20> The virtual COM connection wasn't working, but I was able to ping from VS12 to Win7Dbg and back so I created a network configuration using the raw IP addresses. This seemed to work, copying a bunch of stuff from VS12 to Win7 during the provisioning phase. Win7 ran a bunch of remotely controlled installation scripts, and VS12 reported partial success. I restarted Win7. This involves network debugging on a public network, which is not ideal.
13:30> Still fails to connect. Possibly using the wrong interface on VS12? I ran msvsmon.exe from the VS2015 folder on Win7. This tried (for a very long time) to sign into VisualStudio.com. I think this may have downloaded the entire VS2015 installer without telling me. :( OK, this is somewhat amusing: I had clicked on "msvsmon.exe", but because File Explorer was hiding the file extensions I had actually clicked on "msvsmon.exe.xml". This was registered to Visual Studio 2015, which started, then kicked off the installer. So by clicking a single file I launched a full download and install of a 7GB program just to view a text file. Wow.
13:50> OK, I am switching tactics. Since I now have VS15 and WDK10 installed on Win7, I should be able to rebuild and run NetMon locally on Win7. I won't be able to step through it, but at least I will be able to test without risking the VS12 machine. Git is already installed but the repo is not set up. I need to configure the git keys, copying the configuration in \\VS12\C:\CL\git\.ssh.
14:30> OK, getting git to work was a real pain. The githqchip.rsa
key was not being recognized. TL;DR version:
> cd S:\Src\HQ\Dev\SB\Chip
Chip> path=C:\CL\Git\bin;%PATH%
Chip> bash
$ eval "$(ssh-agent -s)"
Agent pid 4064
$ ssh-add /c/CL/git/.ssh/githqchip.rsa
Identity added: blah blah
$ exit
Chip> git clone ssh://hq-chip@aws.chip4.net/git/src/HQ/Dev/SB/Chip
Cloning into 'Chip'...
remote: Counting objects: 9490, done.
...
14:35> Unfortunately, I cloned into the wrong directory. I needed to be above the 'Chip' directory. The fast internet connection allows me to clone again in less than a minute. (Note to self: I need to split my git repos into a dedicated repo per project. Keeping everything in a single repo is a bad idea for many reasons.)
14:39> I now have the latest NetMon source code on Win7.
NetMon> git checkout netmon
I'm not sure what that Visual Studio installer actually installed. I still don't have the developer environment!
14:45> I am running the VS15 installer on Win7 (again?) It starts in modify/repair mode but shows that the C++ compilers are not installed.
=====
09:30> Starting fresh, I took another run at connecting to Win7. It connected when I selected the middle option 'DebugEntry'! However, it is running very slowly -- taking many minutes to boot up. I took some time to create a page from the BCDedit help text. BCDedit
11:45> Win7 went to sleep while I was editing the BCDedit page, which caused VS15 to lockup in VS12. Win7 woke up but VS15 did not even after waiting many minutes. I reset Win7, VS15 then reported that it hit a breakpoint but was still unresponsive to UI. I gracefully restarted VS12 and left Win7 in the Boot Manager menu. I restarted VS15 and tried to connect to Win7. This time I received an error message, "Could not start debug session, error 80070005: Access is denied." I tried running VS15 as Admin, but the target machine was not configured for that user. I am not bringing the whole system down and restarting the host machine (Pogo).
12:00> I was able to connect after restarting everything. I do not need to launch VS15 as Admin to debug a target machine. I am assuming the problem was an orphaned handled to the com_1 pipe inside the VMware engine. If the connection is made, it occurs about 20 seconds after releasing Win7 from the Boot Manager screen. It then takes about 5 minutes to boot to the desktop "Please wait...", and the desktop should appear about a minute later.
12:05> I am using the VS15 embedded WinDbg. I copy only the binaries to the target and set the breakpoint on VS12. I am setting a breakpoint on DriverEntry. I see the breakpoint in the Debugger Immediate Window almost immediately, but then the VS15 UI locks up, complains about a timeout, then goes away forever. Recovering from this requires rebooting everything.
I am now going to try using the standalone WinDbg. Hold it! Just as I was about to hit the reset button VS15 awoke from its coma. This was 3:40 minutes after launching the target process. The target did not halt on the breakpoint I set from the VS15 UI, it stopped on the hard DbgBreakPoint() call in DriverEntry(). It takes just over 2 seconds to step over a line, but the debugger is working!
:)
12:12> I set a breakpoint in Process:ProcProbeLibraries() and let it go. The breakpoint triggers after about 6-7 seconds. I try to step through the function, but I keep hitting the breakpoint. This is either a debugger problem or I am re-entering ProcProbeLibraries on multiple threads/processes. A re-entrancy problem is something I feared, but hoped I could defer until a bit later. It may be that I need to address it now. However, at this point I am just trying to walk the module list. I will set a simple global flag to prevent re-entrancy and deal with it later.
To drop the debug session and restart:
Letting the target sleep seems like a minor thing, but it causes big problems. The target wakes up OK, but the debugger hangs on the first breakpoint. Even worse, Visual Studio locks up hard and requires rebooting the VS12 system. Even worser, it orphans the handle to the com_1 named pipe and VS12 won't be able to reconnect to the target even after it reboots. Even worsest, this may require rebooting the host system, VS12, and the target system to recover.
Success! This concludes yesterday's task of reviving Win7 and debugging the NetMon kernel module on a remote system.
Oct 25 2015
I was caught by Windows Update this morning. It took over two hours to reboot Win7 as it installed 53 updates.
Installing VS2015 on Win7 was not necessary and a waste of time. Since I did not need to install VS2015, I didn't need to expand the disk. I wasted an entire day because I picked the wrong Boot Manager option (out of 3 possible choices).
I am wondering if debugging could be accelerated by switching from virtual serial port (nominally 115Kbits/sec) to using a virtual ethernet switch (nominally 100Mbits/sec). This MSDN page might help. I will need to archive both VS12 and Win7 VMs before making any configuration changes.
Update: While VS15's kernel debugger has improved slightly since VS13, WinDbg is still vastly superior. Skip trying to use VS15's internal kernel debugger, it will only waste huge amounts of time. WinDbg has a much steeper learning curve but it doesn't tie up the editor and compiler while debugging, is much more responsive, wastes far less time, and has a vast set of debugging commands specially tailored for the Windows kernel.
Nov 1 2015
Doh! I was reorganizing my files and I needed to free up space on my external drive. I saw Win7 and deleted it since it was now an obsolete copy of the VM on Pogo's internal drive. I later realized there was no version on Pogo, I had been using the external drive. Oops. The most recent backup of Win7 was a year old. I seem to have an incredible knack for saving worthless crap and neglecting to save the stuff that matters. Fortunately, there isn't really much, if any, original information on Win7 so the year-old backup should be fine. It just means I will be downloading 60+ Windows Updates the next time I try to use it. I need to boot it up before going to sleep.
This minor disaster prompted me to build an "Archives" database and web page to better track what is on all these external drives. UPDATE: The Archives database was a complete waste of time because I never bothered to update it when copying files around. The pragmatic solution is to simply make a note in my daily journal.
Nov 4 2015
129 Windows Updates installed. I started the updates last night and left it running overnight. In the morning the "please wait" swirly was still moving but the system was hung on 116 of 129. I let it swirl for another 15 minutes before hitting the reset button. The system restarted three or four times before making it to a stable desktop. Three more trips through the "Check for Updates" and restart cycle and I finally made it to "There are no new updates." I believe Win7 has been resurrected and is back in business.