November 30 2016

Today's task: Try to send arbitrary data through the item log interface. The goal is to send a bitmap image from a Windows client and reconstitute it in the Viewer. This is a feature I have wanted since the earliest versions of DbgOut, as one of the most vexing problems is trying to debug graphics operations.

This will be a significant modification to the data flow. I need to make sure all my current changes to the Linux branch as committed, merge it into the master branch, then create a new LogData branch.

Finished code:

Design

1. Client app calls DbgBitmap(HBITMAP hBitmap)
2. DbgBitmap() extracts the raw bytes from the bitmap and constructs a BITMAP header.
3. DbgBitmap() uploads the data to the Server in a series of 1KB DATA/BITMAP commands. Each chunk contains an index byte count to allow the Viewer to reconstruct the bitmaps from the chunks. Data items are always terminated (closed) with a chunk of 0 bytes.
4. Server stores the DATA items in the item fifo. This will require some code changes to assign the server-unique DataID to the item.
5. Relay downloads the DATA items from the server and stores them in the datastore Items.bin. This should require no code changes.
6. Viewer scans the datastore Items. For each DATA item it creates a single new DATA display line for each unique item. This is rendered as a single line in the log window with an appropriate icon.
7. When the user double-clicks a DATA line in the log window, a new child window is created to display the full object.
8. The DATA window reconstitutes the bytes for the entire data object by scanning the datastore items, displays a hex dump of the raw bytes, and attempts to render the object (ie bitmap).

Step 1: DATA item

This the data structure that is used to communicate the item from the client to the Server and how it will be stored in the DataStore/Item.bin file.

This is the existing item header:

#define ITEM_MARKER 0xCD //DbgItemHdr_s.Marker #define ITEM_SIZE_MAX 2048 //Maximum size (bytes) of an item typedef struct DbgItemHdr_s { //Universal item header UINT8 Marker; //Must be ITEM_MARKER UINT8 Type; //One of ITEM_ UINT16 ByteCt; //Size of item, including header UINT16 SeqCt; //Item sequence count, used to detect data drops UINT16 ClntID; //Local client ID UINT64 Tick; //Local CycleCt } ITEMHDR;

I am already treating everything as opaque data objects, so this step is trivial. Just add a new ITEM_DATA type.

Step 2: Data Flow

This describes how the data item is handled by each component: Client, client interface library, Server, and Relay.

Each bitmap begins with a DbgItemData_s.nPart=0, which indicates the beginning of a data object and that DataHdr is valid. The data header item will have ByteCt of 0, and contain no object bytes. The object data will be contained in subsequent DbgItemData_s records with the matching DataID values and incrementing nPart indexes.

DbgBitmap() will need to pay attention to DbgItem.h:ITEM_SIZE_MAX to make sure it does not exceed to maximum allowed size.

NOTE: This scheme requires that the complete data object be present when DbgBitmap() is called, so that it can fill out the header accurately. Alternatively, I could omit PartCt and ByteCt from the header and rely on a final closing record where nPart>0 and ByteCt==0. This would allow open-ended, possibly long-term, data objects to be handled by exactly the same code. This makes the Viewer code only slightly more complex as it will need to tally the bytes counts until it finds the terminator record, but it needs to scan the records anyway.

The parts for each data object must be recorded in order, but there may be other records interleaved (including other data objects). DataID must therefore be assigned by the Server (not DbgBitmap()) since only the Server can assure that it will be unique across all clients. The assigned DataID is returned to the client in the DbgItemData_s struct so that it can be set in all subsequent items.

The Server must detect items with ITEMHDR.Type==ITEM_DATA and DbgItemData_s.nPart==0. For these items, the Server sets DbgItemData_s.DataID= gDataID++. No other code changes should be needed.

DbgBitmap() then sends the actual bytes, incrementing DbgItemData_s.nPart. Note that the first record with actual data bytes will have DbgItemData_s.nPart==1. The data object must be closed with a final record where DbgItemData_s.nPart>0 and DbgItemData_s.ByteCt==0.

The Relay does not need to do anything special.

Step 3: Viewer

The viewer reads the items from a static Item.bin file. It is assumed that the Item.bin file will not change, although it may be appended.

The Viewer will display the data items as a single line in the log items list, identified by a special icon. The user can double-click the item to expand it into a viewer window that will display the raw bytes as a hex dump and also try to render the object, such as a bitmap image.

DataItem

Viewer/DataItem.cpp will be the interface to manage data items, including a fast lookup index.

DataItem::BytesSet() will read the DbgItemData_s record from Item.bin. If nPart==0, it will create a new data index record. If nPart>0, it will search the index for a matching DataID and update the ByteCt. The return value will be the current ByteCt for the item, which should always be 0 for the first record of the item.

struct DataIndex_s { UINT8 Type; //One of DBGITEMDATA_ UINT32 ByteCt; //Data bytes UINT64 FilePos; //Location in Item.bin of nPart==0 };

DataItem::TextGet() will return a TmpBuf containing the text that should be presented in the log items list. This should contain the data type as text, total bytes, and any user-defined text. pType will contain the DBGITEMDATA_ type.

DataItem::InfoGet() will return a TmpBuf containing the DbgItemData_s header. This should contain enough information to allow ItemView to know how to render the object.

DataItem::BytesGet() will return a MemAlloc() buffer containing only the entire raw byte stream for the item. The caller must call MemFree().

DataStore

Viewer/DataStore.cpp needs to detect records where ITEMHDR.Type==ITEM_DATA call DataItemBytesSet(hDataItem,&nData,nIndex,FilePos) to add it to the data index. DataItemBtyesSet() returns 0 (DBGERR_OK) to indicate the creation of a new data item, and DataStore will create a placeholder INDEX with INDEX.Pos being the nData index rather than the actual Item.bin index. DataItemBytesSet() returns >0 to indicate data was appended to an existing item, in which case DataStore does nothing and moves on to the next record.

ItemList

Viewer/ItemList.cpp needs to handle double-clicks on an item where INDEX.Type==ITEM_DATA by creating a new DataView object for the item index.

DataView

Viewer/DataView.cpp will use DataIndex to collect all the data bytes for the object into a single coherent buffer, then try to reconstruct the original object (ie bitmap) and render it to the window. The ItemData window should also present a hex dump of the raw data.

This is the only time when the actual data bytes are read from Item.bin.

Implementation

ExWindow

I start by creating a test bitmap in the ExWindow project, which is a one-minute project.

I load the bitmap during MainWnd::Create2(). If it loads, I call DbgBitmap() to log it.

int MainWnd::Create2(void) { int Err= DBGERR_OK; BITMAP Bitmap; hbmTest= LoadBitmap(0,L"BMP_TEST"); if(!GetObject(hbmTest,sizeof(Bitmap),&Bitmap)) { hbmTest= 0; } else { rTest={ 0,0,Bitmap.bmWidth,Bitmap.bmHeight }; DbgBitmap(hbmTest,"Test bitmap"); } if(!(hThread= CreateThread(0,0,ThreadProc,this,0,&ThreadID))) { Err= Error(DBGERR_SYSCREATE,"MainWnd:Create2: Unable to create thread."); } else if(WaitForSingleObject(hRunning,100000)) { Err= Error(DBGERR_SYSCREATE,"MainWnd:Create2: Window failed to start."); } return(Err); }

I also want to display the bitmap in the window, so I add a WM_PAINT handler to paint an offscreen image containing the bitmap.

BOOL MainWnd::MsgPaint(void) { PAINTSTRUCT Pnt; if(BeginPaint(hDlg,&Pnt)) { RECT *pR= &Pnt.rcPaint; if(DoRedraw) Redraw(Pnt.hdc); BitBlt(Pnt.hdc,pR->left,pR->top,RWID(*pR),RHGT(*pR),hdcImg,pR->left,pR->top,SRCCOPY); EndPaint(hDlg,&Pnt); } return(1); }

Redraw() creates the offscreen image:

void MainWnd::Redraw(HDC hdcPaint) { if(!hdcImg) hdcImg= CreateCompatibleDC(hdcPaint); if(!hdcSrc) hdcSrc= CreateCompatibleDC(hdcPaint); if(!hbmImg) hbmImg= CreateCompatibleBitmap(hdcPaint,RWID(rClnt),RHGT(rClnt)); SelectObject(hdcImg,hbmImg); if(!hbrFill) hbrFill= CreateSolidBrush(RGB(10,192,10)); FillRect(hdcImg,&rClnt,hbrFill); if(hbmTest) { int x1= (RWID(rClnt) - RWID(rTest))/2; int y1= 10; SelectObject(hdcSrc,hbmTest); BitBlt(hdcImg,x1,y1,RWID(rTest),RHGT(rTest),hdcSrc,0,0,SRCCOPY); } DoRedraw= false; }

Finally, I need to reduce the size of the text edit control so it doesn't collide with the bitmap image.

BOOL MainWnd::MsgResized(void) { GetClientRect(hDlg,&rClnt); int EditTop= hbmTest ? RHGT(rTest)+12 : 0; SetWindowPos(hEdit,0,0,EditTop,RWID(rClnt),RHGT(rClnt)-EditTop,SWP_SHOWWINDOW); if(hbmImg) { DeleteObject(hbmImg); hbmImg= 0; DoRedraw= true; } return(1); }

The complete code:

Now I can see the test bitmap in the window.

ClntLib

DbgBitmap() will only be supported in the user-mode Windows client library. The first step is to create the declaration in DbgOut.h.

/*************************************************************************/ /** Data items. **/ /*************************************************************************/ #if defined(DBGOUT) && DBGOUT>0 ... #ifdef _WINGDI_ EXTERNC int DbgBitmap(HBITMAP hBitmap, const char *Label); #endif ... #else //if DBGOUT ... #ifdef _WINGDI_ static __inline int DbgBitmap(HBITMAP hBitmap, const char *Label) { return(DBGERR_OK); } #endif

Create ClntUser/ItemData.c and add a stub for DbgBitmap():
#include <Windows.h> #include "DbgOut.h" #include "DbgCmd.h" #include "Globals.h" #pragma message(__FILE__":Optimizer disabled.") #pragma optimize("",off) int DbgBitmap(HBITMAP hBitmap, const char *Label) { int Err= DBGERR_OK; return(Err); } //EOF: ITEMDATA.C

I should now be able to uncomment the call to DbgBitmap(), build the ExWindow project and see the test image bitmap, although nothing is being logged yet.

DbgBitmap()

Now that I know I have a valid bitmap, I can extract the raw bytes and create the bitmap item header.

Complete code:

Server

I need to handle DBGCMD_DATA to assign the DataID, replace the command header with an item header, and write the items into the item FIFO.

Viewer

Handling the data items in the Viewer is the biggest task. I need to handle the ITEM_DATA records during the initial DataStore indexing to create an index of data items. I need to render the data items in the log listbox and handle double-clicks by creating a new DataView window. The DataView then needs to combine all the individual ITEM_DATA records for the DataID into a single coherent block of bytes, draw the hex dump, and try to render the bitmap.

The task of rendering the hex dump is simplified by the DrawText object. This lets me print text using printf with embedded escape characters for all the formatting.

Testing

November 2 2016

I have chunked in the first draft of the complete code, including a crude hex dump window. Now I need to test it.

1. Boot up the Win10Target VM. I have made changes to the Server driver, so I don't want to blow up my development system.
2. Leave Win10Target idling on the login screen.
3. Configure the ExWindow project for remote debugging. Project Properties > Debugging > Remote Windows Debugger
4. Launch WinDbg on VS12 using DbgOut\1102\Test\WinDbg.bat and wait for it to connect.
5. Log in to Win10Target and make sure the remote debug server is running -- it will randomly crash and disappear while idle.
6. On VS12, configure the Relay project for local debugging, set the command line to
   --server LISTENTCP:192.168.0.204 --datastore FILE:DataStore --notimestamp
   and launch a new instance of Relay to run on VS12.
7. On VS12, configure the Relay project for remote debugging and launch a new instance of Relay to run on Win10 Target.
8. On VS12, launch a new instance of ExWindow to run on Win10Target.
9. Let ExWindow run until some data (~150KB) has been written to the datastore on VS12. Exit ExWindow, Win10Target\Relay, and VS12\Relay.
10. Launch a new instance of Viewer on VS12.
11. The ITEM_DATA should be detected during the DataStore scan.
12. DataIndex should create a DATA object for it.
13. The DATA ByteCt should be correct.
14. I should see the DATA item listed in the log view.
15. I should be able to double-click the DATA item in the log view to create a DataView window.
16. I should see the hex dump of the bitmap bytes.

First glitch: WinDbg did not connect to Win10Target. I am using the VMware virtual subnet in the (perhaps mistaken) belief it would provide the best performance with near-zero latency. The IP address for the VS12 machine has changed, fixed by updating the configuration on Win10Target:
bcdedit /dbgsettings net hostip:192.168.114.130 port:49999

Second glitch: ExWindow failed to build due to unresolved symbol MemAlloc and MemFree. I had replaced these with DbgOutMemAlloc and DbgOutMemFree but apparently neglected to rebuild the x64 version. Rebuilding requires shutting down all the debug processes, and because the DbgOut11.sys driver doesn't unload properly this requires rebooting Win10Target.

Third glitch: The data store is corrupt. No call to ItemData(). I expect the ITEM_DATA to be early in the log, and since the corruption occurred at byte 80008 of 85594 I am assuming the corruption has nothing to do with ITEM_DATA. The next question is whether the ITEM_DATA was actually written into the log. Viewer didn't crash, so I am able to open the log window and see that the corruption actually occurs much earlier, after the second text item, make the ITEM_DATA a prime suspect.

The first item should be "MainWnd: Msg 20:WM_SETCURSOR" not "20:WM_SETCURSOR". Since the problem appears in the first visible item, it is worth stepping through the Scan() code to look at the bytestream.

The first item in the datastore is an invisible ITEM_CLNT_CREATE, and the next item is the first visible line, which is corrupted. This makes me think I am not calculating the offset from one item to the next correctly. This is most likely a bug introduced when I switched from using ITEMHDR to ITEM in the internal DataStore API.

bool DataStore::ItemRead(UINT64 Pos, ITEM *pDst, UINT DstSz) { DWORD ExtraCt,ReadCt; LARGE_INTEGER lPos; bool IsOk= 0; if(hItem) { lPos.QuadPart= Pos; //Hopefully the optimizer removes this two-step. SetFilePointerEx(hItem,lPos,0,FILE_BEGIN); //memset(pDst,0xCD,DstSz); if(ReadFile(hItem,pDst,sizeof(*pDst),&ReadCt,0) && ReadCt>=sizeof(*pDst)) { if(pDst->Hdr.Marker!=ITEM_MARKER || pDst->Hdr.ByteCt<sizeof(ITEMHDR)) { //Corrupt? } else if(pDst->Hdr.ByteCt <= sizeof(ITEMHDR)) { IsOk= 1; } else if(pDst->Hdr.ByteCt > DstSz) { Warn(DBGERR_TOO_BIG,"DataStore:ItemRead: Item %X [%llu] is too big! (%u > %u)",pDst->Hdr.Type,Pos,pDst->Hdr.ByteCt,DstSz); IsOk= 1; } else { ExtraCt= pDst->Hdr.ByteCt - sizeof(pDst->Hdr); if(ReadFile(hItem,&pDst->Raw[sizeof(pDst->Hdr)],ExtraCt,&ReadCt,0) && ReadCt>=ExtraCt) { IsOk= 1; } } } } return(IsOk); }

The first ReadFile() should be sizeof(pDst->Hdr). This fixed the corruption, but there is still no ITEM_DATA.

I used hexdump to look at the raw Item.bin bytes. I did not see and ITEM_DATA header (CD 0B). I suspect it is not actually being written. Did it fail in DbgBitmap() or in the Server?

I relaunch everything to step through the DbgBitmap() code.

int DbgLink::Send(struct CmdHdr_s *pCmd) { int Err= DBGERR_OK; DWORD WriteCt; if(pCmd->ByteCt

The DBGCMD_DATA was failing because pCmd->ByteCt was 4. This was because of a stupid bug in DbgBitmapHeader():
pCmd->H.ByteCt= sizeof(ItemSz);
This should be "pCmd->H.ByteCt= ItemSz;".

DataID is not being returned from the first DBGCMD_DATA command. The server is returning error -2003:DBGERR_BADCMD. So it appears I neglected to update Win10Target to the latest version of DbgOut11.sys. This happened because I removed DbgOut11.sys from the remote deployment list for Relay, because DbgOut11.sys doesn't unload when Relay exits and cannot be overwritten. The unloading bug is a real nuisance. The fix is easy, reboot Win10Target and explicitly deploy the remote Server project.

Fourth glitch: Win10Target locked up during reboot, and locked up WinDbg as well. This is where debugging using VMs pays off, I simply reset Win10Target and since WinDbg is "just" a user-mode process I can kill and restart it.

DataID is now being set by the server, but I am neglecting to save it in DbgBitmapHeader().

Now I have ITEM_DATA records in the DataStore. Back to debugging Viewer locally.

I am neglecting to return 1 from DataIndexBytesSet() when I create a new item.

DataFind() failed to find the existing DATA record. Because I neglected to add it to DataList in DataAdd().

This led to discovering a mistake I repeated in several places: DataList is an array of DATA pointers not complete structures. Where I had
DATA *pData= &LISTPTR(DATA,DataList)[nData];
should be
DATA *pData= LISTPTR(DATA*,DataList)[nData];

Now it appears the ITEM_DATA records are being written properly, read and parsed properly, and the DataIndex is being built.

Now ItemList is barfing on the ITEM_DATA record.

I had added the ITEM_DATA handler to the simple text renderer (used to copy text to the clipboard), not the draw renderer.

The ITEM_DATA Tick is not being set in the item header.

DataStore is reporting only 3 items, so it must have bailed after the ITEM_DATA. ItemData() is returning -1015:DBGERR_NOT_FOUND. This occured on nPart=11, which points the finger at DataGrow(). I am neglecting to write the new pData back into DataList.

The log view is now correct, including the tick for the DATA item.

I can double-click the DATA item to create the DataView window, but the call to DataStoreItemBytes() fails with -1103:DBGERR_BAD_ARG3. This was a simple bug in DataStoreItemBytes().

There is no DataView window because I forgot to create the dialog resource. I also forgot to create the hEvent.

I can see the DataView window, it has the correct label and ByteCt but I don't see any hex dump. This was because I forgot to add a newline to the end of each row and DrawText only draws text when at each newline. Now I can see the hex dump. The data looks suspect because it repeats, but that may be correct since I unfortunately put a big border of a solid color around the bitmap.

To make the pixels easier to recognize, I hack the test bitmap. I vaguely remember that bitmaps may be stored "bottom-up" so I hack pixels on the top and bottom.

I relaunch everything and now I see the modified pixels in the hex dump. They are the top pixels, so the bitmap is saved top-down.

I am now confident the ITEM_DATA is working properly, I just need to reconstitue the bitmap and draw it in the window. Then the finishing touches will include creating a child windows for the hex dump and image, adding a scrollbar to the hex dump, and a zoom function to the image.

I spent about 5 hours debugging (and documenting) to have everything (except the bitmap renderer and the "prettifying") working.

I neglected to preserve the Bitmap info in DataIndex, then to write it out in DataIndexItemGet().

I added some code to create and render the bitmap in the DataView window, and ....

SUCCESS!

Next Steps

This whole process took about two days, from design through to first working version. Now it opens the door to yet another time sink...

• Scroll the window to see the entire hex dump. DONE
• Copy the hex dump to the clipboard as text. DONE
• Save the bitmap to a file. DONE
• Resize large bitmap images to fit into the width of the window. DONE
• Nicer fonts DONE
• Handle bitmaps with alpha channels (Use DirectDraw instead of the oh-so-easy GDI).
• Compress the data.

I found ByteScout that offers a free bitmap visualizer. It only mentions .NET. Graphics Debugger Visualizer has the source code to do the same thing. Another Visualizer at CodePlex.

Dec 3 2016

I added a vertical scrollbar. The code was fairly straight-forward, but trying to get the scroll position to line up with the actual image is very frustrating. The text painting functions use "logical units", GetClientRect() returns "dialog units", and the bitmap uses "device units".

After resizing the dialog window:
BOOL DataView::DlgResized(void) { RECT rNew; GetClientRect(hDlg,&rNew); rPixels= rNew; MapDialogRect(hDlg,&rPixels); Debug(DBG_DRAW,"DataView:DlgResized: rNew[%dx%d] rPixles[%dx%d]",rNew.right,rNew.bottom,rPixels.right,rPixels.bottom); DBG: DataView:DlgResized: rNew[447x286] rPixles[671x465]

After scrolling to the bottom of the window:
DBG: DataView:DlgScrollY: Y[24276] Msg[4] Pos[24276] H[465] Extent[24717] DBG: DataView:DlgScrollY: Y[24276] Msg[8] Pos[0] H[465] Extent[24717] DBG: DataView:UpdateScroll: Pos[6069] Max[6179] Page[111]
This does not show the bottom of the image.

After scrolling by line to the bottom:
DBG: DataView:DlgScrollY: Y[24717] Msg[8] Pos[0] H[465] Extent[24717] This tells me that 24717 is the actual bottom of the image.

The question is why can't I thumb-scroll down to 24717? UpdateScroll is setting nMax to 6179 (6179*ScrollScaler=24716), but the maximum scroll position is 24276 (24276/ScrollScaler=6069). 6179-6069=110. So it appears the maximum thumb position is nMax-nPage.

If I set Scroll.nMax to TextExtent.cy/ScrollScaler + Scroll.nPage, thumbing to the bottom moves the bottom pixel to the top of the window. This isn't quite what I want, I want the bottom of the scroll to bring the bottom pixel to the bottom of the window.

I was over-thinking the problem. If I just use the dimension from GetClientRect() and set nMax= TextExtent.cy everything seems to work as desired.

December 4 2016

I finished the DataView window by adding a menu, the ability to copy the hex dump to the clipboard as text, and the ability to save the bitmap to a file.