Tutorial 21: Pipe

In this tutorial, we will explore pipe, what it is and what we can use it for. To make it more interesting, I throw in the technique on how to change the background and text color of an edit control.
Download the example here.

Theory:

Pipe is a communication conduit or pathway with two ends. You can use pipe to exchange the data between two different processes, or within the same process. It's like a walkie-talkie. You give the other party one set and he can use it to communicate with you.
There are two types of pipes: anonymous and named pipes. Anonymous pipe is, well, anonymous: that is, you can use it without knowing its name. A named pipe is the opposite: you have to know its name before you can use it.
You can also categorize pipes according to its property: one-way or two-way. In a one-way pipe, the data can flow only in one direction: from one end to the other. While in a two-way pipe, the data can be exchanged between both ends.
An anonymous pipe is always one-way while a named pipe can be one-way or two-way. A named pipe is usually used in a network environment where a server can connect to several clients.
In this tutorial, we will examine anonymous pipe in some detail. Anonymous pipe's main purpose is to be used as a communcation pathway between a parent and child processes or between child processes.
Anonymous pipe is really useful when you deal with a console application. A console application is a kind of win32 program which uses a console for its input & output. A console is like a DOS box. However, a console application is a fully 32-bit program. It can use any GUI function, the same as other GUI programs. It just happens to have a console for its use.
A console application has three handles it can use for its input & output. They are called standard handles. There are three of them: standard input, standard output and standard error. Standard input handle is used to read/retrieve the information from the console and standard output handle is used to output/print the information to the console. Standard error handle is used to report error condition since its output cannot be redirected.
A console application can retrieve those three standard handles by calling GetStdHandle function, specifying the handle it wants to obtain. A GUI application doesn't have a console. If you call GetStdHandle, it will return error. If you really want to use a console, you can call AllocConsole to allocate a new console. However, don't forget to call FreeConsole when you're done with the console.
Anonymous pipe is most frequently used to redirect input and/or output of a child console application. The parent process may be a console or a GUI application but the child must be a console app. for this to work. As you know, a console application uses standard handles for its input and output. If we want to redirect the input and/or output of a console application, we can replace the handle with a handle to one end of a pipe. A console application will not know that it's using a handle to one end of a pipe. It'll use it as a standard handle. This is a kind of polymorphism, in OOP jargon. This approach is powerful since we need not modify the child process in anyway.
Another thing you should know about a console application is where it gets those standard handles from. When a console application is created, the parent process has two choices: it can create a new console for the child or it can let the child inherit its own console. For the second approach to work, the parent process must be a console application or if it's a GUI application, it must call AllocConsole first to allocate a console.
Let's begin the work. In order to create an anonymous pipe you need to call CreatePipe. CreatePipe has the following prototype:

CreatePipe proto pReadHandle:DWORD, \
       pWriteHandle:DWORD,\
       pPipeAttributes:DWORD,\
       nBufferSize:DWORD

pReadHandle is a pointer to a dword variable that will receive the handle to the read end of the pipe
pWriteHandle is a pointer to a dword variable that will receive the handle to the write end of the pipe.
pPipeAttributes points to a SECURITY_ATTRIBUTES structure that determines whether the returned read & write handles are inheritable by child processes
nBufferSize is the suggested size of the buffer the pipe will reserve for use. This is a suggested size only. You can use NULL to tell the function to use the default size.

If the call is successful, the return value is nonzero. If it failed, the return value is zero.
After the call is successful, you will get two handles, one to read end of the pipe and the other to the write end. Now I will outline the steps needed for redirecting the standard output of a child console program to your own process.Note that my method differs from the one in Borland's win32 api reference. The method in win32 api reference assumes the parent process is a console application and thus the child can inherit the standard handles from it. But most of the time, we will need to redirect output from a console application to a GUI one.

Create an anonymous pipe with CreatePipe. Don't forget to set the bInheritable member of SECURITY_ATTRIBUTES to TRUE so the handles are inheritable.
Now we must prepare the parameters we will pass to CreateProcess since we will use it to load the child console application. One important structure is the STARTUPINFO structure. This structure determines the appearance of the main window of the child process when it first appears. This structure is vital to our purpose. You can hide the main window and pass the pipe handle to the child console process with it. Below is the members you must fill:

cb : the size of STARTUPINFO structure
dwFlags : the binary bit flags that determine which members of the structure are valid also it governs the show/hide state of the main window. For our purpose, you should use a combination of STARTF_USESHOWWINDOW and STARTF_USESTDHANDLES
hStdOutput and hStdError : the handles you want the child process to use as standard output/error handles. For our purpose, we will pass write handle of the pipe as the standard output and error of the child. So when the child outputs something to the standard output/error, it actually passes the info via the pipe to the parent process.
wShowWindow governs the show/hide state of the main window. For our purpose, we don't want the console window of the child to show so we put SW_HIDE into this member.

Call CreateProcess to load the child application. After CreateProcess is successful, the child is still dormant. It is loaded into memory but it doesn't run immediately
Close the write pipe handle. This is necessary. Because the parent process has no use for the write pipe handle, and the pipe won't work if there are more than one write end, we MUST close it before reading the data from the pipe. However, don't close the write handle before calling CreateProcess, your pipe will be broken. You should close it just after CreateProcess returns and before you read data from the read end of the pipe.
Now you can read data from the read end of the pipe with ReadFile. With ReadFile, you kick the child process into running mode. It will start execution and when it writes something to the standard output handle (which is actually the handle to the write end of the pipe), the data are sent through the pipe to the read end. You can think of ReadFile as sucking data from the read end of the pipe. You must call ReadFile repeatedly until it returns 0 which means there are no more data to be read. You can do anything with the data you read from the pipe. In our example, I put them into an edit control.
Close the read pipe handle.

Example:

.386
.model flat,stdcall
option casemap:none
include \masm32\include\windows.inc
include \masm32\include\user32.inc
include \masm32\include\kernel32.inc
include \masm32\include\gdi32.inc
includelib \masm32\lib\gdi32.lib
includelib \masm32\lib\user32.lib
includelib \masm32\lib\kernel32.lib
WinMain PROTO :DWORD,:DWORD,:DWORD,:DWORD
.const
IDR_MAINMENU equ 101         ; the ID of the main menu
IDM_ASSEMBLE equ 40001
.data
ClassName            db "PipeWinClass",0
AppName              db "One-way Pipe Example",0 EditClass db "EDIT",0
CreatePipeError     db "Error during pipe creation",0
CreateProcessError     db "Error during process creation",0
CommandLine     db "ml /c /coff /Cp test.asm",0
.data?
hInstance HINSTANCE ?
hwndEdit dd ?
.code
start:
    invoke GetModuleHandle, NULL
    mov hInstance,eax
    invoke WinMain, hInstance,NULL,NULL, SW_SHOWDEFAULT
    invoke ExitProcess,eax
WinMain proc hInst:DWORD,hPrevInst:DWORD,CmdLine:DWORD,CmdShow:DWORD
    LOCAL wc:WNDCLASSEX
    LOCAL msg:MSG
    LOCAL hwnd:HWND
    mov wc.cbSize,SIZEOF WNDCLASSEX
    mov wc.style, CS_HREDRAW or CS_VREDRAW mov wc.lpfnWndProc, OFFSET WndProc
    mov wc.cbClsExtra,NULL
    mov wc.cbWndExtra,NULL
    push hInst
    pop wc.hInstance
    mov wc.hbrBackground,COLOR_APPWORKSPACE
    mov wc.lpszMenuName,IDR_MAINMENU
    mov wc.lpszClassName,OFFSET ClassName
    invoke LoadIcon,NULL,IDI_APPLICATION
    mov wc.hIcon,eax
    mov wc.hIconSm,eax
    invoke LoadCursor,NULL,IDC_ARROW
    mov wc.hCursor,eax
    invoke RegisterClassEx, addr wc
    invoke CreateWindowEx,WS_EX_CLIENTEDGE,ADDR ClassName,ADDR AppName,\ WS_OVERLAPPEDWINDOW+WS_VISIBLE,CW_USEDEFAULT,\ CW_USEDEFAULT,400,200,NULL,NULL,\ hInst,NULL
    mov hwnd,eax
    .while TRUE
        invoke GetMessage, ADDR msg,NULL,0,0
        .BREAK .IF (!eax)
        invoke TranslateMessage, ADDR msg
        invoke DispatchMessage, ADDR msg
    .endw
    mov eax,msg.wParam
    ret
WinMain endp
WndProc proc hWnd:HWND, uMsg:UINT, wParam:WPARAM, lParam:LPARAM
    LOCAL rect:RECT
    LOCAL hRead:DWORD
    LOCAL hWrite:DWORD
    LOCAL startupinfo:STARTUPINFO
    LOCAL pinfo:PROCESS_INFORMATION
    LOCAL buffer[1024]:byte
    LOCAL bytesRead:DWORD
    LOCAL hdc:DWORD
    LOCAL sat:SECURITY_ATTRIBUTES
    .if uMsg==WM_CREATE
        invoke CreateWindowEx,NULL,addr EditClass, NULL, WS_CHILD+ WS_VISIBLE+ ES_MULTILINE+ ES_AUTOHSCROLL+ ES_AUTOVSCROLL, 0, 0, 0, 0, hWnd, NULL, hInstance, NULL
        mov hwndEdit,eax
    .elseif uMsg==WM_CTLCOLOREDIT
        invoke SetTextColor,wParam,Yellow
        invoke SetBkColor,wParam,Black
       invoke GetStockObject,BLACK_BRUSH
        ret
    .elseif uMsg==WM_SIZE
        mov edx,lParam
        mov ecx,edx
        shr ecx,16
        and edx,0ffffh
        invoke MoveWindow,hwndEdit,0,0,edx,ecx,TRUE
    .elseif uMsg==WM_COMMAND
       .if lParam==0
            mov eax,wParam
            .if ax==IDM_ASSEMBLE
                mov sat.niLength,sizeof SECURITY_ATTRIBUTES
                mov sat.lpSecurityDescriptor,NULL
                mov sat.bInheritHandle,TRUE
                invoke CreatePipe,addr hRead,addr hWrite,addr sat,NULL
                .if eax==NULL
                    invoke MessageBox, hWnd, addr CreatePipeError, addr AppName, MB_ICONERROR+ MB_OK
                .else
                    mov startupinfo.cb,sizeof STARTUPINFO
                    invoke GetStartupInfo,addr startupinfo
                    mov eax, hWrite
                    mov startupinfo.hStdOutput,eax
                    mov startupinfo.hStdError,eax
                    or startupinfo.dwFlags, STARTF_USESHOWWINDOW+ STARTF_USESTDHANDLES
                    mov startupinfo.wShowWindow,SW_HIDE
                    invoke CreateProcess, NULL, addr CommandLine, NULL, NULL, TRUE, NULL, NULL, NULL, addr startupinfo, addr pinfo
                    .if eax==NULL
                        invoke MessageBox,hWnd,addr CreateProcessError,addr         AppName,MB_ICONERROR+MB_OK
                    .else
                        invoke CloseHandle,hWrite
                        .while TRUE
                            invoke RtlZeroMemory,addr buffer,1024
                            invoke ReadFile,hRead,addr buffer,1023,addr bytesRead,NULL
                            .if eax==NULL
                                .break
                            .endif
                            invoke SendMessage,hwndEdit,EM_SETSEL,-1,0
                            invoke SendMessage,hwndEdit,EM_REPLACESEL,FALSE,addr buffer
                        .endw
                    .endif
                    invoke CloseHandle,hRead
                .endif
            .endif
        .endif
    .elseif uMsg==WM_DESTROY
        invoke PostQuitMessage,NULL
    .else
        invoke DefWindowProc,hWnd,uMsg,wParam,lParam ret
    .endif
    xor eax,eax
    ret
WndProc endp
end start

Analysis:

The example will call ml.exe to assemble a file named test.asm and redirect the output of ml.exe to the edit control in its client area.
When the program is loaded, it registers the window class and creates the main window as usual. The first thing it does during main window creation is to create an edit control which will be used to display the output of ml.exe.
Now the interesting part, we will change the text and background color of the edit control. When an edit control is going to paint its client area, it sends WM_CTLCOLOREDIT message to its parent.
wParam contains the handle to the device context that the edit control will use to write its own client area. We can use this opportunity to modify the characteristics of the HDC.

    .elseif uMsg==WM_CTLCOLOREDIT
        invoke SetTextColor,wParam,Yellow
        invoke SetTextColor,wParam,Black
        invoke GetStockObject,BLACK_BRUSH
        ret

SetTextColor changes the text color to yellow. SetTextColor changes the background color of the text to black. And lastly, we obtain the handle to the black brush which we return to Windows. With WM_CTLCOLOREDIT message, you must return a handle to a brush which Windows will use to paint the background of the edit control. In our example, I want the background to be black so I return the handle to the black brush to Windows.
Now when the user selects Assemble menuitem, it creates an anonymous pipe.

            .if ax==IDM_ASSEMBLE
                mov sat.niLength,sizeof SECURITY_ATTRIBUTES
                mov sat.lpSecurityDescriptor,NULL
                mov sat.bInheritHandle,TRUE

Prior to calling CreatePipe, we must fill the SECURITY_ATTRIBUTES structure first. Note that we can use NULL in lpSecurityDescriptor member if we don't care about security. And the bInheritHandle member must be TRUE so that the pipe handles are inheritable to the child process.

invoke CreatePipe,addr hRead,addr hWrite,addr sat,NULL

After that, we call CreatePipe which, if successful, will fill hRead and hWrite variables with the handles to read and write ends of the pipe respectively.

                    mov startupinfo.cb,sizeof STARTUPINFO
                    invoke GetStartupInfo,addr startupinfo
                    mov eax, hWrite
                    mov startupinfo.hStdOutput,eax
                    mov startupinfo.hStdError,eax
                    mov startupinfo.dwFlags, STARTF_USESHOWWINDOW+ STARTF_USESTDHANDLES
                    mov startupinfo.wShowWindow,SW_HIDE

Next we must fill the STARTUPINFO structure. We call GetStartupInfo to fill the STARTUPINFO structure with default values of the parent process. You MUST fill the STARTUPINFO structure with this call if you intend your code to work under both win9x and NT. After GetStartupInfo call returns, you can modify the members that are important. We copy the handle to the write end of the pipe into hStdOutput and hStdError since we want the child process to use it instead of the default standard output/error handles. We also want to hide the console window of the child process, so we put SW_HIDE value into wShowWidow member. And lastly, we must indicate that hStdOutput, hStdError and wShowWindow members are valid and must be used by specifying the flags STARTF_USESHOWWINDOW and STARTF_USESTDHANDLES in dwFlags member.

invoke CreateProcess, NULL, addr CommandLine, NULL, NULL, TRUE, NULL, NULL, NULL, addr startupinfo, addr pinfo

We now create the child process with CreateProcess call. Note that the bInheritHandles parameter must be set to TRUE for the pipe handle to work.

invoke CloseHandle,hWrite

After we successfully create the child process, we must close the write end of the pipe. Remember that we passed the write handle to the child process via STARTUPINFO structure. If we don't close the write handle from our end, there will be two write ends. And that the pipe will not work. We must close the write handle after CreateProcess but before we read data from the read end of the pipe.

                        .while TRUE
                            invoke RtlZeroMemory,addr buffer,1024
                            invoke ReadFile,hRead,addr buffer,1023,addr bytesRead,NULL
                            .if eax==NULL
                                .break
                            .endif
                            invoke SendMessage,hwndEdit,EM_SETSEL,-1,0
                            invoke SendMessage,hwndEdit,EM_REPLACESEL,FALSE,addr buffer
                        .endw

Now we are ready to read the data from the standard output of the child process. We will stay in an infinite loop until there are no more data left to read from the pipe. We call RtlZeroMemory to fill the buffer with zeroes then call ReadFile, passing the read handle of the pipe in place of a file handle. Note that we only read a maximum of 1023 bytes since we need the data to be an ASCIIZ string which we can pass on to the edit control.
When ReadFile returns with the data in the buffer, we fill the data into the edit control. However, there is a slight problem here. If we use SetWindowText to put the data into the edit control, the new data will overwrite existing data! We want the data to append to the end of the existing data.
To achieve that goal, we first move the caret to the end of the text in the edit control by sending EM_SETSEL message with wParam==-1. Next, we append the data at that point with EM_REPLACESEL message.

invoke CloseHandle,hRead

When ReadFile returns NULL, we break out of the loop and close the read handle.

Iczelion's Win32 Assembly Homepage