Android Layer && LayerStack

In SurfaceFlinger, the basic logic unit to show is a Layer (which corresponds to one ViewRootImpl in app process). The contents users see is a set of Layers with depth information. With depth info, some Layer displayed in front, while some in back, and this depth info is the z value. The larger, the nearer to user. WindowManagerService can adjust the z value through SurfaceControl.setLayer, which conveys this info to SurfaceFlinger so create different effect. For example, user can enable StrictMode violation flash in developer option, once there is violation, WMS creates a Layer, and make it visible for a while, then make it invisible. To prevent other Layer from hiding the flash effect, the z value of StrictMode Layer is adjusted so that it would be displayed on top to get attention.

The above works perfect for Layers showing on single display. When multiple displays are supported, we need a way to know which Layer to show on which display, hence the LayerStack. LayerStack is an attribute for both a Layer object and a display. When a Layer has same LayerStack as a display, it would be showed on that display. Effectively, LayerStack creates an depth space for a display, so that each display has its own depth space. One immediate use case for LayerStack is mirror mode. When a user plugin a HDMI to device, both displays shows the same content (mirror mode). WMS just set the LayerStack for HDMI display to the same value as panel, so that when SF composes contents for HDMI, it picks up same set of Layers as primary display.

Display LayerStack
In framework, a display device is described by a subclass of DisplayDevice. There are several different subclass of it, let's make it simple, using LocalDisplayDevice as an example. A LocalDisplayDevice means a physical device that can connect to the tablet or phone. Internal panel and HDMI are examples of LocalDisplayDevice. When a display is connected, framework creates an object of LocalDisplayDevice, and enumerates a LayerStack for this device (now LayerStack is same as display id, but they do not need to be the same), then using this LayerStack, it creates an associated LogicalDisplay object. One LogicalDisplay associates with one LocalDisplayDevice, and LogicalDisplay decides the LayerStack to use, then tell SurfaceFlinger through LocalDisplayDevice, LocalDisplayDevice probably can be think of an interface to SF for display related setting. So, basically if you connect HDMI on your device, you'll have 2 LogicalDisplay, each with its own LayerStack. But in mirror mode, LayerStack of both display is the same, what happened ? The trick is in DisplayManagerService.configureDisplayInTransactionLocked()

    private void configureDisplayInTransactionLocked(DisplayDevice device) {
        final DisplayDeviceInfo info = device.getDisplayDeviceInfoLocked();
        final boolean ownContent = (info.flags & DisplayDeviceInfo.FLAG_OWN_CONTENT_ONLY) != 0;

        // Find the logical display that the display device is showing.
        // Certain displays only ever show their own content.
        LogicalDisplay display = findLogicalDisplayForDeviceLocked(device);
        if (!ownContent) {
            if (display != null && !display.hasContentLocked()) {
                // If the display does not have any content of its own, then
                // automatically mirror the default logical display contents.
                display = null;
            }
            if (display == null) {
                display = mLogicalDisplays.get(Display.DEFAULT_DISPLAY);
            }        
        }
        ....
        display.configureDisplayInTransactionLocked(device, info.state == Display.STATE_OFF);        ....
    }

When mirror mode is in use, FLAG_OWN_CONTENT_ONLY is not set for LogicalDisplay of HDMI, so the LogicalDisplay used to configure HDMI monitor is that of internal panel (see above code), that's why in that case, the LayerStack in both display in SF are the same, hence showing the same contents.

LayerStack@Layer
Layer is the concept used in SurfaceFlinger. It has different face in different places. A Layer in SF has a corresponding WindowState in WMS, and a corresponding ViewRootImpl object in app process. When a ViewRootImpl object is created, it has an associated display, and this display decides the LayerStack value to be assigned by WMS and used in SF. In WMS, each WindowState object has its own WindowStateAnimator object, which ask SF to create a Layer object for it. Below code is from WindowStateAnimator.

    SurfaceControl createSurfaceLocked() {
        ....
        // Start a new transaction and apply position & offset.
        SurfaceControl.openTransaction();
        try {
            ....
            try {
                ....
                final DisplayContent displayContent = w.getDisplayContent();
                if (displayContent != null) {           
                mSurfaceControl.setLayerStack(displayContent.getDisplay().getLayerStack());
            }
            ....
        } finally {
            SurfaceControl.closeTransaction();
        }
       }

WMS maintains a DisplayContent object to store the list of WindowState object to show in this display. From DisplayContent, it can get the display object, then get the LayerStack of it, and assign this to the Layer object in SF.

SurfaceFlinger State Update

SurfaceFlinger主要將State分成兩種, 一種是ComposerState, 另一種是DisplayState. ComposerState是跟Layer有關, DisplayState則是跟Display (Monitor)有關. 這兩種State會被儲存在SurfaceFlinger::State的物件裡.

    struct State {
        LayerVector layersSortedByZ;
        DefaultKeyedVector< wp, DisplayDeviceState> displays;
    };

layersSortedByZ儲存的是排序過的所有Layer物件, displays則是所有Display的State. layersSortedByZ是先以LayerStack排序(由小到大), 如果LayerStack相同, 再以Layer的Z做排序(由小到大).

State的變化由client的request開始, client必須透過ISurfaceComposerClient的interface與SF溝通. client可以是NDK app, 也可以是Java app. client在下command時是這樣子的, 它必須將request包在openGlobalTransaction與closeGlobalTransaction裡, 中間的每一個set function會更新ComposerState或DisplayState, 然後統一在closeGlobalTransaction時全部交給SurfaceFlinger::setTransactionState. 所以, 嚴格的來說, SurfaceFlinger::setTransactionState是SF對client的唯一interface.

    SurfaceComposerClient::openGlobalTransaction();
    surfaceControl->setXXX(...);
    surfaceControl->setYYY(...);
    SurfaceComposerClient::closeGlobalTransaction();

這裡有一個Threading Model要注意一下. SurfaceFlinger::setTransactionState是run在client所使用的binder thread. 讓我多解釋一下這句話. client與SF通常是分屬不同Process, 當client呼叫ISurfaceComposerClient的function時, 它是執行在SF的process裡, 但是這是透過binder做到的, SF裡有binder thread pool, 所以當client呼叫ISurfaceComposerClient時, binder會從SF的binder thread pool取得一個thread來執行, client本身的thread這個時候通常是被block的. 所以, SurfaceFlinger::setTransactionState是執行在SF的binder thread, 而SF本身state的update, 以至於後續的動作是在SF的main thread裡執行, 而main thread通常是在VSYNC-sf開始的時候才執行, 下面這張圖或許可以讓你比較清楚一點.

在VSYNC-sf發生時, SF的main thread會讀取內部的Layer與DisplayDeviceState物件, 為了避免SF在讀的時候, client正在更新state, SurfaceFlinger::mStateLock會用來避免這件事.

SurfaceFlinger裡有好幾個thread, 其中一個是main thread, 也就是SurfaceFlinger主要執行的thread, 另外有兩個VSYNC thread, 一個用來產生VSYNC-sf, 另一個產生VSYNC-app, 在這裡我們主要著重在VSYNC-sf, 簡單稱為VSYNC-sf thread. VSYNC-sf thread的requestNextVSync必須被呼叫, VSYNC-sf thread才會在VSYNC-sf發生時, queue一個INVALIDATE的message給main thread, main thread才會被叫醒, SF的state才會更新, 反應在使用者面前.

VSYNC-sf的requestNextVsync是在MessageQueue::invalidate()中被呼叫到的, Client (SF Client, 透過ISurfaceComposerClient)或main thread是使用setTransactionFlags()-->signalTransaction()-->MessageQueue::invalidate()這條路徑.

Double State Tracking

SurfaceFlinger有兩個State物件, 分別是mCurrentState與mDrawingState. mCurrentState指的是新要求的State, 但還沒反應在使用者面前, mDrawingState則是指目前已經反應在使用者面前. 但是在某個時間點(commitTransaction), mCurrentState的內容會被複製到mDrawingState. 利用這兩個State, 就可以知道有什麼需要更新. 除了SF本身外, 每一個Layer也有mCurrentState與mDrawingState, 具有相同的意義.

Layer
Layer代表Activity裡一個ViewRootImpl的畫面, 每一個ViewRootImp都有相對應的Layer, Android的畫面就是由多個Layer所組合而成, 這些Layer有前後關係(用Z值作區分), 因此, 有些Layer會蓋在某些Layer之上, 有些Layer是透明或半透明, SF將這些Layer組成一個畫面顯示在使用者面前.

當一個Layer產生時, 它會產生一個handle, 這個Handle是要傳回給client, 當client要調整這個Layer屬性時, 需要將這個handle傳到SF, SF才有辦法找到相對的Layer物件. ($TOP/frameworks/base/core/java/android/view/SurfaceControl.java). handle是一個簡單BBinder物件

當這個handle被destroy後, LayerCleaner::~LayerCleaner()會被執行, 再去呼叫SurfaceFlinger::onLayerDestroyed()做後續處理.

每一個ViewRootImp的畫面都是由client所繪製的, SF只是負責將它組合顯示在使用者面前, 因此, 一個Layer產生後第一次被使用時, SF會產生一個BufferQueue, Producer是MonitorProducer物件, Consumer則是SurfaceFlingerConsumer物件, Producer的interface會傳回給client, client可以利用producer interface將畫好的GraphicBuffer放入BQ, 然後由SF使用.

    void Layer::onFirstRef() {
        ...
        // Creates a custom BufferQueue for SurfaceFlingerConsumer to use
        sp producer;
        sp consumer;
        BufferQueue::createBufferQueue(&producer, &consumer);
        mProducer = new MonitoredProducer(producer, mFlinger);
        mSurfaceFlingerConsumer = new SurfaceFlingerConsumer(consumer, mTextureName);
        mSurfaceFlingerConsumer->setConsumerUsageBits(getEffectiveUsage(0));
        mSurfaceFlingerConsumer->setContentsChangedListener(this);
        mSurfaceFlingerConsumer->setName(mName);
        ....
    }

MonitoredProducer

MonitoredProducer是一個很簡單的Pass-Through class, 所有的Producer interface都是直接呼叫createBufferQueue()所產生的producer object. 它有一個cleanup的功能, 當MonitoredProducer被destroy時, 它會post一個message給SF去做clean的工作, 其實只是將Producer從SurfaceFlinger::mGraphicBufferProducerList中移除而已.

SurfaceFlingerConsumer
SurfaceFlingerConsumer是GLConsumer的subclass, 這個consumer的用途可以將MonitoredProducer所放入的GraphicBuffer取出當做Texture使用, 當它的updateTexImage()被呼叫時, 它會從BQ裡取出Buffer供SurfaceFlinger::RenderEngine使用. 一般而言, 當Producer將Buffer放入BQ時, Consumer的onFrameAvailable()會被執行, 但是在上面的code裡有一行

    mSurfaceFlingerConsumer->setContentsChangedListener(this);

這一行會使得Layer::onFrameAvailable()被執行, 而不是SurfaceFlingerConsumer::onFrameAvailable. 當Layer::onFrameAvailable()被執行時, 會對Layer::mQueuedFrames累加, 這個變數代表這個Layer有幾個畫面已經更新, 但還沒被SF處理.

    void Layer::onFrameAvailable() {
        android_atomic_inc(&mQueuedFrames);
        mFlinger->signalLayerUpdate();
    }

SurfaceFlinger::signalLayerUpdate()會通知在下一個VSYNC-sf時, 將SF的main thread叫起來更新畫面, 稍後會看到.

ComposerState
ComposerState包含client的pointer與一layer_state_t的結構. layer_state_t::surface存的是Layer的handle, 也就是上面提到, 當Layer被create時, 它會產生一個Handle物件傳回給client. Client要對這個Layer進行操作必須將這個handle傳回SF, SF會利用Client::getLayerUser()從client所有的layer裡找出來. 一個client可以有多個layer, 最常見的case就是一個Activity叫出Dialog, 這個Dialog也是一個Layer, 因此這個client就有2個layer.

    struct ComposerState {
        sp client;
        layer_state_t state;
        status_t    write(Parcel& output) const;
        status_t    read(const Parcel& input);
    };

ComposerState裡主要的是layer_state_t, 這個class會攜帶client所要改變的屬性. layer_state_t是一個public interface, 用來做為client與SF的溝通, SF有自己的data structure (Layer class), 因此, 當SF收到新的ComposerClient (經由setTransactionState), 會將它的內容反應在Layer物件裡, SurfaceFlinger::setClientStateLocked()負責將layer_state_t轉換到Layer, 然後在VSYNC開始時, 才由SurfaceFlinger::handleTransaction()做進一步的處理.

Display
在SurfaceFlinger裡, Display可以分成兩種, 一種是Physical Display, 也就是一個實體的Monitor或Panel接在機器上, 另一種是Virtual Display, 像是WifiDisplay或Chromecast, 它不是一個實體的Monitor連接著你的機器.

SurfaceFlinger利用幾個data structure來管理Display.

SurfaceFlinger::mBuiltinDisplays
這是一個table, 每個entry都是一個BBinder物件, 每一個BBinder代表一個Physical Display的Token, Virtual Display的Token不會存在這裡.

SurfaceFlinger::State::displays
這是一個mapping table, 利用Display的Token, 可以取得該Display的DisplayDeviceState, 所有的Display的DisplayDeviceState (包含Virtual Display)都可以從這裡取得. DisplayDeviceState記錄該Display的狀態, 它的內容是根據client傳進來的DisplayState而設定的.

SurfaceFlinger::mDisplays
這也是一個mapping table, 利用Display的Token, 可以取得該Display的DisplayDevice物件.

在SF裡, 每一個Display, 不管是Physical或Virtual都有一個Token, 可以把它想成是個ID用來辨識不同的Display. Physical Display的Token是BBinder class, Virtual Display則是DisplayToken class. DisplayToken是BBinder的subclass, 所以它也可以以BBinder的形態儲存.

Token只是一個類似ID的作用, 真正在記錄或處理Display的則是一個叫DisplayDevice的class. SurfaceFlinger::mDisplays記錄著每一個Display(包含Virtual)的DisplayDevice物件, 利用Display的Token可以找到相對應的DisplayDevice.

DisplayState
DisplayState是client跟SF溝通的一個結構, SF會將這個結構的內容反應在每個Display的DisplayDeviceState裡, 當SF main thread執行時, 它會比較mCurrentState與mDrawingState來決定是否有display hot plug/unplug發生, 也會更新一些屬性, 像是width, height, projection matrix等等.


DisplayDevice
不管前面的Flow如何, SF最終要將相關的State更新到DisplayDevice物件裡. 針對每一個Display, 除了會有一個DisplayDevice外, SF還會為每一個Display產生一個BufferQueue. DisplayDevice物件在被create時, 會利用eglCreateWindowSurface產生一個EGLSurface當作BQ的Producer. Physical與Virtual Display在這方面有點不同, 請見下圖.

Producer是EGLSurface, 因為它主要是用在GLES composition. EGLContext是使用SurfaceFlinger的RenderEngine.

DisplayDevice比較像是一個PlaceHolder放Display相關的資料, SF在稍後更新Layer或HWComposer的狀態時, 會用到裡面的資料.

Wrap Up
在看SF的code, 最好能掌握一個原則, SF的main thread才是主導State的關鍵, 而且它只有在VSYNC-sf發生時才會被叫醒, 其餘時間它是在sleep的狀態. 另外, client主要是透過setTransactionState將client state (ComposerState && DisplayState)轉換成SF內部的state (Layer && DisplayDeviceState). SF main thread只從內部的state來比較是否有任何新的request, 然後才去處理. 它的流程大概是像下面這個樣子

• Client (binder thread)呼叫setTransactionState傳入ComposerState與DisplayState, setTransactionState會做幾件事
• 利用setClientStateLocked()與setDisplayStateLocked()將ComposerState與DisplayState反應在Layer與DisplayDeviceState裡.
• 利用setTransactionFlag將SF main thread叫醒.
• 如果client要求synchronous mode, setTransactionState就會一直等到main thread將state處理完才return.因此, client使用的binder thread會被block住直到main thread執行完handleTranslactionLocked.
• SF main thread被叫醒後, 會利用handleTransactionLocked()檢查各個Layer與DisplayDeviceState. 接下來我們會看handleTransactionLocked()

handleTransactionLocked()
這個function看起來似乎比較長, 但實際上它的邏輯不複雜, 我們可以將它分成幾個部份來看

Layer Update
檢查每一個Layer是否有任何的更新, 如果有, 就利用Layer::doTransaction()對該Layer做更新.

Layer::doTransaction()跟handleTransactionLocked()的意義是一樣的, 差別在於doTranaction是對Layer做state的檢查, 但handleTransactionLocked()是針對整個SF做state的檢查. 把它想成一層一層的管理, 就會比較容易理解.

Display Update
在這裡會比較mCurrentState與mDrawingState裡的displays, 從這兩個State裡display的不同, 可以得知是否有hot plug/unplug的發生.

• 如果一個Display在mDrawingState.displays, 但不在mCurrentState.displays, 代表該Display發生unplug. 需要呼叫EventThread::onHotplugReceived()通知其它component有display unplug.並將DisplayDevice物件從SurfaceFlinger::mDisplays中移除.
• 如果一個Display在mCurrentState.displays, 但不在mDrawingState.displays裡, 代表該Display發生hot plug. 除了要利用EventThread::onHotplugReceived()通知其它componenet外, 還必須產生一個新DisplayDevice放到SurfaceFlinger::mDisplays裡. 而且必須為這個Display產生BufferQueue, 並更新DisplayDevice的一些屬性, 像是layerStack, projection matrix等.
• 如果一個Display同時出現在mCurrentState.displays與mDrawingState.displays, 表示這個Display可能有些屬性改變, 像LayerStack, projection matrix等.

Dirty Region Update
由於Activity可能會結束, 因此Layer也會被刪除, 這有可能會影響到畫面上可看見的區域. 透過mCurrentState與mDrawingState的比較, 可以知道哪些Layer被刪除了, 這些被刪除的Layer的顯示區域必須設成Dirty, 要重畫.

上面紅色區域就是用來計算被移掉的Layer的可見區域, 然後交給invalidateLayerStack()來更DisplayDevice裡的dirty region. 如果我們看一下invalidateLayerStack(), 它會檢查被移除Layer的LayerStack是否與DisplayDevice的LayerStack相同, 才會更新DisplayDevice的dirty region. LayerStack是用來決定一個Layer是否要在一個Display上顯示出來. 我們會再另外討論這個東西.

在handleTransactionLocked()的最後會有一個commitTransaction()的動作, 這個很重要, 因為它會將mCurrentState的內容複製到mDrawingState裡, 這代表在這之後, 如果要知道哪些State有更新, 要去參考mDrawingState, 而不是mCurrentState. 因為從這之後, mCurrentState就會繼續接收client新的state update.