SQLite3 support and inspection of free pages

The upcoming 1.0.0 version of the Profiler introduces support for SQLite3 databases.

SQLite table

You’ll see that even viewing large tables is pleasantly fast. The SQL table control is available to the Python SDK as well: it can either be created via createView or inside a custom view with the tag sqltable.

Once the sql table view is created, it offers the following methods:

    getSQLColumns() -> NTString
    getSQLCondition() -> NTString
    getSQLTable() -> NTString
    setSQLTable(NTString const & table, NTString const & columns=NTString(), NTString const & condition=NTString()) -> bool
    setSQLTable(NTString const & table, NTString const & columns=NTString()) -> bool
    setSQLTable(NTString const & table) -> bool
    setSQLTableSelectVisible(bool b)
    setSQLite3Object(CFFObject obj)

So it’s possible to display a particular table in it or offer the possibility to the user to choose the table via setSQLTableSelectVisible.

The database can be accessed as well. The Profiler exposes its internal SQLite code in the homonymous module. It differs from the standard Python implementation and it matches the C API. For instance, to enumerate the table names in a database we can use this code:

from Pro.SQLite import *

db = obj.GetHandle() # retrieves the internal SQLite handle, never to be closed!

ret, stmt = sqlite3_prepare(db, "SELECT name FROM sqlite_master WHERE type = 'table'")
if sqlite3_step(stmt) == SQLITE_ROW:
    print(sqlite3_column_text(stmt, 0))
    sqlite3_finalize(stmt)

The handle returned by GetHandle grants only read access. In fact, to maximize speed and avoiding copy operations, the Profiler replaces the virtual file-system of the SQLite database in order for it to read directly from the CFFObject.

The exposed C API can be used to open external databases as well and will be used to access the main report database file in order to give plugins the capability to store and retrieve their own data.

Free pages inspection

When the database file contains free pages, it will be reported in the summary. Free pages usually contain deleted data and can therefore be of interest for forensic purposes.

Free pages

The image above shows a test database I’ve created. In it I created a few tables, and inserted some records containing repeated values (but keeping each record different). Then I deleted a specific record containing ‘1’s. The result is that the database now contains free pages and when inspecting them with the Profiler we can see a big part of the original data.

Keep in mind that data contained in free pages can be incomplete and is scattered. The free pages data can be retrieved programmatically as well through the method GetFreePages.

Stay tuned as there’s much more coming soon!

News for version 0.9.9

The new 0.9.9 version of the Profiler is out with the following news:

added support for docked views in the main window
added scanning and rload (report load) hook notifications
partially exposed custom views to Python
exposed addEmbeddedObject method to Python
exposed NTContainer find methods to Python
improved importing of anonymous records (C11)
– added recognition of volatile keyword in types
– moved the message box constants to the Pro.Core module
– added tools view
– added quoted-printable decoding filter
added format quota calculator extension
added experimental EML attachment detection extension

Improved importing of anonymous records

C11 supports anonymous records like the following:

struct test {
    union {
        struct {
            unsigned int a;
            unsigned int b;
        };
        struct {
            unsigned int c;
            unsigned int d;
        };
        struct {
            unsigned int e;
            unsigned int f;
        };
    };
};

Notice that not only is the union anonymous but even its substructures are. The Header Manager is now capable of correctly importing this code. As usual anonymous types will be renamed (both their type and name).

Creating undetected malware for OS X

We have discovered a way to defeat current anti-malware solutions. We will publicly disclose the full details of the issue in a few weeks.

In the meantime, we’re more than happy to confidentially disclose the information with interested organizations (either security vendors or known companies which could benefit from it). Just send an email to: info@icerbero.com

-----BEGIN PGP PUBLIC KEY BLOCK-----
mQENBE1j8U0BCACm3tMNVVDb4gIEGdPYq3le5gzBngN43J7SXvGH6nlDnG6s/zS1
lBtecoqvtgXlS9KDonzq4KR0AfcEQh8ziwCgRbkgfQUxyIvJxt+cxW2zblnP37UQ
AuwhqO/Sc1yG3cT8wFSoaiF+tXJca879WEvimTyaoZSlXb3JuKt4UmDYbrOSLfDL
vd1rJ59R7GE5B2ThnSKDU/8pSvYVMKJdq0ArM8Nwg7gmUaBwpsvtEaGFB0gh3kBv
C/clsY8MR+MOBVI2f+95kekLIrUmOKzjXp5GkTgt5a6hqobU8jkVixN/KqgnT4Aj
LGPIyZkKgo/735SCT0JuWJKSHqJ+tw4aYUt7ABEBAAG0HkNlcmJlcm8gVUcgPGlu
Zm9AaWNlcmJlcm8uY29tPokBPgQTAQIAKAUCTWPxTQIbIwUJCWYBgAYLCQgHAwIG
FQgCCQoLBBYCAwECHgECF4AACgkQO3RcR15Gr56YXQf7BIKai3Y2i1xz4KAXfWmP
bsfuzHV4ZSsx8o1rnihtmXfN6qbPR1ySEfP/XgRXLZ2coerMYtx/Ydr4KnXlD1Sa
K7rGdTpqlwZ46p9RqtliFGELeglmHzD5ZCxawmHXf14OvTFgJKYqztFjBOvYMthI
xn5hdx/AqixCky5BMjdh7909cUP5Xzi0oYlpHcmiBsdPx8LPMh+DqGg9W0iahgnd
X+E0dW0dOEpg101esGMsHGaSbxw0+5ybh8XCIAl/50GLCVQWSE/8hxJnNoQy3AI/
P/d2olyGFVYbFVm+MUoRpx0zWgGP90n/Q9toGwl4qhhviyEl6vg6syYTuruhRcx7
D7kBDQRNY/FNAQgAskW9XYOXUd+DnEqGJywltTDpxSwwpCrfnqJG90YimVkK396G
ZG8uI5AnGqJ/+gThvgAMTY826WwlDP3DOyhmv1Iq7hKXDh9w2O5q8a1nsdaiGKws
7RBJ04xgfciifZRueGdEioiAFS5YmDLjdrBh6rX+6UfXTkbv1x1qodn1R9wFPxxS
nadpKwskG4YszNeViJxHMZTmnuKH9AOvCH7qiyWERNejeLRy1yFXVwD2HnCEjCNT
Loa3HvO5aJDT4Lww/w0McLPU0Tso5qQlXKk/I0C/llGD87rzuDffBswPQYfn2FkI
bHT5wdYh8Si+tA0oLI/bjRO254iFHDVgT/Vm3wARAQABiQElBBgBAgAPBQJNY/FN
AhsMBQkJZgGAAAoJEDt0XEdeRq+e+D4H/0W4oPHGv04y6KcuAR7XbgoXQ5fJVghY
XeKuYXD95WMT3W3PyoCirst9dX1MeJJ/wxi7dBCjT0iBbeb7mDERBQLi7L3hJnpg
wz1tokLb0QL+HNKIYZ8PsuuW3yQsbjSu1hCsCqNFe9nY3wkEDa3TWjjk5i1ejnnb
PCvGTOO/siwXGgZq7YWvoafCsdgbAwW8G6pO9BjZrrbDMMgFtQLWHLNBzDHTpWL3
BqjLlYisENQAO63FSAcu1ubhzFtIcVsjW8cgAxHQy4nN2RJHv23il+/PLsHquElP
gG4qSk8PudeEQUhFLLANRCSQ5yYlBhv4hJGGdAvYvYZQC36Nljg5WHI=
=SD9C
-----END PGP PUBLIC KEY BLOCK-----

EML attachment detection and inspection

The upcoming 0.9.9 version of the Profiler includes some very useful SDK additions. Among these, the addEmbeddedObject method (to add embedded objects) and a new hook notification called ‘scanning’. The scanning notification should be used for long operations and/or to add embedded objects. In this post we’ll demonstrate these new features with a little script to detect attachments in EML files.

EML attachments

One of the advantages of using the Profiler is that we are be able to inspect the sub-files of the attachments as well. The screenshot above shows a PNG contained in an ODT attachment. Nice, isn’t it?

But the nicest part is how little code is necessary to extend the functionality of the Profiler. These are the lines to add to the user hook configuration file:

[EML: detect attachments]
file = eml.py
scanning = detectEmlAttachments

And this is the Python code:

from Pro.Core import INVALID_STREAM_OFFSET

def detectEmlAttachmentsCb(offset, npattern, sp):
    c = sp.getObjectStream()
    # hdr range
    m = c.findFirst("\n--".encode("ascii"), 0, offset, False)
    hdrstart = 0 if m.offset == INVALID_STREAM_OFFSET else m.offset
    m = c.findFirst("\r\n\r\n".encode("ascii"), offset)
    hdrend = c.size() if m.offset == INVALID_STREAM_OFFSET else m.offset
    # make sure it's an attachment
    m = c.findFirst("Content-Disposition: attachment".encode("ascii") , hdrstart, hdrend - hdrstart)
    if m.offset == INVALID_STREAM_OFFSET:
        return 0
    # data range
    datastart = hdrend + 4
    m = c.findFirst("\r\n\r\n".encode("ascii"), datastart)
    dataend = c.size() if m.offset == INVALID_STREAM_OFFSET else m.offset
    # retrieve file name (if any)
    name = "no_name"
    m = c.findFirst('name='.encode("ascii"), hdrstart, hdrend - hdrstart)
    if m.offset != INVALID_STREAM_OFFSET:
        namestart = m.offset + 5
        namedel = "\r\n"
        if c.read(namestart, 1) == '"'.encode("ascii"):
            namedel = '"'
            namestart = namestart + 1
        m = c.findFirst(namedel.encode("ascii"), namestart)
        if m.offset != INVALID_STREAM_OFFSET:
            namesize = min(m.offset - namestart, 200)
            name = c.read(namestart, namesize).decode("utf-8")
    # add attachment
    sp.addEmbeddedObject(datastart, dataend - datastart, "?", name, "")
    return 0

def detectEmlAttachments(sp, ud):
    sp.getObjectStream().find(detectEmlAttachmentsCb, sp, "Content-Transfer-Encoding: base64".encode("ascii"))

That’s it. Of course, this is just a demonstration, to improve it we could add support for more encodings apart from ‘base64’ like ‘Quoted-Printable’ for instance.

Some email programs like Thunderbird store EML files by appending them in one single file. In fact, as you can see, the screenshot above displays the attachments of an entire Inbox database. 😉

EML attachment types

Also notice that in the code the addEmbeddedObject method is called by specifying a base64 decode filter to load the file. We can, of course, specify multiple filters and Lua ones as well. This makes it extremely easy to load files without having to write code to decode/decrypt/decompress them. The “?” parameter leaves the Profiler to identify the format of the attachment.

Format quota calculator

In the upcoming 0.9.9 version of the Profiler it will be possible to create docked views even in the context of the main window. This feature combined with custom views is extremely useful if we want to create custom reports at the end of a scan.

Some time ago I needed a little script to calculate the format quotas of files in a specific directory and their sub-files: we’ll use this sample to demonstrate the new features. For example we could use it to determine what kind of files and in what percentage the System32 directory on Windows contains. Or we could use it to determine the quotas of files in a Zip archive. To make it even more useful, the script now asks the user before the scan to enter the nesting range to consider. For example the value ‘0’ means all levels (starting from 0). If we want to calculate the quotas of top level files only, we must insert ‘0-0’ (start-end). The files contained in a Zip archive can be calculated with the value ‘1-1’ and if we want to include their sub-files we must insert ‘1’.

System32 quotas

We’re probably going to include the script in the upcoming release. But in case we don’t, in order to try it out, add the following lines to the hooks configuration file:

[Format Quota Calculator]
file = quotas.py
init = typeQuotaCalcaulatorInit
end = typeQuotaCalcaulatorEnd
scanned = typeQuotaCalcaulatorScanned

And create a ‘quotas.py’ file in your ‘plugins/python’ user directory with the following content:

from os import path
import random

def generateColor():
    c = ""
    for i in range(3):
        c = c + "%0.2X" % ((random.randint(0, 200) + 300) >> 1)
    return c

def typeQuotaCalcaulatorInit():
    random.seed(0)
    # ask for nesting levels to consider
    from Pro.UI import ProInput
    ns = ProInput.askText("Format Quota Calculator (nesting level: from(-to))", "0")
    lstart = 0
    lend = -1
    if ns != None:
        ns = ns.split("-")
        if len(ns) > 0:
            lstart = int(ns[0])
        if len(ns) > 1:
            lend = int(ns[1])
    return { "lstart" : lstart, "lend" : lend, "total" : 0, "quotas" : { } }

def typeQuotaCalcaulatorEnd(ud):
    from Pro.UI import proContext, ProView
    from html import escape
    prec = "%.2f"
    mbsize = 1024 * 1024
    u = ud["total"] / 100
    # prepare content
    s = "Total size: " + (prec % (ud["total"] / mbsize)) + " MBs\n"
    ui = ""
    for k,q in ud["quotas"].items():
        ps = (prec % (q / u))
        ss = (prec % (q / mbsize))
        s = s + "\n" + k + ": " + ps + "% (" + ss + " MBs)"
        ui = ui + ""
    ui = ui + ""
    # display view
    ctx = proContext()
    v = ctx.createView(ProView.Type_Custom, "Format quotas")
    v.setup(ui)
    v.getView(1).setText(s)
    ctx.addView(v)

def typeQuotaCalcaulatorScanned(sp, ud):
    # check nesting
    nesting = sp.scanNesting()
    if ud["lstart"] > nesting or (ud["lend"] >= 0 and ud["lend"] < nesting):
        return
    c = sp.getObjectStream()
    fmt = sp.getObjectFormat()
    # if we didn't recognize the file, use extension as format identifier
    # we could also use an external signature db...
    if fmt == "":
        fmt = path.splitext(c.name())[1]
        if len(fmt) > 0:
            fmt = fmt[1:] # skip dot
    if len(fmt) == 0:
        fmt = "?"
    else:
        fmt = fmt.upper()
    # add to quotas
    size = c.size()
    ud["total"] = ud["total"] + size
    if not fmt in ud["quotas"]:
        ud["quotas"][fmt] = 0
    ud["quotas"][fmt] = ud["quotas"][fmt] + size

Remember to activate the hook from the UI before running a scan.

Of course, the view will be displayed even after an individual file scan in the workspace.

PDF quotas

In order to improve the script, we could use an external signature database for those file formats not recognized automatically.

This is a perfect example of the capabilities to extend the functionality of the Profiler. While there’s yet no estimated release date for the upcoming version, keep in tune as we hope to publish very interesting stuff soon.

Custom Views

The upcoming 0.9.9 version of the Profiler will partially expose the use of custom views. These views are used internally by the Profiler to create complex graphical UIs using short XML strings. While at the moment extensions can use PySide to create complex UIs, it’s better to avoid it if possible, since it involves an extra dependency and also because PySide might not be ported to Qt 5 in the future.

But let’s see a code snippet:

from Pro.UI import *

ctx = proContext()
v = ctx.createView(ProView.Type_Custom, "Debug Directory")
v.setup("<ui><vsplitter><table id='0'/><hex id='1'/></vsplitter></ui>")
ctx.addView(v)

These few lines will display the following view:

Empty custom view

Controls can be organized in layouts (hlayout/vlayout), splitters (hsplitter/vsplitter) and tabs (tab). These elements are called containers. Available controls are: label, pie, plot, table, tree, hex, text and media.

More controls will be available in the future and not all of the current ones can be used as it is. Some controls make sense only in combination with a callback to be notified about changes of the state of the control. The notification system will be made available to Python as well in the future, but it made sense to release a partial solution in the meantime, because many views don’t require notifications and only need a way to display information at the end of an operation.

Let’s see for example how to make use of the UI above to display information.

Custom view

This code replicates the Debug Directory UI in Portable Executables.

from Pro.UI import *

ctx = proContext()
obj = ctx.currentScanProvider().getObject()
dbgdir = obj.DebugDirectory().MakeSingle()
dbgdata = ctx.currentScanProvider().getObjectStream()
dbgdata.setRange(*obj.DebugDirectoryData(dbgdir))

v = ctx.createView(ProView.Type_Custom, "Debug Directory")
v.setup("<ui><vsplitter><table id='0'/><hex id='1'/></vsplitter></ui>")
v.getView(0).setStruct(dbgdir)
v.getView(1).setData(dbgdata)
ctx.addView(v)

Elements in a view can have attributes. We’ve only seen the id attribute used to identify the embedded controls. There are two kind of attributes: shared attributes and individual ones. Only controls have these shared attributes: width, height, min-width, max-width, fixed-width and fixed-height. If a c is prefixed to the width/height word, then the size can be expressed in characters. e.g.: fixed-cwidth=’10’. Additionally, since version 1.3, there’s also wfixed and hfixed. Both are booleans which, if true, set the fixed size policy.

Here’s a list of individual attributes for controls and containers.

  • ui
    • bgcolor (e.g. ffffff)
  • hlayout/vlayout (hl/vl)
    • margin
    • spacing
    • align (hcenter, vcenter, center, top, left, bottom, right)
  • hsplitter/vsplitter (hs/vs)
    • sizes/csizes (separated by -)
  • tab
    • index
    • titles (separated by 😉
  • label
    • bgcolor (e.g. ffffff)
    • select (bool)
    • margin
  • text
    • readonly (bool)
    • linenr (bool, show line number)
    • hline (bool, highlight current line)
    • hword (bool, highlight current word)
    • wrap (bool)
  • combo (since version 1.3)
    • edit (bool)
    • text (string, only if editable)
  • btn (since version 1.3)
    • text (string, only if editable)
  • check (since version 1.3)
    • checked (bool)
    • text (string, only if editable)
  • tline (text-line, since version 2.5)

While this post doesn’t present many usage examples, we’ll try to show additional ones in future posts.