News for version 0.9.8

Since 0.9.7 has been a massive release with lots of changes, we dedicated the new 0.9.8 version of the Profiler to improve things and fix minor bugs. Here’s the change list:

– improved support for Windows 8.1 PEs
– added language options to Header Manager
– improved anonymous types renaming logic
– improved TrueType font disassembler
– many small improvements
– fixed some minor bugs

Since some improvements are PE related, PE Insider has been updated as well.

Enjoy!

PE Insider

It is always nice to give something back to the community and although this is unfortunately not always possible, we’re happy to announce the release of PE Insider, a free PE viewer which shares the same codebase for inspection as Cerbero Profiler and hence supports the entire PE specification and is incredibly fast and stable. We’re always very busy, but I was finally convinced by Ange Albertini to create this utility. 😉

PE Insider

The utility clearly stands light-years away in terms of functionality compared to the Profiler, but it does have some things which go beyond the simple format inspection including MSIL disassembly, navigation, ranges and resource preview.

Of course there’s room for improvement, but in the meantime here’s a first version. Enjoy!

P.S. To keep up-to-date with news regarding this utility either subscribe to our twitter account or follow the blog.

News for version 0.9.7

The new 0.9.7 version of the Profiler is out with the following news:

introduced C++ class/struct parsing with Clang
introduced headers, layouts and manual analysis in hex mode
exposed all the above to the Python SDK
added capability to turn into a portable application
– added SHA-3 hashes
– updated Qt to 4.8.5
– updated OpenSSL
– behavior change: displaying table flags now requires a double click

Enjoy!

Dissecting an ELF with C++ Types

While there are more interesting targets which could be manually analyzed with the new features provided in the Profiler, I decided to write a small post about ELF, also because official support for ELF will be added sooner or later.

Let’s start by importing the types contained in ‘elf.h’. You’ll probably find this header in ‘/usr/include’. Everything we’re interested in is in this file, so we can avoid importing other stuff. I added some predefines in order to avoid includes:

Then I pasted ‘elf.h’ into the Header Manager after the HEADER_START directive and clicked on ‘Import’.

ELF types import

We now have a header (elf) with all the types we need to start the manual analysis.

Since this is just a demonstration I didn’t do a full analysis of the ELF format. I limited the scope to finding the imported symbols and their strings.

ELF analysis

Every ELF starts with a _Elf64_Ehdr header (Elf32_Ehdr for 32-bit files, in this case it’s a 64-bit ELF). The header specifies the offset, number and size of the sections (we’ll just assume the standard 0x40 size here). The ‘name’ field of sections is just an index into a ‘SHT_STRTAB’ section whose index is specified by the header. The contents of a section are specified by its type, so finding the symbol table is pretty straight-forward. In this ELF we have a SHT_DYNSYM section. This section is just an array of _Elf64_Sym structures. Again, their ‘st_name’ field is just an index into another SHT_STRTAB section (the interval in the screenshot named ‘.dynstr’).

As already mentioned in the previous post, we can create a layout programmatically as well:

Moreover, the imported types can be used to do other operations not related to layouts. For instance let’s write few lines of code to print out the symbol names for this ELF:

The output will be:

Rememebr that the advantages of using CFFStructs rely not only in their dynamism or easiness in displaying them graphically, but also security. Contrary to a structure pointer in C, there’s no risk of crash when accessing members in a CFFStruct.

Today some final tests will be performed on the new version and if everything goes well, it will be released tomorrow or the day after. So stay tuned!

C++ Types: Under the Hood

In this post we’re going to explore the SDK part of the Profiler associated to imported structures and also all the C++ internals connected to the layout creation of structures/classes.

At first I thought about subdividing the material into several posts, but at the end it’s probably better to have it all together for future reference.

Layouts

In the SDK a Layout is the class to be used when we need to create a graphical analysis of raw data. While we can create and handle headers from the UI, it is also possible to do it programmatically.

Creating a layout is straightforward:

The data can be associated to a structure (or array of structures) as well. Please remember that the name of a header is always relative to header sub-directory of the user directory. Saving the layout is not necessary: it’s automatically saved in the project.

Attaching a layout to a hex view is also very easy:

Of course, layouts can be used for operations not related to graphical analysis as well.

Headers

Headers are part of the CFF Core and as such the naming convention of the CFFHeader class isn’t camel-case.

A CFFHeader represents an abstract database in which structures/classes and other things are stored. While we won’t use most of its methods, some of them are very useful for common operations.

Let’s say we want to retrieve a specific structure from a header and use it.

The output of this snippet is:

We can specify the following options when retrieving a structure:

These are the same options which are available from the UI when adding a structure to a layout.

When options are not specified, they default to the default structure options of the object. It’s possible to specify the default structure options with this method:

We’ll see later the implications of the various flags.

When I said that a CFFHeader represents an abstract database, I meant that it is not really bound to a specific format internally. All it cares about is that data is retrieved or set. The standard format used by headers is SQLite and you’ll need to use that format when creating layouts associated to structures. However, when using structures from Python it can be handy to avoid an associated header file. When the number of structures is very limited and you don’t need write or other complex operations, structures can be stored into an XML string. In fact, the internal format of structures is XML. Let’s take a look at one:

We can inspect the format of a structure stored in a header from the Header Manager in the Explore tab by double clicking on it. But we can also avoid creating a header altogether and output the schema of parsed structures directly when importing them from C++. Just check ‘Test mode’ and as ‘Output’ select ‘schemas’.

Output schemas

Let’s import a simple structure such as:

The output will be:

To use this structure from Python we can write the following code:

As you can see it’s very simple. I’ll use this method for the examples in the rest of the post, because they’re just examples and there’s no point in creating a header file for them.

Pointers

As a rule of thumb if a structure contains a pointer (or a vtable pointer) it is always a good idea to specify the desired size. When the size is omitted both in the explicit options and in the default structure options, the size will be set to the default pointer size of an object, which apart for PEObjects and MachObjects will always be 32bits.

Endianness

When endianness is not specified it will be set to the default of the object. While internally it’s already possible to have individual fields with different endianness, an extra XML field attribute to specify it will be added in the future.

Arrays

The first thing to say is that there’s a difference between an array of top level structures and an array of fields. Creating a top level array of structures is easy:

The support of arrays is somewhat limited. Multidimensional arrays are only partially supported, in the sense that they will be converted to a single dimension. For instance:

Or in XML:

Will be convrted to:

Also notice that to access an array element in a CFFStruct the syntax to use is not “a[15]” but “a.15”, e.g.:

Sub-structures

The only thing to mention about Sub-structures is that complex sub-types are always dumped separately, e.g.:

In XML:

In Python:

The output:

Being a separate type, we can also use ‘A::Sub’ without its parent.

A new thing we’ve just seen is the presence of multiple structures in a single XML header. I’ve pasted the whole Python code once again just for clarity, in the next examples I won’t repeat it, since the Python code never changes, only the header string does.

Unions

Unions just like sub-structures are fully supported. The only thing to keep in mind is that when we have a top level union, meaning not contained in another structure, such as:

Then to access its members it is necessary to add a ‘u.’ prefix. The reason for this is that CFFStructs support unions only as members, so the union above will result in a CFFStruct with a union member called ‘u’.

Anonymous types

Anonymous types are only partially supported in the sense that they are given a name when imported. A type such as the following:

Results in the following xml:

As you can see a ‘_Type_’ + number naming convention has been used to rename anonymous types. The first character (‘_’) in the name represents the default anonymous prefix. This prefix is customizable. If a typedef is found for an anonymous type, then the new name for that type will created by using the anonymous prefix + the typedef name.

Bit-fields

Bit-fields are fully supported.

Output:

The unnamed field at the end represents the unused bits given the field size, in this case we have an ‘int’ type and we’ve used only 5 bits of it.

There are significant differences in how compilers handle bit-fields. Visual C++ behaves differently than GCC/Clang. Some of the differences are summarized in this message by Richard W.M. Jones.

Another important difference I noticed is how bit fields are coalesced when the type changes, e.g.:

Without going now into how they are coalesced, the thing to remember is that the Profiler handles all these cases, but you need to specify the compiler to obtain the correct result.

Namespaces

Namespaces are fully supported.

Results in:

Moreover, just as in C++ we can use namespaces to encapsulate #include directives.

This will cause all the types declared in ‘Something’ to be prefixed by the namespace (‘N::’). This can be very handy when we want to include types with the same name into the same header file.

Inheritance

Inheritance is fully supported.

XML:

Output:

Same with multiple inheritance:

Output:

VTables

The presence of virtual table pointers in structures which require them is fully supported. Let’s take for instance:

XML:

Output:

Let’s see an example with multiple inheritance:

Output:

When virtual tables are involved it is very important to specify the compiler, because things can vary a great deal between VC++ and GCC/Clang.

Virtual Inheritance

Virtual inheritance is fully supported. Virtual inheritance is a C++ feature to be used in scenarios which involve multiple inheritance with a common base class.

Let’s take the complex case of:

Output (Visual C++):

Output (GCC):

As you can see the layout differs from Visual C++ to GCC. Another thing to notice is that members of virtual base classes are appended at the end. There’s a very good presentation by Igor Skochinsky on C++ decompilation you can watch for more information.

Field alignment

Field alignment is an important factor. Structures which are not subject to packing constraints are aligned up to their biggest native member. It’s more complex than this, because sub-structures influence parent structures but not vice versa. Suffice it to say that there are some internal gotchas, but the Profiler should handle all cases correctly.

Packing

When a packing constraint is applied, fields are aligned to either the field size or the packing whichever is less. A packing constraint of 1 is essential if we want to read raw data without any kind of padding between fields. For instance, PE structures in WinNT.h are all pragma packed to 1, so we must specify the same packing when using them.

Templates

And for the end a little treat: C++ templates. Let’s take for instance:

XML:

We can specify template parameters following the C++ syntax:

Output:

So, even nested templates are supported. 😉

C++ Types: Introduction

As announced previously, the upcoming 0.9.7 version of the Profiler represents a milestone in the development road map. We’re excited to present to you an awesome set of new features. In fact, the ground to cover is so vast that one post is not nearly enough. Throughout this week I’ll write some posts to cover the basics and this will allow for enough time to beta test the new version before reaching a release candidate.

Let’s start with an awesome image:

Presentation

Does it look like a Clang based tool to parse C++ sources and extract type information? If yes, then that’s exactly it!

To sum it up very briefly, the Profiler is now able to extract C++ types such as classes and structures and use these types both in the UI and in Python.

Add structure dialog

Of course, there’s much more to it. The layout of C++ types is a complex matter and doesn’t just involve supporting simple data structures. This post is just an introduction, the next ones will focus on topics such as: endianness, pointers, arrays, sub-structures, unions, bit-fields, inheritance, virtual tables, virtual inheritance, anonymous types, alignment, packing and templates. Yes, you read correctly: templates. 🙂

And apart from the implications of C++ types themselves, there’s the SDK part of the Profiler which will also require some dedicated posts. In this introduction I’m going to show a very simple flow and one of the many possible use cases.

You probably have noticed that the code in the screenshot above belongs to WinNT.h. Let’s see how to import the types in this header quickly. Usually we could parse all the headers of a framework with a few clicks, but while Clang is ideal to parse both Linux and OS X sources, it has difficulty with some Visual C++ extensions which are completely invalid C++ code. So rather than importing the whole Windows SDK we just limit ourselves to a part of WinNT.h.

I have added some predefines for Windows types (we could also include WinDef.h):

Then I just copied the header into the import tool. Usually this isn’t necessary, because we can set up the include directories from the UI and then just use #include directives, but since we need to modify the header to remove invalid C++ extensions, it makes sense to paste it.

The beginning of the code:

Did you notice the HEADER_START macro?

This tells our parser that the C++ types following this directive will be dumped into the header “WinNT.cphdr”. This file is relative to the header directory, a sub-directory of the user data directory. A HEADER_END directive does also exist, it equals to invoking the start directive with an empty string. To give you a better idea how these directives work take a look at this snippet:

If you specify the “#” string in the start directive, the types which follow will be dumped to the ‘this’ header. This is a special header which lives in the current project, so that you can pass the Profiler project to a colleague and it will already contain the necessary types without having to send extra files.

Back to the importing process, we click on ‘Import’ and that’s it. If Clang encounters C++ errors, we can fix them thanks to the diagnostic information:

Diagnostic information

We can explore the created header file from the ‘Explore’ tab.

Explore header

Now let’s use the header to analyze a PE file inside of a Zip archive.

Add structure to layout

Please notice that I’m adding the types with a packing of 1: PE structures are pragma packed to 1.

What you see applied to the hex view, is a layout. In a layout you can insert structures or intervals (a segment of data with a description and a color).

A layout can even be created programmatically and be attached to a hex view as we’ll see in some other post. The implementation of layouts in the Profiler is quite cool, because they are standalone objects. Layouts are not really bound to a hex view: a view just chooses to be attached to a layout. This means that you can share a single layout among different hex views and changes will reflect in all attached views.

Multi-view layout

And while I didn’t mention it, the table view below on the left is the layout inspector. Its purpose is to let you inspect the structures associated to a layout at a particular position. Since layouts allow for overlapping structures, the inspector shows all structures associated in the current range.

Multi-structure inspection

But what if you go somewhere else and return to the hex view? The layout will be gone. Of course, you could press Ctrl+Alt+L and re-attach the layout to the view. There are other two options: navigate back or create a bookmark!

Bookmark

The created bookmark when activated will jump to the right entry and associate the layout for us. Remember that changing the name of a layout invalidates the bookmark.

That’s all for now. And we’ve only scraped the surface… 🙂